cloudsoft.io

AMP upgrade

This guide provides all necessary information to upgrade AMP for both the RPM/DEB and Tarball packages.

Backwards Compatibility

AMP version 4+ runs inside a Karaf container. When upgrading from version 3.x, this update changes the mechanisms for launching AMP. This will impact any custom scripting around the launching of AMP, and the supplying of command line arguments.

Use of the lib/dropins and lib/patch folders will no longer work (because Karaf does not support that kind of classloading). Instead, code must be built and installed as OSGi bundles.

Upgrading

  • Use of RPM and DEB is now recommended where possible, rather than the tar.gz. This entirely replaces the previous install.

  • CentOS 7.x is strongly recommended over CentOS 6.x (note: the RPM is not fully supported on CentOS 6.x)

Upgrade from 4.x to AMP 5.x

For upgrade, the biggest difference between AMP 4.x and 5.x is that the catalog now uses bundles throughout. When something is added to the catalog, it is added as an OSGi bundle, and the .jar file is added to persisted state in the /bundles sub directory.

In contrast, with AMP 4.x catalog items are added in /catalog. Any bundles referenced from a catalog’s brooklyn.libraries section will also have been copied to the persisted state’s /bundles directory.

  1. Stop AMP:

    # CentOS 7 / RHEL
    sudo systemctl stop amp
    # CentOS 6 and older
    sudo initctl stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

  2. Important! Backup persisted state and custom configuration, in case you need to rollback to a previous version .

    1. By default, persisted state is located at /var/lib/amp.

      On first launch. the persistence dir is configured with the environment variable export AMP_PERSISTENCE_DIR=.... This updates the persistenceDir configuration in the file /etc/amp/org.apache.brooklyn.osgilauncher.cfg. Alternatively, and subsequently, this file can be modified directly. The persistenceLocation can also be configured in that file. The persistence details will be logged in /var/log/amp/amp.info.log at startup time.

    2. Configuration files are in /etc/amp.

  3. Upgrade AMP:

    1. Download the new RPM/DEB package

    2. Upgrade AMP:

      # CentOS / RHEL 7 
      sudo yum upgrade cloudsoft-amp-karaf-5.x.x-noarch.rpm
      

      Note this creates a new directory at /opt/amp-5.x.x, and updates the symbolic link at /opt/amp to point at this. The previous AMP directory is left as-is.

      If there are conflicts in configuration files (located in /etc/amp), the upgrade will behave differently based on the package you are using:

      • RPM: the upgrade will keep the previously installed one and save the new version, with the suffix .rpmnew. You will then need to check and manually resolve those. In particular, check for /etc/amp/config.properties.rpmnew

      • DEB: the upgrade will ask you what to do.

  4. (This step is only required if you are using custom OSGi bundles containing Java code compiled against an older version of AMP. If such bundles have been added explicitly via br catalog add mybundle.jar or are referenced in a catalog’s brooklyn.libraries, then steps must be taken to update the Java bundles. These will need to be recompiled for the new AMP release.)

    1. Prepare the new releases of your OSGi bundles containing Java code, uploaded to stable URLs (e.g. using Artifactory).

      1. Recompile the bundle against the latest version of AMP.

      2. Add instructions to the bundle’s MANIFEST.MF to say that it upgrades older versions of that bundle, and any catalog items it contains (see Upgrading Blueprints and Bundles):

        AMP-Catalog-Upgrade-For-Bundles: *
        AMP-Catalog-Force-Remove-Bundles: *
        AMP-Catalog-Force-Remove-Legacy-Items: *
        
    2. Add the new versions of these OSGi bundles (which will thus replace older versions in the persisted state /bundles dir):

      1. Prepare a .bom file that references the new versions of your Java OSGi bundles. For example:

        brooklyn.catalog:
        version: "5.0.0"
        
        brooklyn.libraries:
        - https://example.org/foo-1.1.0.jar
        - https://example.org/bar-1.1.0.jar
        
      2. Add this to the initial catalog, for example by saving the file as /opt/amp/catalog/catalog-custom.bom and adding a reference to it at the end of /opt/amp/catalog/catalog.bom.

    3. Update links in your legacy /catalog persisted state:

      1. Prepare a configuration file, which lists the old and new bundle URLs. For example:

        https://example.org/foo-1.0.0.jar=https://example.org/foo-1.1.0.jar
        https://example.org/bar-1.0.0.jar=https://example.org/bar-1.1.0.jar
        
      2. Use the amp-upgrade-bundles.sh script to update the persisted state to reference the new bundle versions:

        chmod u+x amp-upgrade-bundles.sh
        
        export AMP_PERSISTENCE_DIR=/path/to/persisted-state-v2
        export BUNDLE_MAPPINGS_FILE=/path/to/bundle-list.conf
        
        ./amp-upgrade-bundles.sh
        
      3. Check the stdout and stderr of the script to ensure it did the transformations expected (e.g. no typos in the bundle list configuration file).

    4. If you have deployed bundles that contain a catalog.bom which explicitly reference such Java bundles in the brooklyn.libraries, these references will also need to be updated. Please contact Cloudsoft support for assistance, if required.

  5. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure /etc/amp/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      # CentOS / RHEL 7
      sudo systemctl start amp
      

      (If AMP fails to start, consult the troubleshooting section below)

    3. Check whether you have rebind ERROR messages in /var/log/amp/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /var/log/amp/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      # CentOS / RHEL 7
      sudo systemctl stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in /etc/amp/org.apache.brooklyn.osgilauncher.cfg

  6. Start AMP:

    # # CentOS / RHEL 7
    sudo systemctl start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

    Confirm that there are no errors reported in the log (/var/log/amp/amp.info.log).

Troubleshooting:

  • AMP fails to start, writing to /var/log/amp/karaf.log the message “SEVERE: Could not launch framework” … “Could not resolve mvn:org.apache.felix/org.apache.felix.framework/5.6.1”.

    This means there is an invalid configuration file. Mostly likely, the upgrade encountered conflicts in the configuration file /etc/amp/config.properties.rpmnew, and created a file with suffix “.rpmnew”, leaving the old file in place.

    If so, consider replacing /etc/amp/config.properties with /etc/amp/config.properties.rpmnew, but ensure that any manual customization of the previous config.properties is also applied to the new version of the file.

  • AMP fails to read the persisted state (having performed the state tranformation step above).

    Ensure the persisted state (e.g. at /var/lib/amp) is owned by amp:amp.

  • AMP’s yum upgrade fails with an error like:

    Transaction check error:
    file /opt/amp from install of cloudsoft-amp-karaf-5.0.0_1.noarch conflicts with file from package cloudsoft-amp-karaf-4.2.0-1.noarch
    

    These upgrade instructions only apply when upgrading from AMP 4.3 onwards. If upgrading from earlier AMP releases, see the instructions in the sections below.

  • AMP upgrade fails on CentOS 6.

    Use of CentOS 7 is strongly recommended, instead of CentOS 6. However, if using CentOS 6, there are the following differences:

    • To start and stop AMP, instead use sudo initctl start amp and sudo initctl stop amp.

    • Instead of using the yum upgrade ... command, use sudo rpm -Uvh --nopreun cloudsoft-amp-karaf-5.x.x-noarch.rpm.

  1. Stop AMP:

    ./bin/stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

  2. Important! Backup persisted state and custom configuration, in case you need to rollback to a previous version .

    1. By default, persisted state is located at ~/.brooklyn/brooklyn-persisted-state. The persistenceDir and persistenceLocation are configured in the file ./etc/org.apache.brooklyn.osgilauncher.cfg. The persistence details will be logged in ./log/amp.info.log at startup time.

    2. Configuration files are in ./etc/. Any changes to these configuration files will need to be re-applied after reinstalling AMP.

  3. Install new version of AMP:

    1. Download the new tarball zip package

    2. Install AMP:

      tar -zxf cloudsoft-amp-karaf-5.x.x-xxxxxxxx.xxxx.tar.gz
      cd cloudsoft-amp-karaf-5.x.x-xxxxxxxx.xxxx
      
    3. Restore any changes to the configuration files (see step 2).

  4. (This step is only required if you are using custom OSGi bundles containing Java code compiled against an older version of AMP. If such bundles have been added explicitly via br catalog add mybundle.jar or are referenced in a catalog’s brooklyn.libraries, then steps must be taken to update the Java bundles. These will need to be recompiled for the new AMP release.)

    1. Prepare the new releases of your OSGi bundles containing Java code, uploaded to stable URLs (e.g. using Artifactory).

      1. Recompile the bundle against the latest version of AMP.

      2. Add instructions to the bundle’s MANIFEST.MF to say that it upgrades older versions of that bundle, and any catalog items it contains (see Upgrading Blueprints and Bundles):

        AMP-Catalog-Upgrade-For-Bundles: *
        AMP-Catalog-Force-Remove-Bundles: *
        AMP-Catalog-Force-Remove-Legacy-Items: *
        
    2. Add the new versions of these OSGi bundles (which will thus replace older versions in the persisted state /bundles dir):

      1. Prepare a .bom file that references the new versions of your Java OSGi bundles. For example:

        brooklyn.catalog:
        version: "5.0.0"
        
        brooklyn.libraries:
        - https://example.org/foo-1.1.0.jar
        - https://example.org/bar-1.1.0.jar
        
      2. Add this to the initial catalog, for example by saving the file as /opt/amp/catalog/catalog-custom.bom and adding a reference to it at the end of /opt/amp/catalog/catalog.bom.

    3. Update links in your legacy /catalog persisted state:

      1. Prepare a configuration file, which lists the old and new bundle URLs. For example:

        https://example.org/foo-1.0.0.jar=https://example.org/foo-1.1.0.jar
        https://example.org/bar-1.0.0.jar=https://example.org/bar-1.1.0.jar
        
      2. Use the amp-upgrade-bundles.sh script to update the persisted state to reference the new bundle versions:

        chmod u+x amp-upgrade-bundles.sh
        
        export AMP_PERSISTENCE_DIR=/path/to/persisted-state-v2
        export BUNDLE_MAPPINGS_FILE=/path/to/bundle-list.conf
        
        ./amp-upgrade-bundles.sh
        
      3. Check the stdout and stderr of the script to ensure it did the transformations expected (e.g. no typos in the bundle list configuration file).

    4. If you have deployed bundles that contain a catalog.bom which explicitly reference such Java bundles in the brooklyn.libraries, these references will also need to be updated. Please contact Cloudsoft support for assistance, if required.

  5. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure ./etc/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      ./bin/start amp
      
    3. Check whether you have rebind ERROR messages in ./log/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /opt/amp/log/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      ./bin/stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in ./etc/org.apache.brooklyn.osgilauncher.cfg

  6. Start AMP:

    ./bin/start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

Upgrade from AMP 4.3 onward

  1. Stop AMP:

    # CentOS 7 / RHEL
    sudo systemctl stop amp
    # CentOS6 and older
    sudo initctl stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

  2. Important! Backup persisted state and custom configuration, in case you need to rollback to a previous version https://docs.cloudsoft.io/operations/upgrade.html#rollback.

    1. By default, persisted state is located at /var/lib/amp.

      The persistence dir is configured with the environment variable export AMP_PERSISTENCE_DIR=... Alternatively, persistenceDir can be configured in the file /etc/amp/org.apache.brooklyn.osgilauncher.cfg. The persistenceLocation can also be configured in that file. The persistence details will be logged in /var/log/amp/amp.info.log at startup time.

    2. Configuration files are in /etc/amp.

  3. Upgrade AMP:

    1. Download the new RPM/DEB package

    2. Upgrade AMP:

      # CentOS / RHEL 7 
      sudo yum upgrade cloudsoft-amp-karaf-4.x.x.rpm
      

      Note this creates a new directory at /opt/amp-4.x.x, and updates the symbolic link at /opt/amp to point at this. The previous AMP directory is left as-is.

      If there are conflicts in configuration files (located in /etc/amp), the upgrade will behave differently based on the package you are using:

      • RPM: the upgrade will keep the previously installed one and save the new version, with the suffix .rpmnew. You will then need to check and manually resolve those. In particular, check for /etc/amp/config.properties.rpmnew

      • DEB: the upgrade will ask you what to do.

  4. (This step is only required if your persisted state references Java OSGi bundles in the catalog items’ libraries section, which need to be updated/recompiled for the new AMP release. The steps below will update the persisted state to use the new versions of the OSGi bundles.)

    1. Prepare the new releases of your OSGi bundles, uploaded to stable URLs (e.g. using Artifactory).

    2. Prepare a configuration file, which lists the old and new bundle URLs. For example:

      https://example.org/foo-1.0.0.jar=https://example.org/foo-1.1.0.jar
      https://example.org/bar-1.0.0.jar=https://example.org/bar-1.1.0.jar
      
    3. Use the amp-upgrade-bundles.sh script to update the persisted state to reference the new bundle versions:

      chmod u+x amp-upgrade-bundles.sh
      
      export AMP_PERSISTENCE_DIR=/path/to/persisted-state-v2
      export BUNDLE_MAPPINGS_FILE=/path/to/bundle-list.conf
      
      ./amp-upgrade-bundles.sh
      
    4. Check the stdout and stderr of the script to ensure it did the transformations expected (e.g. no typos in the bundle list configuration file).

  5. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure /etc/amp/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      # CentOS / RHEL 7
      sudo systemctl start amp
      
    3. Check whether you have rebind ERROR messages in /var/log/amp/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /var/log/amp/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      # CentOS / RHEL 7
      sudo systemctl stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in /etc/amp/org.apache.brooklyn.osgilauncher.cfg

  6. Start AMP:

    # # CentOS / RHEL 7
    sudo systemctl start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

    Confirm that there are no errors reported in the log (/var/log/amp/amp.info.log).

  7. Add the new AMP catalog items (as per the release notes “Upgrade Instructions”):

    1. Download the br tool (i.e. from the “CLI Download” link in the web-console)

    2. Login with br: br login http://localhost:8081 <user> <password>

    3. Update the catalog: br catalog add /opt/amp/catalog/catalog.bom

Troubleshooting:

  • AMP fails to start, writing to /var/log/amp/karaf.log the message “SEVERE: Could not launch framework” … “Could not resolve mvn:org.apache.felix/org.apache.felix.framework/5.4.0”.

    This means there is an invalid configuration file. Mostly likely, the upgrade encountered conflicts in the configuration file /etc/amp/config.properties.rpmnew, and created a file with suffix “.rpmnew”, leaving the old file in place.

    If so, consider replacing /etc/amp/config.properties with /etc/amp/config.properties.rpmnew, but ensure that any manual customization of the previous config.properties is also applied to the new version of the file.

  • AMP fails to read the persisted state (having performed the state tranformation step above).

    Ensure the persisted state (e.g. at /var/lib/amp) is owned by amp:amp.

  • AMP’s yum upgrade fails with an error like:

    Transaction check error:
    file /opt/amp from install of cloudsoft-amp-karaf-4.5.1_1.noarch conflicts with file from package cloudsoft-amp-karaf-4.2.0-1.noarch
    

    These upgrade instructions only apply when upgrading from AMP 4.3 onwards. If upgrading from earlier AMP releases, see the instructions in the sections below.

  • AMP upgrade fails on CentOS 6.

    Use of CentOS 7 is strongly recommended, instead of CentOS 6. However, if using CentOS 6, there are the following differences:

    • To start and stop AMP, instead use sudo initctl start amp and sudo initctl stop amp.

    • Instead of using the yum upgrade ... command, use sudo rpm -Uvh --nopreun cloudsoft-amp-karaf-4.x.x.rpm.

  1. Stop AMP:

    ./bin/stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

  2. Important! Backup persisted state and custom configuration.

    1. By default, persisted state is located at ~/.brooklyn/brooklyn-persisted-state. The persistenceDir and persistenceLocation are configured in the file ./etc/org.apache.brooklyn.osgilauncher.cfg. The persistence details will be logged in ./log/amp.info.log at startup time.

    2. Configuration files are in ./etc/. Any changes to these configuration files will need to be re-applied after reinstalling AMP.

  3. Install new version of AMP:

    1. Download the new tarball zip package

    2. Install AMP:

      tar -zxf cloudsoft-amp-karaf-4.x.x-xxxxxxxx.xxxx.tar.gz
      cd cloudsoft-amp-karaf-4.x.x-xxxxxxxx.xxxx
      
    3. Restore any changes to the configuration files (see step 2).

  4. (This step is only required if your persisted state references Java OSGi bundles in the catalog items’ libraries section, which need to be updated/recompiled for the new AMP release. The steps below will update the persisted state to use the new versions of the OSGi bundles.)

    1. Prepare the new releases of your OSGi bundles, uploaded to stable URLs (e.g. using Artifactory).

    2. Prepare a configuration file, which lists the old and new bundle URLs. For example:

      https://example.org/foo-1.0.0.jar=https://example.org/foo-1.1.0.jar
      https://example.org/bar-1.0.0.jar=https://example.org/bar-1.1.0.jar
      
    3. Use the amp-upgrade-bundles.sh script to update the persisted state to reference the new bundle versions:

      chmod u+x amp-upgrade-bundles.sh
      
      export AMP_PERSISTENCE_DIR=/path/to/persisted-state-v2
      export BUNDLE_MAPPINGS_FILE=/path/to/bundle-list.conf
      
      ./amp-upgrade-bundles.sh
      
    4. Check the stdout and stderr of the script to ensure it did the transformations expected (e.g. no typos in the bundle list configuration file).

  5. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure ./etc/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      ./bin/start amp
      
    3. Check whether you have rebind ERROR messages in ./log/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /opt/amp/log/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      ./bin/stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in ./etc/org.apache.brooklyn.osgilauncher.cfg

  6. Start AMP:

    ./bin/start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

  7. Update the catalog, using the br command:

    1. Download the br tool (i.e. from the “CLI Download” link in the web-console)

    2. Login with br: br login http://localhost:8081 <user> <password>

    3. Update the catalog: br catalog add /opt/amp/catalog/catalog.bom

Upgrade from AMP 4.1 to 4.3 or Later

  1. Stop AMP:

    # CentOS 7 / RHEL
    sudo systemctl stop amp
    # CentOS6 and older
    sudo initctl stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

  2. Important! Backup persisted state and custom configuration.

    1. By default, persisted state is located at /opt/amp/.brooklyn/. The persistenceDir and persistenceLocation are configured in the file ./etc/org.apache.brooklyn.osgilauncher.cfg. The persistence details will be logged in ./log/amp.info.log at startup time.

    2. Configuration files are in ./etc/. Any changes to these configuration files will need to be re-applied after reinstalling AMP.

  3. Delete the existing AMP install:

    1. Remove AMP package:

      # CentOS / RHEL
      sudo yum erase cloudsoft-amp-karaf
      
    2. On CentOS 7 run sudo systemctl daemon-reload

    3. Confirm that AMP is definitely not running (see step 1 above).

    4. Delete the AMP install directory: sudo rm -r /opt/amp

  4. Make sure you have Java 8 installed. By default CentOS images come with JRE6 which is incompatible version for AMP. If CentOS is prior to 6.8 upgrade nss: yum -y upgrade nss

  5. Install new version of AMP:

    1. Download the new RPM/DEB package

    2. Install AMP:

      # CentOS / RHEL
      sudo yum install cloudsoft-amp-karaf-4.x.x-xxxxxxxx.rpm
      
    3. Wait for AMP to be fully started (i.e. its web-console to be responsive).

  6. Restore the persisted state and configuration.

    1. Stop the amp service

      # CentOS 7 / RHEL
      sudo systemctl stop amp
      # CentOS 6 and older
      sudo initctl stop amp
      

      Confirm that AMP is no longer running (see step 1).

    2. Restore the persisted state directory, and any changes to the configuration files (see step 2). Ensure owner/permissions are correct for the persisted state directory, e.g.: sudo chown -r amp:amp /opt/amp/.brooklyn

  7. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure ./etc/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      # CentOS 7 / RHEL
      sudo systemctl start amp
      # CentOS 6 and older
      sudo initctl start amp
      
    3. Check whether you have rebind ERROR messages in ./log/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /opt/amp/log/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      # CentOS 7 / RHEL
      sudo systemctl stop amp
      # CentOS 6 and older
      sudo initctl stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in ./etc/org.apache.brooklyn.osgilauncher.cfg

  8. Start AMP:

    # CentOS 7 / RHEL
    sudo systemctl start amp
    # CentOS 6 and older
    sudo initctl start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

  9. Update the catalog, using the br command:

    1. Download the br tool (i.e. from the “CLI Download” link in the web-console)

    2. Login with br: br login http://localhost:8081 <user> <password>

    3. Update the catalog: br add-catalog /opt/amp/catalog/catalog.bom

Same instructions as above.

Upgrade from AMP 3.x to 4.x

Upgrading from AMP 3.x to 4.x follows much the same pattern as a standard manual upgrade, these steps are listed below. Please note that it can be substantially more complex if custom Java code has been used (in the ./lib/dropins/*.jar or ./lib/patch/*.jar), which is referenced in the persisted state. These jars must be converted to OSGi bundles. Rebinding to the old persisted state will not work until these bundles are available. The additional steps below describe how to handle this scenario but may be ignored if custom java has not been used.

The instructions below assume that AMP 4.x is installed via the tgz. For RPM and DEB, use the alternative commands for starting and stopping AMP as detailed in Get Amp Running.

  1. Stop AMP 3.x

    ps aux | grep java
    kill <JAVA_PID>
    
  2. Important! Backup persisted state and custom configuration.

    1. By default, persisted state is located at ~/.brooklyn/brooklyn-persisted-state/data. The persistence details will be logged in amp.info.log at startup time.

    2. Configuration files are in ~/.brooklyn/brooklyn.properties. Any changes to these configuration files will need to be re-applied after reinstalling AMP, see configuring AMP for more information.

  3. Install the new version of AMP:

    1. Make sure you have Java 8 (strongly preferred). By default CentOS images come with JRE6 which is incompatible version for AMP. If CentOS is prior to 6.8, also upgrade nss: yum -y upgrade nss

    2. Follow the instructions on Get Amp Running, but do not yet start AMP.

  4. Manually add the OSGi bundles.

    We do this by adding the v2 catalog item(s) that include the libraries section(s). This will cause the bundles to be installed into the karaf data directory.

    cd /opt/amp
    export AMP_PERSISTENCE_DIR="`pwd`/empty_persistence"
    ./bin/start
    # wait for AMP to be up-and-running, then...
    br login http://localhost:8081 admin password
    for i in /path/to/custom-bundles/*.bom; do
      br catalog add $i
    done
    ./bin/stop
    

    Note! What to do when this step failed because of an error in the bundle imported from catalog items above. Uninstall the bundles installed from previous imports of catalog items and then reimport the catalog to actually install fixed bundle.

  5. Add the bundles to the “whitelist”. This allows classloading from these bundles, even if the persisted state does not tell us which bundle the class is in.

    Near the end of ./bin/setenv, change the WHITELIST to include the customer-specific bundle symbolic names as a regex. For example:

    WHITELIST='org.apache.brooklyn.*|io.cloudsoft.*|io.brooklyn.*|com.acmecorp.*'
    

    Note these are the symbolic names of the OSGi bundles. A convention is for this name to match the package name prefix within the bundle, but that is not required.

  6. Re-start AMP with the historic persisted state

    ./bin/stop
    export AMP_PERSISTENCE_DIR=/path/to/persisted-state/
    ./bin/start
    

    Check that AMP comes up without errors.

    AMP will re-write each file in the persisted state. For each classname, it will include the bundle name as a prefix.

  7. Add the new catalog item versions (that include the libraries sections), and stop AMP

    for i in /path/to/custom-bundles/*.bom; do
      br catalog add $i
    done
    
    ./bin/stop
    
  8. Remove the old catalog item version(s) from the catalog

    Create a transformations file that contains all of the old (e.g. v1.0) catalog items that are being replaced. For example in a file named upgrade-catalog.yaml:

    - deletions:
        catalog:
        - org.my.package.tomcat:1.0
    

    Using AMP 3.5.1, run the copy-state command to delete these files:

    cd /opt/amp
    export AMP_PERSISTENCE_DIR=/path/to/persisted-state
    export AMP_PERSISTENCE_DIR_TRANSFORMED=/path/to/persisted-state-v2
    ./bin/amp copy-state --persistenceDir ${AMP_PERSISTENCE_DIR} --destinationDir ${AMP_PERSISTENCE_DIR_TRANSFORMED} --transformations upgrade-catalog.yaml
    
  9. Update all existing entities etc to point at the v2 catalog items:

    OLD_CATALOG_ID=org.my.package.tomcat:1.0
    NEW_CATALOG_ID=org.my.package.tomcat:2.0
    PERSISTENCE_DIR=/path/to/persisted-state-v2
    
    for subdir in entities enrichers policies feeds locations; do
      for tag in catalogItemId catalog.id; do
        find ${PERSISTENCE_DIR}/${subdir}/ -type f -print0 | xargs -0 -n1 sed -i.bak -e "s/<${tag}>${OLD_CATALOG_ID}<\/${tag}>/<${tag}>${NEW_CATALOG_ID}<\/${tag}>/"
      done
    done
    find . -name "*.bak" -delete
    grep -R -E "${OLD_CATALOG_ID}|${NEW_CATALOG_ID}" ${PERSISTENCE_DIR}/*
    
  10. Remove the custom-bundles from the whitelist

    Restore AMP 4.x’s ./bin/setenv to the default whitelist.

  11. Start AMP 4.x, pointing at the transformed persisted state

    cd /opt/amp
    export AMP_PERSISTENCE_DIR=/path/to/persisted-state-v2/
    ./bin/start
    

Troubleshooting:

  • If there are rebind errors after removing the custom bundles names from the whitelist, re-add these to the whitelist and try again.

  • If there are rebind errors for classes org.jclouds.compute.*, this could be an indication that an old 3.x version is in use. Try first rebinding using AMP 3.5.1, which will transform the persisted state so that these objects are not persisted.

  • If there are classloading errors for classes contained in the whitelisted custom bundles, use the karaf client to list the bundles (e.g. bundle:list -t 0 -s | grep acmecorp). Confirm that the bundles are installed correctly, and that the symbolic names match the regex used in ./bin/setenv.

Rollback

This section applies only with you are using the RPM/DEB packages. To perform a rollback to a previous AMP version, please follow these instructions.

Note there are no “forwards compatibility” guarantees, that persisted state generated with a new version of AMP will work with older versions of AMP. Therefore persisted state written by a newer version of AMP will not necessarily work when rolling back to an older version of AMP.

  1. Stop AMP:

    # e.g. on CentOS 7 / RHEL
    sudo systemctl stop amp
    

    If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>, or sudo kill -9 <JAVA_PID>.

  2. Restore the persisted state and custom configuration from the backup you made during the previous upgrade.

    1. By default, persisted state is located at /var/lib/amp.

      The persistence dir is configured with the environment variable export AMP_PERSISTENCE_DIR=... Alternatively, persistenceDir can be configured in the file /etc/amp/org.apache.brooklyn.osgilauncher.cfg. The persistenceLocation can also be configured in that file. The persistence details will be logged in /var/log/amp/amp.info.log at startup time.

    2. Configuration files are in /etc/amp.

  3. Downgrade AMP (to 4.3.x or later):

    # e.g. for CentOS / RHEL
    sudo yum downgrade cloudsoft-amp-karaf-4.x.x.rpm
    

    Note this creates a new directory at /opt/amp-4.x.x, if necessary, and updates the symbolic link at /opt/amp to point at this. The previous AMP directory is left as-is.

    Check if there are any conflicts in configuration files (located in /etc/amp), for example if there are any files with the suffix .rpmnew, and resolve those conflicts.

  4. Validate that the new release works, by starting in “HOT_BACKUP” mode.

    1. Before starting AMP, reconfigure /etc/amp/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

    2. Start AMP:

      # e.g. on CentOS 7 / RHEL sudo systemctl start amp

    3. Check whether you have rebind ERROR messages in /var/log/amp/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /var/log/amp/amp.debug.log If you do not have such errors you can proceed.

    4. Stop AMP:

      # e.g. on CentOS 7 / RHEL
      sudo systemctl stop amp
      
    5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in /etc/amp/org.apache.brooklyn.osgilauncher.cfg

  5. Start AMP:

    # e.g. on CentOS 7 / RHEL
    sudo systemctl start amp
    

    Wait for AMP to be running (i.e. its web-console is responsive)

    Confirm that there are no errors reported in the log (/var/log/amp/amp.info.log).

Migrating AMP to a Different Server

Moving AMP to a different server has much in common with upgrade. The steps are described below, which assume that you are running AMP 4.3 or later. If upgrading AMP at the same time, see the additional steps described in the appropriate upgrade instructions.

  1. On the old server:

    1. Stop AMP:

      # CentOS 7 / RHEL
      sudo systemctl stop amp
      # CentOS6 and older
      sudo initctl stop amp
      

      If this does not stop it within a few seconds (as checked with sudo ps aux | grep karaf), then use sudo kill <JAVA_PID>

    2. Make a copy of the persisted state and custom configuration.

      1. By default, persisted state is located at /var/lib/amp.

        The persistence dir is configured with the environment variable export AMP_PERSISTENCE_DIR=... Alternatively, persistenceDir can be configured in the file /etc/amp/org.apache.brooklyn.osgilauncher.cfg. The persistenceLocation can also be configured in that file. The persistence details will be logged in /var/log/amp/amp.info.log at startup time.

        If using an external object store (e.g. S3), you do not have to copy the persisted state. The new AMP server can instead be configured to point at the same object store.

      2. Configuration files are in /etc/amp.

  2. On the new server:

    1. Install AMP, but do not start the service.

    2. Apply any configuration file customizations from the old server, on the new server.

    3. If using file-based persistence, unpack the persisted state to the appropriate location.

    4. Validate that the new release works, by starting in “HOT_BACKUP” mode.

      1. Before starting AMP, reconfigure /etc/amp/org.apache.brooklyn.osgilauncher.cfg and set highAvailabilityMode=HOT_BACKUP. This way when AMP is started, it will only read and validate the persisted state and will not write into it.

      2. Start AMP:

        # e.g. on CentOS 7 / RHEL
        sudo systemctl start amp
        
      3. Check whether you have rebind ERROR messages in /var/log/amp/amp.info.log, e.g. sudo grep -E "WARN|ERROR" /var/log/amp/amp.debug.log If you do not have such errors you can proceed.

      4. Stop AMP:

        # e.g. on CentOS 7 / RHEL
        sudo systemctl stop amp
        
      5. Change the highAvailabilityMode back to the default (AUTO) by commenting it out in /etc/amp/org.apache.brooklyn.osgilauncher.cfg

    5. Start AMP:

      # CentOS 7 / RHEL
      sudo systemctl start amp
      

      Wait for AMP to be running (i.e. its web-console is responsive)

      Confirm that there are no errors reported in the log (/var/log/amp/amp.info.log).

How to stop your service

On systemd:

systemctl stop amp

On upstart:

stop amp

Web login credentials

  • User credentials should now be recorded in ${karaf.home}/etc/brooklyn.cfg

  • AMP will still read them from both ${karaf.home}/etc/brooklyn.cfg and ~/.brooklyn/brooklyn.properties

  • Configure a username/password by modifying brooklyn.cfg. An example entry is:

brooklyn.webconsole.security.users=admin
brooklyn.webconsole.security.user.admin.password=password2

Persistence

If you have persisted state you wish to rebind to, persistence is now configured in the following files:

For example, to use S3 for the persisted state, add the following to brooklyn.cfg:

brooklyn.location.named.aws-s3-eu-west-1:aws-s3:eu-west-1
brooklyn.location.named.aws-s3-eu-west-1.identity=<ADD CREDS>
brooklyn.location.named.aws-s3-eu-west-1.credential=<ADD CREDS>

To continue the S3 example, for the persisted state, add the following to org.apache.brooklyn.osgilauncher.cfg:

persistenceLocation=aws-s3-eu-west-1
persistenceDir=<ADD HERE>

AMP should be stopped before this file is modified, and then restarted with the new configuration.

Note that you can not store the credentials (for e.g. aws-s3-eu-west-1) in the catalog because that catalog is stored in the persisted state. AMP needs to know it in order to read the persisted state at startup time.

If binding to existing persisted state, an additional command is required to update the existing catalog with the AMP 4.0 versions. Assuming AMP has been installed to /opt/amp (as is done by the RPM and DEB):

br add-catalog /opt/amp/catalog/catalog.bom

All existing custom jars previously added to lib/plugins (e.g. for Java-based entities) need to be converted to OSGi bundles, and installed in Karaf. The use of the “brooklyn.libraries” section in catalog.bom files will continue to work.

High Availability

High Availability mode can not be specified via a CLI parameter (as was the case in v3.x) but should be configured in org.apache.brooklyn.osgilauncher.cfg.

For details on how to configure HA, see the HA tutorial.

Upgrading Blueprints and Bundles

You can install and deploy new versions of blueprints at any time. AMP tracks multiple versions of the blueprints you install, as can be seen in the catalog.

Defining and Forcing Upgrade Paths

Bundles can declare what bundles and types they can upgrade, and they can also force the removal of installed bundles and types on startup/rebind. Forcing can be useful when upgrading AMP to replace any installed bundle not compatible with the newer version of AMP.

To add these definitions, use the following headers in the bundle’s OSGi META-INF/MANIFEST.MF:

  • AMP-Catalog-Force-Remove-Bundles
  • AMP-Catalog-Force-Remove-Legacy-Items
  • AMP-Catalog-Upgrade-For-Bundles
  • AMP-Catalog-Upgrade-For-Types

The most common patterns are to indicate that a bundle can replace all previous versions of itself and all types therein with types in the current bundle of the same name, using:

AMP-Catalog-Upgrade-For-Bundles: *

And you can indicate that previous bundles should be uninstalled, forcing the above upgrades, with:

AMP-Catalog-Force-Remove-Bundles: *

The above items can also take a range syntax, e.g. "*:[1,2)" when releasing a 2.0.0 to restrict to versions equal to or greater than 1.0.0 but less than 2.0.0. (Note that ranges must be quoted.) Entries can also take comma-separated lists, and in the case of replacements, they can define explicit renamed targets using sourceNameAndVersionRanges=targetNameAndVersion entries.
These fields are defined in full in the BundleUpgradeParser’s javadoc.

Upgrading the Version of Deployed Blueprints

New versions of blueprints are not automatically applied to existing deployments from older versions. This requires a rebind using the above techniques, or programmatic intervention: please ask on the mailing list for more information (and to help us identify the most common wishes in this area!).

Upgrading Systems Under Management

Blueprints can encode update processes for the systems they describe. The mechanisms for updating systems vary, depending whether it is stateless or stateful, whether following an immutable pattern (replacing components) or doing it on box (traditional, possibly taking systems out of action while upgrading), and whether applying an upgrade to many resources on a rolling fashion (repaving, blue-green). For this reason there is not a one-size-fits-all upgrade pattern to use in blueprints, but there are some common patterns that may be applicable:

  • Defining an upgrade effector on nodes, and on a cluster to apply to all nodes
  • Using a config key such as version which can be updated and reapplied
  • Exposing a deploy effector to pass files that should be run, such as WAR files, and invoking this effector with newer versions of WAR files to install

There are many more, and if you’ve written some good pieces to share, please consider contributing them so others can take advantage of them!