Skip to content

Instantly share code, notes, and snippets.

@andrewmkhoury

andrewmkhoury/AEM6.x Migration Instructions.md

Last active June 20, 2023 15:29
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save andrewmkhoury/280c6d0ac1db03b57c58 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save andrewmkhoury/280c6d0ac1db03b57c58 to your computer and use it in GitHub Desktop.
Download ZIP
AEM6.x Migration
Raw
AEM6.x Migration Instructions.md

AEM6.x Migration

  • How to Rebuild a Pre-Existing AEM+Mongo Cluster
  • _or migrate from Tar to MongoDB or MongoDB to Tar"
  • or migrate from 5.6.1 or 6.x to 6.x via data migration instead of jar upgrade
  1. (Mongo to Mongo only) Remove one replica from the replica set and delete/recreate the db
  2. Remove the replica node from the set: http://docs.mongodb.org/master/tutorial/remove-replica-set-member/
  3. Validate that no other nodes in the set consider that node to be part of the set anymore. Log into each node in the cluster via mongo shell and run rs.status() to validate that the node that was taken out is not in the cluster any longer.
  4. Drop the aem database on that node http://docs.mongodb.org/manual/reference/command/dropDatabase/
  5. Re-add the database with the correct user permissions (do not add the mongo instance back to the replica set)
  6. Remove one of the AEM cluster nodes from the load balancer or dispatcher config (whichever config is managing the active cluster nodes)
  7. Stop the AEM instance on the inactive node
  8. On the same server install a fresh AEM6.x instance pointed at the new emptied mongo database and install hotfixes (for AEM6.0 install Oak 1.0.27 or later, AEM6.1 install Oak 1.2.11 or later) * 6.0 - https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html * 6.1 - https://helpx.adobe.com/experience-manager/kb/aem61-available-hotfixes.html
  9. Skip step 2 below
  10. Install a fresh AEM6.x instance and all the recent recommended hotfixes

  1. Optimize the instance using the configurations here - https://helpx.adobe.com/experience-manager/kb/performance-tuning-tips.html

  2. Prep the systems for data migration

    1. Prep source system if migrating from CQ5.x version to AEM6.x:
      1. Run consistency check and fix: https://helpx.adobe.com/experience-manager/kb/RepositoryInconsistency.html
      2. Remove Same Name Sibling nodes: https://helpx.adobe.com/experience-manager/kb/find-sns-nodes.html
    2. Prep the destination 6.x system:
    3. Apply any customizations to the AEM start script and restart AEM
    4. Disable workflow launchers for DAM by going to the launcher tab under workflow console url: http://host:port/libs/cq/workflow/content/console.html * Disable all launchers that use "DAM Update Asset" and "DAM Metadata Writeback” workflow models.
    5. Disable OSGi components to reduce overhead. * Install ACS Commons (on the newly installed environment): http://adobe-consulting-services.github.io/acs-aem-commons/features/osgi-disabler.html * Disable these OSGi components using the osgi-disabler: ``` com.day.cq.dam.core.impl.event.DamEventAuditListener com.day.cq.replication.audit.ReplicationEventListener com.day.cq.wcm.core.impl.event.PageEventAuditListener com.day.cq.tagging.impl.TagValidatingEventListener com.day.cq.dam.core.impl.DamChangeEventListener
    ```

  3. (6.x to 6.x migration only) On the old/source instance, package out-of-the-box indexes that were disabled or modified and install those. Do not install custom indexes as it is faster to install them after. The objective here is to get any disabled indexes disabled.

  4. Go to CRXDE on the destination instance, browse to /oak:index/lucene and add a String[] property excludedPaths excludedPaths=[/var, /jcr:system, /etc/workflow/instances]

  5. Perform repository maintenance * Mongo (only) Run the "Revision Clean Up" task - Since revision GC will only clean up revision that are older than 24 hours by default then to make it clean up more recent revisions do the following:

    1. Stop all AEM cluster nodes
    2. On each node, modify the DocumentNodeStore config file under crx-quickstart/install and set the versionGcMaxAgeInSecs property to 3600 which is 1 hour
    3. Start the leader AEM node
    4. Go to this url _/libs/granite/operations/content/maintenance/window.html/mnt/overlay/granite/operations/config/maintenance/granite_daily
    5. Start Revision GC
    6. Monitor the logs until it is done
    7. See here for relevant log messages https://gist.github.com/andrewmkhoury/39b69daf5a097b53937e
    8. Start the other AEM cluster nodes * Tar (only) - Backup the segmentstore and run offline compaction

  6. Monitor the logs to validate when Revision GC completes successfully. * grep VersionGarbage error*

  7. Migrate the data from the old AEM cluster node to the new one

  1. Migrate users and groups (If users were not impored automatically via LDAP) Package users and groups (2 separate packages) on the old system (excluding admin and anonymous OOTB users)

  2. Go to CRXDE lite app /crx/de/index.jsp and log in as admin user (on the old system)

  3. Go to "Tools" => "Query"

  4. In the bottom "Query" box enter this query to find the admin user: /jcr:root/home/users//element(*,rep:User)[@rep:principalName="admin"]

  5. Click "Execute" and copy the path of the admin user node in the results to a text file

  6. Repeat step 3 with a query for anonymous user: /jcr:root/home/users//element(*,rep:User)[@rep:principalName="anonymous"]

  7. Click "Execute" and copy the path of the anonymous user node in the results to a text file (so now you should have two paths, one for "admin" and one for "anonymous")

    For example:

    • /home/users/Q/QY5FIMXeQIbGpwZtQ3Dv – admin user on the system where I am creating the package
    • /home/users/K/Kj1406Qo9IDODc_nk5Ib – anonymous user on the system where I am creating the package

  8. Go to the "Package Manager", http://host:port/crx/packmgr/index.jsp, and log in as admin

  9. Create a package "users"

  10. Add a filter to the package config for /home/users with these exclude rules (on the /home/users filter):

    • exclude /home/users/.*/.tokens
    • exclude /home/users/Q/QY5FIMXeQIbGpwZtQ3Dv
    • exclude /home/users/K/Kj1406Qo9IDODc_nk5Ib
    • exclude /home/users/a/admin
    • exclude /home/users/a/anonymous
    • exclude /home/users/system
    • exclude /home/users/geometrixx
    • exclude /home/users/media
    • exclude /home/users/projects
    • exclude /home/users/mac

  11. Build the package

  12. Download the package

  13. Unzip the package zip file on your computer

  14. Open the file META-INF/vault/filter.xml in a text editor

  15. Add mode="merge" to the <filter ...> tag, for example:

```
<?xml version="1.0" encoding="UTF-8"?>
<workspaceFilter version="1.0">
  <filter root="/home/users" mode="merge">
    <exclude pattern="/home/users/.*/.tokens"/>
    <exclude pattern="/home/users/Q/QY5FIMXeQIbGpwZtQ3Dv"/>
    <exclude pattern="/home/users/K/Kj1406Qo9IDODc_nk5Ib"/>
    <exclude pattern="/home/users/a/admin"/>
    <exclude pattern="/home/users/a/anonymous"/>
    <exclude pattern="/home/users/system"/>
    <exclude pattern="/home/users/geometrixx"/>
    <exclude pattern="/home/users/media"/>
    <exclude pattern="/home/users/projects"/>
    <exclude pattern="/home/users/mac"/>
  </filter>
</workspaceFilter>
```
  1. Re-zip the modified package contents so it includes the change
  2. Create a "groups" package that contains a filter rule /home/groups
  3. Repeat steps 11-14 for the groups package
  4. (Upgrade only) If performing migration to newer AEM version then install a fresh local AEM instance (with nosamplecontent) and install the users package then the groups package there. Then perform an in-place upgrade on that instance. After upgrading then repackage the users again then the groups again.
  5. Install the users package on the new system
  6. Install the groups package on the new system
  7. Install the application and other required files
  8. Install the application including all configurations
  9. Package the replication agents from production, install the package on the new instance and disable the agents
  10. Package any extra items like /etc/designs and other items
  11. Validate any backend and third party integrations
  12. Install custom lucene property indexes, make sure all indexes have been applied and fully indexed
  13. Do testing (functional and load testing) to validate that the application works perfectly
  14. Create package using a search for pages that changed since the copy was done
  15. Install either one of these tools for packaging the changed content:
  16. Create package using a search for tags that changed since the copy was done. Use this query for tags (but change the date): //element(*, cq:Tag)[@jcr:created > xs:dateTime('2015-09-16T00:00:00.000-05:00')]
  17. Create package using a search for assets that changed since the copy was done. Use this query for assets (but change the date): //element(*, dam:AssetContent)[@jcr:lastModified > xs:dateTime('2015-09-16T00:00:00.000-05:00')]
  18. Create package using a search for pages that changed since the copy was done. Use this query for pages (but change the date): //element(*,cq:PageContent)[@cq:lastModified >= xs:dateTime('2015-09-16T00:00:00.000-05:00')]
  19. Install all the packages to the AEM instance (install the tags first, then the assets then pages)
  20. Account for any other data that might of changed (for example, new users added to the system or other content)
  21. Remove the disabling of OSGi component config that we did in step 3 above to re-allow audit trails and tag validation.
  22. (Mongo to Mongo only) Remove another replica from the original mongo cluster, wipe out the database there and instead add it to the new replica from step A.
  23. Remove the replica node from the set: http://docs.mongodb.org/master/tutorial/remove-replica-set-member
  24. Validate that no other nodes in the set consider that node to be part of the set anymore
  25. Drop the aem database on that node http://docs.mongodb.org/manual/reference/command/dropDatabase/
  26. Add the node as a replica http://docs.mongodb.org/master/tutorial/expand-replica-set/
  27. Do testing (functional and load testing) to validate that the application works perfectly
  28. Validate that the environment is fully in sync and, if needed, repeat step D above with a later date / time as needed
  29. Perform repository maintenance
  30. Take a short outage window and do the cut over to the environment by simply clearing the dispatcher cache and pointing it to the new environment. If no dispatcher is used, then point the load balancer to the new AEM environment instead of the old.

Alternative upgrade approach for CQ5.x to AEM6.x upgrade using Oak level crx2oak tool documented here: http://dev.day.com/content/ddc/en/gems/deep-dive-into-aem-upgrade-process.html

@anupdugar
Copy link

anupdugar commented Nov 19, 2015

Regarding "Disable OSGi components to reduce overhead"
The link (http://adobe-consulting-services.github.io/acs-aem-commons/features/osgi-disabler.html) from acs commons advises that we need to mention pid of the components.

/apps/mysite/config/com.adobe.acs.commons.util.impl.ComponentDisabler.xml

<?xml version="1.0" encoding="UTF-8"?>
<jcr:root xmlns:sling="http://sling.apache.org/jcr/sling/1.0" xmlns:cq="http://www.day.com/jcr/cq/1.0"
    xmlns:jcr="http://www.jcp.org/jcr/1.0" xmlns:nt="http://www.jcp.org/jcr/nt/1.0"
    jcr:primaryType="sling:OsgiConfig"
    components="[pid1,pid2]"
    />

components is an array of the OSGi component PIDs to disable.

Is this correct? Wouldn't pids differ from instance to instance? Or just adding component names as array would work?

@andrewmkhoury
Copy link
Author

andrewmkhoury commented Dec 16, 2015

Hi @anupdugar The PIDs will match from instance to instance as long as the component you are disabling is not a factory instance. You can find the pid by clicking on the component in the /system/console/components UI. Make sure to re-enable the components after the migration.
By the way, you can also configure the list of OSGi components to disable via the /system/console/configMgr/com.adobe.acs.commons.util.impl.ComponentDisabler UI. Then the config would get saved in the repo under /apps/system/config/.

@anurag-aem
Copy link

anurag-aem commented Aug 12, 2016

Great post

@andrewmkhoury
Copy link
Author

andrewmkhoury commented Aug 15, 2016

Thanks @anurag-aem If you are using this documentation then I just updated it with some extra details for migration from CQ5.x.

@alexkli
Copy link

alexkli commented Feb 24, 2017

Tag validation can be disabled in http://localhost:4502/system/console/configMgr/com.day.cq.tagging.impl.JcrTagManagerFactoryImpl (instead of disabling the TagValidatingEventListener via osgi-disable).

@akrivitzky
Copy link

akrivitzky commented Apr 6, 2017

Hi Andrew. Thanks for the instructions. I was confused by the wording in Step 17 of Migrating Users and Groups. It led me into the weeds for a bit. Once I realized my misinterpretation, I forked this, clarified the instructions, and improved the formatting. I don't see how you can do a pull request with Gist, but you can see it here:
https://gist.github.com/akrivitzky/5b59cee567062149713e88340e8135ec
You might want to review and update yours. Thanks!

@andrewmkhoury
Copy link
Author

andrewmkhoury commented Apr 13, 2017

@alexkli and @akrivitzky thanks for the updates. I have updated the gist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment