Simplified Index Management

Overview

Note: this feature is currently in Beta mode.

If you have any issues or questions, please open a ticket.

Simplified Index Management simple way to define custom indexes and customize out-of-the-box (OOTB) indexes, using one JSON file. There is no need to copy definitions, or to explicitly define versions. Customizations of index definitions are automatically merged with the latest out-of-the-box index, and if needed, a new index version is automatically created.

Indexes created using simplified index management have a property "mergeInfo" that links here. For more information about indexing, see: Content Search and Indexing.

Run Modes

There are two modes of operation. While the diff.json file is the same in both modes, the configuration and usage of simplified index management is slightly different.

Immediate Mode

Pipeline Mode

Immediate Mode

This is used when running AEM locally, or when using RDE (Rapid Development Environments) for AEM Cloud Service, without pipeline. In this mode, the changes are applied immediately, and the pipeline is not used.

Requirements: AEM as a Cloud Service 2025.03 (release 24893) or later, with Apache Jackrabbit Oak version 1.92 or later.

Changes in CRX/DE to the diff.index / diff.json file immediately create a new version of the respective index, and immediately reindexing starts in the environment.

ⓘ Notice

If the respository is large, reindexing can take some time. Do not use this mode for large repositories. Specially, do not use this mode for DEV environments in AEM as a Cloud Service.

Configuration

Open CRX/DE.
Navigate to the path /oak:index.
Create a new node named "diff.index" of type "oak:QueryIndexDefinition".
Then set the property "type" to "disabled".
Create a new node named "diff.json" of type "nt:file".
Copy the content of the diff index into this file.
Save the changes.
Wait a few seconds, and refresh.
You should see a new version of the respective index that was created.

Pipeline Mode

This is used for cloud service, for Dev, State, and Prod environments with a pipeline.

The feature is enabled for all AEM releases (this is because the "diff.index" is processed outside of AEM).

Configuration

In the Git repository of your program, in the directory ui.apps/src/main/content/jcr_root/_oak_index, create a new directory named diff.index. Inside it, create a new file called .content.xml, with the following content. You will not need to change this file later. It is needed currently for compatibility with the existing tooling. (The "type" property is set to "lucene", otherwise the pipeline run will fail. The includedPaths and queryPaths need to be set to a non-existent path, to avoid that existing tooling will build a large index.)

    <?xml version="1.0" encoding="UTF-8"?><jcr:root
        xmlns:jcr="http://www.jcp.org/jcr/1.0"
        xmlns:nt="http://www.jcp.org/jcr/nt/1.0"
        jcr:primaryType="nt:unstructured"
        type="lucene" includedPaths="/same" queryPaths="/same" async="async">
    <diff.json jcr:primaryType="nt:file"/></jcr:root>

Then create a new file called diff.json in the same directory. Copy the content of the diff index into this file. Also, in the file /ui.apps/src/main/content/META-INF/vault/filter.xml, add the following line:

        <filter root="/oak:index/diff.index" />

Deploy the Diff Index

Run the pipeline to deploy the diff index. This will merge the diff index with the latest out-of-the-box index, and if needed, a new version is automatically created.

Tools

Index Definition Analyzer: create a diff index from existing indexes.
Diff Index Verifier: verifies the correctness of the diff index.

Step by Step Migration Guide

This guide assumes you already have an AEM as a Cloud Service program with custom or customized indexes.

Create the Diff Index

To create the diff index, open the AEM as a Cloud Service "Developer Console". Select "Status Dump": "Oak Indexes", and "Output Format": "JSON". Click "Get Status", and then click on the "Download" button at the top right. This will open the raw JSON file in a new tab. Select the whole content, and copy it to the clipboard. Then open the Index Definition Analyzer. and paste the content into the "Index Definitions (JSON)" textarea. Click "Analyze". On the right side, you will get the diff index. Verify it contains the changes you want to apply. If you manually changed the diff index, you can verify it by opening the Diff Index Verifier.

Example diff.json File

The following is an example of a diff.json file. It contains a customization for the damAssetLucene index, and a fully custom index named "custom.slingFolderNodeNames". For fully custom indexes, the name needs to contain a dot. The index name does not contains the "/oak:index/" prefix. Also, fully custom indexes need to have an "includedPaths" property that is not empty, and set to a path that doesn't include "/apps" or "/libs". It is best to set the "queryPaths" to the same value as the "includedPaths".

{
    "damAssetLucene": {
        "indexRules": {
            "dam:Asset": {
                "properties": {
                    "test1": {
                        "name": "test",
                        "propertyIndex": true
                    }
                }
            }
        }
    },
    "custom.slingFolderNodeNames": {
        "async": [ "async" ],
        "compatVersion": 2,
        "evaluatePathRestrictions": true,
        "includedPaths": [ "/content/dam" ],
        "queryPaths": [ "/content/dam" ],
        "type": "lucene",
        "indexRules": {
            "sling:Folder": {
                "indexNodeName": true
            }
        }
    }
}

Versioning and Removal

Each merged index gets a name like <baseName>-<productVersion>-custom-<customerVersion>. When a new merge produces a different result, the customer version is incremented.
A mergeInfo and mergeChecksum property are added to every merged index.
Old versions of the same base index are removed after the new version is created.
If a previously merged index (one with a mergeInfo property) is no longer referenced in the diff, the merged index is removed.

Merge Rules

The following rules apply when merging a diff with an OOTB index:

Top-Level Property Merging

The following rules apply to the properties at the top level (properties of the index):

includedPaths, queryPaths, and tags are merged (union of old and new values) rather than overwritten, if the property already exists. The properties includedPaths and queryPaths may not be added if they don't exist yet (to avoid making the index more restrictive).
In addition to the above, the following properties may also not be added to an existing (OOTB) index: selectionPolicy, valueRegex, queryFilterRegex, excludedPaths. They are dropped with a warning.
Other new properties that don't exist on the target are added.
Properties that already exist may not be overwritten.

Example:

{
    "acmeAssetsLucene": {
        "tags": [ "additionalTag" ],
        "indexRules": {
            "acme:Asset": {
                "properties": {
                    "newProperty": {
                        "name": "newProperty",
                        "propertyIndex": true
                    }
                }
            }
        }
    }
}

Indexed Property Rules

When adding a new child under a properties node, the merger first tries to match by name value: if an existing child has the same name, the diff is applied to that child (even if the child node name differs).
Similarly, it tries to match by function value for function-based index properties.
If no match is found, a new child node is created.
The following properties may not be added to an existing child node, and may not be updated: isRegexp, index, function, name. This prevents changing how an existing property is indexed.
Existing property values may not be overwritten, except for boost and weight.

Merging Details

In addition, the following rules are used when merging. In most cases, they are irrelevant, but listed here for completeness.

Before comparing indexes, a cleaned and normalized version is produced: properties like reindex, reindexCount, refresh, seed, :version, :nameSeed, :mappingVersion, merges, mergeInfo, and mergeChecksum are stripped.
Oak-style string prefixes (str:, nam:, dat:) are normalized away for comparison.
The jcr:primaryType property is silently ignored at the top level.
Technically, rules in the "Indexed Property Rules" also apply to other child nodes, however in practice these properties are only useful in the properties child node.
All jcr:uuid properties are removed (new UUIDs are generated when needed).
Property order is ignored during comparison (properties are sorted alphabetically), but child node order is significant.
A SHA-256 checksum is computed on the merged definition to detect changes efficiently across runs.

Help

If you have any issues or questions, please please open a ticket.