Setting up Yadle: Indexing: Increase Indexing Speed

For each Indexing Path added, an individual process is created as shown on the Indexing Activity Insight page. It is possible (and useful!) to create additional overlapping indexing paths pointing to different lower level directories in order to spawn more processes. This in turn provides a narrower scope for the indexing processes and thus can complete scanning more quickly.

For example, let’s say you have mounted two network storage devices under the Data Root /yadle. These are mounted as /yadle/storage1 and /yadle/storage2. The most basic Indexing setup would be a single Indexing Path pointed at /yadle. This will create a single process that scans all files under this directory. A better approach would be to create additional indexing paths specifying both /yadle/storage1 and /yadle/storage2. To achieve even faster indexing speeds, you can create additional Indexing Paths for /yadle/storage1/sub-directoryX, /yadle/storage1/sub-directoryZ, /yadle/storage2/sub-directoryW, and /yadle/storage2/sub-directoryY.

In the previous example, there would be a total of 7 added Indexing Paths:
  • /yadle
  • /yadle/storage1
  • /yadle/storage2
  • /yadle/storage1/sub-directoryX
  • /yadle/storage1/sub-directoryZ
  • /yadle/storage2/sub-directoryW
  • /yadle/storage2/sub-directoryY

We recommend monitoring system resource consumption as more indexing paths are added. There will be a balance in the number of Indexing Paths (processes) your system can handle effectively.

During initial indexation, it is a good idea to create many indexing paths as described. As progress is monitored and initial scans completed (See View Indexing Activity) – Indexers can be disabled while leaving the main Data Root indexer running – only looking for new files or changes. This is not necessary, however can reduce the load on the underlying system.


Obtain Near Real-Time Scans

If there are specific directories that undergo frequent file additions or changes, add those as separate Indexing Paths to achieve near real-time scans.

As an example, let’s say that you have just created loads of content for a project and are preparing to ingest into your organization’s storage device. Adding the project folder as a new Indexing Path will allow Yadle to quickly detect these files and add them to the catalog.

Yadle provides the flexibility to allow you to manage what get’s indexed, in addition the speed which indexing can occur.