Setting up Yadle » Scanning Files » Increase Scanning Speed
For each Scan Path added, an individual process is created as shown on the Scanning Activity Insight page. It is possible (and useful!) to create additional overlapping Scan Paths pointing to different lower level directories in order to spawn more processes. This in turn provides a narrower scope for the scanning processes and thus can complete scanning more quickly.
For example, let’s say you have mounted two network storage devices under the Data Root /yadle. These are mounted as /yadle/storage1 and /yadle/storage2. The most basic Scanning setup would be a single Scan Path pointed at /yadle. This will create a single process that scans all files under this directory. A better approach would be to create additional Scan Paths specifying both /yadle/storage1 and /yadle/storage2. To achieve even faster scanning speeds, you can create additional Scan Paths for /yadle/storage1/sub-directoryX, /yadle/storage1/sub-directoryZ, /yadle/storage2/sub-directoryW, and /yadle/storage2/sub-directoryY.
In the previous example, there would be a total of 7 added Scan Paths:- /yadle
- /yadle/storage1
- /yadle/storage2
- /yadle/storage1/sub-directoryX
- /yadle/storage1/sub-directoryZ
- /yadle/storage2/sub-directoryW
- /yadle/storage2/sub-directoryY
We recommend monitoring your system resource consumption as more Scan Paths are added. There will be a balance in the number of Scan Paths (processes) your system can handle effectively.
During initial scanning of your files, it is a good idea to create many Scan Paths as described. As progress is monitored and initial scans completed (See View Scanning Activity) – Scans can be disabled while leaving the main Data Root scanner running – only looking for new files or changes. This is not necessary, however can reduce the load on the underlying system.
Obtain Near Real-Time Scans
If there are specific directories that undergo frequent file additions or changes, add those as separate Scan Paths to achieve near real-time scans.
As an example, let’s say that you have just created loads of content for a project and are preparing to ingest into your organization’s storage device. Adding the project folder as a new Scan Path will allow Yadle to quickly detect these files and add them to the catalog.
Yadle provides the flexibility to allow you to manage what gets scanned, in addition to the speed which scanning can occur.