References » Reprocess All Files » Agent microservices

In some scenarios, it may be necessary to run the agent microservices against all files and not only based on file changes or new files. When this special set of microservices are started, each one will reprocess every relevant file scanned by Yadle. Upon completion, the individual microservices will stop and exit automatically.

Why Reprocess All Files?

It is typically not necessary to invoke this set of containers, however there are cases where it makes sense to do so.

Some examples of when it may be necessary to run the agent microservices against all files:
  • There is a new filetype which Yadle supports generating a thumbnail for.
  • Thumbnails were not generated for some 3D file types due to not enough memory to render, or Maya ssh was not configured to create thumbnails for .ma and .mb file types.

In each of these scenarios, the files would have been scanned by the Yadle agent, however they would not have any thumbnails. There also won’t be any change events to trigger the thumbnailing microservice to reprocess. Thus, the agent microservices need to reprocess every file.

When invoking this set of agent microservice containers (running in “allfiles_mode”), the original microservices (running in “changes_mode”) will continue to process new files and changes. Once the microservices running against all files have completed, the containers will stop automatically.

If you have been instructed by someone at Yadle to start this set of microservices, refer to the steps below.

Starting Agent Microservices On All Files

Step 1: Start Yadle agent microservices in “allfiles_mode”.

Go to agent microservices directory and run start_yadle_all script.

$ cd /opt/yadle/microservices/agent
$ sudo ./start_yadle_all

Step 2: Check for running Yadle containers.

You should now see the allfiles_mode containers running alongside the other Yadle containers. Confirm this by checking Docker status.

$ sudo docker ps -a

The following Yadle containers should now be running in addition to the standard Yadle containers. Refer to the Starting and Stopping Yadle to see the list of standard Yadle containers.

NAMES
<org>_allfiles_mode_red-thumbnails_1
<org>_allfiles_mode_exif_1
<org>_allfiles_mode_ocr_1
<org>_allfiles_mode_video-thumbnails_1
<org>_allfiles_mode_medium-thumbnails_1
<org>_allfiles_mode_small-thumbnails_1
<org>_geo_allfiles_mode_geometry-thumbnails_1

Stopping Agent Microservices On All Files

The allfiles_mode containers will exit one by one as they complete the re-processing of every scanned file with Yadle. If it’s necessary to stop the allfiles_mode agent microservices before they have completed there is a separate stop_yadle_all script for that. Restarting the allfiles_mode containers again will resume where they each left off. They will not start over from the beginning.

Step 1: Stop Yadle agent microservices in “allfiles_mode”.

Go to agent microservices directory and run stop_yadle_all script.

$ cd /opt/yadle/microservices/agent
$ sudo ./stop_yadle_all

Step 2: Check for no running allfiles_mode containers.

Confirm there are no running allfiles_mode Yadle containers by checking Docker status.

$ sudo docker ps -a