Installing Yadle » Self-Hosted » Installation Guide

For Self-Hosted deployments, there are four components that will be installed:

  1. Yadle server: Provides the Web app, backend database, API, and ElasticSearch.
  2. Yadle server microservices: Performs all AI and non-AI file tagging, applies filetype icons for files that don’t receive thumbnails, and generates image sequence scrubbable thumbnails.
  3. Yadle agent: Scans files.
  4. Yadle agent microservices: Extracts embedded metadata and generates all thumbnail previews.
Note:
  • The Yadle Self-Hosted installer is a collection of bash scripts that will execute the steps needed to download, deploy, and configure Docker containers on your host machine.
  • A pre-install check will run first, making sure the host system has all required packages installed and necessary system resources.
  • The installer can be run multiple times without issue. Each time it is run, it examines and determines the state of the container, repairing or configuring any elements that are not correct.
  • The scripts will output many messages. Progress messages will be displayed as the Docker images are downloaded, containers started and configured.
  • Any problems will be displayed in red text and the scripts will stop.
  • Successful installation will output docker status and “Installation Completed”.

1. Download Yadle Self Hosted Installer

a. The Yadle selfhosted installer can be downloaded from https://download.yadle.com/installer/yadle_installer.tar.gz or using curl in terminal.

$ curl -O https://download.yadle.com/installer/yadle_installer.tar.gz

b. Once downloaded, extract the contents of yadle_installer.tar.gz.

$ tar -xzf yadle_installer.tar.gz

2. Copy Install Config File to Yadle Self-Hosted Installer Directory

Yadle will provide you with a install_config file that contains installation parameters and keys specific to your organization.

Copy the install_config file to the extracted Yadle installer directory.

$ cp install_config ./yadle_installer/

3. Copy Certificate and Key to SSL Directory

Yadle Self-Hosted deployments are configured by default to use https protocol for secure communication with the web app, API server, and database.

a. Create the ssl directory that will hold your certificate and key.

$ sudo mkdir -p /opt/yadle/servers/config/ssl

b. Copy your .crt certificate to the ssl directory. The certificate file must follow this naming convention: <org>.com.chained.crt.

$ sudo cp <your_certificate_name>.crt /opt/yadle/servers/config/ssl/<org>.com.chained.crt

c. Copy your key .pem to the ssl directory. The key file must follow this naming convention: privatekey.pem.

$ sudo cp <your_key_name>.pem /opt/yadle/servers/config/ssl/privatekey.pem

d. Generate a dhparam.pem file in the ssl directory by running the following command:

Note:
  • This can take several minutes to complete.
$ sudo openssl dhparam -out /opt/yadle/servers/config/ssl/dhparam.pem 4096

4. Run Yadle Servers Installer

The Yadle server installer will execute two separate scripts in the following order:
  1. Pre-install check
  2. Install Yadle servers

The script must be run as a sudoer or root user.

$ sudo ./install_yadle_servers

5. Run Yadle Server Microservices Installer

The script must be run as a sudoer or root user.

$ sudo ./install_yadle_microservices --selfhosted --install_type=server

6. Run Yadle Agent Installer

The Yadle agent installer will execute two separate scripts in the following order:
  1. Pre-install check
  2. Install Yadle agent

The script must be run as a sudoer or root user.

A. Standard Yadle agent installer. 
$ sudo ./install_yadle_agent
B. Yadle agent installer for Qumulo file data platforms. 
$ sudo ./install_yadle_agent --qumulo
Note:
  • You can pass the following command line options or enter them when prompted by the installer.
  • --qumulo_host=<IP/FQDN of a Qumulo node>
  • --qumulo_user=<username to access Qumulo API>
  • --qumulo_password=<password for user>

7. Run Yadle Agent Microservices Installer

The script must be run as a sudoer or root user.

$ sudo ./install_yadle_microservices --selfhosted --install_type=agent

8. Confirm running Yadle containers

Run the following command to verify all Yadle containers have been started and are running. If running separate server and agent nodes, then containers prefixed with ‘agent’ should be on the agent node, and containers prefixed with ‘server’ should be on the server node. The command must be run as sudoer or root user since Docker is being invoked.

$ sudo docker ps -a

The following containers should be listed in output:

NAMES
<org>_agent_changes_mode_video-thumbnails_1
<org>_agent_changes_mode_small-thumbnails_1
<org>_agent_changes_mode_exif_1
<org>_agent_changes_mode_ocr_1
<org>_agent_changes_mode_medium-thumbnails_1
<org>_agent_changes_mode_red-thumbnails_1
<org>_agent_geo_changes_mode_geometry-thumbnails_1
<org>_yadle_agent 
<org>_server_changes_mode_imagehash_1
<org>_server_changes_mode_tags_1
<org>_server_changes_mode_aitags-colossus_1
<org>_server_changes_mode_aitags-krell_1
<org>_server_changes_mode_filetype-icons_1
<org>_server_changes_mode_filesequence-thumbnails_1
<org>_server_changes_mode_normalization_1
<org>_yadle_es
<org>_yadle_servers

9. Mounting File Systems to Scan with Yadle Under Data Root

After successful installation of the 4 Yadle components (server, server microservices, agent, agent microservices), the next step is to mount the desired file systems to be scanned by Yadle under the specified Data Root Path (default is /yadle).

Let’s imagine you have three network volumes that you want to scan with Yadle:

/nas/images
/server/data/videos
/ssd_array/admin/local

The above three network volumes are mounted on the Yadle data root path /yadle with the following mount points:

/nas/images              —> /yadle/images/
/server/data/videos     —> /yadle/videos/
/ssd_array/admin/local   —> /yadle/local/
Note:
  • The Yadle agent only requires read access to mounted filesystems. Best practice to mount as read-only.
  • It is recommended to add these mounts to /etc/fstab so that they are mounted each time the Yadle agent host system starts.
  • Whenever additional filesystems are mounted under the Data Root, the Yadle agent and agent microservices need to be restarted (See Step 10).

10. Restart Yadle agent and agent microservices.

After mounting the desired file systems to be scanned by Yadle, restart both the Yadle agent and agent microservices. This is necessary for the containers to be able to access the newly mounted file systems.

a. Restart Yadle agent:

$ cd /opt/yadle/agent
$ sudo ./stop_yadle
$ sudo ./start_yadle

b. Restart Yadle agent microservices:

$ cd /opt/yadle/microservices/agent
$ sudo ./stop_yadle
$ sudo ./start_yadle

Once both agent and agent microservices have been restarted, proceed to the Setting up Yadle section of the online documentation to begin configuring Yadle Scan Paths.