Setting up Watchful Hub

Watchful Hub is a server-side component that enables features like collaboration, S3 integration, and MLOps platform integration. It can be run directly on a machine, within a docker container, or even orchestrated as part of a workload on a scheduler (like kubernetes). If you're running Watchful Hub, you are likely in contact with Watchful Support - your support point-of-contact can help you with setting it up properly for your environment.

With Watchful Hub, you and your team members have the ability to collaborate on the same project. This means you can share anything within that project, such as datasets, classes, hinters, and hand labels. Moreover, each team member can push and pull their progress to the Watchful Hub. Here, we will walk through the most basic Watchful Hub setup processes:

By default, Watchful-Hub accepts traffic on 0.0.0.0 which makes it easy to run, but is inherently insecure unless run behind a firewall or VPC. This design choice was made to let Watchful Hub play nicely with existing security infrastructure without requiring much ongoing duplicated effort.

We STRONGLY recommend that Watchful-Hub be run ONLY in a trusted environment with network access correctly limited to only those who need it, and should not be made accessible ‘publicly’ to the outside internet. e.g it should never be accessible from a public IP address. However, if you do need to run Watchful-Hub in an untrusted environment while making it accessible and secure (e.g: with TLS), there are a few options:

In order for Watchful-Hub to function properly, you will need to set a few environment variables prior to running:
CUSTOMER_ID : this value will be provided to you by your Watchful representative.
WATCHFUL_KEY : this value will be provided to you by your Watchful representative.
WATCHFUL_SECRET : this value will be provided to you by your Watchful representative.

Additional optional environment variables are listed below.

To authenticate and acquire the Watchful-Hub docker image, you will first need to install the AWS-cli (https://aws.amazon.com/cli/)

Then add an entry to your ~/.aws/credentials file: e.g

[company-name]
aws_access_key_id =  <omitted, WATCHFUL_KEY>
aws_secret_access_key = <omitted, WATCHFUL_SECRET>

setting [company-name] to an appropriate name for your organization.

Then it is a matter of issuing the following commands in a terminal:

Running Watchful-Hub In Docker

$ aws ecr get-login-password --profile company-name --region us-west-1 | docker login --username AWS --password-stdin 610410161133.dkr.ecr.us-west-1.amazonaws.com
$ docker run --name watchful-hub \ 
-d \
-p 9005:9005 \
-e CUSTOMER_ID=<omitted, provided by Watchful> \
-e WATCHFUL_KEY=<omitted, provided by Watchful> \
-e WATCHFUL_SECRET=<omitted, provided by Watchful> \ 
610410161133.dkr.ecr.us-west-1.amazonaws.com/watchful-hub:latest

Running Watchful-Hub On a VM

## transfer Watchful-Hub to your VM
scp watchful-hub <virtual-machine-ip>:/usr/local/bin
## log on to your VM and start the Watchful-Hub service:
ssh <virtual-machine-ip>
cd /usr/local/bin && nohup ./watchful-hub &

Watchful-Hub should be running on the host machine on port 9005 in both cases (VM or Docker)

Where Files are Stored on Watchful-Hub

Watchful-Hub stores project files and datasets in $USER/remote/
In Docker this will default to /root/remote
You can mount a volume to point to this directory by running the container with the flag
-v $HOME/watchful-hub:/root/remote

Environment Variables

Environment Variable

Description

Required

CUSTOMER_ID

Provided to you by your Watchful Representative. Necessary to receive updates.

Yes

WATCHFUL_KEY

Provided to you by your Watchful Representative. Necessary to receive updates.

Yes

WATCHFUL_SECRET

Provided to you by your Watchful Representative. Necessary to receive updates.

Yes

AWS_CUSTOMER_BUCKET

Necessary for exporting datasets to AWS S3.

No - to run Hub.
Yes - for exporting to S3.

AWS_ACCESS_KEY_ID

Necessary for exporting datasets to AWS S3. This credential key must have write access to AWS_CUSTOMER_BUCKET.

No - to run Hub.
Yes - for exporting to S3.

AWS_SECRET_ACCESS_KEY

Necessary for exporting datasets to AWS S3. This credential secret must have write access to AWS_CUSTOMER_BUCKET.

No - to run Hub.
Yes - for exporting to S3.

AWS_REGION

Necessary for exporting datasets to AWS S3. This is the region AWS_CUSTOMER_BUCKET is located in. e.g. US-EAST-1.

No - to run Hub.
Yes - for exporting to S3.


Did this page help you?