Hand Labeling

Creating Hand Labels in Watchful

Watchful has two methods of hand labeling Focus Mode and Query Hand Labeling. Best Practices for these features are to use Hand Label Focus mode as the primary method as it contributes to the base rate calculation and validation and utilize Query Hand Label Mode as spot checking for canary values.

Hand Label Focus Mode

Typically, the first step of using Watchful is to hand label to initialize a base rate. This gives Watchful a sense of how likely it is to run into candidates of a given class. You can start this process by navigating to the Hand Label tab on the top nav bar.

Full Text Classes

To hand label Full Text classes, simply select a class (or create one by typing a class name into the class selection), view the candidate displayed, and hit either "Yes" or "No" depending on whether or not this candidate belongs to the class. Once the candidate is hand labeled, a new, randomly chosen, unlabelled candidate will be displayed.

NER Classes (Not available in Watchful Community Edition)

If you have selected an NER class from the class selection dropdown the UI will change allowing for you to select as many examples of the class in the block of sample text provided by Watchful. Note: Rather than displaying discrete rows of data for NER hand labeling, Watchful samples one kilobyte of text from the dataset.

1. Begin by either clicking or highlighting the spans that belong to the class.
2. After highlighting a span, click the plus sign (+) to create the entity.
3. Continue until all examples are created.
4. Removing a label can be done by reselecting the label and clicking the trash icon to delete.
5. Click the Submit button

Hand Label History (Full Text Classes)

Within each Full Text class in Hand Label Focus Mode a hand label history is tracked. You may go back at any point and correct any hand label(s) here by clicking on the candidate in the Hand Label History side bar and reselecting Yes /No. The hand label will update in the side bar with the new value.

Query Results Hand Labeling (Full Text Classes)

Query Result Hand Labeling is intended to be used as “canary values” based on the current query results and will be treated as validation examples in Watchful. We do not recommend using this as your only method of hand labeling as this mode will not affect base rate and instead to utilize the Focus Hand Label mode that applies random sampling to calculate base rate. This mode is best utilized as a method of fine-tuning for rare examples of the class, leveraging the Confusion Index, and spot checking mistakes.

Note: Query results hand labeling is only available for Full Text classes.

To toggle this mode click the lightning bolt icon next to the Query Results table header. You can select Yes/No for any candidates that are appropriate. We do not require you to label the entire page.
Click the same expanded toggle to exit the mode.

📘

Hand Labeling Best Practices

We only recommend hand labeling for base rate calibration until the peak of the base rate chart has stopped fluctuating.

Imported Hand Labels

See section in Datasets

Best Practices

• Spend as little time hand labeling as possible. Hand labeling simply slows down the process of labeling for a class. It's useful to help calibrate the base rate, or to assess label quality, but outside of that it is strictly slower than building hinters
• Stop labeling for base rate once the graph stops shifting drastically. Watchful has a reasonable estimation of the base rate once the peak of the graph stops shifting drastically as you hand label.