Using the New Wildcard URL Search API to Monitor Sitewide Annotation Activity

In New Search Capability: Find All Annotations on Any Website, Katelyn Lemay introduced a new capability developed by Hannah Stepanek: wildcard URL search. Katelyn’s sample query, https://hypothes.is/search?q=url:https://elifesciences.org/*, populates the Hypothesis dashboard with (currently) 2300 annotated documents. Here are some other fun examples:

New York Times stories in 2017 (q=url:https://www.nytimes.com/*2017*)

New York Times opinion columns in 2017 (q=url:https://www.nytimes.com/*2017*opinion/*)

As Katelyn notes, these wildcard queries are based on a new API endpoint, wildcard_uri, which is a key enabler for services that monitor entire websites, or subsites, and report new activity.

There isn’t yet an official Hypothesis service that does that kind of monitoring and reporting, but we’ve upgraded a do-it-yourself solution, h_notify, to make use of wildcard_uri and the results have been enlightening. For example, here’s some chatter between Katelyn and Nate Angell regarding the above-mentioned post:

The quoted selection — “Go to https://hypothes.is/ if you want” — is crossed out because Katelyn has implemented Nate’s suggestion. And that interaction produced these alerts in an internal Slack channel.

The Slack channel is fed by an instance of h_notify that watches our website for Hypothesis activity and posts annotations or replies to Slack. Because the watcher knows the internal Slack IDs corresponding to handles like @katelyn and @nateangell, those Slack users were notified when @mentioned in the annotations. It’s a really powerful way to collaborate on the web content they’re responsible for.

If you want to try this for yourself, start with examples.py and tailor as needed. For @mention support, create a slack_namemap.json file as shown in the README. Ideally you’ll run the code on a server, but in a pinch you could even run it on an always-on-and-connected desktop computer.