A Hypothesis-powered Toolkit for Fact Checkers

Overview

In Annotating the wild west of information flow we sketched one of the ways annotation can help combat the plague of fake news. The approach we imagine there—an annotation-powered toolkit that supports an emerging standard for fact checking—remains a thought experiment. But journalists aren’t the only ones who need to master the critical thinking skills and digital literacies required of fact-checkers. These skills and literacies are now required of everyone, and not only to gauge the credibility of news. We all are fishing in seas of information for facts to support evidence-based professional practices.

How do we teach people to fish? Hypothesis is collaborating with one effort to do that. The Digital Polarization Initiative (Digipo) is a template for a college course that will lead students through exercises to analyze and fact-check news stories. The pedagogical approach, described here by project leader Mike Caulfield, is evolving. In parallel we’ve been evolving a toolkit to help students research and organize the raw evidence for the analyses they’ll be asked to produce. Annotation is a key component of the toolkit, which is implemented as a Chrome extension that works closely with Hypothesis.

Here’s a screencast that shows the toolkit in action:

Related Annotations

Here’s an example of a Digipo claim to be investigated:

The North Carolina Republican Party sent out a press release boasting about how its efforts drove down African-American turnout in this election. (see New Yorker)

Assume that a student begins with none of the background knowledge required to evaluate this claim, which has its roots in the 1965 Voting Rights Act and the 2013 Supreme Court decision, in Shelby vs. Holder, that relaxed the Act’s requirement that historically racist states seek federal approval to modify their voting laws. The first order of business is to marshal some evidence. Hypothesis is ideal for this purpose. It creates links that encapsulate both the URL of a page containing found evidence, and the evidence itself — that is, a quote selected in the page.

There’s a dedicated page for each Digipo investigation. It’s a wiki, so students could manually include Hypothesis links. But fact-checking is tedious work, and they’ll benefit from any automation that helps them focus on the analysis.

So we include a Hypothesis widget that displays annotations matching the wiki id of the page. Here’s a standalone Hypothesis view that gathers all the evidence tagged with digipo:analysis:north_carolina_voter_suppression. And here’s that view included as a Related Annotations section in the wiki page:

That’s helpful, but it would be onerous to require students to copy and paste the correct tag in order to populate the widget. To streamline the workflow, the Digipo extension adds a Tag this Page helper. A student who is viewing the New Yorker article that is the seed of this investigation invokes it like so:

The Digipo extension presents a list of active Digipo investigations:

When the student selects a name, the Digipo extension creates a Hypothesis note with that name as a tag, then follows a Hypothesis direct link to the annotation:

The annotated page will now appear in the Related Annotations widget. Students can add notes to the page, and discuss it, in the annotation layer.

The Timeline

For a complex investigation with a long history, it’s helpful to arrange the evidence on a timeline. So the Digipo toolkit also injects a Timeline section into the wiki page. As with Related Annotations, the Timeline is populated by means of Hypothesis tags. Again it would be unrealistic to expect students to add tags manually in a prescribed format. So the Digipo extension provides another helper, Assign Publication Date, which is available when a date is selected on a page to be added to the Timeline:

When the student clicks Assign Publication Date the Digipo extension creates a new note, with the correct date tag — in this case, digipo:date:2016-12-12 — and again follows the resulting Hypothesis direct link. If the investigative team needs to discuss the choice of date, that discussion can happen in the annotation layer.

Here’s an example of a Timeline made from annotations with auto-assigned date tags:

Footnotes

See Footnotes in action:

Here a student has used a Hypothesis direct link to cite evidence in the form of a quote from a source page. (Hovering over the link text “boasted” reveals that the link begins with https://hyp.is/.)

A reader can follow that link to view the cited quote in context:

The Digipo site also adds a footnote that includes the quote directly in the wiki page. A reader who clicks the (1) link lands in the Footnotes section of the wiki page:

This section of the wiki page gathers quotes cited elsewhere, by way of Hypothesis direct links, and presents them as footnotes linked from the text of the investigation page.

More tools for fact checkers

The Related Annotations, Timeline, and Footnotes sections of the wiki page are supported by Hypothesis-powered options on the right-click menus of the Digipo extension. But not all of those options involve Hypothesis. Some encapsulate best practices for fact checkers.

Suppose a student is investigating a claim made on this bipartisanreport.com page, but has never heard of the site. How to orient? One helpful strategy is to Google the site. But a basic search for bipartisanreport.com will mainly find pages there. The student is looking for diverse and credible perspectives on the reputation of that site, so wants to exclude self-referential pages.

There’s an advanced query that does that: bipartisanreport.com -site:bipartisanreport.com. Here’s the result:

We can see at a glance that a couple of fact-checking sites (whose reputations can be checked in the same way) have reviewed bipartisanreport.com.

Savvy web citizens know that bit of Google syntax; student fact-checkers should too; the Google this Site helper makes it handy.

Here’s another example. Suppose you are investigating the claim, made here, that celebrities plan to stage an alternative concert during the 2017 inauguration. In this example part of that claim is selected, and the Digipo helper is ready to help:

And here’s the advanced Google query that the helper performs:

The query asks whether any of a set of fact-checking sites have evaluated this claim. In this case, you can see at a glance that Snopes has already looked into it.

The full Google query looks like this:

the event in Miami on Inauguration Day (site:www.sourcewatch.com OR site:www.factcheck.org OR site:hoax-slayer.com OR site:www.truthorfiction.com OR site:opensecrets.org OR site:www.politifact.com OR site:snopes.com OR site:www.washingtonpost.com/news/fact-checker/ OR site:digipo.io)

Next steps

Fact checking is hard work. A single investigation can require students to find, organize, and present evidence found online in dozens of HTML or PDF pages. The Digipo toolkit provides a set of helpers that enable students to power through the grunt work of assembling evidence so they can focus on producing well-written investigations.

As instructors begin to assign this work to students, we’ll continue to evolve the Digipo toolkit in order to streamline the bookkeeping and help students maintain that focus. In parallel, we plan to tailor variants of the toolkit to other fact-checking workflows.