Feedback, conversations, and interviews. What our users taught us about Paced Social Annotation before it was a thing.

Hi there to our instructors, designers, admins and more! As Hypothesis prepares to release a new feature, I thought I’d give you a little peek into what happens behind the scenes when we’re trying to figure out the best tools to build.

The idea for a new feature can come from a lot of different places: looking at market trends, looking at our usage data, or just our imaginations. But the best place we get new feature ideas is from our users. Our newest feature, Paced Social Annotation, came directly from the requests we get, even if it wasn’t called that at first. Here’s a little look into how a feature moves from requests to a real plan. 

How we get feedback

Hypothesis is close with our educator community, and we hear from you in a number of ways. And we listen! We get feedback through our twice-a-year email satisfaction survey, our support tickets, and the conversations we have with instructors, instructional designers, and other folks at colleges and universities. When we get this feedback, we record it in a variety of ways, but we try to preserve the actual text as much as we can. This is helpful down the road when we think we see a trend; it’s useful to verify that people are talking about what we think they’re talking about.

Wait, what are they talking about?

Sheer numbers aren’t the only factor in deciding what gets built, but they help. Along the way we’ll classify our data, hoping we’re categorizing it in a way that’s accurate but still generalizable enough to find trends. And when we looked over data spanning a few years, we found an unexpected trend. Categorizing the data had one piece of feedback stand out, which can be roughly summarized as “I want students to annotate without seeing each other’s annotations, and then later on I want to show them each other’s annotations”. Sometimes this came up as discussion board functionality, sometimes it was more explicitly stated as the ability to hide student annotations, and sometimes it was described in less direct terms that still seemed to boil down to a hide and reveal feature.

Hide…

So project Hide/Reveal started up. We gathered up the requests with contact information attached to them and started reaching back out by email, asking if we could grab some of their time for an interview. I want to pause for a moment and say that user interviews are probably the most enjoyable part of my role at Hypothesis, because I get to hear directly about what’s going right, and what’s not, with our product in their classroom. Though it might be the most enjoyable part because it’s usually going right. :)

On these calls we cover a lot of ground. I try to leave room to hear what they want to say, while also getting into some specific areas. Getting to explore what instructors are trying to solve starts at the feedback level I mentioned earlier, but digging in one to one with someone really helps make the problem clearer, and exposes the commonalities and differences that will let us design something that’s useful for lots of different people. We can also get past the problem and into the feature itself a bit: how do they envision using a hide and reveal feature in their class? What exactly would they like to be able to do when the annotations are hidden, and what should students be able to do? Do they need it to work with other integrations (like our LMS Groups integrations, or our Auto-grading feature)?  We might meander a bit along the way; when we do I get to learn more about how they’re using Hypothesis in their class already, and what other features they’d like to see. (I think the lesson here is that if you’d like to put your ideas for Hypothesis right into our heads, reach out and do an interview with us!)

…and reveal

So, across the interviews, what did I learn? 

Instructors wanted to find new ways to encourage student replies, and to encourage students to revisit the document in the assignment. By focusing on their own annotations in the first read, and by only being able to add their replies on a subsequent read, we’re hoping to reduce the cognitive load a bit on students and help them focus on the particular stage of the assignment they’re working on.

I also found that the interviewees wanted to scaffold the annotation process for students by letting them annotate in the private way they were used to before opening this up to actual social annotation. The transition from annotating entirely privately to annotating in front of your teacher is already a big step. Including your peers in that equation can feel like a lot for students. Giving instructors flexibility in setting expectations for students (assignment 1: teacher’s eyes only; assignment 2: delayed sharing; assignment 3: pure social annotation with everything shared up front) would be a big win here.

Finally, they also wanted to make sure that students had a chance to take a pass at a document on their own before being asked to think about their peers’ thoughts, sparking more original individual work. Part of this is LLM-resistance. It’s harder to copy others’ work when you can’t see it. Part of it was the cognitive load I mentioned earlier. Part of it might be close reading strategies, or just more pedagogical flexibility in the classroom.

We also discussed some technical nitty gritty: how many additional passes should students take through the document (pretty much just one), how should it interact with grading (there’s a need for in-app due dates), and how should the reveal mechanism work (manual is ok, calendar was preferred which is why it’s on our roadmap to be delivered as soon as we can). 

That’s a very high level summary, but it’s an important step since we’re going to end up with a single feature, hopefully without too many different buttons. Because while I may have, as a child, imagined myself one day standing in front of a sci-fi apparatus made entirely of blinking lights and switches, in reality apps are usually a lot friendlier when you don’t have to work your way through fifty options on a settings page just to get an assignment working.

The end beginning.

I have extreme gratitude for everyone who gave feedback on this in one form or another, because it led us to a plan. A well-fleshed out plan that would lead us to actually building the feature. That involves things like resource documents, engineering plans, wireframes, etc. Stuff we can maybe discuss another day, if you’re interested.

If you are interested, or if you’d like to give us your feedback, please send us a message. We’ll be in touch!

Happy teaching!

Michael DiRoberts

Head of Product, Hypothesis