How to Trust a Crowd Without Believing Anyone

Every crowdsourced dataset has the same original sin: it asks you to trust strangers.

Not always maliciously. Someone misremembers, fat-fingers a number, reports the wrong semester, or inflates a grade to make a course they liked look better. Do it a few thousand times and what you have isn't really data anymore, just a large pile of guesses formatted to look like it.

I ran into this while building UniStats, a site where Hebrew University students could look up a course before signing up for it and see the things the course catalog never tells you: the real average, the grade distribution, the standard deviation. The catalog says "Introduction to Economics B." UniStats said: "The average was 73, and the distribution had a brutal left tail." That's the number that actually decides your semester.

To be useful, UniStats needed near-total coverage. A beautiful interface with 12 courses would be trivia; a searchable table with most of the university would be infrastructure. That meant thousands of submissions. And the moment you invite thousands of people to submit numbers, you have to answer the question that kills most crowdsourced projects: why would anyone believe the numbers?

Answering that question is what shaped the whole design, and it created a harder problem underneath.

The trap: a form that asks students for grades

The obvious build is a form. "Enter the course, enter the average, enter the standard deviation, hit submit." It works in an afternoon and it's worthless by the weekend.

• People guess. Nobody remembers the standard deviation of a course they took two years ago. They'll type something plausible.
• People fight over it. A form is an argument waiting to happen: two students remember the same course differently and now you're storing two contradictory numbers with no way to tell which is real.
• You can't audit any of it. A number typed into a box has no provenance. It could be the registrar's official figure or something someone invented on the bus. Once it's in your database they look identical.

The problem isn't the students. It's that a free-text form asks a human to reproduce a statistic from memory, and people are bad at that in the same way a language model is bad at arithmetic: they produce something that sounds right and often isn't. So I stopped asking students for the numbers.

The unlock: the numbers already existed - on a page only the student could see

Here's the thing about Hebrew University's student system at the time. Every student had a personal grades page. And that page didn't just show your grade in a course - it showed the official course statistics the registrar had computed: the average, the distribution, the standard deviation, per semester. The exact data UniStats wanted was already sitting there, authoritative, computed by the university itself.

There were only two problems. It was buried one course at a time behind a login. I had no access to it. Only the student did.

So the whole architecture fell out of a single hard constraint: I couldn't reach the data, but the person sitting in front of the screen could. So I put the code where the data was - inside the student's own logged-in browser session - via a bookmarklet.

A student never types a grade. They click a bookmarklet on their own university page, and it copies the numbers the registrar already computed and displayed. Every figure on UniStats originated as a statistic shown by the university's own system - not a memory, not a self-report.

You drag a link to your bookmarks bar once. Then, sitting on your personal grades page, you click it. It reads the course statistics straight out of the page you're already looking at and sends them to UniStats. One click, no typing, no login for me to steal, no server of mine ever touching the university's site. The student's authenticated browser session does the one thing only it can do; my code just reads what's on the screen.

I kept the boundary deliberately simple: the code ran only when a student chose to run it, on a page they were already allowed to see, and it extracted only aggregate course statistics - never personal grades or identifiers.

That single decision is what made the whole thing trustworthy. There was no normal path for a student to invent a number. The intended path wasn't "student reports a number" - it was "student carries an official number from one system to another," instead of inventing it from memory.

Three tensions I had to design around

1. Trust vs. privacy

This is the tension the whole project lived inside, and the two sides genuinely pull against each other.

To trust crowd data, your instinct is to collect more: who submitted it, when, what their own grade was, enough to cross-reference and catch fraud. To protect the people submitting, you must collect less - because the page the bookmarklet runs on is the most sensitive page a student has. It shows their own grades, right next to the course averages I wanted.

So I drew a hard line through the middle of that page. The bookmarklet reads only the aggregate course statistics - numbers that describe a course, not a person - and never the student's own grades, name, or ID. Nothing that could identify who clicked the button ever leaves their browser. A submission isn't "Yaron got a 78 in Economics"; it's "Economics, 2013 semester, average 73.3, here's the distribution" - a fact about the course that's identical no matter which of the fifty students in it clicked submit.

That constraint turned out to help trust rather than fight it, because of the next tension.

2. Coverage vs. verification

I needed two things that normally trade off. Coverage - close to every course, which means encouraging as many submissions as possible. And verification - confidence that each submission is real, which usually means being suspicious of every new one.

Because submissions are copied from the source instead of typed, those two goals stopped fighting and started reinforcing each other. Fifty students take a given course; each one's page shows the same official statistics. So when several of them click the bookmarklet, I get the same course reported multiple times - and I can cross-check the copies against each other, and against what the university structure guarantees. Agreement is the signal. A submission that matches the others is confirmed; an outlier that disagrees gets flagged instead of silently overwriting the truth. This didn't make fraud impossible - a determined person can always tamper with their own browser - but it made the normal path honest, and abnormal submissions easier to detect.

That flipped the usual crowdsourcing curse. Normally more contributors means more noise to police. Here, more submissions meant more corroboration - every extra student who shared a course made that course's numbers more trustworthy, not less. Growth and reliability pointed the same direction, which is why chasing coverage was safe. It's how the site got past 86% of undergraduate courses without turning into a swamp of contradictory guesses.

3. Reading a page that was never meant to be read by a machine

The registrar's page was built for a human to glance at, not for my bookmarklet to parse - and it showed. It was Hebrew and right-to-left, so numbers and labels didn't sit where naive scraping code expects. The same course wore different names and numbers over the years, so "the same course" arrived as three unrelated rows that had to be merged into one history. And statistics were sliced by semester, so a single click carried several data points to pull apart and label - without ever touching the personal grade one column over.

None of this is visible from a description of the page. It only showed up once I was elbow-deep in the actual markup the university served - the renamed courses, the RTL number ordering, the semester slices all crammed into one table built for eyeballs, not parsers. You can't plan for any of that from the outside. You find it by reading what's really there.

What happened after it worked

The part I didn't see coming arrived once the coverage was real. The university itself got in touch - not to shut it down, but to ask about making it official. I started noticing people who worked inside the university using it to look things up, the registrar's own numbers coming back to them through my little site. And the student body wanted to take it over and run it themselves. A tool I'd built precisely because I couldn't get at the registrar's data had somehow become something the people who owned that data were relying on too. That's the moment a side project stops feeling like a workaround and starts feeling like it filled a hole that was actually there.

If you're crowdsourcing anything

The move that mattered most was removing the text field. A number copied from an authoritative source carries its provenance with it; a number typed from memory never does. So wherever you can, let people carry a fact rather than reproduce one, and put your code where the data already lives, the way the bookmarklet ran inside the one session that could actually see the numbers.

The rest followed from that. Because submissions were copies rather than opinions, agreement between them became a usable trust signal, and more contributors meant more corroboration instead of more cleanup. And collecting the bare minimum, aggregate course stats and never personal grades, didn't cost me any trust, because trust was already handled at the source.

A form would have gotten me a database of what 20,000 students thought their courses' grades were. The bookmarklet got me a database of what the grades actually were, assembled by the same 20,000 students without any one of them having to be believed.