Building Your Form and Database

ProPublica built on the software tool we created for our large-scale partnerships and released it open source. It’s meant to be a turnkey solution that you can use to build your own project without having to start from scratch or hire a developer. It's called Collaborate, and it's available here.

The project is a collaboration with the Google News Initiative, which also provided financial support. The software currently supports Google Forms, which is the tool we’ll use for our examples. It also supports data submitted using Screendoor, as well as data in Google Sheets spreadsheets you have access to and CSVs.

From here on out, this guide is going to assume your project is based on crowdsourced tips.

The foundation of a crowdsourced data project is your tip submission form, so you need to think very carefully about what data points you need, the phrasing of your questions and the terminology you use. The wording of your questions requires reporting just like anything else you publish.

We highly recommend doing research and consulting with experts about those three points. ProPublica’s Ken Schwencke based the form we use in ProPublica’s Documenting Hate project on other crime victimization surveys, and he talked with social scientists who helped make a form that could support serious reporting down the road.

And just like anything else ProPublica publishes, the form went through editors, the copy desk and legal review.

We also recommend keeping the form brief and simple; if there are too many questions, people will give up before they finish.

Try to get it right the first time. Although as a technical matter you can change the form later, it adds lots of complexity to do so. If you’re not careful, changing your form might require that you split your respondents into groups whose answers really can’t be compared because the questions were worded differently (or they answered different questions altogether).

That having been said, there may be good reasons to — very carefully — change your form later on. Changing the form could help take advantage of feedback loops and to respond to reporting plans becoming more focused or to new facts coming to light. “I am a fan of updating and tweaking as reporting moves forward on different projects,” Ariana Tobin, ProPublica’s engagement editor, said. “You want to be transparent about it, of course.”

At a minimum, you will likely want to capture:

  • Dates, locations and descriptions of the thing that happened.

  • The tipster's name and contact information.

  • Space to include additional links or attachments.

You should also consider any categories that you can use for filtering within the database of responses. For example, Documenting Hate allows tipsters to choose location type (school, private property, etc.); reasons they were targeted; and whether they reported the incident to police.

You should also think carefully about the language on the page that comes before the form. Keep it brief, but be clear about what you plan to do with submissions, how you’ll follow up and what you’ll do to ensure tipsters' privacy.

For example, here's the form we use for Documenting Hate, and here's the form we used to report on separated families at the border. Both are available in Spanish.