[Editors note: Today, our own Dr. Pamela Gay is giving a presentation at a meeting on citizen science in Tucson, Arizona. Where possible, she’ll be live streaming from the meeting. Here is her planned talk.]

This talk is a story of doing citizen science, discovering the power of coupling with edutainment, and of trusting ourselves, while building a relationship of trust with our scientists, citizen scientists, and team members.

Since this is a conference on lessons learned, I’m going not only tell you what works, but also what fails.

Within CosmoQuest, we define Citizen Science as science that is enabled through everyday people who make meaningful contributions that enable science that couldn’t otherwise be done. Citizen Scientists, within the context of Ethical Philosophy are Knowledge Producers, and if we do our jobs right, they have the ability to also be knowers who understand what they are contributing toward.

For those of you who don’t know me, my background is as an astronomer who deals with big data sets and writes software. I also like things that are variable. I started my career at the age of 16, studying active galaxies and their variability at the Special Astronomical Observatory’s 6m. I went on to use VLA data to analyze TTauri stars at 17, and then spent the next 7 years giving my research over to pulsating variable stars. In 2003, a move to Boston got me involved with the American Association of Variable Star Observers and eventually converted me into a mentor to their RR Lyrae crowd, and a Board Member from 2005-2011. In working with the AAVSO, I realised the limitless potential of these volunteers and the science they can accomplish with sufficient time, determination, and support. 

During this time, fully online citizen science was becoming part of astronomy with StarDust@home in 2006 and Galaxy Zoo in 2007. Astronomy across the world was also uniting to plan the 2009 International Year of Astronomy. As chair of the IYA New Media Task Force I traveled the world meeting people and building programs. One of these people was Chris Lintott, and it was while drinking coffee in the elevator lobby at Oxford Astrophysics that we came up with the idea to build a platform that would make spinning up a citizen science projects as easy as spinning up a blog with WordPress. Two large proposals later, with me as PI with NASA, and him as PI with a foundation in the UK, the Zooniverse was born.

While the bulk of my time is generally split between developing software for astronomy and new media, and producing content and research with that code, one of my passions and skills researching motivates people to spend their spare time and spare money doing science. As part of my research for the Zooniverse, I mentored  graduate students Georgia Bracey and Jordan Raddick studying the motivations of Galaxy Zoo users, and graduate students Bracey and Shanique Brown to study the motivations of Moon Zoo participations.

For these projects, neither of which was accompanied by substantial educational content, we found that motivations centered on wanting to contribute, being interested in the topic, and wanting to make a discovery. In this table the motivations are listed from most common to least common. Learning was only motivating for a couple percent, and community – something that is important to me – wasn’t of importance to hardly anyone. 

This hints at a link between design and motivation.

Beyond studying why people participated, we also looked at why people left. Put simply, they felt lost, were afraid they were making mistakes, and didn’t sense they were part of an active community.

With Moon Zoo, these insecurities were reflected in the data we were getting; data we couldn’t use.

As liason to the Moon Zoo team, I got an ear full from the LRO PI on how he hated how the data was presented, and another earful from folks upset that the boulders icon looked like a pile of poo. And no one was happy with the results.

After a team meeting at LPSC 2011, we realized changes were needed. My staff worked to create educational content and a better Moon Zoo tutorial, but change was slow and there was a lot of concern that educational content would distract people from doing science.

The problems we were having with Moon Zoo were a concern for a lot of people. We had projects planned for Mercury, Vesta, Mars, New Horizons, and HST. As an experiment, my team in Edwardsville created a second software package – that php based Citizen Science software that can be setup in moments that I’d dreamed of. We used it to launch Ice Hunters, and with a wordy interface that let users know when their results were confirmed by other users, we were able to move past the problems our research had pointed out. But – we’d fragmented the Zooniverse codebase into two different software languages and design philosophies. 

At the same time this was happening, Fraser Cain was on my case that citizen science sites weren’t inviting participants into the ivory tower – we weren’t giving them the seminars, the resources, the mentoring that we give our junior colleagues. We were, to use Kantian ethics terms, using them as merely a means to an end, and that isn’t ethical. 

And I listened. In September 2011, Fraser and I brainstormed what a citizen science site could look like if we use new media technologies to give our volunteers as close to the same experience to our students as possible.

After a long conversation with Chris Lintott, and with funding from Messenger, Dawn, LRO, and HST, as well as two ROSES grants, we launched CosmoQuest in January of 2012. The original idea was to be a sister to the Zooniverse that took on the harder, more tedious tasks associated with surface science.

And as Fraser and I planned, we launched with opportunities to learn side by side with opportunities to do science.

Our Goals were 4-fold

1. Treat volunteers as much like we’d treat our students as possible.
2. Build an enduring community that is responsive to changing needs. This means never assuming we know what’s best for our volunteers better than they know what’s best for themselves. We will use iterative design, and work to constantly improve our outcomes by constantly improving our content and using new technologies.
3. We will always be honest and choose what is more ethical rather than more popular. There are a lot of things I could say to our people – like “You’re now an official mission team scientist”  – that would increase click throughs. There are a lot of things I could do, like not requiring logins, that would increase participation. But these click throughs and participation numbers are at the risk of betraying trust as people realise they aren’t listed anywhere as mission team members, or see that their data is being used with out them having ever given consent or given a name that can receive credit.
4. Finally – everything we do will be toward accomplishing science, either advancing research about worlds, or in the development of new machine learning algorithms.

In meeting our goals, we ended up with a wordy site, filled with examples and instructions, and linked to educational content.

And it worked. Using more statistics than I could have done, Stuart Robbins examined variations in mapping between a group of 8 professions and a myriad of volunteers. When we compared the aggregate results of both groups, there was 1 to 1.01 correspondence.

We’ve been able to repeat these results with project after project.

And, our goal of building a community of people who come back over and over again was also fulfilled… for awhile. In this graph, we show the percentage of users each week who are new, returning after 1week, 4 weeks, 12 weeks, half a year, or after a year or more. During the first year, you can see how the long term users fill in as time ticks by, you can also see an abrupt loss of our old community in what turns out to be November of 2016. 

These kinds of violent community changes don’t always show up in a plot of images versus time, because it’s possible to convince your community to just do more images if you develop better communications. This doesn’t showup in a plot of users per day, because you can get better at recruitment.

This graph… When your goal is to build a community that lasts, this sudden change stabs you in the soul. And this is where the lesson of trust comes into play.

This is where I’m going to step back and show you what happens when you combine sociology and data science.

This graph plots user number versus time, with each person’s contributions plotted out horizontally. The slop of the left edge represents the rate at which people join, with a vertical line showing a massive user influx, and a flat line meaning a dead community. The size and color of the points reflects how many images per day a person processed, with large and red being ideal.

In looking at these 6 years of data, we identified the source of every inflection and change in behavior. I can tell you we succeeded when we leveraged press releases, when we showcased science, when we had good design, and when we partnered media content with citizen science.

Our project faltered when we stopped doing what was right for the community: those times when we were doing what funders demanded instead. When our bottom line became our bottom line we bottomed out.

I don’t have enough time to go into all we learned, but I want to focus on the single most important lesson we learned – listen to your users and beware of paid by the hour consultants. (See point 10 in image above).

In 2015 we were awarded a Cooperative Agreement to grow CosmoQuest to serve NASA’s needs. We were told that before our site could be linked to by NASA, we had to make it beautiful. Since we had an evaluation firm hired to test our design, we asked for their input. 

This is what we started with.

Our evaluation company said that in 2016, the new standard was to go to nearly monochrome layouts, to remove all possible text, and to open apps like our citizen science app in float-over divs instead of in their own browser tabs. 

Following their recommendations, we switched to this. We weren’t sure it was the right thing to do – it just felt wrong – but we knew we needed to get listed on NASA’s citizen science pages if we were going to be competitive with the other NASA projects – our grass roots campaigns wouldn’t be enough – so we did as asked, got ourselves posted, and put out a press release.

And our entire community went away. Even our huge influx of new users went away.

It’s hard to remember in the moment that we are the experts, and in matters of taste, we are the ones to trust. I was there, innovating fully online citizen science in the beginning, and I was the one writing the research papers on motivations of users while working on the websites and the science.

I now have this quote hanging above my desk. It’s from director Brian Grazer: “You have to trust yourself, not research. Not testing. Testing helps, but you have to trust your own taste. If your taste says something isn’t any good, don’t let research rationalise that out of its own truth.”

We lost site of our goals. We lost trust in ourselves. We lost our community.

Under normal circumstances we would have cursed, worked our asses off to redo the site as fast as we could, sent out an apology to our old users, and everything would have been fine. We’ve broken things before and gone down with Amazon. We know how to recover.

But we were in a weird situation, and to risk TMI, my entire team had just mass quit our jobs and was switching institutions due to our team being subjected to a hostile work environment, including title IX and X complaints being made. 

This institutional transfer led to us getting told we couldn’t change anything on our site without NASA HQ approval, and having all our educational programs defunded. Our new institution, concerned about consequences to them if we screwed up, decided the day before we were supposed to launch a new project, that we couldn’t launch anything without our outside evaluation firm approving of what we did – that’s the same firm who suggested our catastrophic design. I was required to convert a programmer’s worth of salary into an expanded evaluation budget, and was mandated to use additional donations to my project to have the evaluation firm project manage the launch of the project (don’t hire a usability firm by the hour to project manage what they’ll be testing… which you will also pay for by the hour). That project just went terribly.

Put simply, we were fucked.

And this is when we learned that if you use all your non-grant funded podcasts to daily and weekly beg people to do citizen science, you can actually build a new audience. 

That’s point 11 on the graph.

In the musical Hamilton there is a line I have written on my white board: dying is easy, it’s living that’s hard.

As a project, we decided to do the thing that is hard. We decided to live.

And when we realised we couldn’t meet our goals, that our team scientists and staff weren’t being paid on time, that we couldn’t do right by our volunteers, and that we were only keeping things going by begging people on new media… my entire team quit their jobs a second time. That is when we came to PSI.

Unfortunately, in the 3 days between putting in my resignation and Mark being able to look into transferring my grant, our funding was cancelled.

But we’re still alive. Yes, living is hard, but we are starting to thrive at PSI.

We’re now working on a paper to document everything we learned. The working title is “An overly honest look at the factors impacting citizen science participation.”

We also have a paper on the ethics of citizen science that was submitted yesterday, and a paper on CosmoQuest motivations that is going in as soon as final formatting is done.

Today, we are funded through donations, corporate sponsors, and a contact with the OSIRIS-REx mission. We are returning to our wordy roots, and as a result, we once again see our community staying and sciencing. 

We’re still not fully there, but we’re getting there. With our newest project Bennu Mappers, we’re seeing people complete images that take 15 minutes to an hour to map out, and the work they’re doing is good. This image is from a volunteer who joined during a fellow streamer, Skylias’, live stream. He shared it to ask “is this right” and I’m happy to say, yes Joe, this is right.

We found our community.

As I’m writing this, I can see that 5,845 people (of which, at least 6 are me running tests) have registered for Bennu mappers. Of those folks, 3,002 saw just how many rocks are in every image on Bennu, and went on to accept the challenge of mapping them all. Most folks have done between 10 and 50 images, and a few percents of have done more than 1000 images! To those people – they are our heroes. All told, our community marked:

• 3,214,993 rocks,
• 1,158,474 boulders,
• 26,500 craters.

And the data is good.

This meeting is supposed to be a meeting on lessons learned. Well, if you want to do hard problems, if you want to do tedious science, and if you want to build a community that will stay with you for a few years instead of a few images… you have to educate them, you have to build a community of trust and that includes listening to your audience and being responsive. Sometimes, it means quitting your job and losing all your funding so that you can get on track, and make sure a spacecraft called OSIRIS-REx can find a safe place to grab a soil sample.