Archives For Data Updates

Earlier this week, we sent out a tweet as sort of a fishing expedition: “Political scientists: if you’ve collected precinct-level election results in your work & are willing to share, get in touch!”

kankakee-2000-resultsMike Sances did. An assistant professor of political science at the University of Memphis, Sances let us know that as part of his dissertation research he had obtained precinct-level results files for many counties in Illinois. In some cases, the files went back more than a decade. Even better, he was willing to share them with us.

Thanks to Sances, and to the National Science Foundation, which funded his research, we’ve begun posting the files – many PDF documents but some HTML and text files – to our Github repository for Illinois sources.

We won’t have every county, nor every election, but this generous contribution to the project is exactly the sort of thing we were hoping to see when we asked. While OpenElections will be processing results data for state and federal races, we’re also posting these files to allow people to find local race results as well – and to avoid requesting the same materials that Sances did for his research. These are public documents; they should be available to the public.

And we’re renewing our call to political scientists: if you have precinct-level election results, either in original format or that you’ve digitized, we’d love to hear from you about making them a part of OpenElections.

 

 

The Price of OpenElections

December 25, 2015

As we wrap up OpenElections’ work in 2015, we’d like to give you an update on how we’ve spent not only our time but also the money that we’ve gotten, particularly from the John S. and James L. Knight Foundation’s Knight News Challenge. Most of that money that we’ve spent since mid-2013 has gone to salaries for our project manager and a single developer. Neither of the project’s co-founders has been paid for working on OpenElections, and we’ve tried to keep our operations pretty lean.

While our initial grant funding from Knight is nearly exhausted, we’ve made good progress and will keep going. In the past few months we’ve added a few more states (Louisiana, Missouri and Virginia) and we have volunteers working on Wisconsin, Georgia and Oregon, among others. We’ve revised our volunteer documentation to make it easier to understand what we’re doing and how you can help.

In most states, getting county-level data isn’t too much of a problem; that data is usually available online, if not always in native electronic formats. County-level data is usually freely available as well, but we’ve always wanted to develop a resource that can offer precinct-level results where they are available. Here’s why: while counties can be homogenous, precincts are even more distinct and smaller political units and lend themselves to more sophisticated analysis. Candidates and their campaigns care about precinct results. Journalists and researchers should do the same.

Some states make precinct-level data available for free, which is a great service to the public. They include Louisiana, Maryland, Wyoming, West Virginia, Virginia, North Carolina and Florida, among others. Some states, like Pennsylvania, Colorado and Utah, charge a nominal fee for precinct results. But for other states, precinct results are only available county-by-county, and that takes both time and money. We’ve written about Oregon in the past, and we’d like to offer it as an example of the price of precinct results.

The bad news is that it’s not uniform, even within a state. In Oregon we’ve spent more than $1,000 to obtain precinct-level results covering elections 2000-2014, although in many cases we don’t have all of those years. Some counties literally don’t have results to give us before 2010. Crook County was unable to find precinct results for the 2010 general and 2012 primary elections, while a number of counties don’t have elections from before 2008. In other cases, price was a factor: we’ll only have precinct results for 2010-2014 from Tillamook County because the clerk there charged us $222.75 for results for those years. Lake County charges $50 an hour for pulling the results files and another $0.25 a page for copying them. We’ve yet to receive those results, so we don’t know what the final cost for Lake will be.

The good news is that when we do request election results that aren’t freely available online, we’re posting them on our Github site in state-specific repositories. That way other organizations or individuals won’t have to repeat our processes and/or pay for results that we’ve already gotten. We want you to use what we’ve gathered, whether that’s CSV files or original PDFs. That’s our holiday gift to you. We’ll be back at it in 2016, when there are more elections coming up, we hear.

Offline Election Results

September 7, 2015 — Leave a comment

In some states, getting election results data is pretty easy (looking at you, Maryland, Wyoming and Florida, to name three). In others, it’s a matter of going county-by-county, as we’ve previously written. But in most of these cases, we’ve been dealing with results files that are available online.

But what about cases where the results aren’t on the Internet?

Continue Reading...

The Oregon Paper Trail

June 14, 2015

IMG_3129

A few weeks ago, we began requesting precinct-level election results from counties in Oregon. The Secretary of State maintains county-level results, typically in electronic PDFs, so to get down to precinct level we need to ask county clerks across the state. Many of them post precinct results on their sites, but some don’t, so we emailed a few to ask for results from 2000-2014. In doing so, we were prepared to pay reasonable fees for them, as the Oregon Revised Statutes permit.

Local officials were quick to get back to us in every case, and their responses were straightforward. Here’s an example, from Art Harvey, the Josephine County clerk and recorder:

The reports you are interested in are available in PDF format.

The cost would be $10.00 per election.

Other counties charged fees ranging from $25 (Umatilla County) to $45 (Wasco County) to $86.50 (Linn County, which sent us paper print-outs of election results that we’ll be scanning). And then there’s Tillamook County, where Tassi O’Neil, the county clerk there, has set a price of $664 for PDF copies of precinct-level results for elections from 2000-2014.

We wondered how that price was calculated, so we asked. Ms. O’Neil responded:

The fee for each election is $3.75 locate fee and then .25 cents per page.  That is the fee if it is a paper copy or if we send it in a PDF.  That is the charge that the Oregon Revised Statues say that we can/or should charge.

That is true, but there are two points here: One is that members of the public are being charged for pages of an electronic document. There are no paper copies involved here. The other is that the Oregon Revised Statutes also say this:

The custodian of any public record may furnish copies without charge or at a substantially reduced fee if the custodian determines that the waiver or reduction of fees is in the public interest because making the record available primarily benefits the general public.

As OpenElections is a non-profit effort dedicated to publishing machine-readable election results that can be freely used by anyone, we’re pretty sure that our project primarily benefits the general public. We’ve asked for such a waiver or reduction of the $664 and are awaiting a reply. Oregon law also permits us to appeal a denial of a fee waiver or reduction to the Attorney General, and we will be pursuing that option should it become necessary.

In the meantime, we’ve been converting Oregon PDF results to CSVs and will continue to do so. There are plenty of ways for you to contribute to that effort, and we welcome any suggestions or advice on our dealings with Oregon officials.

Scraping Nevada

January 28, 2015

ICYMI, Derek Willis wrote a piece for Source about his experience scraping Nevada precinct results. Check it out!

By Derek Willis

OpenElections is nearly two years old, and we’re not nearly done yet. In most states we still have a lot of work to do.

As our initial funding from the Knight Foundation winds down, we wanted to provide an update on where the project is and our plans going forward. The first thing to know is this: OpenElections is here to stay. Our timetable has expanded, and we’re looking at other sources of money to boost our capacity to process election results data, but the work we’ve done so far and your interest in it has convinced us of the need.

When we started, Serdar and I had between us years of experience working with election results data in multiple formats. We both worked at news organizations that routinely dealt with different types of data and various election systems.

We still have been surprised by the diversity of results that we’ve found. States like Pennsylvania, North Carolina and Florida have consistent and reliable data across time. Other states, like Arkansas, Colorado and Washington, have different formats and systems depending on the year. Then there are states like Mississippi and New York, which have required significant investments of time and effort.

In practice, that has meant a lot of work within individual states in order to load and process data from 2000 onward. Those efforts have taken more time than we anticipated, for two reasons. First, we have found that states have switched the systems and software they use to publish election results, in some cases multiple times in the past 15 years. We have found some abstractions – we released a separate library to handle states that use Clarity’s software – but in many cases this meant writing several different custom parsers for a single state.

Second, machine readable data is not a universal standard, and for many states it is a recent addition to their practices. This isn’t a criticism as much as it is a statement of reality. Officials from nearly every state we’ve been in contact with have been helpful and even supportive of the project. But we’re also not too far removed from all-paper elections, either.

In response to these factors, we’ve made some adjustments. The main one is to publish “raw” results data from states even before we standardize offices, candidates and parties. We think having election results in a fairly consistent format across a number of years is pretty useful, so we’re not going to wait until everything is done to release that. This week we’ve published raw results in North Carolina, Florida, Pennsylvania and (for recent elections) Mississippi. You can download these from our site or clone them from GitHub depending on your needs. We’ll continue to follow that path as we work on standardization.

Along the way we’ve been very fortunate to have had contributions from volunteers, who both gathered information about the state of election results and also contributed code to the project. We can’t thank all of you enough for your interest and contributions. This would be a much longer road without them, and we hope that you’ll stay involved.

We’d also like to recognize the people who have lived this project with us for most of the past two years. Geoff Hing has been the main point of contact for web development volunteers and has written the bulk of the code that powers the results loading and data display portions of the project. Geoff began a new job at The Chicago Tribune this week, although he’ll still be involved with OpenElections as a volunteer. We’re extremely grateful for his efforts.

Many more of you have emailed with or spoken to Sara Schnadt, the project manager for OpenElections. She’ll be with us through the end of the year as we plan our next steps, and her organizational skills, creative thinking and ability to wrangle two co-founders living on separate coasts has made OpenElections possible.

Investigative Reporters & Editors, a source of training and inspiration for journalists for decades, has made things easy for us by handling the accounting and grant management tasks. Both Serdar and I are proud to be “graduates” of IRE, and we’re thankful for their support of OpenElections.

The goal of OpenElections – to provide access to machine-readable, standardized election results – remains the same as when we began. The path to reach that goal is now a lot clearer than it was two years ago, and with your help we’ve learned a lot about how to get there. We’ll keep moving forward, and invite you to stay involved.

 

 

Eating Our Dog Food

July 15, 2014

By Derek Willis

When Serdar and I first talked about building a national collection of certified election results, we had a very specific audience in mind: the two of us. It seemed like every two years (or more frequently), one or both of us would spend time gathering election results data as part of our jobs (me at The New York Times, Serdar then at The Washington Post). We wanted to create a project that both of us could use, and we knew that if we found it useful, others might, too.

Precinct comparison

The New York Times

In the world of software development, using your own work is called eating your dog food, and we’ve done just that. While we’re nowhere near finished, I am happy to report that OpenElections data has proven useful to at least half of the original intended audience. Both last week and this week, The Upshot, a new politics and policy site at The Times that I work on, used results data from Mississippi collected by OpenElections to dig into the Republican primary and runoff elections for U.S. Senate. The analyses that Nate Cohn did on voting in African-American precincts would not have been possible using the PDF files posted by the Mississippi Secretary of State. We needed data, and we (and you) now have data.

We’ve completed data entry of precinct-level results for the 2012 general election and the 2014 Republican primary runoff elections, plus special elections from 2013, and we’re working on converting more files into data (we just got our first contributions from volunteers, too!). These are just the raw results as the state publishes them; we haven’t yet published them out using our own results format (but that’s coming soon for Maryland and a few other states). We provide the raw results for states that have files requiring some pre-processing – usually image PDFs or other formats that can’t be pulled directly into our processing pipeline.

The Mississippi example is exactly the kind of problem that we hoped OpenElections would help solve, and it’s only the beginning for how election results data could be used. Once we begin publishing results data, we’d love to hear how you use it, too. In the meantime, if you have some time, there’s more Mississippi data to unlock!

Home Screenshot

As we get the first few states’ data processed and ready to release, we are building an interface to deliver it to you, and to show our progress as we go. The live site (above) now shows metadata work to date, and the volunteers involved. Soon you will be able to toggle between this view and a map of the current condition of results data (below). Clicking on each state will show you details on the most cleaned version of available data.

Data Map

The color coding on this new map will change as we get more states online with ‘raw’ results, and fully cleaned data (what you see here is hypothetical and just to illustrate how the map will work). When we say ‘raw’, we really mean results that reflect the data provided by state elections officials. These results are only available at the reporting levels provided by the states and fields like party and candidate names have not been standardized.  These results do have standardized field names, so you will be able to more easily load and analyze the data across states. We will get as many states fully cleaned as we can, but our baseline goal for this year is to wire up and make available most of the data in a ‘raw’ state.

As we build the data interface, we would love to know what you think. Is the terminology we are using clear to you? Is the interaction clear? Is there anything else you would like to see here?

Download Page

If you click on the ‘Detailed Data’ link on the data map page, you will get to this download page showing all the races for the state you have chosen. You can download results at a variety of reporting levels, depending on what is available for this state. We will include rows in this view for all processed data (clean and raw) as well as any races we haven’t processed yet, just so that you know they exist.

Above the download table there is a slider that both gives you an overview of all the races available for a state, and a way to select just a specific date range for which to browse detailed results. You can filter results by race type – such as President, Governor, State Legislative races, etc.. If there are any other ways that you need to access the data, or if anything about this interface could be clearer, please let us know!


We will be building out a preliminary version of this interface in the next couple of weeks, and will revise it further based on what we hear from you.  

To tell us what you think, comment on interface elements here, or email us at openelections@gmail.com.

Announcing FEC Results

November 18, 2013

By Derek Willis

The Federal Election Commission isn’t the first place that reporters might expect to find election results. The agency is known for providing data about the financing of campaigns – who is giving and getting money – and for deciding how to enforce campaign finance laws.

fec

But election results? There’s no direct link to the FEC’s section on congressional and presidential results from the site’s home page, but every two years the commission compiles certified results for primary and general elections from official sources, and then publishes spreadsheets and PDFs containing the data.

Today we’re announcing the release of FecResults, a Ruby gem for retrieving and parsing the data contained in those files, as well as a JSON API for that data.

OpenElections, like the FEC, collects official results data from state election officials, and we’re looking for as much detail as we can get. The FEC data, by contrast, is just concerned with the race-wide totals for House, Senate and presidential contests. So why would we make the FEC results data available?

One reason is that for many users, race-wide data for federal elections is useful enough, and some of the breakdowns that the FEC provides for presidential races are particularly useful (state-by-state electoral and popular vote totals, for example). Another reason is that we see this data as helping us fulfill the mission of OpenElections, which is not only to publish election results data but to make it easier to link to other kinds of information.

The FEC results data (2004 example) includes unique identifiers for most candidates within each election, which are assigned by the FEC for fundraising purposes. Most state election results sites don’t include this identifier, since it has little use for state authorities. Publishing the FEC data will enable us to use it as a reconciliation service so that we can connect state-produced data with the federal ID system, and thus with federal campaign finance data where possible. Having the certified totals is also a way to check our state data gathering processes, too.

Some technical details: the FecResults gem makes use of the RemoteTable library, which handles all kinds of tabular data in files on the Internet. Since the spreadsheet files aren’t always consistent from year to year, we needed to customize the process of grabbing and parsing the data into Ruby objects. A separate gem, FecResultsGenerator, publishes the data as static JSON files, which are then published via GitHub Pages.

We had a great turnout last Saturday for our first Metadata Sprint. Volunteers joined us from cities around the country including Brooklyn, Chicago, Louisville, Milwaukee, Minneapolis, Philadelphia, and Phoenix – plus our bi-coastal OpenElections team hosting from D.C., San Francisco and L.A..

We worked diligently for three hours and between 13 of us collectively entered 193 elections into our system, making progress on 15 states, and almost doubling the number of elections entered so far.

Ahead of the pack were Milwaukee Journal Sentinel news app developer Allan James Vestal, with 29 elections entered, and MIT Electronics and Computer Science student and news app developer Joanna Kao, with 22 elections.

A big thank you to both of your for being OpenElections Rock Stars!

JamesJoanna

@allanjvestal                         @joannaskao

We also had lively discussions about the quirks in electoral processes in various states including stumbling across a 2011 race in Arkansas in which one person voted from the Hot Springs airport, all sorts of unusual candidate names (real and otherwise) including ‘Grand-Pa’ Goshorn in Arizona in 2010, and various colorful stories about candidates and races from the beats of reporters in the group.

We also got great feedback on our Metadata admin, and flushed out some great edge cases for the nuanced and wildly varied ways that people vote from state to state that we hadn’t accounted for in our admin. Surveying local electoral processes across the country is a very colorful and telling process. While we have a better idea of the scale of the task in front of us, we also learned that we can do it. A reminder: we still have several states up for adoption, so if you’re interested in joining the effort, let us know.

Thank you to all of you who came out! Let’s do it again soon!