Archives For Using the data

When we released our initial dashboard for downloading election results in July, we wanted to make it easy for anyone to grab CSV files of raw results with just a browser. We’ve continued adding states to our results site, the latest being North Carolina, Florida and — for a few elections — Mississippi. Pennsylvania will be on the way soon.

But we also wanted our results to be usable by developers as well, and we’re taking advantage of Github to help make that easier. Each time that we publish raw results data, which hasn’t been standardized beyond geography — we publish it first to a GitHub repository for that state. For example, you can find a repository for Mississippi results that can be cloned and/or accessed via API, avoiding manual downloads. The naming convention for the repositories is the same: openelections-results-{state}, and you might find partial results for states that don’t yet appear on the download map (like Iowa) because they’re still in progress.

Screen Shot 2014-12-22 at 2.18.09 PM

Using GitHub has two advantages for us — it helps to maintain a history of published changes, of course — but GitHub Pages also provides a filesystem for storing the raw CSVs that power the results site downloads. And should we need to move the CSV downloads to another location, we can do that, too. All of this underscores our commitment to using existing standards and practices rather than inventing new ones.

So if you were looking for election results CSVs as part of your holiday plans, we’ve got two ways to get them. Enjoy, and Happy New Year!

Hack with us at NICAR!

January 15, 2014

We are organizing an OpenElections hackathon on the Sunday of NICAR, from 10am til 6pm in the Chesapeake room.

If you are here at NICAR, and would like to join us after all the other activities, we would love to have you. We are looking for coders as well as anyone interested in helping out, and spending the day with our team. Here’s how you can be involved:

Track One – Build Our Interface
Help us build the first draft of a public-facing map-based interface for OpenElections. The map will show the status of the project metadata and data acquisition, as well as scraper development. In the finished version, the interface will also give users access to results data.

Track Two – Election Results Data Scraper Development
Help us extend our core scraper architecture to create a series of custom scrapers that account for the idiosyncrasies in how each state structures data, stores it, and makes it available.

**Our docs for this process are now up on our site. Look here to see what would be involved with joining in**

Track Three – Documentation and Use Cases
Help us flesh out the guides that articulate all of the processes volunteers need to know to work with us, as well as the documentation that other developers will need in order to build on our work. Also, or alternately, come by and give us your use cases! We will be collecting descriptions of how all kinds of journalists use elections data now, and how you would like OpenElections to work for you.

Your time and expertise would be most appreciated either all or part of the day.

Thank you!

The OpenElections Team

By Sara Schnadt

Panelists Sara Schnadt, Dan Melton, Elise Hu, and Anthea Watson Strong.
Image: Phil Tenser via Twitter

OpenElections was invited to participate on a session about turning civic data into civic narrative at this year’s recent Online News Association Conference. The purpose of the session was to survey the current states of affairs of transparency in civic data sets, and how journalists are using them. Hosted by Anthea Watson Strong from Google’s civic data initiative, the panel included myself representing OpenElections and Census Reporter, Dan Melton of Granicus (and former CTO of Code for America), and Elise Hu, tech and culture reporter at NPR (and part of the team that founded the Texas Tribune).

Some of the key questions that came up in this open-discussion-based session were:

  • How has the amount of available civic data changed in the past ten years, and how well are journalists making use of it?

  • What is the current state of available data  – how much of it is clean and standardized -vs-  messy and inconsistent?

  • What does a robust civic data ecosystem look like, and what are it’s components?

  • How are newsrooms adopting open source values and best practices in software and app development as well as data cleaning and analysis?

  • What can algorithms do that journalists can’t? (bubble up patterns for lead gen)

  • What can journalists do that algorithms can’t? (identify and tell a good story)

  • How can we make data sets and their interfaces ‘sexy’ so that we entice more journalists and the public in general to become more data literate?

  • Since the internet ‘trends for entertainment’ how do we deliver substantive stories online alongside the more popular entertaining ones when getting to content online is so targeted?

  • Is crowd-sourcing data an effective strategy for journalists? When is it more and less effective and how much quality-control is necessary?

  • Is access to large amounts of civic and social data, and tools to hyper-personalize it, making the culture more or less civic-minded, more or less individualistic?

  • How do you tell compelling stories with data?

  • How can aesthetics help make data more relatable and articulate the dynamics within it?

  • Is data the chicken or the egg when creating a story (do you start with an idea and then find data to support it, or the other way around)?

Illustration: Graham Clark, ONA Student Newsroom, ONA Illustrated feature

There was lively discussion on all of these subjects between the panelists and the audience (which included many journalists with deep experience on the topic). There seemed to be a general consensus that this moment is more exciting than it is challenging, and that next steps include: moving beyond civic data transparency to consistently standardized and easily available data (where OpenElections comes in); widely available tools for exploring and analyzing data; and an increased data literacy among journalists and the general public (which can be facilitated by well-designed data visualizations and great data-driven storytelling).

To learn more, you can listen to the recorded session, and follow along with the slides.
(Also see credits and further links for my slides).

An Improved Metadata API

October 23, 2013

derekBy Derek Willis

It’s not new, but it is improved. The Election Metadata API that we first released at the end of August has been revamped to make it easier to access specific objects and to provide more detail.

Rewritten using Tastypie, the API now provides a list of endpoints for individual states (for example, here is Maryland). That gives developers and users a roadmap for what kinds of information are available.

A common request could be for the elections in a given state. Continuing with Maryland, here are all the elections in the Old Line State contained in the OpenElections database. The responses include not only details about the elections but also about the source of the data (the organization in our parlance) and the state object itself.

While the initial release of the API provided the ability to filter elections by year, this updated version allows the use of specific dates to restrict the elections. For example, users can supply a date range in order to find elections held in Maryland in 2012. To drill down to only general elections, use the race_type filter. You can check out all of the possible filters at our GitHub repository.

For any request involving an object like election or organization or state, the response will include a meta object that includes the number of records returned.

The Metadata API is essentially dogfood for the project – it is mostly used to help other parts keep track of what we know about the availability and scope of election results. But if you find it useful, or have improvements you might like to see, let us know on GitHub, in our Google Group or on Twitter.

craig gilbert headshotOE: What is your beat and what kinds stories do you report on?

CG: I cover Washington for the Milwaukee Journal Sentinel but also write a blog about elections, public opinion and political trends with a pretty heavy Wisconsin focus. It’s called The Wisconsin Voter. The majority of my work is for the blog and most of what goes into the blog also runs in the print edition of the paper.

OE: How often do you use elections results in your stories, and why?

CG: I use election data all the time. We’ve had an inordinate number of elections in Wisconsin since 2010, including an explosion of recall fights without any real historical precedent in this country. Wisconsin is also typically a presidential battleground. I use election data to write about political trends nationally and inside the state. I write about turnout trends, and about the election process. Right now I’m doing a long-term research project for the paper in conjunction with a 6-month fellowship at the Marquette Law School. It’s about political polarization, and I’m relying pretty heavily on election data to make the case that metropolitan Milwaukee is an unusual and extreme example of a metro region polarized by party and deeply divided geographically between city and suburbs, with Democratic and Republican voters increasingly clustered in very partisan and very separate communities. I’m comparing voting patterns in Milwaukee to the other major metropolitan areas in the U.S. The project will look at how this came to be, and at the consequences of this kind of division.

OE: What processes do you go through now to access elections results? 

CG: I’ve purchased a lot of historical national and state data from Dave Leip’s US Election Atlas, which is incredibly useful, but only goes down to the county level. I recently discovered that Harvard and Stanford have a burgeoning project to publish precinct-level election data for most states. There’s obviously the election data that’s made available by state and county governments.

OE: How long does this take, and how many people are involved? 

CG: In the case of many of these sources, getting the data into a form that’s convenient for analysis, and building election databases around the numbers, can be labor-intensive. I generally do it myself, although for my current project I’m working with a political scientist, Charles Franklin, who is at the Marquette Law School.

OE: Do you feel like the information that you are able to access now is comprehensive and reliable for your purposes?

CG: No, not comprehensive. There are gaps, mainly when you get down below the county level and want to work with data at the level of municipalities and voting wards. The differences between how states report this information are vast. Wisconsin is pretty good, fortunately for me. You can get a lot of geographic detail. But some states are horrible. In many states, only county data is available. It’s amazing to me how difficult it is to get presidential elections results for many major American cities (as opposed to counties), for example. Some cities like Milwaukee and Chicago post this data routinely. Others don’t and all you get is data for the county they are located in.

OE: How would a project like OpenElections change your process?

CG: Anything that would standardize the data and provide the kind of geographic detail – wards and municipalities – mentioned above would be very helpful.

OE: Would our project make it possible for you to do more with election results. How?

CG: You could do a lot more to compare voting trends and patterns from state to state, city to city, etc. You could also write about voting trends at the local level, which is where people live. People don’t think of themselves as residents of counties, but of towns, cities and neighborhoods.

OE: Do you have any particular requests for our project that would make your work more effective?

CG: It would be nice if the election data were coded by county for sure, and coded in a way that allowed you to aggregate by municipality and congressional district. It would be nice to have results for president, governor and senator included in the project. And of course, all in Excel.

OE: In an ideal world, how far back would you like to have historical results available?

CG: In an ideal world, the 1970s, but that’s asking a lot. The 1990s would be nice.

Craig Gilbert is the Milwaukee Journal Sentinel’s Washington Bureau Chief and author of “The Wisconsin Voter” political blog. Gilbert has covered national and state politics for the paper since 1990, and has written extensively about the electoral battle for the crucial swing states of the upper Midwest. He is currently a fellow at the Marquette Law School working on a research project on polarization. He was a 2009-10 Knight-Wallace fellow at the University of Michigan, where he studied public opinion, survey research, voting behavior and statistics. He previously worked for the Miami Herald, the Kingston (NY) Daily Freeman and was a speechwriter for New York Sen. Daniel Patrick Moynihan. Gilbert has a B.A. in History from Yale University.