Introducing MadCrime, a Django-based scraper of incident reports released by the Madison Police Department
tl:dr - I can finally say I made a Django application, and it’s live on the web and available on GitHub. For now I’ll call it MadCrime, and it scrapes Madison Police Department incident reports and displays the content to the user.
It’s been more than a year in the making, but I can finally say I built and deployed a Django project.
The project itself isn’t overly complicated, but it has a high ceiling and most of all has given me some confidence in my practice habits and ability to retain concepts.
For now I’m calling it MadCrime, and its purpose is to scrape Madison Police Department incident reports, store them in a database and display the content to the user with a map of the incident. It also offers access to the JSON of the incidents using Tastypie.
There are limitations; some from the data, more based on my abilities. For instance, suspect, arrest and victim details are entered into one field, and I would need to either manually separate out, or figure out how to parse out the details into separate fields. More importantly, this table is not the comprehensive list of Madison Police Department incident reports, but rather a snapshot that is released to the public.
But with the possibility of the open data ordinance being passed and implemented that might change, so in some terms the life span of MadCrime as it is right now is likely limited. Which is why it was important for me to “get this out the door” somewhat quickly and call it version 1 as opposed to sitting on it while learning how to implement search, or make the main display sortable.
The code is on GitHub, and I have a nice long list of potential improvements, including adding search functionality and separating out information like suspect information that is contained in one singular field in the incident report.
The idea to scrape police incident reports and map from CityCampMadison last month, but the follow-through and execution has been informed and inspired by so many different projects and thought-provoking discussions, many of which I’ve mentioned in previous posts:
-
HackingMadison, which is dedicated to “highlighting the civic projects, datasets, resources and people that make Madison better.”
-
Christopher Groskopf’s Tyler Sirens project.
-
Ben Welsh’s story-writing algorithm proposals in his talk at the 13th International Symposium on Online Journalism, and web scraping walkthrough, not to mention his Crime L.A. project.
-
Andy Boyle’s Firetracker walkthroughs.
-
Kevin Schaul’s Web scraping with Django tutorial.
-
Jonathan Stray’s discussion last month around bringing a new model for crime reporting.
-
My days time covering crime and courts in Chicago’s South Suburbs while working as Illinois Editor for The Times.
This little program won’t write the story itself, had the data been available I could have used a tool like this to create shells for briefs that could have been edited down so I could move on to focus on larger stories.
And with some simple search capabilities, maybe even find some similarities between crimes and suspects.
