Is open data really that open?

Is open data really that open?

The importance of open data

More countries open up their government data. That’s good news. Governments are therefore more transparent, it thrives commercial initiatives and creates new insights which makes our world a better place. However I notice a trend where Open Data is not that open.

Crime map Toronto

Toronto police offers a crime Indicator Map which includes crimes like robbery and theft. Valuable information for decision makers and insurance companies to improve the safety of the city of Toronto. However the data provided doesn’t stimulate this at all. First of all the data can only be accessed as a viewer. In order to perform analytics and discover trends you need to have access to the raw data. Secondly the map only shows crimes of the past week. This makes this tool practically worthless. You need trends over several years in order to know whether parts of the city are getting more safe or just less safe.

The Dutch Police

The national police of the Netherlands handed us their crime statistics of the past year. Way better than the crime data of Toronto but it also doesn’t tell the whole story. Only a few crimes were being reported and postal code 4 level zones with less than a 1,000 people were left out. Car thefts were only reported at postal code 2 level. I know the data is available at neighborhood level because in the past we received the data at this detailed level. We analyzed and processed this data for an insurance companies. They didn’t use the data to increase their premium. No. They started crime awareness campaigns. With the current data restrictions these campaigns become less effective.

What is happening? First of all I notice it is fear. The crime data for instance is far from perfect. We have seen that ourselves when processing the data. Police agents on duty make mistakes when entering crimes into police databases. Wrong addresses for instance can make crimes disappear in a database. When making data public these mistakes can be discovered. Many years ago we were being discussed in the Dutch Parliament. Sounds flattering. Doesn’t it? Several chiefs of Police didn’t like that we showed police crimes to the public. They believed that the chief of Police and the Mayor should decide on whether to declare a neighborhood safe or not. Based on what. A hunch? Statistics aren’t perfect but they don’t lie. Every data analyst knows data isn’t flawless. However it is far better than a hunch.

I like data analytics and the enormous possibilities big data has. We can really change things for the good. We can better understand patterns so we know how to make neighborhoods more safe, improve traffics flows in cities and better understand diseases. However when you don’t tell the whole story things can go horribly wrong. Wasn’t it former president Bush who invaded Iraq because he thought Iraq had nuclear weapons? Data analytics and satellite images showed no proof. Islamic State (IS) is a bad result of that decision.

Don’t be afraid and let’s make sure governments and commercial companies work together in making this world an awesome place to live and work by being transparent and open! This way we can act upon intelligence and not a hunch.