It’s been a while since our last blog post, but not because we’ve been standing still. To the contrary, we have been busy working, building and thinking about how The DataTank can be a platform that not only publishes data into several data formats, but also a platform that can enrich data with ontologies, visualise data, and how all those different features can be loosely coupled.

The result can be seen in our repository which holds a set of packages that provide the different features a DataTank can provide. By doing this, we allow the user to have its own configuration that fits its needs, and allow other PHP developers to use our tools to perform certain functionality, which is not necessarily Open Data related. For the people who are used to our old repository, or for people who haven’t played with our code just yet, a small introduction is in place. Our The DataTank goal is simple: providing a tool that allows to publish data on the web. However, there are multiple ways of doing that. Reading data from data sources, and formatting them into a certain format for example are two different features, enriching them with ontologies is another one. Therefore, we have split those features in packages. Each package has its own goals and purpose and can contribute to publishing data in its own way. If you have no clue where to start, check out this repository.

Data processing, the early days.

In a previous blogpost we have mentioned that we were going to work on projects with Westtoer, and the Flemish Government. The Westtoer project has almost come to an end now, and it is time for us to focus on the next big thing: opening Flanders! Therefore, we are currently working on some final enhancements and issues and we should be finished by the end of May in a stable release. After that we will gather all of our long-term visions and ideas and pour them into a roadmap, aligned with the needs of the Flemish government, so stay tuned.

The DataTank took another step in making source code more reusable by starting to use composer. This means we now have decent package managing by packagist, which allows any PHP5.3+ developer to reuse any package in another project.

One of these package is negotiators, which just reached its first stable release.

There are two negotiators: a language negotiator and a content negotiator. The first one negotiates the language, the second one the format, each by reading the accept-language and accept header. The language header can be overridden by a GET parameter though.

More information on github: https://github.com/tdt/negotiators

Kind regards,

Pieter

It has been a while since you have heard from us, however, some open software analysis tools have noticed something was stirring with The DataTank repository!

The DataTank commits of the past month.

Since the last few weeks of 2012 and the first of 2013 a lot of work has been done to The DataTank project. Why is this? Let’s say The DataTank has had a great start of 2013.

We are very proud to announce that The DataTank is going to be used in a number of open data projects this year. These projects will allow the further development of The DataTank software and integration with semantic web technologies.

A first project where The DataTank software is being used is a project funded by Westtoer. This project will create a data hub for data sources, such as weather information and touristic routes. The novelty of this project enables the introduction of large data sources and enables lifting the data to the 5th star of linked open data, a promise we have made since the beginning of The DataTank. With this project we have had the unique opportunity to fully revision our architecture and came up with a more modular and professional approach than before.

This project is realized by combined forces of We Open Data, a working group of OKFN Belgium,  iMinds, and Wunderkraut Belgium. The beauty of this project is that the entire solution has to be open source and must provide structure so that 3rd parties can use it as well. The requirements and deliverables have been made by Marc Portier. Marc has a great vision on open data and made it so that every piece of the Westtoer project is completely open source, ready for other open data projects and enthusiasts to use. This will hopefully snowball the rate of adoption of the developed framework and will allow us to create a stable community around it.

Next to this great project, we received some other good news. Minister of innovation Ingrid Lieten shows her support to create an open data platform for Flanders. iMinds has the lead of this project and will use The DataTank as part of its open data platform solution.

These two projects allow The DataTank to be developed in an environment where academic requirements meets business requirements and become more aligned with its core purpose, namely opening data through an open software solution.

For further information please visit our new DataTank repository. Our previous repository, containing both version 1.0 and 2.0, will be supported until the summer, but we will help everyone updating their current instance to a new one. We have worked hard to make sure there is backward compatibility and you can export and import your configuration.

To conclude this post we would like to thank digipolis, the city of Ghent, the city of Antwerp and iRail (our open transport working group) for using our software in pioneering open data projects such as open data gent, open data antwerpen and data.iRail.be. These pioneer projects provide incentives for students and businesses to use open data and transform that data into information and knowledge. Furthermore, they have provided the first steps towards a community that builds further on The DataTank software outside of the We Open Data working group.

- Jan Vansteenlandt

The DataTank can be used to create a RESTful web-service in almost no time. Together with the Belgian incubating chapter of the Open Knowledge Foundation we have implemented The DataTank for a hackaton for the city of Ghent. Apps For Ghent an event with students, hackers, professionals and policy makers. Together they discuss open data in their own way. Students and hackers get to participate in a hackaton with beautiful prices, professionals get to help and interview the students, the policy makers get to speak about their dos and don’ts, and so on.

We started by checking out the data sets available at http://data.gent.be. Some datasets were structured in nice CSV files or XLS files which just needed one PUT request to get plugged into The DataTank. Soon we saw that a lot of files on the site were formatted as KML. As we did not have a generic way to add those to The DataTank, we quickly implemented this in 200 lines of code.Soon we had http://data.appsforghent.be up and running, providing a RESTful API for a lot of datasets. All that was left for us to do was creating some new formatters like the map, the grid and the charts (thanks to Lieven Janssen), and creating the portal. The portal is a small PHP script which reads the self documenting resource http://data.appsforghent.be/TDTInfo/Resources to generate hyperlinks.

Click on the picture for a live version

As the city of Ghent has De Lijn as a bus company, and NMBS as a railway company, we did 2 final PUT requests to the server to add remote resources from The DataTank at http://data.iRail.be. Now you can use the iRail API as if it were Apps For Ghent resources.

The advantages of an API

During the event developers and users of the data can point the city at errors in the datasets. Changing a value in the datasets is easy as all applications will be using the Apps For Ghent API.

During the hackaton there will be live statistics on how many times each resource get hit. Because you score less creativity points if you use a dataset that gets used by everyone, participants will be able to preemptively choose datasets that get used less.

For now, join the fun at http://www.appsforghent.be

- Pieter

Bronze

In the speech Neelie mentions the Open Data portals of the UK, Denmark and France – who last week opened data.gouv.fr – as heroes in Open Data work. And indeed they are heroes! But if we take a look at Tim Berner’s Lee suggested 5 star scheme for Linked Open Data we see that most of the Open Government Data portals currently available on the web only reach the second star. I would give them the bronze medal but why won’t we go for gold?

Gold

Well that’s precisely where The DataTank comes in to help. We see the 5 star scheme as a step-by-step process and The DataTank as the tool that can assist you to reach the fifth star.

Get Started

In our Get Started section you can see how we open up proprietary data formats (excel and shape files) from data.gov.uk and dati.gov.it into a RESTful API which returns Open Standards such as JSON, XML and RDF:

• Creating an excel resource
• Creating a shape resource

Now it’s your turn!

Why not give it a try for yourself? Install a datatank instance and open up some data from Open Government Data portals in Europe:

• Austria
• Belgium
• Denmark
• France
• Ireland
• Italy
• Norway
• Spain
• The Netherlands
• United Kingdom