Painless Loading Of National Open Data Sources

John Birkett (Brent Council)

12:30 on Thursday 19th September (in Session 5, starting at noon, EMCC: Room 2)

Show in Timetable

Description: We describe a painless, automated process of loading cumbersome collections of open data that leads from data ordering to ready-to-use in a day, using some simple scripting and Postgres/PostGIS, with minimal officer input!

OpenData is a useful resource but it can involve large and cumbersome files. Brent wanted to make use of nationwide postcode data through CodePoint Open, but staff resources could never be allocated to the time-consuming task of manipulating, loading, and managing the data. A solution proposed was to develop a batch loading process incorporating the manipulating and management stages utilising Brent’s Postgres/PostGIS spatial database. The data can now be ordered, downloaded, manipulated, loaded and go live within one working day – requiring only around 20 minutes of officer time! We will guide you through our process which takes a large cumbersome dataset and gives you useful outputs, including for example postcode sectors, change files, and incorporating vertical streets. A similar process is used for Environment Agency data. We will share the pitfalls the can be discovered and how they can be overcome, for example multiple projections, non-topological data, etc.