Oceans of information, open sourced.
Site purpose
The Ocean Survey 20/20 (OS20/20) programme is a whole of Government initiative, which aims to provide New Zealand with better knowledge of its ocean territory, including our exclusive economic zone, Continental Shelf, and the Ross Sea Region.
The Ocean Survey 20/20 portal was developed to enable expert and public access to data, information and reports from the Bay of Islands OS20/20 survey, as well as support the addition of further OS20/20 surveys when their data becomes available. As of 2011, the Bay of Islands survey is the broadest marine survey ever carried out in New Zealand, and was funded through LINZ, who contracted NIWA to carry out the research.
What we did
SilverStripe Ltd were contracted by NIWA to build a web-portal to help the general public and specialist users access the data that's being gathered. The web portal allows end users to get full access to a vast array of information stored in distributed databases (around four terabytes' worth). This information is easily discoverable via a custom metadata interface, in an easy to understand way, as well as via a powerful web mapping application that enables access to data in a geographical context.
There were a number of other challenges for us in building the site beyond just the sheer volume of the data. Content includes 20,000 images, geospatial station data, PDFs, spreadsheets, and zip files. The integration of HD video is also supported.
As an open source company, SilverStripe Ltd is committed to open data. As a Crown Research Institute, NIWA is committed to an open data model, and an extensive user of open source tools. We saw that typical methods of data delivery were inadequate - costly, time-consuming, difficult to use, and often out of date. Together we decided to deliver a better solution.
The traditional approach to government web mapping sites in NZ has been to purchase licenses for expensive Geographic Information System (GIS) software and develop purpose built web applications. One of the innovative goals of this project was to ensure that the entire site; all code, content and all the application and software libraries it is built on, are open source, and communicate using open standards so the entire portal comprises a suite of reuseable, interoperable components that can be reused together or separately, with zero purchase or licensing costs. Yet, through SilverStripe Ltd, any potential users who wish to, can still have a full commercial support contract. This approach minimises costs, maximises reusability and offers the best of both the commercial and open source models.
What users can do now
Most government websites are known for not making access to data very easy. In the past, when agencies have released data, they have tended to do it in proprietary formats, and distributed by DVDs posted in the mail. Data was often formatted in such a way it took extensive digging around to get what you needed. We were determined that the Ocean Survey 20/20 site would be a better way of doing things.
The site extends an open data model in that not only is the survey data discoverable and downloadable, but so are presentations, reports, maps, and other information that accompanies the data, which makes information from the survey directly useful to the general public. Map layers are not only available in the portal, but can be connected into desktop GIS and web mapping applications, allowing rich manipulation and live updates for advanced users.
Instead of requiring users to visit all the different agencies involved in the survey to request different sets of data, then needing to bring the results together themselves, OS20/20 allows users to get all the information they need in one place, and puts it together automatically. It gives users access to terabytes of data hosted in disparate data centres, and delivers it in real time.
The web interface is not only intuitive, it's also built to allow later enhancement so that future research surveys (which the portal already supports) can add in additional elements which may be needed. Data is presented using layers on maps that can be individually enabled or disabled, allowing the viewer to define the context themselves, eliminating or enabling datasets at will.
The web mapping client within the portal was developed specifically for the portal, integrating the open source web-mapping client OpenLayers with the SilverStripe CMS as a web mapping module, and will be available as a standard SilverStripe CMS module. What's particularly exciting about the mapping is that users can view it at nearly any size - the map is not constrained by anything other than the user's browser; it can be extremely large if the user desires it - something most mapping applications inadequately support. Atlas, an image server to deliver high quality photography and videos and ESRI ArcServer, licensed to NIWA, are the two proprietary applications used. The portal uses other open source solutions to provide storage and web-service capabilities, supporting Open-Geospatial Consortium standards. Most of the open source components are supported by the Open Source Geospatial Foundation.
Alongside with the mapping component, NIWA and SilverStripe Ltd have developed a catalogue service, enabling the website user to search for the raw data, reports or download datasets directly. It provides access to data, reports, maps, photographs & presentations thus providing both data as well as interpretations and information from experts explaining the data. The portal also enables website users to stream the data into other applications via the OGC web-services which are discoverable via the catalogue.
Measures of success
- Ocean Survey 20/20 was a finalist in the government category of the New Zealand Open Source Awards.
- As the entire framework the portal is built around is reuseable, several agencies are already looking at adapting and using it with their data to meet their needs. The use of open standards to provide map layers and metadata content, has meant that third parties have been able to access map data directly from NIWA servers and embed it in their own applications. With some datasets running into terabytes, the sharing of data this way and on this scale has not been done in New Zealand before.
- The New Zealand Ministry of Fisheries has already embedded map layers from this portal in their National Aquatic Biodiversity Information System, as well as in desktop mapping systems used by Ministry staff in their policy & management role.
