Landcare Research - Informatics Team

High performance computing helps saves the NZ stick insect

medyckyj-scottd — Tue, 17 Apr 2012 22:27:02 +0000

With the increasing use of next generation DNA sequencing technologies as a regular part of biological research, vast amounts of DNA sequence data are being generated at an ever increasing rate. But with this added capability and power comes technical challenges around data management, processing and analysis. Genomics work carried out in Landcare is no exception when it comes to encountering such problems. An example of these challenges is found is some recent work led by Landcare Research scientist Thomas Buckley and carried out by post-doctoral researcher Alice Dennis and PhD student Luke Dunning.

Alice and Luke are interested in the functional coding regions of genomes of stick insect species native to New Zealand. Until recently processing their insect DNA sequence data collected from the next generation sequencing-by-synthesis platforms took one whole week per individual; even when using a fast multi-core desktop Linux machine with plenty of RAM.

The length of time to process the sequence data significantly slows down their research program and limits the number of such processing steps that can be undertaken within any given project. To improve on this problem Dan White (Informatics team) worked with Alice and Luke to use the computing resources within the National e-Science Infrastructure (NeSI) to shift their memory absorbing processes to NeSI’s high performance computing (HPC) resources which Landcare Research has access to as a member of NeSI. In comparison to the resources available at Landcare Research, the NeSI HPC resources that are now available include an 80 processor computer with each processor having 12 cores and 96 GB of RAM and access to 200TB of disk storage. For the stick insect case above early tests have shown a reduction in processing time from one week to just over 3 hours; a dramatic and significant improvement! Further improvements are expected as we explore the options of processing multiple files simultaneously and by chaining multi step sequence analyses together in an automated process.

· High powered computing saves the NZ stick insect: With the establishment and increasing use of next generation DNA sequencing technologies as a regular part of biological research vast amounts of DNA sequence data are being generated at an ever increasing rate. But with this added capability and power comes technical challenges around data management, processing and analysis. Genomics work carried out in Landcare is no exception. An example of these challenges is found is some of the work carried out by postdoctoral researcher Alice Dennis, and PhD student Luke Dunning, who work with Thomas Buckley. Alice and Luke are interested in the functional coding regions of genomes of stick insect species native to New Zealand. But processing their insect DNA sequence data collected from the next generation sequencing-by-synthesis platforms took one whole week per individual; even when using a fast multi-core desktop Linux machine with plenty of RAM. This type of delay significantly slows down their research program and limits the number of such processing steps that can be undertaken within any given project. To improve on this problem Dan White (Informatics) worked with Alice and Luke, and the computing resources within the National e-S

High powered computing saves the NZ stick insect: With the establishment and increasing use of next generation DNA sequencing technologies as a regular part of biological research vast amounts of DNA sequence data are being generated at an ever increasing rate. But with this added capability and power comes technical challenges around data management, processing and analysis. Genomics work carried out in Landcare is no exception. An example of these challenges is found is some of the work carried out by postdoctoral researcher Alice Dennis, and PhD student Luke Dunning, who work with Thomas Buckley. Alice and Luke are interested in the functional coding regions of genomes of stick insect species native to New Zealand. But processing their insect DNA sequence data collected from the next generation sequencing-by-synthesis platforms took one whole week per individual; even when using a fast multi-core desktop Linux machine with plenty of RAM. This type of delay significantly slows down their research program and limits the number of such processing steps that can be undertaken within any given project. To improve on this problem Dan White (Informatics) worked with Alice and Luke, and the computing resources within the National e-Science Infrastructure (NeSI) to shift their memory absorbing processes to NeSI’s high powered computing (HPC) resources which LCR has access to as a member of NeSI. In comparison to the resources available at LCR, the NeSI HPC resources that were now available include an 80 processor computer with each processor having 12 cores and 96 GB of RAM and access to 200TB of disk storage. For the stick insect case above early tests have shown a reduction in processing time from one week to just over 3 hours; a dramatic and significant improvement! Further improvements are expected as we exploring the options of running multiple files simultaneously and by chaining multi step sequence analyses together in an automated process.

cience Infrastructure (NeSI) to shift their memory absorbing processes to NeSI’s high powered computing (HPC) resources which LCR has access to as a member of NeSI. In comparison to the resources available at LCR, the NeSI HPC resources that were now available include an 80 processor computer with each processor having 12 cores and 96 GB of RAM and access to 200TB of disk storage. For the stick insect case above early tests have shown a reduction in processing time from one week to just over 3 hours; a dramatic and significant improvement! Further improvements are expected as we exploring the options of running multiple files simultaneously and by chaining multi step sequence analyses together in an automated process.

Share/Bookmark

Kaipara Harbour – Update to website and online atlas

medyckyj-scottd — Wed, 07 Mar 2012 08:51:48 +0000

The Informatics development team have released a major upgrade to the Integrated kaipara Harbour Management group (IKHMG) website and on-line atlas – http://www.kaiparaharbour.net.nz/.

The IKHMG was established to promote integrated management and inter-agency coordination and kaitiakitanga of the Kaipara Harbour and its catchment for creating a healthy and productive Kaipara harbour. The redesigned website brings a new look and organisation to key information on the group’s core objectives; activities; and interests ,and provides new access to maps and the groups knowledge-base allowing information and data that describes the Kaipara Harbour to be explored.

This work was collaboration between the Landcare Research Informatics group and Te Uri o Hau through funding from the Te Uri o Hau Settlement Trust.

Share/Bookmark

Report on the New Zealand AuScope Spatial Information Services Stack (SISS) workshop

medyckyj-scottd — Wed, 18 Jan 2012 03:42:59 +0000

Written by Alistair Ritchie.

In November 2011 Landcare Research co-presented a workshop on the AuScope Spatial Information Services Stack (SISS) with CSIRO in Wellington. SISS is a suite of software tools designed to enable the sharing of information between computer systems distributed across the internet.

The workshop was hosted by NIWA and was sponsored by NIWA, GNS Science and Landcare Research. The NZ Geospatial Office was instrumental in the organisation and promotion of the workshop. The workshop attracted the attention of senior figures within the Open Geospatial Consortium (OGC), and a blog representing their view of the day can be found at http://www.opengeospatial.org/node/1513.

One of the great challenges facing the environmental informatics community is the management of what has become known as the ‘data deluge’. The increasing complexity of the environmental problems faced by modern society has required the capture and processing of large volumes of relevant data. No one organisation has sole responsibility for the storage and management of these data, or the necessary processing and analysis. Instead this is spread across a wide variety of organisations (government, academic and private), each with their own information systems but also a need for access to information and computational resources maintained elsewhere.

These systems must be networked, and in a way that allows translation between the languages of different information systems. The internet is an ideal platform for these ‘interoperable spatial data infrastructures’ (SDIs), but the protocols of the internet needs to be extended to define consistent structures for information, and the behaviour of systems that deliver or process the information.

Bodies such as the International Standards Organisation (ISO) and the Open Geospatial Consortium (OGC) have defined technical specifications for the network information systems (web services). They have also defined a syntax for representation of geospatial data that has been adopted by a growing number of scientific communities and extended into data models that meet their specific needs.

Well documented standards are only a part of the solution, there must also be a concrete means of implementing them. The Australian National Collaborative Research Infrastructure project addressed this by funding AuScope – a not for profit company, run by CSIRO – to develop a variety of tools to enable the conduct of research on a grid of networked computer systems. The SISS is one of their key products and incorporates the following tools:

Web feature and maps services (WFS, WMS – Geoserver)
Web coverage services (WCS – THREDDS, ERDDAP)
Web catalog services (CS/W – GeoNetwork)
Vocabulary services (SISSVoc)
Unique identifier services (SISS URI resolver)
Information modelling (FullMoon)

A full description of the SISS, and an explanation of the purpose of these tools, can be found at the project’s web-site (http://siss.auscope.org) and in the workshop summary and presentations (https://www.seegrid.csiro.au/wiki/Siss/SissWorkshopNZ2011). It builds on existing open-source technologies, focussing on making enhancements where necessary and only developing new tools where no candidates already exist.

There is a web portal, specifically constructed to demonstrate the capability of the data infrastructure (http://portal.auscope.org). Its content reflects AuScope’s earth science focus and its active support of the deployment of the stack at all the Australian geological surveys. SISS should not, however, be considered a tool set for earth science only; it is being adopted by other domains. It is also a proposed means of implementing the local SDI defined by the New Zealand Geospatial Office.

Landcare Research has a long-standing interest on these SDIs, and a number of staff with significant expertise in the area. Alistair Ritchie presented at the workshop, having worked with AuScope since the project’s inception. In his experience, the success of SDI’s depends on well-organised communities willing to make an effort to define and implement infrastructures specific to the their needs. The workshop had a large turnout with 42 people, from 18 organisations from government, academia and the private sector. However, interest is not enough. The challenge for the New Zealand information community is to translate this interest into an actual, functioning infrastructure with collaborative communities of do-ers all willing to support each other are essential.

Share/Bookmark

New web mapping application tool launched – Our Environment

medyckyj-scottd — Tue, 17 Jan 2012 03:56:21 +0000

Following on from the successful launch in September of last year of the S-Map Online web mapping web site, the Informatics team recently unveiled a new and free online mapping tool which will make it easier for businesses, government, researchers and the public to better understand the natural environments that underpin New Zealand’s economy and society.

Our Environment Home Page

Our Environment (http://ourenvironment.scinfo.org.nz) is a new web mapping tool which provides easy access to information on New Zealand’s land environments and land resources. Based on “Google Maps” style navigation the mapping tool makes it easy to search, view and query environmental information. Using Our Environment users can

Use interactive maps to learn about their local environment and regional variations.
Select particular data themes such as vegetation, land suitability or surface rock type to view as maps.
View detailed information from a particular data theme.
Obtain further details about a particular theme (metadata).
Create high quality, custom PDF maps for printing.

Our Environment uses data from nationally significant databases such as the Land Resource Inventory System (LRIS), the Land Cover Database (LCDB) and other national datasets and is designed to support applications at local, regional and national scales. Further data themes and features will be added as time goes by.

The Our Environment Mapping Application

The Our Environment mapping application is built using the new geospatial-portal-software platform for visualising geospatial data and for querying datasets. Designed and written by the Informatics team the platform is based on open standards and open source software (including PostGIS, MapServer, Geo Tools, MapCache (formerly mod-geocache) and Open Layers). Taking an across Landcare view of the future requirements for delivery of environmental science data through rich data visualisations, the platform has been designed so that it can be used, and extended if required, to roll out portal applications with a location-based focus. S-map Online was the first such portal, Our Environment is the second. Other Landcare datasets have been already identified as being suitable for delivery through new instances of the web mapping application.

Share/Bookmark

New Zealand Virtual Herbarium launches

medyckyj-scottd — Fri, 23 Sep 2011 03:15:54 +0000

The New Zealand Virtual Herbarium (www.virtualherbarium.org.nz) has been launched by the New Zealand National Herbarium Network with Informatics having a significant input to the website and the underlying database.

The NZVH is a collaborative project involving 11 herbaria in New Zealand, developed under the auspices of the New Zealand National Herbarium Network, with financial assistance from Terrestrial and Freshwater Biodiversity Information System (TFBIS) Programme. It provides free internet access to over 700,000 records held by 11 botanical collections from Auckland to Dunedin. The records relate to over 1.4 million specimens of mosses, lichens, liverworts, ferns, algae and seed plants that have been collected in New Zealand from 1769 (Captain Cook´s first voyage ) to the present day, and cover all regions of the New Zealand mainland, as well as the offshore islands.

These herbaria records can now be searched simultaneously across the participating herbaria and results viewed as distribution maps or download. Each herbarium specimen is accompanied by information on its correct botanical name, where and when it was collected, by who, and also often ecological details.

While the larger collections have comprehensive records, until now users could not just go to their “local” herbarium to get all the information they needed. The New Zealand Virtual Herbarium pools all the available records in a ‘one stop shop’ making it easy for users to obtain the most complete picture for the species they are interested in, and generate distribution maps at the click of a button. It also provides these data in a consistent format and standard.

The project to make these data accessible was implemented by the Informatics team using a platform that was initially developed for Australia’s Virtual Herbarium by the Council of Heads of Australasian Herbaria. Informatics undertook a major rebuild of the software and incorporated some new features to make it easier to use. Improvements were also made to the harvesting of records from each of the participating New Zealand herbaria; NZVH consolidates all of these data into a single access point, providing a nation-wide view of botanical information.

Share/Bookmark

FOSS4G and OGC TC, Colorado, USA – September 2011

medyckyj-scottd — Sat, 17 Sep 2011 19:34:16 +0000

FOSS4G is taking place in Denver, USA this year and four staff from the Informatics team are attending for the week.

FOSS4G is the global conference focused on Free and Open Source Software for Geospatial, organized by OSGeo. It attracts the best Open Source geospatial software projects and their developers making it the gathering of the tribes. FOSS4G addresses the needs of users and developers alike but this year also intends to consider how business models can be used to address the needs of small and large organisations. Its a busy week with technical workshops, parallel sessions, panel discussions, lightening presentations and BOF. Some 800+ delegates are expected to attend.

The Informatics team are active users of open source software, and geospatial open source software in particular (PostGIS, Geoserver, MapServer, OpenLayers, Mod_GeoCache and so on) and FOSS4G is a great opportunity to catch up on the latest developments, engage with developers and users of the software and learn about large and mature projects.

The week after FOSS4G, members of the Open Geospatial Consortium will just up the road from Denver in Boulder for a technical planning meeting. Two of the Informatics staff, Science Leader Robert Gibb and Team Manager David Medyckyj-Scott will be staying on in Colorado to attend. It will be another busy week for alongside the meetings of the standards working groups and domain working groups there are a series of parallel events including a Ocean/Met/Hydro Water Cycle Summit and a GEOSS Workshop XLIII: Sharing Climate Information and Knowledge.

Share/Bookmark

S-map Online – an interactive mapping application for New Zealand soils data

medyckyj-scottd — Thu, 01 Sep 2011 05:58:37 +0000

Informatics has launched a new online tool to make it easier for land managers, business, scientists and the public to better understand the soils that underpin New Zealand’s economy, land use and ecology.

The home page of S-map Online

S-map Online (http://smap.landcareresearch.co.nz) is a web service which provides easy access to information on New Zealand’s soils. It’s based on “Google Earth” style navigation to make it easy to search, view and query, and allows a user to:

explore interactive maps of soil properties such as soil drainage and available water,
learn about the soil in their backyard or paddock,
view detailed information about soil classes or attributes,
create custom PDF soil maps for printing, and
for specific locations, download soil factsheets which provide more detailed knowledge of soil properties and information relevant to a variety of potential uses.

S-map Online is based on a digital soil spatial information system that is being populated by staff from Landcare’s Soils and Landscapes team with data to provide consistent and comprehensive national soil data layers to support applications at local, and regional to national scales. Soil factsheets have been available on request over the web for sometime and this facility has now been linked to a mapping application.

The mapping application

As well as access to detailed soils information, the S-map Online web service has excellent, high quality topographic and terrain maps including maps based on LINZ topographic data. These allow the user to not only locate an area of interest with a high degree of accuracy but also provide a geographic context for the soils data.

Although S-map coverage is not yet complete, there is sufficient coverage to make the service a useful resource for many people. Work continues to extend S-map to the Environment Waikato and Environment Canterbury territorial areas, the soft-rock hill country in the central North Island and parts of Northland, Gisborne, Hawkes Bay, Manawatu, Wairarapa, Golden Bay and the West Coast.

S-map Online has been built using a new geospatial-portal-software platform for visualising geospatial data and for querying attributes of datasets. Designed and written by the Informatics team and based on open standards and open source software (including PostGIS, MapServer, Geo Tools, mod-geocache and Open Layers), the platform will be used to roll out other web applications with rich data visualisations capabilities over the next few years.

Share/Bookmark

Landcare wins contract for new editions of the New Zealand Land Cover Database (LCDB)

medyckyj-scottd — Thu, 01 Sep 2011 05:20:17 +0000

The Informatics team along with colleagues from the Soils and Landscapes team have successfully secured the LCDB3+ contract.

Land Cover Database (LCDB) is a digital map of the surface of New Zealand created using satellite imagery. It contains detailed information on categories of land cover and boundaries, and is a record of land cover changes over time. Two completed editions (LCDB1 and LCDB2), which show the state of New Zealand’s land cover in 1996-1997 and 2000-2001 respectively, have become critical to central, regional and local government, industry and research institutions. The information is used for land, water and biodiversity management, pest control and monitoring, wildfire threat and risk analysis, and environmental monitoring and reporting.

The creation of new land cover databases and related research has been identified as an important priority by Ministry for the Environment (MfE), Department of Conservation (DOC) and regional councils. The Government will give $1 million a year over the next four years to the creation the new land cover databases and will involve the use of satellite technology, geospatial mapping techniques and advanced computing systems.

Landcare Research has been commissioned to deliver two further editions of LCDB and, a parallel research programme will develop improved ways of generating land cover information. LCDB3, the map of land cover as at 2008, should be released in July 2012. LCDB4, which will use imagery acquired over the summers of 2011/12 and 2012/13, is expected to be released in July 2014.

Share/Bookmark

OurFuture – Sharing stories about New Zealand’s environmental research

medyckyj-scottd — Thu, 01 Sep 2011 03:25:14 +0000

OurFuture is an interactive website which brings together projects and activities that benefit New Zealand’s natural environment and resources. OurFuture is aimed at making connections between research and action, and between people and projects and places. The OurFuture site allows partner organisations to contribute stories about activities that benefit New Zealand’s natural environment and resources in sustainable ways. Once signed up, contributors can write and edited stories together with tags, images, embedding video and maps. Stories can also be grouped into collections creating a cohesive package based on a particular theme (e.g., ‘Ecosystem research’, ‘Coastal restoration projects’, ‘Northland schools environmental projects’).

A page from the OurFuture website

OurFuture is coordinated and funded by Landcare Research, in partnership with a range of organisations and groups who contribute their stories. The site was developed by the Landcare Research Informatics team and Swirling World and is a Microsoft MVC based web development integrating a dynamic Google maps service and can be thought of as a light weight, spatially-aware Content Management System (CMS).

Visit OurFuture at: www.ourfuture.net.nz

Share/Bookmark

An update for the Landcare Research SINDI (Soil INDIcators) website

medyckyj-scottd — Thu, 01 Sep 2011 03:08:29 +0000

Screenshot of the SINDI service

For landowners who use SINDI, the upgrade creates a more user friendly and interactive site to assess and learn about soil quality. In this new release management and soil quality information has also been updated to reflect current best management practice and a link to the S-map Online website has been added so that landowners can obtain additional soils information (soil maps and soil factsheets) for their area. In addition to updated information for landowners, Regional Councils can now upload their own soil quality data and compare it with the soil quality characteristics contained within SINDI through a map interface. This data will be combined with the approximately 1,000 soil sample sites across New Zealand that have been added to the system to continue to improve the information available.

This work was made possible through the New Zealand EnviroLink fund administered by the Ministry of Science and Innovation (MSI).

The Informatics team together with Landcare soil science staff, has recently undertaken an update of the SINDI (Soil INDIcators) website – sindi.landcareresearch.co.nz. SINDI was created to interpret a specific set of dynamic soil quality characteristics to inform landowners of the general condition of their soil and suggest suitable management practices to remedy soil degradation. For landowners who use SINDI, the upgrade creates a more user friendly and interactive site to assess and learn about soil quality. In this new release management and soil quality information has also been updated to reflect current best management practice and a link to the SMAP website has been added so that landowners can obtain additional soils information (soil maps and soil factsheets) for their area. In addition to updated information for landowners, Regional Councils can now upload their own soil quality data and compare it with the soil quality characteristics contained within SINDI through a map interface. This data will be combined with the approximately 1,000 soil sample sites across New Zealand that have been added to the system to continue to improve the information available.

pgrade creates a more user friendly and interactive site to assess and learn about soil quality. In this new release management and soil quality information has also been updated to reflect current best management practice and a link to the SMAP website has been added so that landowners can obtain additional soils information (soil maps and soil factsheets) for their area. In addition to updated information for landowners, Regional Councils can now upload their own soil quality data and compare it with the soil quality characteristics contained within SINDI through a map interface. This data will be combined with the approximately 1,000 soil sample sites across New Zealand that have been added to the system to continue to improve the information available.

Share/Bookmark