New version of the COMPADRE & COMADRE portal is coming up
by Rob Salguero-Gomez on Mar 20, 2017Just like winter, a new version of COMPADRE & COMADRE is coming too. This work has been going on for a couple of years now, but we are starting to make some tangible progress and we are excited to share where the project stands at the moment. We assume that you are familiar with the COMPADRE Plant Matrix Database and the COMADRE Animal Matrix Database in their current format. Navigating to http://www.compadre-db.org/ presents you with information about the databases, the team behind them and a direct download link to a single R-data file for either COMPADRE or COMADRE. There is also a table for the ‘waiting list’ of species that are soon to be added to the sister databases, but the current format does not offer searchable menus to explore the data before downloading them.
The current COMPADRE & COMADRE data portal
Behind the scenes, we have been building a new way of presenting and accessing COMPADRE and COMADRE, whereby you’ll be able to explore the data through the data portal. Here is a quick rundown on some of the features:
1. Query, then download:
You will be able to query the database so that before you download the data, you can work out whether the data are useful to you. We will have developed tables to search by species and publication and will be extending this to advanced filtering through all metadata so that the user gets just what s/he needs.
Table for searching by species, able to filter by growth type and search terms
2. Explore demographic info across the tree of animals and plants:
Come and explore among multiple kingdoms, orders and families, worm your way through the tree of life and see what species are related to one another, and where we have (and not) demographic data. Currently, we are finalising a click-through explorer gathering a short summary from Wikipedia, but there are plans to integrate the incredible OneZoom taxonomic explorer into the user interface.
Click-through taxonomic explorer
3. Educational pages for all ages, sizes and stages:
We are dedicated to teaching all st/ages about the wonders of demography and connecting maths with the living environment. For every species page, we are developing a georeferenced interface to show the locations of the populations of the study species. We also include species occurrence data from GBIF to give a general view of the distribution of the species. In addition, we provide relevant metadata about the species to be able to contextualise the matrix model.
Map showing the locations of studied populations, the small yellow squares are GBIF occurrence data.
4. More hierarchy, but easier to manage and navigate:
The database’s hierarchical structure is being incorporated into how we present the data. Beneath the species information, each population is listed with summary statistics about how many matrices it contains, its ecoregion and for how long it was studied. Expanding this reveals each of the matrices that were produced from studying this population. Expanding the matrices shows the matrix model in full, with clickable tabs to select whether you wish to view the full matrix, fecundity matrix, survival matrix or clonal matrix.
The contents of a single population with multiple matrices. One matrix has been expanded to show the values in the matrix.
5. Giving credit where it is due:
At the bottom of the page, you can see details about the publications from which the MPMs were taken from, and directly link to the original journal article via DOI. We hope that the users of COMPADRE and COMADRE will continue to credit the tremendous work done by the original sources of the demographic information.
A list of all publications contributing data for Cirsium pitcheri
Intended users of the database
We hope that this project will make the database more accessible to a wider range of users. These improvements to the COMPADRE & COMPADRE data portal resource are aimed at:- Students and entry-level population modellers who want to explore matrix populations models (MPMs).
- Scientists who aren’t primarily studying population dynamics or demography but are working with a study species and would like to know more about their study species.
- Comparative demographers who are looking to use a large chunk of the data available but might want to examine a particular matrix or species as to whether they will include it in this analysis. The user interface will provide an easy way of viewing the data and linking to the literature source.
- Ecologists who are looking to construct MPMs but would first look at previously constructed matrices for similar species or study motivations to help guide their study design.
- School groups who are interested in life-cycles and we want to hook them into a life long obsession of COMPADRE and COMADRE.
- Other managers of ecological databases that might want to share data and collaborate.
The Compadrino Zone
The compadrinos are the real workforce behind the demographic data. These are a group of MSc, BSc, PhD and postdocs students supervised by the core committee. They carefully inspect published papers, extract relevant information, email authors for extra information and clarification, and help implement the error-checking before the data go live on the open-access portal. The total COMPADRE & COMADRE workload is split amongst nodes located at different institutions across the globe. This poses a logistical challenge as to how to manage the data which have been currently been tackled with cumbersome spreadsheets.
A form for entering information about a publication, with an auto-fill from DOI javascript button.
The work we are doing will also improve the behind-the-scenes data digitisation workflow. Compadrinos will now have access to easy-to-use friendly web forms and be able to use the user interface to help identify errors. We will also be able to track changes to the database over time and make it a more streamlined, enjoyable, and less error-prone experience. That’s all well and good but as a user of the database, why should you care about this? These improvements should result in an increased flow of data from publications to database, leading to a greater quantity of data, expanding the taxonomic breadth of the database.
We hope that once the new portal has been instituted, we will move onto a new functionality: providing users with the ability to upload their data directly. The hope is for COMPADRE & COMADRE to become the DRYAD of stage-structured demography. Of course, all data that th users will upload will still need careful curation, and every datum will continue to go through our error-checking and data standardisation protocols.
Technical details
Finally, a few details of the project:- The newest iteration of the COMPADRE and COMADRE database will be hosted on a server at the University of Exeter and mirrored among all nodes of the COMPADRE/COMADRE digitalization network.
- The web application is built on a python-based framework called Flask and the database is being migrated from the original R-data object into a MariaDB database.
- Following the open-access philosophy behind COMPADRE & COMADRE, the present also is an open source project, and the source code is available on our Github repository: https://github.com/Spandex-at-Exeter/demography_database
Timeline
Now we’ve whetted your appetite, you’re probably wanting to know how long before you can delve into this exciting new resource? Our first milestone is to finalise the process by which compadrinos input and process data to make sure everything functions as expected and the data are flowing from publication to the database without a hiccup. This is our focus for the next couple of months. Following that, we will change focus to improving the user interface and data download tools for end users, hoping to launch late 2017. Simon (@simon_rolph) and Danny (@dannylbuss) |
| Danny and Simon hard at work |
Social Media
