Wilderlab API User Manual
Environmental DNA (eDNA) testing generates a lot of data. Over the past three years our clients have detected over a million different DNA sequences from thousands of different species. While our primary method of disseminating data is in xlsx format, static result tables do not reflect continuous updates to taxon IDs due to the availability of new reference sequences. Furthermore, static result tables are not always shared between departments, resulting in fragmented data and reporting.
To remedy this, Wilderlab has developed an API (Application Programming Interface) that enables clients to access their data in real time, facilitating the export of eDNA results to environmental data storage platforms such as Kisters and Hilltop.
This tutorial provides a brief introduction to accessing the connect.wilderlab
API using the R programming language and the the wilderlab R package.
Please contact us
if you would like us to generate API access keys for
your organisation, or if you would like any further help.
Wilderlab API Instructions
1. Installation
The wilderlab R package contains functions for importing
and exporting eDNA data. To download the package from GitHub, ensure that
devtools is installed. The insect package will also be required later on in this tutorial:
if(!("devtools" %in% list.files(.libPaths()))) install.packages("devtools")
if(!("insect" %in% list.files(.libPaths()))) install.packages("insect")Then run:
devtools::install_github("wilderlabnz/wilderlab")
library(wilderlab)2. Load access keys
When signing up to the Wilderlab API, your unique log in information
will be securely sent to you. This will include three access keys: an
API access key id, key; a secret access key,
secret; and a X-API-Key, xapikey.
Copy and paste this unique information into the appropriate slots in the
following code to load them into your R session.
key <- "*****************"
secret <- "***************************************"
xapikey <- "***************************************"3. get_wilderdata function
Description
Wrapper functions for creating URLs and authorisation headers to download job, sample, taxa, and record information from the connect.wilderlab API.
Usage
get_wilderdata(tb, key, secret, xapikey, JobID = NULL)Arguments
| Argument | Description |
|---|---|
tb |
a character string specifying the table required.
Accepted values are jobs, samples,
taxa, and records. |
key |
a string specifying the API access key for the client account. Please contact info@wilderlab.co.nz if you would like access keys generated for your account. |
secret |
a string specifying the API secret access key for the client account. Please contact info@wilderlab.co.nz if you would like access keys generated for your account. |
xapikey |
a string specifying the X-API-Key value for the client account. Please contact info@wilderlab.co.nz if you would like access keys generated for your account. |
JobID |
a 6 digit integer specifying a Wilderlab job number. Required only for accessing the records table. |
Details
The Wilderlab API is designed for clients to access up-to-date eDNA
data for their internal data storage platforms and geospatial
applications. Clients can access their job, sample, taxon, and eDNA
records data at any time by querying the API with a valid URL and
authorisation header. The get_wilderdata function is a
wrapper that enables these URLs and headers to be compiled with minimal
effort.
4. Get data tables
Pulling information from the job, samples,
and taxa tables is very straightforward. All the
information associated with a unique API access key is accessed as a
whole:
jobs <- get_wilderdata("jobs", key = key, secret = secret, xapikey = xapikey)
samples <- get_wilderdata("samples", key = key, secret = secret, xapikey = xapikey)
taxa <- get_wilderdata("taxa", key = key, secret = secret, xapikey = xapikey)Pulling from the records table is done on a specified
JobID thus multiple calls must be made to get a full records table.
After accessing the jobs table, the relevant JobIDs can be
iterated over, the corresponding records pulled through the API, and
finally the gathered records combined into a complete records table. The
following code chunk is an example of how this can be done.
records <- vector(mode = "list", length = nrow(jobs))
for(i in seq_along(records)){
records[[i]] <- get_wilderdata("records", JobID = jobs$JobID[i],
key = key, secret = secret, xapikey = xapikey)
}
records <- do.call("rbind", records)5. Fill lineage information
The final step in completing the records table is to add lineage information and sample metadata. This is an optional step but can improve interpretation of the results.
library(insect)
tdb <- taxa[, 1:4]
colnames(tdb) <- c("taxID", "parent_taxID", "rank", "name")
lineages = insect::get_lineage(records$TaxID, tdb)
records$phylum <- sapply(lineages, "[", "phylum")
records$class <- sapply(lineages, "[", "class")
records$order <- sapply(lineages, "[", "order")
records$family <- sapply(lineages, "[", "family")
records$genus <- sapply(lineages, "[", "genus")
records$species <- sapply(lineages, "[", "species")
records$Latitude <- samples$Latitude[match(records$UID,samples$UID)]
records$Longitude <- samples$Longitude[match(records$UID,samples$UID)]
records$ClientSampleID <- samples$ClientSampleID[match(records$UID,samples$UID)]Example output
For the purposes of this manual, we will use example access keys as follows. This information will allow us to produce an example of what to expect from connecting to the Wilderlab API. Feel free to try out the API with these access keys first to ensure it is working as expected.
key <- "AKIATVYXGCYLWADFJVEX"
secret <- "SiDvZFUFXlCXK/jeBtHrfRPWMmb8veW6q5+ULuyx"
xapikey <- "7CCm580l5vgeKbalwIEy565uFhbEudTauAq80B38"Jobs
Getting information from the jobs table:
jobs <- get_wilderdata("jobs", key = key, secret = secret, xapikey = xapikey)
jobs JobID SubmissionDate ContactName ContactEmail PurchaseOrder JobReference JobNotes NumberOfSamples TestsRequired MakeDataPublic PassCode JobStatus InvoiceNo
1 601833 2021-07-04 Shaun Wilkinson api@example.com NA Passive sampler validation experiment NA 3 AP 1 W1E0638 Complete INTERNAL
2 601834 2021-07-04 Shaun Wilkinson api@example.com NA Passive sampler validation experiment NA 3 AP 1 W1E0638 Complete INTERNALSamples
Getting information from the samples table:
samples <- get_wilderdata("samples", key = key, secret = secret, xapikey = xapikey)
samples SID JobID UID CollectionDate CollectedBy ClientSampleID Latitude Longitude VolumeFilteredML DeploymentDuration EnvironmentType TICI TICINoSeqs TICIQuantile TICIVersion ClientNotes AccountName MakeDataPublic Report
1 8145 601833 507875 2021-07-03 Shaun Wilkinson Ruakokoputuna C3 -41.312665 175.449873 1000 River/Stream 104.53 112 0.55 Riverine V1 Control rep 3 Wilderlab NZ Ltd 1 https://s3.ap-southeast-2.amazonaws.com/wilderlab.openwaters/reports/ae59dcd538716094.html
2 8143 601833 507877 2021-07-03 Shaun Wilkinson Ruakokoputuna C1 -41.312665 175.449873 1000 River/Stream 99.81 93 0.50 Riverine V1 Control rep 1 Wilderlab NZ Ltd 1 https://s3.ap-southeast-2.amazonaws.com/wilderlab.openwaters/reports/ae59dcd538716094.html
3 8144 601833 510042 2021-07-03 Shaun Wilkinson Ruakokoputuna C2 -41.312665 175.449873 1000 River/Stream 99.57 171 0.50 Riverine V1 Control rep 2 Wilderlab NZ Ltd 1 https://s3.ap-southeast-2.amazonaws.com/wilderlab.openwaters/reports/ae59dcd538716094.html
4 8146 601834 510897 2021-07-03 Shaun Wilkinson Ruakokoputuna P1 -41.312665 175.449873 24 River/Stream 100.93 426 0.52 Riverine V1 Passive rep 1 Wilderlab NZ Ltd 1 https://s3.ap-southeast-2.amazonaws.com/wilderlab.openwaters/reports/0b6f40128711495a.html
5 8147 601834 510898 2021-07-03 Shaun Wilkinson Ruakokoputuna P2 -41.312665 175.449873 24 River/Stream 101.65 420 0.53 Riverine V1 Passive rep 2 Wilderlab NZ Ltd 1 https://s3.ap-southeast-2.amazonaws.com/wilderlab.openwaters/reports/0b6f40128711495a.html
6 8148 601834 510899 2021-07-03 Shaun Wilkinson Ruakokoputuna P3 -41.312665 175.449873 24 River/Stream 99.46 355 0.50 Riverine V1 Passive rep 3 Wilderlab NZ Ltd 1 https://s3.ap-southeast-2.amazonaws.com/wilderlab.openwaters/reports/0b6f40128711495a.htmlTaxa
Getting information from the taxa table:
taxa <- get_wilderdata("taxa", key = key, secret = secret, xapikey = xapikey)
head(taxa, 10) TaxID ParentTaxID Rank Name CommonName MaoriName
1 1 0 no rank root
2 2 131567 superkingdom Bacteria Eubacteria
3 10 1706371 genus Cellvibrio
4 18 213421 genus Pelobacter
5 20 76892 genus Phenylobacterium
6 22 267890 genus Shewanella
7 29 28221 order Myxococcales Fruiting gliding bacteria
8 31 80811 family Myxococcaceae
9 39 80811 family Archangiaceae
10 40 39 genus Stigmatella Records
Getting information from the records table:
records <- vector(mode = "list", length = nrow(jobs))
for(i in seq_along(records)){
records[[i]] <- get_wilderdata("records", JobID = jobs$JobID[i],
key = key, secret = secret, xapikey = xapikey)
}
records <- do.call("rbind", records)
head(records, 10) HID UID TaxID Rank Name CommonName Group Count
1 3963310 507875 578128 family Holostichidae Ciliates Ciliates 74
2 3963311 507875 584654 genus Anteholosticha Ciliates 69
3 3963318 507875 50949 genus Navicula Diatoms 16
4 3963327 507875 5948 genus Stylonychia Ciliates 10
5 3963332 507875 70185 genus Spongospora Other 7
6 3963343 507877 584654 genus Anteholosticha Ciliates 63
7 3963344 507877 68038 genus Chaetonotus Gastrotrich Other 56
8 3963346 507877 41372 family Chaetonotidae Other 60
9 3963347 507877 109671 species Physella acuta Left handed sinistral snail Molluscs 531
10 3963357 507877 50949 genus Navicula Diatoms 7Adding lineage information and sample metadata information into the records table:
library(insect)
tdb <- taxa[, 1:4]
colnames(tdb) <- c("taxID", "parent_taxID", "rank", "name")
lineages = insect::get_lineage(records$TaxID, tdb)
records$phylum <- sapply(lineages, "[", "phylum")
records$class <- sapply(lineages, "[", "class")
records$order <- sapply(lineages, "[", "order")
records$family <- sapply(lineages, "[", "family")
records$genus <- sapply(lineages, "[", "genus")
records$species <- sapply(lineages, "[", "species")
records$Latitude <- samples$Latitude[match(records$UID, samples$UID)]
records$Longitude <- samples$Longitude[match(records$UID, samples$UID)]
records$ClientSampleID <- samples$ClientSampleID[match(records$UID, samples$UID)]
head(records, 10) HID UID TaxID Rank Name CommonName Group Count phylum class order family genus species Latitude Longitude ClientSampleID
1 3963310 507875 578128 family Holostichidae Ciliates Ciliates 74 Ciliophora Spirotrichea Urostylida Holostichidae <NA> <NA> -41.31267 175.4499 Ruakokoputuna C3
2 3963311 507875 584654 genus Anteholosticha Ciliates 69 Ciliophora Spirotrichea Urostylida Holostichidae Anteholosticha <NA> -41.31267 175.4499 Ruakokoputuna C3
3 3963318 507875 50949 genus Navicula Diatoms 16 Bacillariophyta Bacillariophyceae Naviculales Naviculaceae Navicula <NA> -41.31267 175.4499 Ruakokoputuna C3
4 3963327 507875 5948 genus Stylonychia Ciliates 10 Ciliophora Spirotrichea Sporadotrichida Oxytrichidae Stylonychia <NA> -41.31267 175.4499 Ruakokoputuna C3
5 3963332 507875 70185 genus Spongospora Other 7 Endomyxa Phytomyxea Plasmodiophorida Plasmodiophoridae Spongospora <NA> -41.31267 175.4499 Ruakokoputuna C3
6 3963343 507877 584654 genus Anteholosticha Ciliates 63 Ciliophora Spirotrichea Urostylida Holostichidae Anteholosticha <NA> -41.31267 175.4499 Ruakokoputuna C1
7 3963344 507877 68038 genus Chaetonotus Gastrotrich Other 56 Gastrotricha <NA> Chaetonotida Chaetonotidae Chaetonotus <NA> -41.31267 175.4499 Ruakokoputuna C1
8 3963346 507877 41372 family Chaetonotidae Other 60 Gastrotricha <NA> Chaetonotida Chaetonotidae <NA> <NA> -41.31267 175.4499 Ruakokoputuna C1
9 3963347 507877 109671 species Physella acuta Left handed sinistral snail Molluscs 531 Mollusca Gastropoda <NA> Physidae Physella Physella acuta -41.31267 175.4499 Ruakokoputuna C1
10 3963357 507877 50949 genus Navicula Diatoms 7 Bacillariophyta Bacillariophyceae Naviculales Naviculaceae Navicula <NA> -41.31267 175.4499 Ruakokoputuna C1Issues
If you experience a problem using this software please feel free to raise it as an issue on GitHub.