1 of 21

The OTN Data Centre and OTN Data Nodes

Jon Pye – Director of Data Operations

Hi, my name’s Jon Pye, I lead the data shop here at OTN, and I’m here to lead you all into this week’s Node Manager Training. This is a talk meant to set the stage, by describing what Nodes are meant to do, why OTN has decided to make and support them instead of making one big super-datacenter, and how the software and data management pipelines we have been building can help with the vital and difficult task of coordinating your telemetry community, bringing their data together and producing important products for them, as well as allowing everyone to participate in cross-referencing their data across the entirety of the OTN Node network.

It’s my hope that by the end of the week you’ll have a sense of not only what your Node is and what it can do, but also of how nodes fit into OTN’s mandate from our funders,

the benefits of the Node system over ad-hoc sharing, especially as a community outgrows more organic person-to-person sharing,

and the benefits of being part of a system that is internationally recognized and certified by organizations such as the Intergovernmental Oceanographic Data Exchange, the World Data System, and the CoreTrustSeal community.

We are here also to convey to you that there is an ongoing support and community-building relationship that exists between OTN and its Nodes,

and through the lessons and the exercises, to give you a realistic understanding of the work involved in serving and maintaining a telemetry community.

2 of 21

ABOUT OTN

Global aquatic research, data management and partnership platform headquartered at Dalhousie University in Nova Scotia, Canada.

Since 2008, OTN has been deploying �ocean monitoring equipment and marine autonomous vehicles in key aquatic locations globally.

OTN has established partnerships with a global community of stakeholders to document the movements and survival of aquatic animals.

Knowledge generated by OTN is used to help guide management decisions and the sustainable use of aquatic systems.

OCEAN TRACKING NETWORK

First, let’s go over a bit about what the Ocean Tracking Network does, our mandate and mission from the funders. We’re a global infrastructure network funded by the Canadian Foundation for Innovation and headquartered here at Dalhousie University. We are funded currently as a Major Science Initiative, which means we are an infrastructure program meant to provide scientific infrastructure to our research partners. We provide gear and field support but the data part of OTN is our contribution as digital infrastructure. This is how we’ve been operating since 2015, but we’ve been around since 2008, supporting Canadian and international research by deploying ocean monitoring equipment in key locations around the world and training successive academic generations of movement ecologists and biogeographers. More recently we’ve increased our focus on physical and chemical oceanography and limnography using ROVs and autonomous gliders. Our goal is to establish partnerships across a global community of stakeholders to document the movement of aquatic animals in the changing ocean. The knowledge collected by OTN-affiliated researchers has been useful to policymakers, industry, Indigenous and coastal communities, and the general public.

New-ish news is that we are being invited to change our designation to Major Research Facility, which is equivalent roughly to the US IOOS line-item designation in terms of funding stability and no longer having to constantly prove ourselves on a short reporting cycle. It has some other advantages when it comes to public-private funding matches, but what it represents is a continued evolution of our program here, and a recognition of the hard work that has paid off in previous iterations of OTN into a lasting research infrastructure that the Government of Canada values and wants to see grow.

3 of 21

Digital Infrastructure

Thorough and uniform metadata optimizes discovery and secondary applications for project data

Improve inter-project sharing as well as designing standards for publication

Share resources, capabilities, techniques, not just data

OCEAN TRACKING NETWORK

The way that OTN’s data centre fits in, the way we as data managers help fulfil OTN’s infrastructure mission and provide value to the researchers and organizations that partner with us, is by being a reliable and reproducible example of digital infrastructure. There are a couple of needs and concerns researchers have that we as data managers are well positioned to satisfy.

When researchers are sharing data amongst themselves, combining their data holdings into one large dataset, the key is making sure everyone is recording their metadata and data to a uniform standard. We at OTN have defined and refined a standard that meets as many of the needs of our partners and affiliate data networks while being manageable for field work.
We define, harvest, and publish project level information to publish in catalogues for easy discovery of complementary projects. Generally the more information collected and recorded about a project or study, the more likely we can fill out metadata and search-engine friendly pages to help others discover that project existed, and to find the data once it’s published, adding value whenever new investigators find new applications for the results. For the last decade and more, we have been building and refining metadata sheets that harvest all this necessary information and allow us to be confident that we can represent the project properly both within our own data system and into other more general data systems. This work is ongoing, a recent example would be the tracking of map codes on instrument deployment, and it’s important that we are always listening for new things that need capturing, and looking for ways to make things easier on the people on the deck or on the ground.
By having a well defined set of metadata that can be understood (by humans and by computers), partners will also find it easier to collaborate firsthand, and in telemetry research, you derive so much benefit from sharing data with your neighbors, and the world
Researchers want to compare notes and metadata not just to analyze data, but to plan future deployments, to understand how programmes rely on each other or on network-maintained core lines to supplement observing efforts. It doesn’t hurt that one can implement and share common telemetry data analysis workflows since they will be anticipating the same format of input metadata and data. To maximize that benefit, we partner with folks who are writing analysis packages and help backstop these institutional or volunteer efforts where they are serving the community and filling in important parts of the data pipeline, making sure that our data is fit for their purposes, and confirming their packages can act on Node-structured data.

4 of 21

OTNDC / Node Data Workflow

Labs/Institutes report data to data center, as collected via a variety of platforms and technologies
Standardized ingest into the Databases, QCed data matches animals detected across projects/platforms
Data reported to tag owner, w/ links back to all contributing projects
Data format for detection extracts are expected input into telemetry analysis software in R and Python

OCEAN TRACKING NETWORK

So under the framework of our CFI funding, we as an organization are meant to provide ‘infrastructure support’, and OTNDC is the digital infrastructure support that guides information from the field book to the Postgresql database to the detection report to Zenodo to OBIS. Organization-wide, we get this done in a few other, tangible ways. OTN funds and maintains a set of core acoustic telemetry receiver lines around the world that researchers can leverage and rely on when designing their own studies. We loan out receivers we aren’t able to use or maintain in order to supplement others’ studies. We co-design and help innovate new ways to attach listening equipment to new platforms, like our Slocum and Wave Gliders, and more recently to ocean physics drifters and ROVs, and that helps lower the engineering costs and smooth the way for new research projects to adopt those techniques in turn.

Within the data shop, beyond just providing metadata standards and software, we’re also a human resource. The OTNDC contribution to our infrastructure mandate for us also means being on standby to receive, quality-control, and aggregate the data that’s reported to our databases from individual projects, labs, institutes, citizen science programmes using any or all of these technologies, performing the cross-network matching of the QCed detections, and reporting the most complete picture of that data back to the researchers who deployed the tags. These researchers can then take their network-wide detection report, called detection extracts, which are in a nice tabular data format, and use them as the input into that community-built open source telemetry analysis software.

5 of 21

Why the Node paradigm?

Tobler’s first law of geography:

“Everything is related to everything else. But near things are more related than distant things.”

Data collation and management within regions is the proper-scaled approach to organic collaboration and coordination of effort

Nodes allow regional organizations to cross-reference data and metadata with OTN and provide more data to their end-users

Report-once, track-everywhere from the PI perspective simplifies reporting of metadata and data and keeps a project’s data authoritatively in one place

OCEAN TRACKING NETWORK

Something that I’ve been convinced of in my first few years of supporting the OTN data system and throughout our transition to the Node model of collaboration, is that we are all very susceptible to the first law of geography. Researchers tend to be very interested in what’s happening nearer to their study area, and then, depending on their study animals, perhaps progressively less interested in what’s happening further from it. They also tend to share views on what should happen to their data with their nearer neighbors, are more trusting of their neighbors to safeguard their intellectual property, and are likely to agree on how best to work together to cover more area collaboratively with their listening stations.

And for most study species that are observed using acoustic telemetry, the global scale isn’t going to be the most important scale. So to be optimally relevant and useful to a global community of researchers, we at OTN have to empower more localized communities and collaborations. On the data management front, that means we empower the people who have been tasked by their networks to aggregate and disseminate data, to enable them, enable -you-, to guide their data through the quality control pipeline and aggregate it in exactly the same way that we do it in the OTN shop, lightening the burden that is placed on you as regional data managers and allows the data you receive and aggregate to be represented and trustable all the way up to the global picture.

The reality is that this kind of data aggregation isn’t something you only do once you have already designed a database and coordinated data collection techniques with all your prospective colleagues. It starts most often as a shared network folder among a core group, as a series of emails between researchers, as a promise to scour each other’s data for tag detections that belong to your friends. These methods are a great sign that people trust and value one another, but they do not scale up very well at all, three researchers can keep in touch, but twelve? Twenty? Because the value of sharing detection data grows with everyone who agrees to do so, it’s often the case that the will to collaborate scales up much more quickly than the capacity of any one person to facilitate it. This is where we can use the portability of Nodes to make an impact, facilitating sharing among growing communities and at the same time guaranteeing a standard that can follow any contributor’s data all the way to the global aggregate. And if all of our data sharing agreements are sufficiently aligned, this provides global matching to data providers for a participating Node.

So this is how and why we’ve come to support this strong regional organizing tendency, both at a policy level, and with technology, keeping regional data management regimes empowered and in control of their constituencies' data where it’s possible. To this end, we designed our database so that it’s easy to maintain your own structure, certify and train the communities and operators who have adopted them into participating Nodes, and share and co-develop new additions to the structure and code that we all use to populate, QC and aggregate our data.

We have maintained in-house versions of our database for externally-managed regional communities since 2008 and the beginnings of OTN, first with OTN and the NorthEast Pacific node in North America and the ATAP node in South Africa. Around 2016 we started sharing out the Node software for independent operators in the FACT Network in late 2016. Since then we’ve sought to increase the autonomy and capacity of each regional node, and bring more networks into the system, hosting training sessions like this one to bring new folks on board and to keep node managers up to date on the latest tools and techniques. We try and hold these training events once a year, depending on global circumstances, and global circumstances have been poor recently! So we are here with a larger-than-usual, two-years-in-the-making Node Manager Training this year.

One final thing worth mentioning, is that investigators are no better at staying within borders than their study animals are. From a PI perspective, the intercompatibility of our nodes means that we can support their membership in the many networks and help them to respect the policies that go along with their programme’s activities in each locality. This ends up being a much nicer arrangement for the PI than managing membership in multiple networks. They report their data regionally to their closest collaborators, and the software takes care of ensuring that they benefit from the global network, and that a single, authoritative repository of their data is maintained at the place they are most likely to update and curate it regularly.

6 of 21

OCEAN TRACKING NETWORK

Detection Event

Detection Data

Receiver Deployment

Tagging Activity

Receiver deployments: generally uncontroversial and publishable data. Useful for informing potential collaborators of existing equipment deployed in their intended study areas that could detect their tags.

Detection data: protected but not very informative without associated tagging activity data to add the ‘what’ to the ‘where’ and ‘when’.

Tagging activity: The history of which tags are in which animals, where those animals were released, how long the tag will live, + measurements and observations made at tagging time. �OTN Embargo upon request

= tag life + 2 years

Three main components of acoustic telemetry data

So to get into some detail on how OTN’s data policy affects the way Nodes share data, it’s important to outline how we think about each type of data in turn. At the entrance to the data pipeline, there are generally three big kinds of data (plus project-level metadata), that we all need to process in order to produce animal presence information, and we’ll deal with each of them over the course of the week. The most interesting thing about this graphic to me, is that two out of the three pieces of information that fix any animal detection in space and time come from the humans recording them, often by hand. [CLICK] Receiver deployment history gives us the location for a detected tag. [CLICK] Tagging activity gives us the animal that was carrying the tag with that code at that time, [CLICK] and then the instrument itself records times and tag codes and matches those two bits of information together to give us this animal, at this time, in this place. The tagging metadata is the part of the system that most networks consider to be the sensitive piece, and so for node data systems, it is against each tagged animal record that we can apply an embargo period or end date to easily manage the release of potentially sensitive information into the public sphere. OTN’s embargo switched in 2024 to become opt-in, any researcher can get the two-year embargo but they must request it at project registration time. Those embargoes operate the same as they always have, the time begins to tick down for an individual animal at the moment the tag batteries are estimated to have expired, and by default last 2 years past that tag enddate. Researchers who have received that embargo can waive it at any time to speed up the publication of their data, or apply to our ISAC for an extension to slow it down.

These are OTN’s policies for the OTN node here in Halifax, which is still the largest by most metrics but is being given fierce competition from some of the networks here this week. Networks and their Nodes are not bound to have the same exact policy as OTN, and can design and implement their own rules for publication that will follow the tag information registered in their Node as it collects detections from projects within their Node or from other participating Nodes. The key rule that we all must agree on, that guides how the policies will interoperate and interact across our systems is this: the tagging researcher’s data policy/data sharing agreement will follow it throughout all the affiliated Nodes. So for example, detections of OTN animals in other networks will be bound by the OTN rules on embargo and publication, and in turn detections of FACT animals in OTN are bound by the FACT User Agreement.

7 of 21

What’s in a Node?

Direct inter-compatibility w/ other OTN-style nodes
Detection resolution between nodes
Same QA/QC process and data output formats
Develop new processes alongside OTNDC + comm.
Periodic training sessions and ongoing support for node managers

OCEAN TRACKING NETWORK

Shared tools, common processes

So at a high level, that’s the game. By sharing a common set of processes, we can all run to QC incoming data, and by having the exact same format in the OTN database as in each of these nodes, we can achieve inter-compatibility. The common QA/QC processes give us confidence in making comparisons and matches across these distinct data systems, and the output formats are standard between nodes, meaning they’ll be compatible with the analysis packages that are designed to accept them. That helps us share analysis tools as well among all these disparate communities of researchers.

The other major benefit about OTN having so many collaborators globally using the same system, is that regions will discover new use cases and configurations, new problems and technologies that we will all have to become fluent in. In building capacity and connectivity between you coordinators and database managers, we also get insight into emerging needs of all of our communities and are provided an opportunity to collaboratively tackle these challenges. This makes the extra overhead of open development and rigorous change management worthwhile, to be in the know sooner about new ideas, and to get our system caught up to speed with the help of data folks from around the world.

As a link in the chain between researchers collecting animal presence and movement data and the global understanding of marine biodiversity, OTN and our partner Nodes have the ability to work together across regions and disciplines to securely and accurately move information towards decision makers and help our researchers create change from local to global scales.

But of course OTN likes nodes. We’ve been working on the concept of Nodes for many years now, and there’s now a mature observing system that spans three Nodes and OTN itself on the Atlantic coast of North and Central America

8 of 21

OCEAN TRACKING NETWORK

OTN NODE NETWORK

Federated global biologging observing system

Acoustic array projects providing interoperable coverage

Loaner program partnerships

OTN partnerships with provincial organizations and DFO cover a great deal of the St Lawrence and Gulf of SL

New coastal capacity in the NE Pacific will compliment MigraMar, PATH, PIRAT and OTN historic efforts

Map of historic acoustic receiver coverage (2002-2023)

(original produced for Bangley et al. 2020)

So if you saw the Chuck Bangley paper in 2020, they were touting the recent availability of data across ACT, FACT, and OTN. Since that paper was published, MigraMar has extended the reach of this system from the Mexican Caribbean. Here we have an example of nodes in practice, if we look at this receiver coverage map from last year, we can see how well facilitating this intercompatibility with our partners has allowed researchers to track their animals along some truly amazing journeys. The strong collaborative spirit across the communities of ACT and FACT and OTN in the Atlantic arena, supported by our OTN database nodes, has helped researchers capture and catalogue movements across the entire Northwest Atlantic from the Arctic through the Caribbean.

Success stories like the Atlantic collaboration will continue to make the case for OTN to our funders. In turn, OTN will assist with the continued expansion and growth of these efforts through data sharing collaborations and equipment loans to researcher across the global community, and help coordinate and build new capacities for data sharing, reporting, aggregation, and archival at regional, national and global scales.

9 of 21

FACT Network

Node since 2017

March 2022

Joy Young, PhD

10 of 21

What is FACT?

The FACT Network is a grassroots collaboration that is dedicated to improving the conservation and management of aquatic animals by facilitating data management amongst researchers using acoustic telemetry, providing a community for scientists and building stakeholder partnerships.

Dani: Just as a reminder as we move through this afternoon’s session, when you see a red microphone icon in the lower corner of the slide, that indicates that this part of the presentation is being recorded. [START RECORDING] So for this afternoon we have already begun the welcome. We are going to continue by giving a recap of how we got here, what input we received from the summer FACT meeting and how we have updated the data visualization tool. We will have two panel sessions, one from funders, and another from other regional telemetry networks. We will discuss how we have safeguarded this data and what would happen if someone attempted to use it in a journal article. We are also going to provide you with resources to help get this data visualization tool off the ground in your organization as well as walk you through the sign on process. As we close this afternoon’s session we will have discuss what the final steps are as we come to the end of the grant, have a break out session to capture people’s thoughts and then it will be time to wrap it all up. So to start off our next section, I will turn this over to Bob Ellis. Bob, the floor is yours.

11 of 21

Why did we choose a node?

Our system was broken, unsustainable, and maladapted for the increasing size of our network and needs of our researchers.

OTN offered a working, proven system. No need to reinvent the wheel!

12 of 21

Challenges

Buy in

Collecting new types of data

New can be scary

Dedicated effort

Updating data policies

Logistics

13 of 21

Overcoming Challenges

New can be scary

Buy in

Collecting new types of data

New can be scary

Dedicated effort

Updating data policies

Logistics

Staggered introduction, data workshops

Clear instructions

Benefits outweigh the concern

Partnerships with OTN, SECOORA and the ATN

You just got to do it.

Data Manager funded by the ATN through SECOORA

14 of 21

2,101 Stations

1,272 ‘active’

211.5 Million Detections

92 mil matched

45 mil unmatched

13,630 Deployments

4,338 deployments 2019 - 2022

FACT NETWORK NODE

8,028 Tagged Animals

2,663 active tags

108 Species

Boney fish, sharks & rays, alligators, aquatic turtles

U.S. >$4,700,000

Telemetry assets currently in the water

15 of 21

Connectivity

Big Selling Point!

The FACT node increased in size not only due to the contributions of FACT members, but also the ease of matching tags and detections between networks.

Researchers benefit from:

Larger study area

- Quality controlled (equipment level) data

- Standard format that is easily ingestible into existing tools

- No burden of finding tag/receiver owners and sending data

= More time doing actual science!

16 of 21

Cross-matching

FACT

OTN

ACT

MIGRAMAR

In 7 Data Events…

1,693 unique connections from 347 projects (within FACT and between FACT and other compatible networks.

Includes 120 OTN, 48 ACT, and 6 Migramar projects

17 of 21

COOL STUFF! Informative web tools from aggregated data

The DaViT- an interactive web map that displays animal movement metrics (distribution and range) by species.

*Data are pulled from extracts, requires no extra effort by researchers

www.secoora.org/fact/data-visualization-tool/ (not live)

18 of 21

A Node… you won’t regret it

Me at every presentation talking about nodes for the first two years

19 of 21

Training Schedule

Node Concept

Data Policy

Introductions!

Node Struct. + Outputs

Setup and Install

Data Loading Workflow

Project Metadata

OCEAN TRACKING NETWORK

Monday

Tuesday

Wednesday

Thursday

Friday

https://ocean-tracking-network.github.io/node-manager-training/

Data Loading:

Tagging Metadata

Data Loading:

Deployment Metadata

Detection Loading

Validation

Tag and Detection

Matching

Movers Workflow

Visualization and Reporting

(NMT Dinner)

Fixing Data Errors

The Data Push

OTN “All-Hands Meeting”

Supplementary Notebooks

Notebook Improvement and Development

Tandem Real Data Loading w/ OTNDC

Social Outing

So before we talk about some of the Policy specifics, here’s a schedule for the full week we have planned. The lessons we have prepared are very hands-on, each of you will work as data managers in the same test Node and we have prepared for each of you a folder full of data that you will process and load. Because of this hands on kind of work, some of the lessons may take longer, some may take less time, but generally we will have large blocks of time for each data type, sessions that you can work through the prepared lessons on how to use the software to deal with each data type and in the end, we’ll go through how to generate detection data products from the database for your contributors. We’ll be in this room most of the week except for Tuesday afternoon,. they’ve got an exam in here and so we’ll have an alternate room . We will also come back to help out on Friday if you are interested in getting started with putting data into your own Node. We will walk the folks who have a Node already set up through loading real data into their Node.

One thing I may or may not be able to dive into - the ‘All-Hands’ meeting on Thursday afternoon is actually Fred’s and Sara’s retirement party in the guise of the OTN Holiday Party. We happen to overlap with the secret date and so you’re all invited to join us for the sendoff for our Executive and Scientific Directors. And Kim’s organized a group to go to a hockey game on Friday night, if you’re in town still and doing that it will be after we wrap up for the week but before many of you catch your plane out of here. If you need a ticket, we can try and help you get a hold of one.

So right after this we will have a quick chat about the Data Policies, then I will hand it over to Angela Dini, OTN’s Database Manager to walk us through the structure of the OTN data system and its outputs, then we’ll ensure that you have your installations all taken care of before going into what the data loading workflow looks like in more detail, and hopefully we’ll have time today for the first part of starting a new project in the data system, the loading of the project-level metadata, which will be guided by Shannon Landovskis and Caitlin Bate, who are our Field Operations / Data Acquisition Coordinators (FO-DACs)

20 of 21

Detection Event

Detection Data

Receiver Deployment

Tagging Activity

OCEAN TRACKING NETWORK

Data Policies, Data Sharing Agreements, User Agreements

No documented data sharing agreement / user agreement yet:

N-PACT

RAFT

Updates required for node operation:

iTAG

So, back to this slide here. I’m going to try to use it to illustrate what gets published using the diagram and some network-style overlays.

OTN: Publish receiver deploy locations immediately via GeoServer and ERDDAP. Publish tag release data immediately via ERDDAP, publish animal tagging metadata related to that, if unembargoed, to OBIS. New in 2025: Share species data with receiver operators only if publicly available. Make a private collated ‘detection extract’ for researchers to give them one final dataset they can easily use to edit their stuff.

FACT: Don’t publish receiver deployments, share species data internally with receiver operators even if that data’s embargoed, but include a clause that demands a conversation about collaboration if a conclusion in a report or a paper could not have been reached but for the data that was shared. We call it the ‘but-for’ clause for short.

FACT publishes a subset of their data heavily aggregated and generalized, into a tool called the DaViT. This shows receiver locations to two decimal places of lat/lon, and shapes that represent the weighted detection frequencies per-species and project, per month. This is a compromise position from publishing the receiver locations and animal release locations, because of anticipated and historic trouble with interactions with the ocean-going public.

Many other networks on joining, chose either the OTN-like or FACT-like data policy as their starting point. Notably, ACT, MigraMar, and PIRAT adopted policies that were FACT-like, whereas PATH is more OTN-like in that they have large datasets historically that they are looking to share openly and on short time scales.

Recent efforts to create sensible public data products that update in near-realtime have been undertaken, in part as a way to meet reporting expectations in the US. BOEM is looking to have investigators publish BOEM-supported data right away, and OTN provides a pipeline for those investigators to opt into that sort of publishing, but more recently, offshore wind energy concerns who are looking to use acoustic telemetry as a solution to monitor interactions with their offshore platforms are looking for a way to catalogue their efforts in a public-facing place, leading to small updates to policies allowing for the creation of public geospatial data products from some of the US Nodes.

Some nodes are currently in the process of finalizing their Data Sharing / User Agreements (N-PACT / RAFT), others have existing Agreements that take them most of the way to a FACT-like network, but without the data sharing component (iTAG has the but-for clause and lots of ‘should’ rules but fewer ‘must’ rules and no inter-network language). If the policy updates make them OTN-like or FACT-like or something new, there’s one clause we all have to keep to let this ecosystem operate in harmony:

1 of 21

2 of 21

3 of 21

4 of 21

5 of 21

6 of 21

7 of 21

8 of 21

9 of 21

10 of 21

11 of 21

12 of 21

13 of 21

14 of 21

15 of 21

16 of 21

17 of 21

18 of 21

19 of 21

20 of 21

21 of 21