1 of 20

Edge computing for disaster relief operations

Nicolai Vatne

2 of 20

Scenario

The first few hours after a natural disaster are often mission critical.

A quick and organized response.

Existing infrastructures - “Nødnett”.

Voice-based communication.

Tightly bound to departments, not the complete response.

The first few hours of a natural disaster are mission critical, due to the fact that lives are at stake.

A quick and organized response is necessary to save lives, and giving the response teams a wider range of utilities, can only prove beneficial for the cause.

Here in Norway, we have the emergency response network called “nødnett”. “Nødnett” is based on the TETRA standard, utilizing only voice communication. TETRA is a european standard for both mobile and two-way communication.

Recently, innovations in the field of 5G has pushed researchers to experiment with replacing this solution all together, towards the more favorable aspects that 5G would offer.

Lastly, it is worth mentioning that Nødnett is often bound to departments, making communication between multiple departments harder. In earlier cases limitations as to TETRAs capabilities have became apparent, when multiple departments had to work together.

Nødnett is sepearted in to multiple slices of communication, making communication across departments tough. This is a point that has been emphazised in earlier cases, where nødnetts voice capabilities showed its limitations

3 of 20

Scenario ( before )

Foto: Anders Martinsen, UAS Norway

4 of 20

Goal

Explore the capabilities & opportunities of an ad-hoc network for a HADR operation.

Video and sensor data can offer valuable information for first-responders in the early hours of a natural disaster

Developments in key-technologies such as distributed computing, IoT and 5G drive innovation in Edge computing

Unifying technologies in IoT can offer an extensive infrastructure with massive capabilities

We worked on a lot of different components and architectures, but broken down, we can say that our goal was to explore the capabilities and opportunities of an ad-hoc network for a disaster relief operation.

Touching on that a little deeper, It is also vital to mention the significant work we put in to validating Kubernetes as a possible solution.

Our work was very practical in many aspects, testing out different implementations and theories in terms of how things would operate.

With respect to our scenario, video and sensory data can offer a new valuable source of information for first-responders in the midst of a natural disaster.

When a natural disaster strikes, existing infrastructures are in most cases torn away, making communication difficult for the first responders.

With the recent developments in fields such as 5G, IoT and distributed computing, we believe that opening up more channels of communication instead of simply relying on voice, can improve the efforts of those responding.

Applying these principles to create a improvised ad-hoc networking solution to be deployed as a temporary measure before surrounding infrastructure is restored, could provide new opportunities for the people on the ground.

IoT has contributed to an increased effort in uncovering capabilities that exist on networks, even adopting in concepts such as Edge computing - which could power a new alternative computational platform instead of relying solely on data centres.

Utilizing the collective power and capabilities of IoT devices has been the subject of research for years past.

5 of 20

Background

Internet of things (IoT)

Raspberry Pi

Edge computing

Kubernetes

MQTT

Pub/Sub, Client and Broker

Internet of things or IoT is a umbrella term used to describe the ubiquitous presence of devices connected to the internet. With a massive increase over the last 10 years in the form of smart watches, fridges, actuators and sensors - we are experiencing a new age of computing, where manual checks are often not necessary due to the massive opportunities that lies in the everyday objects we interact with, both in a private homes and professional industry.

Edge computing has sprung out due to innovations in cellular technology, forcing technologists to think of alternative ways for computing - utilizing the already massive infrastructures to perform computations closer to where end-users operate. Edge computing has posed interesting questions for researchers and companies alike, as to how we can best use the existing infrastructures, along with springing out ideas such as ours, of using small single-board devices to host a ad-hoc network for disaster response.

Our work has taken inspiration on earlier research by NATO regarding applicability of IoT and Edge computing. This research has identified IoT as a suitable infrastructure that could provide benefit to military operations such as disaster relief. Additionally, our work has basis in an earlier ICCRTS paper, which is a valueable resource in terms of Kubernetes Capabilities for military usage.

Further research has also identified the MQTT messaging paradigm to play a key role in interoperability between military C2 systems and IoT infrastructures, due to its favorable qualities for both the sender and the receiver.

Even though our work is at a smaller scale than what Is formulated in their research, it still important to further the ideas to prevent solutions from becoming too proprietary in the long run.

MQTT is considered an event driven paradigm, that is based on the publish/subscribe pattern. If we consider a system that gathers massive amounts of information per second - an event driven architecture helps structure and create logical timelines of the information. MQTT messages also offers ease of use in terms of parsing the information to JSON or any other format, more readily used within modern computing.

MQTT is based on the idea of clients and brokers. A broker is the central entity responsible for coordinating the messages between the participating clients on the network. Along with this the broker verifies the credentials of a client and filters messages based on their topic. There are many widely used brokers today such as Mosquitto, HiveMQ and others with the notable differences being capacity in terms of clients.

6 of 20

5G Edge

Extending capabilities to deliver voice, data and video communication
An ad-hoc networking solution
Ability to operate freely until infrastructure is restored

Example: EU-Project 5G Vinni
Open-Standards & Open Source

Moving on to our approach,

Our idea is to extend the capabilities of response teams to also deliver data and video communication. It will function as an ad-hoc networking solution to be deployed until surrounding infrastructure becomes accessible again. With our approach and mindset, we aim for the solution to be able to seamlessly operate with surrounding infrastructure, if need be.

Our approach, combined with a 5G core beacon would essentially creating a bubble on the edge of the network

The EU-project 5G Vinni is a perfect example ofthis, allowing for mobile base-stations to be deployed as need be. 5G vinni is also furthering the idea of having dedicated slices within the networks to allow for a separation of context in terms of utilization, which would be a very efficient way to load-balance the network as more actors participate and more data is distributed.

5G Vinni has done experimental research with their hardware being deployed in cars and boats, to allow for a quick response in cases such as natural disasters.

With the previous research in mind, we aimed for our solution to be based on open-standards and include as much open-source components as possible This is a step in the direction to enable federation across multiple platforms, and with military command and control systems.

7 of 20

Kubernetes & K3s

Open-source orchestration tool
Popular approach in many IaaS solutions
A large variety of distributions
K3s - made for IoT and Edge computing
Still in infancy, with a dedicated open-source community
The idea behind “bare-metal” clusters

Kubernetes is an Open-source orchestration tool to manage containerized environments. Containerized environments are simply a way of deploying applications, where the applications or services are packaged as containers.

It is a Popular approach in many IaaS solutions such as Amazon AWS, Azure and Google Cloud Engine as a means of delivering services and applications that are “Always available”.

There is a large variety of Kubernetes distributions,such as Rancher, Docker Kubernetes and Amazon Elastic Kubernetes Service, we are however focusing on Ranchers implementation called K3s.

K3s has been popularized as an alternative for devices running ARM-64 architectures such as Raspberry Pi.

K3s is still in infancy, as most orchestration tools are used and adapted for other larger desktop grade architectures forcing Rancher to think of new ways to approach utilization and usage.

This has in terms led to a large open-source community, adapting a variety of applications and containers for ARM64 devices, and it is this open-source community that frequently uses and develops for bare-metal clusters.

8 of 20

High-level technical architecture

9 of 20

Hardware

Raspberry Pi

Model 3b - 1GB RAM

Model 4b - 4GB RAM

Early issues

For our work, we used Raspberry Pi’s.

Raspberry Pi is a small computing device with the appearance of a circuit board, costing around 15-40 dollars.

We discovered very early on in development when all of our nodes was of the model 3 type, that unexplainable problems arose. It was not before we actually wrote a script to retrieve memory usage, that I discovered that the upper memory limit of the Master node was being exceeded.

After some more testing with even less components I discovered that It was simply not feasible to run the master node with only 1GBs of RAM, forcing me to get a hold of a Raspberry Pi 4 with 4GBs of RAM.

The reasoning behind the exceeding memory usage is not completely clear, as online documentation for the different components, distributions and frameworks we used stated their memory requirement quite confidently. We did however have a short dialog with one of the lead developers at Rancher. He explained that the continuous development of Kubernetes has been pushed in a direction not particularly favorable for smaller clusters such as ours.

I will continue talking about memory a little later, as we consider it an important aspect of our work.

10 of 20

Installation

Modifications to Raspbian OS

Kernel options
Legacy IP-Tables

K3s installation
Resource shifting

Installing Kubernetes is normally considered a tedious task. K3s has simplified the installation down to a few single commands. Yet, with a wide variety of single board devices and operating systems, its hard to make it run out of the box for every single one.

Installation of K3s on the Raspian OS requires a series of configurations prior to installation, such as enabling kernel options for multiprocessing and reinstalling legacy IP tables in favour of the new ones. As an added percautionary measure, running a headless version of Raspian OS is advised, as K3s can become fairly resource intensive in the long run.

As a side-note for anyone attempting to replicate similar systems or results, there is a major lack of documentation in terms of the various issues you can face developing for Raspberry Pi Kubernetes cluster.

Installing and enabling K3s is a fairly simple task. You start of by curling the directory from the K3s website to the master node. Once the installation is complete, you retrieve the masters IP and token and repeat the process on the desired worker nodes. Within minutes of the completing the operations, the cluster will be operational with its default configuration.

As a final percautinary measure, we manually disabled and removed the various processes related to the HUB and bloatware on the devices to shift the most resources we could towards K3s.

11 of 20

Testing

12 of 20

Memory Utilization

Important aspect to determine validity for Edge computing
K3s <= 500mb
The more workers, the more jobs (processes)
Load Balancers

Traefik
ServiceLB
MetalLB

As a means to determine whether this approach would be feasible, we wanted to measure the metrics of how complex it would be to run this system on the different nodes.

The developers behind K3s lists in their documentation that K3s requires by average around 500mbs of memory with a restricted set of components installed. As we discovered during our initial set-up, this was not the case, and on the next slide - we will show a graph with the metrics we gathered from our system.

Even though we used K3s above the minimum, we noticed the pattern that the more nodes we included as workers, the more processes were initialized on the master node to maintain the coordination efforts between these nodes.

As we briefly touched on a few slides ago, we were always working with the mindset of making the cluster as resource caring as possible.

As an example, During installation, we noticed a significant resource usage from Traefik and serviceLB in terms of both processing and memory, these two softwares are load-balancer and service exposer in Kubernetes. As we did not require its services, we simply opted to remove them during installation, freeing up roughly 10-13 percent of the memory consumption on a nodes. As an alternative, we included a innovative load balancer called MetalLB, as it performs the load-balancing directly on the networking card of the devices and was significantly more resource friendly, as it was built with single board computers in mind.

13 of 20

14 of 20

Configurations (Networks)

Goal: Emulating both weak and strong configurations
Mid-Band to emulate a participating 5G beacon
Tactical, NBWF, CNR for military deployments
20ms latency + Loss model

In our work, we provided some information related to the various network configurations we wanted to test. Ranging from combat radio network to modern 5G networks.

And I just want to briefly cover this before we move on.

Our goal was to emulate both weak and strong configurations, to test the different various and compare the results.

We have had a strong emphasis on the developments in 5G in our work, so we were interesting to see those results in particular - as it could potentially strengthen our idea

Tactical, NBWF and CNR are typically configurations for military deployments - and are significantly weaker compared to 5G - but can also be very relevant, which is why we included them in our testing.

All of our tests were run with a cap at 20ms latency, as we regarded anything above 20ms latency to be a fairly unrealistic for a disaster relief operation, as they are normally performed in zones or within constrained areas.

To better emulate the harsh operating environments in the real world, we used the Gilbert Elliott loss model as a means of emulating loss across the networks. We performed tests with 4 different parameters, and displayed the results in the following graphs.

For this presentation, we want to emphasize the results we got from the 5G configuration, as it would provide a more accurate image for what the future might hold.

Revision 14/7:

Vi brukte alle disse i papers, men pga båndbredde o.l så bruker vi 5G som basis/utgangspunkt

15 of 20

Results - K3s Master node

16 of 20

Results - MQTT broker node

17 of 20

Results - MQTT broker node

18 of 20

Discussion

Packet captures revealed congestion

Extreme loss / very low bw break congestion control algorithms.

Bandwidth too constrained

Combat Network Radio & NATO Narrowband Waveform.

Recommended networks

Mid-Band 5G & Tactical Broadband Network

When we reviewed the results, the significant differences became apparent. Obviously, the higher the bandwidth - the less packets lost which generally leads to the less issues. We noted however, that on the weaker configurations, several packets was lost early on in our recordings, once these packets were lost - many of them had several re-transmissions and lost packets over a series of time.

This led us to believe that we were getting halted by congestion. Congestion is a state that can occur in a network when traffic is so overwhelming that it decreases the overall response. And if a network is less than good in terms of bandwidth, congestion issues can frequently arise - leading to packet loss.

In TCP New Reno, there are two different mechanisms, congestion policy and a window. Within the congestion policy, there are also two different phases, slow start and congestion avoidance.

Congestion avoidance is a conservative policy, that adjusts the window at the end of every round trip, increasing or decreasing based on wheter it received the acknowledgement packets in time or not. If this policy does not receive its acknowledgments, it will continuously decrease the window, potentially even leading to little to no packets getting through

We believe, based on our findings during manually analyzing the packets, that this policy could have played a role in the results we got on the weaker bandwidth configurations.

Alternative measures such as Thin Stream extensions can be tested as a means to reduce the amount of packets lost. Thin stream extensions in TCP is a more adapted solution, which detects thin streams - that often occur in time-sensitive applications - and adjusts the acknowledgements accordingly to prevent extreme latencies and so on.

Alternatively a entirely different congestion control mechanisms can also prove beneficial as to validating the results.

Another alternative that can prove just as likely is that the weaker network configurations are simply too constrained for the system we want to run.

Our assumptions are however that both Mid-Band 5G and Tactical Broadband Network performed up to par, and as to our expectations.

Combat Network Radio and NATO Narrowband Waveform, being the two absolute weakest bandwidth configurations generally had such as a high amount of packets lost that we consider them unsuited, in its current state.

19 of 20

Summary

Conclusion

System performance
Applicable as a ad-hoc solution in a HADR operation

Future Work

High Availability cluster / Real-world simulated test

Our work was oriented around exploring the capabilities of edge computing in a less-than-perfect environment to test its capabilities as an ad-hoc network for a disaster relief operation.

System wise, the entire project proved viable as to how we implemented it and how it functioned at a high level. We face a few problems along the way due to the infancy of containerized software built for ARM64 environments, but the thriving open-source community proved to be a helpful tool to overcome these issues.

Given our networking and memory tests, with the subsequent proposed scenario and high-level architecture, we believe that it could be a solution that would benefit responders during a disaster relief operation, as it would extend todays capabilities in many respects.

The lower bandwidth networks proved to have significant packet loss, as to be expected for network within the kilobyte range. We are under the impression that these bandwidth configurations may be simply too constrained. Further testing and research may be beneficial if this was to be relevant in a real-world scenario to fully determine the lowest cap on bandwidth.

We listed a few directions to take this project further, ranging from a more comprehensive messaging paradigm, better utilizing MQTTs capabilities, along with providing support for video transmission.

I believe that this area of technology will flourish in the coming years, and future developments in the field can only further our idea and help solve some of the problems and difficult areas we faced during our work.

20 of 20

Questions?

nicovatne@gmail.com