2 of 11

Typical ‘data’ journey

Aspects of a ‘data analytics pipeline’

Data

Acquisition

Data

Cleaning

Data

Analysis

Data

Visualisation

Research

Question

Informed

Decision

You may recall this 'typical data journey' diagram we've presented earlier.

So within Aleks' presentations on Data Organisation and OpenRefine, she covered tools and techniques to help with specific parts of this pipeline at the Data Acquisition and Data Cleaning stages, which - in addition to the Data Analysis and Data Visualisation stages - help you to arrive at an informed decision about a research question you have.

You may also recall that we looked at the Bash shell, and that it can help you at any stage in this pipeline.

The cloud, in this respect, is the same
There are many useful services offered by cloud providers that can help you throughout your research process
In essence, if you need more storage space for your GBs or TBs of research data, or you need more compute power, and you have the money to pay for it, Cloud could help - and in some instances it's free
But as we'll see, that comes with some important caveats, and other options may be available, particularly at the institution level, that you may want to consider

3 of 11

What is the Cloud?

The storage and access of data�and programs over the internet

4 of 11

Cloud in context

Institution

Institutional resources

Image: Wannapik

Personal computer

Cloud-based resources

So let's take a look at where the cloud fits as a resource that researchers can make use of

The place you'd usually start is with a computer in your office - or nowadays perhaps, in your home
Originally, if your programs you'd use in your work - for example, for simulation or data analysis, was running too slow, if you had the funds you could buy a more powerful computer [click]
You may also have the opportunity to make use of institutional resources, such as Southampton's IRIDIS supercomputer, or an institution-wide filesystem, which you could access primarily from anywhere on campus [click]

Typically, some of these services are free, some are paid for

Now since the rise of the internet, many bigtime successful internet players like Amazon and Google started to realise that all the computational infrastructure they were using to run their businesses was itself a valuable commodity [click]

And that they've had decades of experience in managing these infrastructures
So they began to sell access to these resources
And gave us the ability to consume more when we needed it [click], and less when we didn't [click back]

And so this is where we find ourselves - with access to tremendous amounts of compute power and storage

So then, the cloud is a set of computer resources somewhere else in the world, with a veneer over the top of it that allows us to access them, to use them, and also to manage how we use them.

But something to note is that as responsible, ethical researchers the further we venture from our personal computers the morewe need to be careful, particularly around how we manage our research data on these systems

5 of 11

Who uses it?

Anyone who delivers internet services at scale

Now you've almost certainly used the cloud already, either as a consumer or creator of content, likely both.

Underpinning a lot of services we use, are cloud providers - businesses supplying computers and storage that are accessible over the internet that host services we use everyday.

And the biggest players at the moment in cloud are Amazon Web Services, Microsoft Azure, and Google Cloud Platform

As a consumer, you've probably heard of Netflix and Spotify

They make extensive use of Amazon's Web Services - or AWS - cloud to provide video and music content
Accessible from anywhere, on more or less any device
There are even fridges you can buy now that have Netflix
And of course, there's Amazon itself, hosted on AWS
GitHub - which John will be telling you about shortly - is also hosted on AWS
And interestingly, although GitHub has been owned by Microsoft since 2018, it's still deployed on AWS and not Microsoft's Azure

And talking of Microsoft, they have OneDrive

Which give you access to more data storage - where you can upload and download your data from anywhere
With a free offering and more space if you pay for it

Also Ebay and Microsoft's Office 365 - all hosted on Microsoft's Azure infrastructure

And how about Google Docs, and Facebook?

They're hosted on Google Cloud Platform

You'll notice a common theme - "accessible anywhere"

This is important to do as a Cloud provider, since Cloud is a thing that is hosted elsewhere and isn't directly on your computer
It needs to be accessible to work!

And it's not just anywhere. The cloud is everywhere.

It's the utility infrastructure of the 21st century that provides the computing power and data we use everyday.

6 of 11

Why Use Cloud Computing?

Compute, storage, capability
Pay for what you use
Choice, flexibility

Reliability, availability
Lower overheads
Collaboration

So why would we - as researchers - use cloud computing?

The key reasons are that it offers something over what you have in your lab, office, or institution, that helps you do your research better or quicker; so perhaps either

More data storage
A more powerful computing resource to run your programs quicker,
Or specific software that gives you the capability to either do something you couldn't before, or do it more efficiently

And a core idea of cloud is that you only pay for what you use

Cloud providers typically offer ways to upload and access your data and programs, and you're able to see exactly how much space and computing power you are using and how much this all costs
If you need to use more, you generally can, quite quickly - but of course with paid services your bill goes up too

Related to that is choice and flexibility

Many cloud providers offer an often bewildering array of computing and data resources you can use, from very small remote computers or storage to the very large and sophisticated
If you need to change from a smaller resource to a larger one - say you need a bigger processor to do things more quickly, you can. And conversely, you can downsize too

Now, many Cloud providers have been around for a long time

So they've generally become very good at providing such resources reliably
This means any programs or data you put on the cloud, or make use of from the cloud, is very likely to be available when you need it

And since you don't need to maintain the infrastructure such as the machines, the operating systems, the electricity, etc. your overheads are lower.

A key advantage is about collaboration

Cloud offers all its services remotely, and there are generally many options that allow you to grant access to others
For example, to other project collaborators who may need access to your Cloud data

7 of 11

Drawbacks of Cloud Computing

Complexity, support
Data ingress, egress
Data privacy

Software licensing
Downtime
Vendor lock-in

So this all sounds great. Why would we not want to use cloud computing?

There are definitely some things we need to consider

Cloud offerings are often bewildering in their complexity, and the learning curve - beyond the trivial - is often steep

Plus, when you run into trouble, seeking support to solve your problems is often difficult

In order for the software in the cloud to use your data, you need to get the data into a cloud resource to access it

And eventually, you'll also need to get the output data out and back to your personal computer
If that's a few MBs or GBs, probably fine. But if it's much larger than that, it may be a problem and take a lot of time (and possibly cost)

Related to this is data privacy

Putting research data on other systems that may reside in other countries, we must be careful depending on the nature of the data
We need to ensure that we are still conforming to the relevant policies and guidelines regarding how we manage research data
But also institutional, national, and even international policies and laws
In the UK, we need to conform to things like GDPR, for example
If we're moving and storing sensitive data around and in the cloud, are we doing this "correctly"?

There's also the legalities of using software in the cloud

For example, if we install a piece of software in the cloud, does the software's licensing conditions allow us to do so?
We may have a University site licence for the software, but does that extend to the cloud?

And whilst cloud providers have a lot of experience in providing reliable infrastructure, and downtime is relatively rare

There could still be problems - an element that is entirely our of our control

There's also the issue of vendor lock-in

When we invest time and resources in learning and using one cloud technology, it could prove difficult and time consuming to move to another if we need to

So we need to bear these things in mind

------

So what's it good for? What's a solid scenario for using Cloud?

Maybe you have quite a lot of data that needs to be processed, and it's going to take a very long time to process it on your own machine
And, of course, you have access and means (either via funds or prepaid subscription) to be able to use a suitable Cloud resource
You have the rights, in terms of the use of this data, and the software you need to process it

So either you've written some code yourself that is computationally very intensive - it takes ages to run
Or maybe you use a software package developed by others to do it, which you are able - legally and technically - to run on the Cloud - or perhaps it's already available on the Cloud resource
But in either case, the software is readily installable and usable on another machine

8 of 11

The flavours of cloud

Infrastructure as a Service - IaaS

Compute servers, data storage, networking

Platform as a Service - PaaS

Programming platform, software components

Software as a Service - SaaS

Full applications, data management

So when you're out there looking at what cloud offers, you'll likely come across a few terms that describe the types of cloud solution that exist.

Now each of these types of offering builds on the previous ones, like a layer cake.

Each layer offers something new.

[click] So at the lowest level, you have Infrastructure as a Service, which you may see abbreviated to IaaS

This level offers compute servers, data, and networking
And only these things - so you need to add to and manage everything on top of that
Usually this is in the form of a machine somewhere you can control and use remotely
So you'd install all the software components you need on it yourself
And also manage any operating system upgrades that are needed

[click] Next, building on IaaS, you have Platform as a Service

So PaaS builds on everything IaaS has
But things like operating system upgrades are done for you, and it can also offer more on top of that
This is perhaps a programming platform of some kind, together with software components
On top of this, you can write your own applications or software

[click] Add lastly, you have Software as a Service

This takes PaaS one step further, and offers full applications that you can just use
So this is things like DropBox, Google Docs, GMail, Microsoft 365
You use the applications they provide

At the end of the day, it depends on how much control you want.

IaaS gives you full flexibility, but you need to manage so much more to get a working piece of software.

At the other end, SaaS is less flexible - you get a specific software service that you can use and that's it.

But if that software is exactly what you want, that's less you have to worry about.

9 of 11

Example: accessing a Cloud "machine" using Bash

So here's a taster of how we could get started with a Cloud resource.

There are many ways, but here's one that builds on the Bash Shell.

Let's assume we're in need of a machine that's going to run a data analysis program we've developed over some data which is going to take some time to run

First, we've created an account with Microsoft Azure

[click] Then, using Azure's website "portal' as they call it, we create a machine we can use

Now this isn't a physical machine. It's actually what's known as a "virtual machine"
There's actually a clever piece of software running somewhere on a real machine that creates and manages this virtual machine
And for our purposes, the virtual machine will appear very much as a physical machine would

When we create this virtual machine, we're able to specify what kind of machine we want

For example, how much memory we want, disk space, operating system, and so on
Since Microsoft has cloud infrastructure all over the world, we can even choose which country and region to run it in - in this case, somewhere in the East US
So in Azure's case, we fill in a web form with these details, and click a button to create it

Once created, it sits somewhere in Microsoft's cloud infrastructure - and now we need to access it

One very common way to do that is via the Bash Shell [click]
We use a command in Bash called "ssh" - or Secure Shell - to access this virtual machine on the command line
And once we've done that, the commands we type are relayed to the virtual machine, and the responses returned

Once we're at this stage, we can control this machine to install programs, upload our data, and run programs over that data

And once complete, we can download that data to our local machine - include in publications, etc.
We can then reuse this machine - the more we use it the more it costs of course
And when finished with it, we can use the Azure portal to destroy it
Or create other ones, for other tasks, and so on...

10 of 11

Where to go next?

Check out your institution's computational and data offerings, e.g. at Southampton

5TB OneDrive storage / MS Teams Shared Storage (and Office 365)
University Research Filestore
https://sotonproduction.service-now.com/serviceportal?sys_kb_id=befee780dbf08890d5c86055ca9619f7&id=kb_article_view
IRIDIS compute cluster - https://www.southampton.ac.uk/isolutions/staff/iridis.page
Virtual Machines - https://knowledgenow.soton.ac.uk/Articles/KB0011901�

Try out cloud Free Trials and Free Tiers

MS Azure, Google Cloud Platform, Amazon AWS
Azure for Students: https://azure.microsoft.com/en-gb/free/free-account-students-faq/�

Study your University's data policies, e.g. at Southampton

https://library.soton.ac.uk/researchdata/policies

Check out your institution's offerings in terms of computational and data resources

Both from institutionally local resources - such as data storage, local supercomputer
And any arrangements they have with cloud providers which give you a good deal, or perhaps free credits
At Southampton, for example, a local data resource is 5TB of data storage for every student and member of staff, provided by Microsoft via OneDrive

Also MS Teams Shared Storage, tailored to collaboration
There's the University's institutional filestore for higher volume data storage
More information here - the website isn't great!
Often difficult to find what you're looking for, they don't make it easy

Plus there's the IRIDIS supercomputer you can get access to for running programs
iSolutions also provide paid-for virtual machines, which are quite cheap - warning that it might take a while to get them set up
But if you have any specific questions, create a ticket on Southampton's ServiceLine to get in touch with iSolutions

You can also try out cloud solutions for free

Probably for the more technically minded, who want to start writing or deploying their own applications on the cloud
There are loads of tutorials to get you up and running
There's Free Tiers, where up to a certain amount of resources used, like compute or data is free
There's also Free Trials that go beyond those Free Tier limits, where you get some free credits to use more of their services within a limited time
And there's MS Azure for Students as well - where you get $100 of Azure credits a year

If you haven't already, I'd also recommend taking a look at your University's research data management policies too

The Southampton ones cover local policies as well as UKRI funding council policies

1 of 11

2 of 11

3 of 11

4 of 11

5 of 11

6 of 11

7 of 11

8 of 11

9 of 11

10 of 11

11 of 11