1 of 105

2 of 105

Who am I? Dr. Chelle Gentemann

Why am I here talking to you?

I’m Dr. Chelle Gentemann. I’ve worked in academia, industry, and nonprofits, mostly doing research for NASA. I’m the lead scientist on a proposal for a new NASA mission. I’m a mom, an oceanographer, and my kids tell me I don’t really ‘work’ because I’m just on my computer chatting with friends, coding, and listening to music. They also complain that I’m not a ‘real’ oceanographer since I just send robots and satellites out to take my measurements. I can’t really argue - it is all true - I love what I do, have fun doing it, and enjoy science adventuring via my internet connection. I was able to complete a cruise off Barbados and deploy 2 more robots to the Arctic during covid. Dr. Perez invited me here to talk about all the different ways we do data analysis in Earth Science to understand the Earth System and how it is changing due to human activities. The datasets you have encountered so far are mostly tabular, perfect for pandas. First I’ll talk about climate change and how we are monitoring in, but most of this data doesn’t fit into tabular datasets, it changes in space and time, so we will also talk about how we can use powerful open source tools, building on our knowledge of pandas, to explore 3-dimensional data. (3:45)

3 of 105

My path…..

1997

M.S. UCSD

1995

B.S. MIT

1998

RSS

2003

U.Miami

2016

ESR

2020

Farallon

2007

PhD U.Miami

2022

NASA

2024

ICSI

4 of 105

My path…..

2012 NASEM CESAS

2013 AGU Falkenberg Award

2016 JPL Mission Invite

2017 NASEM Co-chair OSS policy for NASA

2018 Co-chair NASEM CESAS

2019 Testified House Committee

2021

2022 NASA HQ

$190M Butterfly Mission

NOAA SAB

2025 AGU Open Science Prize

5 of 105

Video credit: NASA SVS

6 of 105

Video credit: NASA SVS

7 of 105

What does the data say about our climate?

8 of 105

9 of 105

Keeling Curve

In 1958 Keeling got a grant to begin monitoring CO2 in Hawaii.
Roger Revelle (a famous scientist) argued that they just needed a snapshot - CO2 was too variable - and another snapshot 20 years later to show that CO2 was increasing
Keeling advocated for precise measurements over time. By the mid-1960s we had both a measurement of the Earth’s breathing and the global increase in CO2.
How would you remake this figure?

https://pubs.acs.org/doi/10.1021/ac1001492

10 of 105

Video credit: NASA SVS

11 of 105

Calculating the Greenhouse Effect

Goal 1: Calculate how the temperature is changing with increasing CO2.

The Planetary energy balance:

Energy (from the Sun) absorbed = Energy emitted

How does CO2 affects this?

The Sun emits radiation that is absorbed by the Earth (~30% is reflected by clouds, ice/snow, desert, this is the albedo)
The Earth emits radiation according to Stephan-Boltzman’s Law: the rate that a body emits radiation (per unit area) is directly proportional to the body's absolute temperature to the fourth power (blackbody radiation)
The emitted radiation doesn’t all go back into space……..

To investigate how carbon dioxide affects the Earth’s temperature, we need to understand how to calculate the Greenhouse Effect.

The planetary energy balance is basically that what comes in must go out. So the Energy from the Sun, travels through space, and arrives at Earth. The energy absorbed by the Earth must equal the energy emitted by the earth. Where does the carbon dioxide come into it?

The Sun emits radiation absorbed by the Earth (some is reflected by the atmosphere/clouds/earth surface, this is called the albedo, or the reflectivity of the Earth

The Earth emits radiation according to Stephan-boltzemans law -- -the rate the a body emits radiation per unit area is directly proportional to the body’s absolute temperature to the 4th power. This is an equation that permeates all of climate science --- the energy emitted is related to the temperature to the 4th.

But…. the emitted radiation doesn’t all go back into space it is absorbed the atmospheric gasses, like CO2 and that is where the greenhouse effect comes into play.

Now, in the next few slides, I’m going to go through the math. I’ll do it one step at a time, but don’t worry, because I’m going to put it all together for you at the end.

First, do you remember what the area of a circle is? Pi * radius squared.

Also, do you remember what the surface area of a sphere is? Four times pi times radius squared.

Remember the Earth can sometimes look like a sphere, sometimes like a circle, it depends…… (9:45)

12 of 105

Calculate energy in

The Sun emits radiation absorbed by the Earth

Energy in equals the incoming sunlight (W/m²) multiplied by the area (m²) to get W

NASA illustrations by Robert Simmon

= Incoming sunlight x area

r

13 of 105

Some of the energy in is reflected by the atmosphere

The Sun emits radiation absorbed by the Earth (some is reflected by the atmosphere)

NASA illustrations by Robert Simmon

r

Albedo = the fraction of radiation reflected back to space by the atmosphere

The amount that gets through:

Reflective surface

14 of 105

Calculate Energy out

B) The Earth emits blackbody radiation

Energy out equals the emitted radiation (W/m²) multiplied by the area (m²) to get W

NASA illustrations by Robert Simmon

= Outgoing radiation x area

r

The equation was actually a data science problem! 1864, John Tyndall measured the color and infrared emission of a platinum filament. Josef Stefan related the emission to the temperature to the fourth. Boltzmann then derived the theoretical model.

Now we need to calculate the Energy out. Again we have radiation times area. The Earth emits radiation according to Stephan-Boltzman’s Law: the rate that a body emits radiation (per unit area) is directly proportional to the body's absolute temperature to the fourth power (blackbody radiation). The equation was actually a data science problem! 1864, John Tyndall measured the color and infrared emission of a platinum filament. Josef Stefan related the emission to the temperature to the fourth. Boltzmann then derived the theoretical model. Stefan used the equation to determine the temperature of the Sun! For the truly curious, it's cool to note that while the law can be derived purely through a classical thermodynamic argument (as Boltzmann did), the precise form of \sigma has h in it, and is thus a quantum expression. This was the problem that connected, at the hinge of 1900, classical physics and the birth of quantum mechanics.

To calculate Energy out, we need to multiply it by the area, but here this isn’t a disk, this is emission from a sphere, so the surface area of a sphere is 4 pi radius squared. (12:15)

15 of 105

Energy in = Energy out

Goal 1: Calculate how the temperature is changing with increasing CO2

NASA illustrations by Robert Simmon

Reflective surface

Planetary energy balance

16 of 105

Calculating temperature

Goal 1: Calculate how the temperature is changing with increasing CO2

What is the Earth’s temperature?

~255 K

~-16 C

~1 F

17 of 105

18 of 105

Our atmosphere is like a blanket

Solve for the greenhouse effect!
What happens to the temperature if we increase the greenhouse effect?
What happens to the temperature if we decrease/increase the albedo?

without an atmosphere

Planetary energy balance

with an atmosphere

19 of 105

Calculating the Temperature dependence on CO2

Goal 1: Calculate how the temperature is changing with increasing CO2

CO2 is ~380ppm (parts per million)

As CO2 increases what happens to the temperature?

20 of 105

Plot the results

Use the equation to calculate the increase in temperature with time due to the increase in CO2

21 of 105

Global warming is a climate crisis

The equation provides solutions to global warming:

Change the planetary albedo to reflect more radiation to space (increase aerosols, clouds, make surface more reflective)

Reduce greenhouse gases (CO2, Methane)

Sun-shades in space?

Back to our equation - Global warming is right here in this equation. The problem and the solution. We need to reduce energy into the system. So, on the left we can work to change the planetary albedo so more radiation is reflected back to space. You could do this by increasing atmospheric aerosols - like a controlled volcanic event, create more reflective clouds, or even change the reflectivity of the Earth surface by creating more ice, making the ice more reflective using glass beads, or painting mountaintops white. Another approach is to look at the right half --- reduce greenhouse gases like CO2, methane, water vapor, by removing carbon, getting the ocean’s phytoplankton to absorb it, planting more trees. Some of these ideas may sound a bit crazy - but we are at the point where all realistic options need to be evaluated. We need to reduce our emissions - that is none negotiable, but to avoid the impacts that are already unfolding and increasing, we need to also work to remove carbon, methane, and increase the albedo. These are scary alternatives - but we can start with small scale experiments, and we have to move forward and invest in this area because the latest climate assessment is frightening and probably everyone in this lecture has been impacted by wildfire, drought, heat, or hurricanes. 1 in 3 americans were impacted by a climate disaster this Fall. (17:10)

22 of 105

Changing by Alisa Singer

"As we witness our planet transforming around us we watch, listen, measure … respond."

www.environmentalgraphiti.org – 2021 Alisa Singer.

AR6 Climate Change 2021:

The Physical Science Basis

You probably here IPCC reports quote in the news. These reports are scheduled, have deadlines for science to be included, and are comprehensive. The 5th assessment is finished and we are now on the 6th, or AR6, the first report out is The Physical Science Basis. The report addresses the most up-to-date physical understanding of the climate system and climate change, bringing together the latest advances in climate science, and combining multiple lines of evidence from paleoclimate, observations, process understanding, and global and regional climate simulations.

Next year, the reports will be on mitigation, then a report in impacts adaptation, and vulnerability, then finally a synthesis report that puts all 3 together. The current report just documents what we see happening in the climate system. Documents with data…. This is data science in action. So what do we see? (18:10)

23 of 105

24 of 105

25 of 105

26 of 105

27 of 105

28 of 105

29 of 105

30 of 105

31 of 105

Image credit: NOAA

Where does all this data come from? So it gets a little warmer….. Do I really care or need to worry about a deg or two of warming? I can’t really tell what the temperature is today to within 1.5 deg Celcius so why worry about it warming that much? It doesn’t sound that bad really, just a little bit warmer, and it is gradual so we can adapt, right? Well, the AR6 report for the first time specifically calls out climate extremes. A lot of the previous research was on how the means were changing, but as we began to see more extreme events, scientists put 2 + 2 together and realized that if you change the mean, you make what used to be extreme events more common. And it turns out that the extreme events are what cause most of the damage to people and property. Hurricanes, flash-floods, wild fires, heat waves in both the ocean and atmosphere, it feels a little bit like the last couple of years we have been under attack. (20:20)

32 of 105

With every increment of global warming, changes get larger

https://www.ipcc.ch/report/ar6/wg1/#FullReport

This is a figure from the report with every increment of global warming changes are amplified they get larger, top figure shows the observed change per one degree of global warming. This is actually from left hand data and on the right hand side now we see the simulated change and this is the change simulated by the models, these sort of comparisons, help us understand what the models are getting right and what the models are getting wrong and where they need to be improved on the next iteration.

One thing that I'll point out is in the North Atlantic Ocean. There's a cool spot in the data that isn’t in the model.

And we think that because the climate models aren't getting the ocean circulation correct. The region with the cool spot is where there is a lot of carbon dioxide and heat absorbed from the atmosphere and put into the ocean’s deep circulation. It's really important to understand where the model isn’t matching the data so that we can look into the physical parameterizations of the model to improve their simulations of our climate. The climate models rely on parameterizations, which are essentially estimates of what's happening. Estimates of the exchange of momentum, the exchange of heat, the exchange of gases, then we use these parameterizations and models to estimate physical processes. Sometimes we get it right, and sometimes we don't.

There's a lot of work still to be done, there's a lot of data still to be analyzed. to help us understand how to improve these models. Now let's look at what the future look like. There is the Earth at 1.5 degrees global warming.

There it is it 2. And there it is 4.

What is your mean temperature, it changes that much, and especially look at the higher latitudes and the Arctic, where it's amplified. (23:50)

33 of 105

… in precipitation

https://www.ipcc.ch/report/ar6/wg1/#FullReport

CC-relationship water vapor - temperature - pressure : 6% per 1C https://www.jbarisk.com/news-blogs/the-physics-of-precipitation-in-a-warming-climate/

And it's not just the temperature, the temperature affects how much water vapor the atmosphere can hold. The Clausius-Clapyron equation relates the energy required to cause a chemical reaction at a given pressure. Warmer temperatures increase the atmospheric moisture content, the humidity, because the atmospheric pressure will not be affected by climate change in the same way as temperature. So, from the Clausius-Clapeyron relationship, atmospheric water content increases by between 6 and 7% per 1 °C. So, as we warm the atmosphere, this heat affects the global hydrological cycle, the way that water travels through the Earth system. So when you change the temperature, you actually change the precipitation as well. And here is the precipitation at 1.5 degrees, two degrees and four degrees.

And many areas are actually much wetter than before, Some are drier, but many are much wetter. What that means is increased flooding in case dramatic rain events,

which in some ways doesn't immediately make sense to people. If it's getting warmer. Aren't we just going to go into drought, and we may have extended droughts, with then heavy intense rainfall over short periods of time so you can still have drought and have an increase in rainfall.

But these changes in rainfall like we saw on the floods in Europe this summer can cause substantial damage to both physical structures and impact on human life.

(26)

34 of 105

Our trajectory from data and models

https://raw.githubusercontent.com/BrodiePearson/IPCC_AR6_Chapter9_Figures/main/Plotting_code_and_data/Fig9_03_SST/Fig9_03_SST.png

our mean is shifting, what we consider normal is shifting

This is a very interesting figure that has a lot in it.

Focus on the center figure.

Now the black line is an average of all the climate models, there's 40 of them, and they have various scenarios to the future and that's why you see the spread on the right hand side of the center figure. the model data, the orange line are the observational reanalysis. These are climate data sets. These data sets, match the models, quite well for the entire period that we have observations.

And you can see the positive trend that starts around 1970 and the increase in positive ocean temperatures that continues through the 2000s to 2020 when the observational record ends, the models continue to match it and then they continue increasing

our mean is shifting, what we consider normal is shifting. (26:30)

35 of 105

Extremes are the new normal

36 of 105

Extremes are the new normal

A decade ago - scientists would argue - we can’t attribute any single weather event to climate.

In the last decade, we have all experienced major shifts in our climate through changes in our local weather and scientists have figured out ‘climate event attribution’

They look at the probability of the occurrence of an event (eg. a temperature extreme) in models run without human-influence and then compare it to the probability in models run with human-influence.

How?

https://www.nature.com/articles/d41586-021-01142-4

https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2021GL092765

37 of 105

Extremes are the new normal

The probability of an event changes

https://www.nature.com/articles/s41467-019-09729-2/figures/1

Model runs with all forcings

Model runs with only natural

How likely is it that there will be a 2 degree anomaly?

Here is the temperature anomaly, which is the temperature minus the mean value, so how is it warming or cooling. Ths is the temperature anomaly from a climate model HADGEM3 from 1960 to 2010. What you see is there is decadal variability. The red lines are model run with all forcings, including anthropogenic. The blue are models run with only natural, not human induced forcing. The blue anomalies are lower. So if you take these data and plot the probability density, you can see how likely it is to get a temperature anomaly. For only natural forcing, it is really rare to get a 1 deg warm or cold anomaly. When you include human induced warming, you see that the mean value is just over 1 deg, and warm anomalies over 2 deg which were never seen in the other runs, are fairly common.

(28:30)

38 of 105

Positive Feedbacks in the Climate System

39 of 105

Goal: Positive feedbacks in the climate system

The Earth is a system - a complicated game of dominos

As air temperature increases, sea ice concentration decreases…..

40 of 105

Goal: Arctic Amplification: a positive feedback

A positive feedback amplifies the initial perturbation

Albedo feedback:

The albedo of sea ice is ~0.5-0.7. Most sunlight is reflected back to space.
The albedo of the ocean is 0.04. Most sunlight is absorbed and warms the seawater.
A warmer ocean melts more sea ice.

reflectivity!

There are two different feedbacks going on and scientists aren’t sure which is the larger contributor… The first is the albedo feedback --- remember (click) albedo is reflectivity.

The albedo is on global average about 0.3 and we talked about that earlier so that's the amount of radiation reflected back into space.

Now the albedo of ocean water, 0.06 the ocean just absorbs all the radiation. Sea ice has an albedo of 0.5-0.7 ---

That's why it's hard to look at snow and ice, it just reflects it all back up to the sky.

Now, as ice melts more radiation is absorbed, because the ocean water absorbs more and it changes the Earth's albedo. More ocean == lower albedo. As we saw earlier, lower albedo, more global warming. So, you get warmer temperatures, it melts more sea ice, You have more ocean, the ocean absorbs more radiation, which increases the temperatures, those increased temperatures melt more sea ice, which results in more ocean, and this is a positive feedback in the climate system. (31:00)

41 of 105

Goal: Arctic Amplification: a positive feedback

A positive feedback amplifies the initial perturbation

Cloud feedback:

As seaice melts, more ocean is exposed and more moisture is absorbed by the atmosphere
More moisture == more clouds
Clouds trap longwave radiation, warming the Arctic, melting more seaice.

42 of 105

Can we look at this ourselves? --- this is global data --- 3D data

….. Latitude, longitude, and time…. How do we handle that?

https://www.nature.com/articles/s41467-019-09729-2/figures/1

Model runs with all forcings

Model runs with only natural

How likely is it that there will be a 2 degree anomaly?

43 of 105

BREAK

44 of 105

Learning objective: climate data analysis

3D data (time,latitude,longitude)
Introduction to Xarray python library
Xarray Probability density functions
Xarray linear regression: calculating mean trends
Xarray global analysis of trends

45 of 105

ERA5 - 5th gen ECMWF atmospheric global climate ReAnalyses

ERA5 combines vast amounts of historical observations into global estimates using advanced modelling and data assimilation systems.
From 1979 - 2019, hourly estimates of atmospheric, land and oceanic climate variables.
30 km global grid, with 137 levels from the surface up to a height of 80 km.

https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5

An ECMWF snow depth analysis for Scandinavia using ERA5 data shows the highest levels in a decade.

https://medium.com/planet-os/era5-reanalysis-data-on-s3-cee2f22422ae

ERA 5 is the 5th generation European Centre for Medium-Range Weather Forecasts atmospheric global climate re-analysis.

What that means is they take VAST amounts of historical observations, put them in a model, and create 30 years of hourly files, globally at 30 km, with 137 vertical levels.

That is some BIG data. This is one of the foundational datasets for scientists - whether you are working on weather or climate. Here is a paper that used ERA5 data looked at snow depth in scandinavia found the highest levels in a decade. A link to the ERA5 dataset and its details. And a link to the ERA5 hosted for free access on AWS. This is one of the most popular datasets to explore our weather and climate and improve our understanding of what is happening, why it is happening, and improve our understanding of the Earth system.

(33:40)

46 of 105

Vast amount of data

The number of observations assimilated in ERA5 has increased from approximately 0.75 million per day on average in 1979 to around 24 million per day by the end of 2018
~14,000 GB (14 TB)
A key dataset used for understanding our weather and climate, but inaccessible to all but a few privileged institutions

47 of 105

Data Frames are not enough: not all data is tabular

48 of 105

Xarray

Xarray introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like multidimensional arrays, which allows for a more intuitive, more concise, and less error-prone developer experience.

‘data variables’ : temperature and precipitation “data variables”

‘coordinate variables’: they label the points along the dimensions.

Let’s think about 3D data. Tables aren’t really efficient representation of big 3D data. You have blocks of data, maybe even multiple variables like temperature and precipitation. When you watch a weather forecast and it shows the weather model, you are looking at a spatial map, in 2D, that moves in time, and that is the 3rd dimension. In the Xarray world, your different observations are called data variables. Then you have coordinates, that label what the points along each dimensions mean. Now, this is where the magic starts --- just like you can use column names to select panda columuar data, you can use the coordinates to select Xarray datasets. This makes for a intuitive, more concise, less error-prone code --- for example you can just ask for the latitude and longitude you want. (37)

49 of 105

Xarray: Dataset

ERA5

Xarray Datasets are essentially groups of DataArrays.

This is really valuable when you are looking at datasets that have multidimensional groups of data, for example, temperature, precipitation, cloud cover.

Some Xarray methods can be applied to all that Dataset contains.

For example, you can subset a Dataset and the subset, interpolate, calculate a mean, and it will do this across all the DataArrays the Dataset contains.

(slow down) students don’t really have 3d data yet. So introduce this carefully and slowly.

Now, Xarrays go a little crazy. Xarray Datasets are groups of DataArrays. What in world does that mean? A dataarray is a single variable. For example temperature or precipitation. Datasets are groups of these variables, so you can have a dataset which is a group of dataarrays that all share the coordinates and dimensions. The subset of ERA5 data we will be working with has 15 variables, like air-pressure, air-tempreautre. Each of these is a data array. But they are grouped into a dataset. Xarray does this because you can apply methods to the entire dataset. You can subset, calculate a mean, or interpolate across all the data variables at once, and this is really powerful.

If you look to the right, I’ve printed out the ERA5 dataset. First you see the dimensions, this tells you how many points are in each direction. There are 504 values in time, 90 in latitude, 180 in longitude. If you look right below that in the coordinates ou can see what those points are. Dimensions tell you the points. Coordinates tell you the values of those points. If you look at the coordinate for time, you can see it goes from 1979 to 2020. It is a datetime64 which is a core data type that makes doing time really easy and great. Latitude goes from -89 to 89, and longitude goes from 0 to 359. Lat and lon are both floating point numbers. So that tells you this is a global dataset from 1979-2020. So you know what it covers, but what data is in there? If you look at data variables, you can see a list of names, their coordinates, so they go time, lat, lon. These are 15 variables all global, 40 year datasets. Finally at the bottom are attributes which tell you about the dataset. These are called global attributes and are different than attributes for individual data variables. This is another feather of xarray, attributes tell you about the data. Global attribute tell you about the entier dataset. Attributes are for everything and now I’ll show you how to understand your data. (39)

50 of 105

Xarray read in a Dataset

3D data
Dimensions: 504x90x180
Coordinates: time,latitude,longitude
15 data variables (in DataArrays)
Attributes

import xarray as xr

ds = xr.open_dataset('./../data/era5_monthly_2deg_aws_v20210920.nc')

ds

51 of 105

Explore the data

Xarray allows you easily explore the data

Look at data attributes

Now we want to explore this dataset, and X ray allows you to easily explore the data when you put d s as the last part of your executable cell, it prints out the dataset. And if you click on the little file marker so it says look at data attributes there. If you click on that, it expands it just like it showed on the right, and it shows you all the attributes for that data array. In this case we're looking at air temperature, and that tells you the attributes for air temperature now, air temperature is actually the two meter, temperature, and it shows you the different names for that, and what units it said, which is in Kelvin, which is very important, of course, so you can actually expand all the different data arrays, or the different variables of data that you have to understand what each of them represents. (40:30)

52 of 105

Coordinates versus dimensions

DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system.
Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data
A dimension is just a name of an axis, like ‘time’

Text from Xarray docs http://xarray.pydata.org/en/stable/user-guide/data-structures.html

53 of 105

DataArray are data variables in a Dataset

A DataArray holds a multi-dimensional information
DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system.
You can explore the data easily using either syntax

ds["air_temperature_at_2_metres"]

ds.air_temperature_at_2_metres

Text from Xarray docs http://xarray.pydata.org/en/stable/user-guide/data-structures.html

54 of 105

Review: DataFrame access: [], loc, iloc

iloc: integer/positional

loc: Labels

[]: flexible, confusing?

Strings, integers - row/column labels
Lists - similar, but always return dataframes
Slices of labels: end-point inclusive!
Boolean arrays: “mask” selection.

Always 0-based, for rows and columns.
Slices as usual, end-point exclusive.
Use carefully (error prone).

[]

List

[]

Numeric Slice

[]

Name

DataFrame

Series

Single Column Selection

Multiple Column Selection

(Multiple) Row Selection

55 of 105

New: DataArray access: [], sel, isel

isel: integer/positional

sel: coordinates

[]: flexible, confusing?

Strings, integers - coordinates*
Slices: end-point inclusive!

Always 0-based
Slices as usual, end-point exclusive.
Use carefully (error prone).

Only for DataArrays

point = ds.isel(time=0,

latitude=26,

longitude=119)

point = ds.air_temperature_at_2_metres[0,26,119]

point = ds.sel(time="1979-01",

latitude=37.125,

longitude=238.875)

Let’s look at those methods again in xarray world.

Ilon,loc become isel& sel.

Here I've selected air temperature at two meters, and then you just put points in.

You have no idea where those points are what they mean, but you can do that, and it's an easy way to print out dataset really easily. I so it's the same thing, I sell you producing positional selecting so you don't know what times zero is. You don't know whether latitude 26 is minus 90 or plus 90, and you don't know what the longitude is also I'll point out unless you notice that it's ISO, you can actually look at that and get really confused thinking that you are selecting latitude 26 and longitude and 119 now so you use a point. Now here you're actually selecting the values, and it's really easy to read and understand exactly what you're looking at.

(42:30)

56 of 105

Find data at a point or in a region:

sel: coordinates

Strings, integers - coordinates*
Slices of coordinates: end-point inclusive!

point = ds.sel(time="1979-01",

latitude=37.125,

longitude=238.875)

region = ds.sel(time="1979-01",

latitude=slice(30,40),

longitude=slice(230,250))

Xarray helps you understand your code.

57 of 105

Xarray has all sort of high-level cool tricks built in

Select data and plot data in one line (using matplotlib)

ds.air_temperature_at_2_metres.sel(time="1979-01").plot()

ds.air_temperature_at_2_metres.sel(

latitude=37.125,

longitude=238.875).plot()

Select a variable

Select coordinate

Apply a method()

Select a variable

Select coordinate

Apply a method()

58 of 105

Methods can be called across a DataArray or a Dataset -- LAZY

Select data and plot data in one line (using matplotlib)

Same thing, but across all variables

ds.air_temperature_at_2_metres.mean("time").plot()

mean_map = ds.mean("time")

mean_map.air_temperature_at_2_metres.plot()

Select a variable

Apply a method() across a coordinate

Apply a method()

Lazy

Now one of the really good neat things about X ray is that methods can be called across a data array. So a single variable, or a data set. And it's done in a lazy way which means it doesn't actually execute it until it has to. So the top is example of selecting a variable, applying a method across it, and then applying another method so here we're taking the mean across all of the time, so this is 40 years. And we're plotting the value so here you see a map of, you know, average air temperatures over the last 40 years. Now, down below, we do the same thing, we calculate mean map as data set dot mean and time. But it's lazy it runs, it just shows that really fast, it doesn't actually calculate that until you ask it to plot it. So then when we call mean map dot air temperature dot plot, it actually calculates the mean across the air temperature and then plots it. These both yield the same values. One is just doing it for an individual data ray and the others doing it for the data set.

(44:30)

59 of 105

Plot the global trend in a variable

Calculate a time series
Take out the annual cycle
Plot the trend

60 of 105

Goal: Calculate time series

Xarray has high level methods like .mean(), .std(), etc.

ave = ds.mean("time")

ave.air_temperature_at_2_metres.plot()

Take the mean across all time

ave = ds.mean(("latitude", "longitude"))

ave.air_temperature_at_2_metres.plot()

Take the mean across all locations

Does that look right?

61 of 105

Goal: Understand what .mean() does

With great power comes great responsibility

ave = ds.mean()

ave

Does that look right?

Take the mean across all coordinates

62 of 105

The map is flat - but the Earth is not - Gaussian grid

Programs aren’t smart - you are - so what went wrong?

Gridded data is nice to work with but what does it represent?

How many grid points are at 90N (the North Pole)?

How many grid points are at the Equator?

What's going on well, the map is flat, but the Earth is not, and programs aren't smart you are. So what went wrong here.

This is one of the really key lessons about working with data, you have to spend some time to understand what you're looking at, if you want to analyze it correctly gridded data is really nice and easy to work with but what does it represent. So think about how many grid points are at 90 North, the North Pole where everything collapses down to a single point.

And how many grid points are at the equator.

We're working on a grid and you can see in the dimensions and coordinates that they're still 180 points at a latitude of 90. And there's 180 points at a latitude of zero at the equator, but each point at the equator, represents a lot more space than the points at the pool.

(49)

63 of 105

Weight your data

Xarray provides the ability to weight your data

weights = np.cos(np.deg2rad(ds.latitude))

weights.name = "weights"

weights.plot()

64 of 105

Goal: Examine average values - weighted version

Xarray methods like .weighted() can be combined with .mean()

ds_weighted = ds.weighted(weights)

weighted_mean = ds_weighted.mean()

weighted_mean

Does that look right?

65 of 105

Goal: Weighted global time series data

You can create means across coordinates: eg. latitude and longitude

ds_weighted = ds.weighted(weights)

weighted_mean = ds_weighted.mean(("latitude", "longitude"))

weighted_mean.air_temperature_at_2_metres.plot()

66 of 105

Take out the annual cycle using .groupby()

Use .groupby on a coordinate

67 of 105

Goal: Calculate annual cycle

Can use .groupby & .mean

annual_cycle = weighted_mean.groupby("time.month").mean()

annual_cycle.air_temperature_at_2_metres.plot()

68 of 105

Put it all together and plot the trend

Can use .groupby & .mean

weighted_mean = ds_weighted.mean(("latitude", "longitude")) #weighted mean time series

annual_cycle = weighted_mean.groupby("time.month").mean() #calculate annual cycle

annual_cycle += annual_cycle.mean() #add back in the mean value

weighted_trend = weighted_mean.groupby("time.month") - annual_cycle

weighted_mean.air_temperature_at_2_metres.plot()

weighted_trend.air_temperature_at_2_metres.plot()

Now let’s put this all together. Calculate your weighted mean byb taking teh mean in latitude and longitude.

Fing yoru annual cycle using groupby and mean.

Calculate yoru weighted trend, group by a month, and subtract that annual cycle.

Now I'm going to add the mean of that back in, because the annual cycle doesn't have any mean it's just the annual cycle so you subtract that you're down to zero. So we're going to add back the mean so it's gonna plot correctly. And then we calculate the time series the time series is just the data set, the mean and latitude and longitude and the trend is then this d s trend in the mean and latitude and longitude and you can plot that right there just to look at it. know you've seen this data, we've calculated a time series, and a trend.

And now we want to talk about how things are changing with these climate extremes.

(61:00)

69 of 105

Goal: Are extremes more likely? PDF analysis.

xr.plot.hist(darray,

bins=bin_array,

density=True,

alpha=.9,

color="b",

)

70 of 105

ERA5 temperature PDFs

bins = np.arange(284, 291)

xr.plot.hist(

weighted_mean.air_temperature_at_2_metres.sel(time=slice("1980","2000")),

bins=bins,

density=True,

alpha=.9,

color="b",

)

xr.plot.hist(

weighted_mean.air_temperature_at_2_metres.sel(time=slice("2000","2020")),

bins=bins,

density=True,

alpha=.85,

color="r",

)

plt.ylabel("Probability Density (/K)")

So here, I've taken our weighted time series that we created for the air temperature two meters, and in blue. I've taken the first 20 years of our data set. And I'm doing plot hist, and now we're an x ray, so we're doing XR dot plot hist which links back to that earlier NumPy histogram routine. We've given it the bins. We've said density is true because we want the probability density function not the histogram. Now, alpha what alpha does is alpha sets the transparency of the data so that we can see one histogram in front of the other. And so the first one is I'm plotting the first 20 years in blue. Right below that now I'm going to xr plot hist and I'm plotting the next 20 years, and their probability density function, and I'm plotting that in red. And what you can see really clearly in that histogram is a dramatic shift to the right of the air temperatures at two meters from the first 20 years to the second 20 years.

(65)

71 of 105

Goal: Can we plot the trend with our ERA5 data?

pfit = ds.air_temperature_at_2_metres.polyfit("time", 1)

pfit.polyfit_coefficients[0] *= 3.154000000101e+16

pfit.polyfit_coefficients[0].plot(cbar_kwargs={"label":"trend deg/yr"})

72 of 105

Goal: How about all fancy on a globe?

import cartopy.crs as ccrs

p = pfit.polyfit_coefficients[0].plot(

subplot_kws=dict(projection=ccrs.Orthographic(0, 55), facecolor="gray"),

transform=ccrs.PlateCarree(central_longitude=0),

cbar_kwargs={"label": "trend deg/yr"},

)

p.axes.coastlines()

73 of 105

Goal: Positive feedbacks in the climate system

The Earth is a system - a complicated game of dominos

As air temperature increases, sea ice concentration decreases…..

74 of 105

Satellite SSTs along the west coast of the US during the 2014-2016 northeast Pacific marine heat wave

Research supported by NASA Physical Oceanography, NASA Ocean Vector Winds Science Team, and NASA JPL

Image Credit: NASA JPL: C. Thompson & J. Hall

75 of 105

Air-Sea & Sea-Air

Stationary high pressure ridge
Winds 2^nd lowest on record

(Bond, 2015)

Reduced mixing

Reduced Ekman transport (wind driven currents)

Ridiculously Resilient Ridge (RRR)

76 of 105

Timeseries of SST

SST Anomaly (K)

Timeseries shown for Blob region, including all monthly data and EOF reconstruction. Data is smoothed.

HadiSST v2 data does not use EOF in it’s construction

Recent data uses AVHRR SSTs, prior to satellite data all in situ obs

More info: http://www.metoffice.gov.uk/hadobs/hadisst/

77 of 105

Photo by Paul Nicklen from the National Geographic, “The Blob that cooked the Pacific” full article at http://www.nationalgeographic.com/magazine/2016/09/warm-water-pacific-coast-algae-nino/

“Number of Starving Sea Lions in California 'Unprecedented‘” Nat. Geo. 2015

“California’s commercial Dungeness crab season to stay closed” SF Chronicle 2016

“Guadalupe fur seals UME 2015” - NOAA

“2015 Large Whale UME in the Western Gulf of Alaska” - NOAA

“California's New Era of Heat Destroys All Previous Records” Bloomberg 2015

“Shellfish harvest closures ordered along Oregon coast due to marine toxins” Daily Astorian 2015

“Cassin’s Auklet 2014 UME” - Henkel, et al. 2015

“Common Murre 2015 UME” - http://beachwatch.farallones.org/

78 of 105

SST and fuel for fishes

The Pacific Decadal Oscillation (upper), and northern copepod biomass anomalies (lower), from 1969 to present. Biomass values are log base-10 in units of mg carbon m^–3.

Figure from: https://www.nwfsc.noaa.gov/research/divisions/fe/estuarine/oeip/eb-copepod-anomalies.cfm#NSC-01

The northern copepod biomass is an index of the amount of energy transferred up the food chain. These fatty compounds appear to be essential for many pelagic fishes if they are to grow and survive through the winter successfully

79 of 105

High Resolution SSTs

High temporal and spatial resolution SST data allow investigations in to high resolution spatial/temporal changes in ocean conditions in regions along the coast which are crucial for fisheries

Image Credit: NASA JPL: C. Thompson & J. Hall

80 of 105

High temporal/spatial

Allow for more accurate investigations into drivers/responses
Upwelling not uniform
How did SSTs along the west coast change during this event?

Image Credit: NASA JPL: C. Thompson & J. Hall

81 of 105

Results

Satellite SST and wind stress show the phenology and extent of the recent record-breaking marine heat wave along the U.S. West Coast
Warm SSTs occurred January 2014 to August 2016, but abated briefly along the coast during the upwelling season
The largest SST anomalies occurred off central and southern California in late 2015 during decreased upwelling-favorable winds

82 of 105

Brief periods of ‘normal’ SSTs

From 2014 – 2016, onshore warm anomalies only weakened for short periods in May-June of each year

83 of 105

Alongshore warming

Onshore anomalies were generally stronger and more persistent in California than further north, with a peak of 6.2 C on 14 September 2015, just south of Point Conception

84 of 105

SSTs 2014-2016

Figure 3. Time series of daily SSTs, smoothed with a 30-day running mean, in the northern (left column) and southern (right column) parts of the CCUS. (a, e) SST 1000 km offshore. (b, f) SST 1 km. In each panel, light grey indicates the envelope of maximum and minimum values during 2002-2013; dark grey indicates the envelope of +/- 1 SD around the mean during 2002-2013; and the black, blue, red, and green lines indicate the mean during 2002–2013 and the values during 2014, 2015, and 2016, respectively. To emphasize anomalies >1 SD from the mean, the data are plotted so that the yearly lines are obscured when within 1 SD of the mean.

85 of 105

Coastal SSTs / winds

86 of 105

Results

The presence or absence of upwelling-favorable winds is not sufficient to judge ecosystem health
2014-2016: A combination of persistent warm SSTs and weaker/shifted upwelling season were associated with substantial ecosystem disturbances
A better understanding of how changes in the ocean impact ecosystem health is needed to understand how forecasted changes in winds may impact future ecosystems

Data: JPL MUR v4 global, daily, 1km multi-scale ultra-high resolution motion-compensated analysis; PMEL Bakun upwelling index, ECMWF ERA-interim 10 m wind

Jim Edson provided his MATLAB code for the COARE 3.5 drag coefficient

Funding: NASA Physical Oceanography, Ocean Vector Winds Science Team, JPL

Charles Thompson, JPL, provided Blue Marble SST image for May 2015

Thanks to:

Gentemann, C. L., M. R. Fewings, and M. García-Reyes (2016), Satellite sea surface temperatures along the West Coast of the United States during the 2014–2016 northeast Pacific marine heat wave, Geophys. Res. Lett., 43, doi:10.1002/2016GL071039.

87 of 105

Winners and Losers: biological impacts

Figure from Cavole, L. M., et al. (2016). "Biological Impacts of the 2013–2015 Warm-Water Anomaly in the Northeast Pacific: Winners, Losers, and the Future." Oceanography 29.

What is the future?

Climate change predicts overall increase in stratification and warming of the Pacific…

Increase in HABs

Changes to species distributions

88 of 105

Image Credit: NASA JPL: C. Thompson & J. Hall

89 of 105

What does the ocean look like now?

90 of 105

What does the ocean look like now?

91 of 105

Biological Pump

92 of 105

Models and data tell us about our weather and climate

Xarray is a powerful tool to analyze climate data.
I’ve just given some simple examples here, but really, we need more eyes on the data, more eyes on the models.
The models use ‘parameterizations’ which are approximations for all sorts of different physical processes. Each parameterization uses coefficients derived fro data science - but the experiments may be limited or imperfect. Often parameters are adjusted to compensate for an error, but then end up causing other issues, and we are all still working on this.
We need more of you, more data scientists working with climate and weather scientists to look at this data, helping to find new discoveries, amplify messages about changes to our climate and their impacts, and build machine learning models to replace old parameterizations.

Xarray is a powerful tool to analyze climate data and I've just given some examples here but really we need more eyes on a data more eyes on the model.

There's a lot still to be discovered. These models use parameterizations which are approximations for all sorts of different physical processes, and each parameterization use coefficients derived from data science, but these experiments may be limited or imperfect.

Often the parameters are adjusted to compensate for an error but then end up causing other issues and we're still working on this, we need more of you more data scientists working with climate and weather scientists to look at this data to find new discoveries and amplify messages about our climate and their impacts, and build machine learning models to replace old parameterizations.

(71)

93 of 105

I’m just an oceanographer

I started off like you, I’m just an oceanographer trying to find my way through the data science world. You are all data scientists maybe trying to find something interesting to work on.
Well, we all need your help. We need minds like yours to help solve the problems our and previous generations have caused. We need your voices, backed by solid data science, to mitigate what is going on, to change our trajectory.
I’ve shown you how to take pandas and leap to another level library, Xarray, you can do tutorials, leap to Scipy, or other libraries. I hope you enjoyed this small tour of Xarray and climate science.

I started off like you, I’m just an oceanographer trying to find my way through the data science world. You are all data scientists maybe trying to find something interesting to work on.

Well, we all need your help. We need minds like your to help solve the problems our and previous generations have caused. We need your voices, backed by solid data science, to mitigate what is going on.

I’ve shown you how to take pandas and leap to another level library, Xarray, you can do tutorials, leap to Scipy, or other libraries. I hope you enjoyed this small tour of Xarray and climate science.

There will be several NASA funded Summer hackweeks in 2022 - details are still being worked out but I’ll post on twitter and forward to Dr. Perez. Please consider applying.

Are there any questions?

(73)

94 of 105

95 of 105

96 of 105

First: find the Maono

97 of 105

98 of 105

Read in a CSV file

Goal 1: Calculate how the temperature is changing with increasing CO2 by using actual CO2 data collected at Mauna Loa. Original (uncleaned) data is here.

file = "./data_d100/monthly_in_situ_co2_mlo_cleaned.csv"

data = pd.read_csv(file)

data.head()

SUGGEST this slide &* next 2 move to end ? do in lab?

99 of 105

Plot the CO2 timeseries

What is going on? Why are their drops in the data?

import matplotlib.pyplot as plt

plt.plot(data["fraction_date"], data["c02"])

100 of 105

Read in a CSV file

file = "./data/monthly_in_situ_co2_mlo_cleaned.csv"

data = pd.read_csv(file,na_values=-99.99)

plt.plot(data["fraction_date"], data["c02"])

101 of 105

Goal: Understand real data is often a hot mess

file = "./data/monthly_in_situ_co2_mlo.csv"

A lot of text describing how to cite the data at the top of the csv file

Even more text

Oh wait! Here is some data, but the column labels are split across multiple rows????

Look at this data, what do you notice? One thing to note is that in the lecture I’ve used the file that has ‘cleaned’ in the filename. The original file has a lot of text at the top describing how to cite the data at the top of the csv file. Also, the column labels are split across multiple rows. It is a mess to read. A really important real-life example is to try and google manao lao co2 data. There are a lot of different places to get the data. In class a lot of times you are just given a dataset to use, but in real life, you ask friends and google. Make sure to try and find the original data, with citation information.

Quick note: in your hub, you can use JupyterLab to view the raw data as either a pure text file or as a CSV table. This can help you later figure out details on how to call pandas' read_csv function so it loads the data correctly.

102 of 105

Goal: Recognize your real friends who are always there for you

Goal: Try to use the original data - you will want that citation info when you decide to publish results

Arguments

Filepath

header='infer'

names=<no_default>

skiprows=None

na_values=None

103 of 105

How do you calculate probability of event occurrence?

A histogram tells you how many times a particular value occurred in your dataset.

import numpy as np

plt.hist(data["c02"], bins=np.arange(310, 360, 1))

plt.ylabel("Occurrence (#)"), plt.xlabel("C02 (ppm)")

Scientists are looking for how extremes are changing using a type of histogram called a probability density function. A histogram tells you how many times a particular value occurred in your dataset. How many times does 320 ppm occur versus 350 ppm. Going back to our carbon dioxide data, the keeling curve is on the right. If we look there, you can tell that most of the data is going to be between 310 and 360, and probably pretty evenly distributed. Let’s look. Numpy has a nice function that creates an array of data, so I’ll import numpy as np. Now I’m using matplotlib’s hist method, giving it the CO2 data and bins from my numpy array, going from 310 to 360. This produces the plot on the right. So, we were right, data goes from 310-360, and is evenly distributed. If you look at the y-axis you can see this is a histogram, it gives the number of occurrences, 6, 7, 8, or so on. Scientist usually like to work in probability, so let’s normalize the data and plot the probability for any given value.

104 of 105

How do you calculate probability of event occurrence?

A probability density function (PDF) tells you the probability of a particular value occurred in your dataset.

plt.hist(data["c02"], bins=np.arange(310, 360, 1), density=True)

plt.ylabel("Probability (1/ppm)"), plt.xlabel("C02 (ppm)")

105 of 105

Extremes are the new normal

A decade ago - scientists would argue - we can’t attribute any single weather event to climate.

In the last decade, we have all experienced major shifts in our climate through changes in our local weather and scientists have figured out ‘climate event attribution’

Probability of 340 ppm?