1 of 80

Join at slido.com�#3153196

ⓘ Start presenting to display the joining instructions on this slide.

3153196

2 of 80

3153196

3 of 80

Climate Data Science and XArray

Data 100/Data 200, Spring 2023 @ UC Berkeley

Guest Lecture: Fernando Pérez

Lecture materials by: Dr. Chelle Gentemann, NASA

3

Lecture 19

4 of 80

About the author of these materials

Dr. Chelle Gentemann

More: @ChelleGentemann

3153196

I’m Dr. Chelle Gentemann. I’ve worked in academia, industry, nonprofits, and for the last 2 decades mostly doing research for NASA. I’m was the lead scientist on a proposal for a new NASA mission, Butterfly. I’m a mom, an oceanographer, and have twins that are now 12. Now I work at NASA headquarters leading the Transform to Open Science Mission. Dr. Perez invited me here to talk about all the different ways we do data analysis in Earth Science to understand the Earth System and how it is changing due to human activities. The datasets you have encountered so far are mostly tabular, perfect for pandas. First I’ll talk about climate change and how we are monitoring in, but most of this data doesn’t fit into tabular datasets, it changes in space and time, so we will also talk about how we can use powerful open source tools, building on our knowledge of pandas, to explore 3-dimensional data. (3:45)

5 of 80

The foundation of climate change: the Greenhouse Effect

3153196

Irish physicist John Tyndall is typically credited with the foundation of climate science for his 1859 publication on heat-trapping gasses, but more recently it is now recognized that Eunice Newton Foote published a paper linking carbon dioxide and planetary warming in 1856, 3 years before Tyndall. Someone needs to tell google that. As I start this lecture on climate change, I wanted to take a minute right at the beginning to acknowledge that throughout the history of science, including the history of climate science, the voices of historically excluded group have been silenced or ignored. Above, is just one example, but many more are documented. And it is these groups that are being impacted more strongly by climate change. I’m here to try and reach out to all of you, to try and bring more voices to climate science, so that instead of arriving at a solution, we arrive at the best solution because the people in the room represent the diversity in our community. (4:45)

6 of 80

Climate Change

A simplified animation of the greenhouse effect. Credit: NASA/JPL-Caltech

3153196

7 of 80

Video credit: NASA SVS

3153196

8 of 80

Video credit: NASA SVS

3153196

9 of 80

What does the data say about our climate?

3153196

10 of 80

Keeling Curve

In 1958 Keeling got a grant to begin monitoring CO2 in Hawaii.
Roger Revelle (a famous scientist) argued that they just needed a snapshot - CO2 was too variable - and another snapshot 20 years later to show that CO2 was increasing
Keeling advocated for precise measurements over time. By the mid-1960s we had both a measurement of the Earth’s breathing and the global increase in CO2.
How would you remake this figure?

https://pubs.acs.org/doi/10.1021/ac1001492

3153196

11 of 80

Video credit: NASA SVS

3153196

12 of 80

Calculating the Greenhouse Effect

Goal 1: Calculate how the temperature is changing with increasing CO2.

The Planetary energy balance:

Energy (from the Sun) absorbed = Energy emitted

How does CO2 affects this?

The Sun emits radiation that is absorbed by the Earth (~30% is reflected by clouds, ice/snow, desert, this is the albedo)
The Earth emits radiation according to Stephan-Boltzman’s Law: the rate that a body emits radiation (per unit area) is directly proportional to the body's absolute temperature to the fourth power (blackbody radiation)
The emitted radiation doesn’t all go back into space……..

3153196

To investigate how carbon dioxide affects the Earth’s temperature, we need to understand how to calculate the Greenhouse Effect.

The planetary energy balance is basically that what comes in must go out. So the Energy from the Sun, travels through space, and arrives at Earth. The energy absorbed by the Earth must equal the energy emitted by the earth. Where does the carbon dioxide come into it?

The Sun emits radiation absorbed by the Earth (some is reflected by the atmosphere/clouds/earth surface, this is called the albedo, or the reflectivity of the Earth

The Earth emits radiation according to Stephan-boltzemans law -- -the rate the a body emits radiation per unit area is directly proportional to the body’s absolute temperature to the 4th power. This is an equation that permeates all of climate science --- the energy emitted is related to the temperature to the 4th.

But…. the emitted radiation doesn’t all go back into space it is absorbed the atmospheric gasses, like CO2 and that is where the greenhouse effect comes into play.

Now, in the next few slides, I’m going to go through the math. I’ll do it one step at a time, but don’t worry, because I’m going to put it all together for you at the end.

First, do you remember what the area of a circle is? Pi * radius squared.

Also, do you remember what the surface area of a sphere is? Four times pi times radius squared.

Remember the Earth can sometimes look like a sphere, sometimes like a circle, it depends…… (9:45)

13 of 80

Calculate energy in

The Sun emits radiation absorbed by the Earth

Energy in equals the incoming sunlight (W/m²) multiplied by the area (m²) to get W

NASA illustrations by Robert Simmon

= Incoming sunlight x area

r

3153196

14 of 80

Some of the energy in is reflected by the atmosphere

The Sun emits radiation absorbed by the Earth (some is reflected by the atmosphere)

NASA illustrations by Robert Simmon

r

Albedo = the fraction of radiation reflected back to space by the atmosphere

The amount that gets through:

Reflective surface

3153196

15 of 80

Calculate Energy out

B) The Earth emits blackbody radiation

Energy out equals the emitted radiation (W/m²) multiplied by the area (m²) to get W

NASA illustrations by Robert Simmon

= Outgoing radiation x area

r

3153196

Now we need to calculate the Energy out. Again we have radiation times area. The Earth emits radiation according to Stephan-Boltzman’s Law: the rate that a body emits radiation (per unit area) is directly proportional to the body's absolute temperature to the fourth power (blackbody radiation). The equation was actually a data science problem! 1864, John Tyndall measured the color and infrared emission of a platinum filament. Josef Stefan related the emission to the temperature to the fourth. Boltzmann then derived the theoretical model. Stefan used the equation to determine the temperature of the Sun! For the truly curious, it's cool to note that while the law can be derived purely through a classical thermodynamic argument (as Boltzmann did), the precise form of \sigma has h in it, and is thus a quantum expression. This was the problem that connected, at the hinge of 1900, classical physics and the birth of quantum mechanics.

To calculate Energy out, we need to multiply it by the area, but here this isn’t a disk, this is emission from a sphere, so the surface area of a sphere is 4 pi radius squared. (12:15)

16 of 80

Energy in = Energy out

Goal 1: Calculate how the temperature is changing with increasing CO2

NASA illustrations by Robert Simmon

Reflective surface

Planetary energy balance

3153196

17 of 80

Calculating temperature

Goal 1: Calculate how the temperature is changing with increasing CO2

What is the Earth’s temperature?

~255 K

~-16 C

~1 F

3153196

18 of 80

3153196

19 of 80

Our atmosphere is like a blanket

Solve for the greenhouse effect!
What happens to the temperature if we increase the greenhouse effect?
What happens to the temperature if we decrease/increase the albedo?

without an atmosphere

Planetary energy balance

with an atmosphere

3153196

20 of 80

Calculating the Temperature dependence on CO2

Goal 1: Calculate how the temperature is changing with increasing CO2

CO2 is ~420 ppm (parts per million - co2.earth/daily-co2)

As CO2 increases what happens to the temperature?

3153196

21 of 80

Plot the results

Use the equation to calculate the increase in temperature with time due to the increase in CO2

3153196

22 of 80

Global warming is a climate crisis

The equation provides solutions to global warming:

Change the planetary albedo to reflect more radiation to space (increase aerosols, clouds, make surface more reflective)

Reduce greenhouse gases (CO2, Methane)

Sun-shades in space?

3153196

Back to our equation - Global warming is right here in this equation. The problem and the solution. We need to reduce energy into the system. So, on the left we can work to change the planetary albedo so more radiation is reflected back to space. You could do this by increasing atmospheric aerosols - like a controlled volcanic event, create more reflective clouds, or even change the reflectivity of the Earth surface by creating more ice, making the ice more reflective using glass beads, or painting mountaintops white. Another approach is to look at the right half --- reduce greenhouse gases like CO2, methane, water vapor, by removing carbon, getting the ocean’s phytoplankton to absorb it, planting more trees. Some of these ideas may sound a bit crazy - but we are at the point where all realistic options need to be evaluated. We need to reduce our emissions - that is none negotiable, but to avoid the impacts that are already unfolding and increasing, we need to also work to remove carbon, methane, and increase the albedo. These are scary alternatives - but we can start with small scale experiments, and we have to move forward and invest in this area because the latest climate assessment is frightening and probably everyone in this lecture has been impacted by wildfire, drought, heat, or hurricanes. 1 in 3 americans were impacted by a climate disaster this Fall. (17:10)

23 of 80

Changing by Alisa Singer

"As we witness our planet transforming around us we watch, listen, measure … respond."

www.environmentalgraphiti.org – 2021 Alisa Singer.

AR6 Climate Change 2021:

The Physical Science Basis

3153196

You probably here IPCC reports quote in the news. These reports are scheduled, have deadlines for science to be included, and are comprehensive. The 5th assessment is finished and we are now on the 6th, or AR6, the first report out is The Physical Science Basis. The report addresses the most up-to-date physical understanding of the climate system and climate change, bringing together the latest advances in climate science, and combining multiple lines of evidence from paleoclimate, observations, process understanding, and global and regional climate simulations.

Next year, the reports will be on mitigation, then a report in impacts adaptation, and vulnerability, then finally a synthesis report that puts all 3 together. The current report just documents what we see happening in the climate system. Documents with data…. This is data science in action. So what do we see? (18:10)

24 of 80

3153196

25 of 80

3153196

26 of 80

3153196

27 of 80

3153196

28 of 80

3153196

29 of 80

3153196

30 of 80

nytimes.com/2023/03/20/climate/global-warming-ipcc-earth.html

(released yesterday, March 20, 2023)

ipcc.ch/report/ar6/syr

3153196

31 of 80

3153196

32 of 80

3153196

33 of 80

Let's check our understanding

This equation:

Shows that all climate change is due to CO2. If we stop CO2 emissions, we can stop climate change.

True
False

3153196

34 of 80

The given equation shows that all climate change is due to CO2. If we stop CO2 emissions, we can stop climate change.

ⓘ Start presenting to display the poll results on this slide.

3153196

35 of 80

Image credit: NOAA

3153196

Where does all this data come from? So it gets a little warmer….. Do I really care or need to worry about a deg or two of warming? I can’t really tell what the temperature is today to within 1.5 deg Celcius so why worry about it warming that much? It doesn’t sound that bad really, just a little bit warmer, and it is gradual so we can adapt, right? Well, the AR6 report for the first time specifically calls out climate extremes. A lot of the previous research was on how the means were changing, but as we began to see more extreme events, scientists put 2 + 2 together and realized that if you change the mean, you make what used to be extreme events more common. And it turns out that the extreme events are what cause most of the damage to people and property. Hurricanes, flash-floods, wild fires, heat waves in both the ocean and atmosphere, it feels a little bit like the last couple of years we have been under attack. (20:20)

36 of 80

With every increment of global warming, changes get larger

https://www.ipcc.ch/report/ar6/wg1/#FullReport

3153196

This is a figure from the report with every increment of global warming changes are amplified they get larger, top figure shows the observed change per one degree of global warming. This is actually from left hand data and on the right hand side now we see the simulated change and this is the change simulated by the models, these sort of comparisons, help us understand what the models are getting right and what the models are getting wrong and where they need to be improved on the next iteration.

One thing that I'll point out is in the North Atlantic Ocean. There's a cool spot in the data that isn’t in the model.

And we think that because the climate models aren't getting the ocean circulation correct. The region with the cool spot is where there is a lot of carbon dioxide and heat absorbed from the atmosphere and put into the ocean’s deep circulation. It's really important to understand where the model isn’t matching the data so that we can look into the physical parameterizations of the model to improve their simulations of our climate. The climate models rely on parameterizations, which are essentially estimates of what's happening. Estimates of the exchange of momentum, the exchange of heat, the exchange of gases, then we use these parameterizations and models to estimate physical processes. Sometimes we get it right, and sometimes we don't.

There's a lot of work still to be done, there's a lot of data still to be analyzed. to help us understand how to improve these models. Now let's look at what the future look like. There is the Earth at 1.5 degrees global warming.

There it is it 2. And there it is 4.

What is your mean temperature, it changes that much, and especially look at the higher latitudes and the Arctic, where it's amplified. (23:50)

37 of 80

… in precipitation

https://www.ipcc.ch/report/ar6/wg1/#FullReport

CC-relationship water vapor - temperature - pressure https://www.jbarisk.com/news-blogs/the-physics-of-precipitation-in-a-warming-climate/

3153196

And it's not just the temperature, the temperature affects how much water vapor the atmosphere can hold. The Clausius-Clapyron equation relates the energy required to cause a chemical reaction at a given pressure. Warmer temperatures increase the atmospheric moisture content, the humidity, because the atmospheric pressure will not be affected by climate change in the same way as temperature. So, from the Clausius-Clapeyron relationship, atmospheric water content increases by between 6 and 7% per 1 °C. So, as we warm the atmosphere, this heat affects the global hydrological cycle, the way that water travels through the Earth system. So when you change the temperature, you actually change the precipitation as well. And here is the precipitation at 1.5 degrees, two degrees and four degrees.

And many areas are actually much wetter than before, Some are drier, but many are much wetter. What that means is increased flooding in case dramatic rain events,

which in some ways doesn't immediately make sense to people. If it's getting warmer. Aren't we just going to go into drought, and we may have extended droughts, with then heavy intense rainfall over short periods of time so you can still have drought and have an increase in rainfall.

But these changes in rainfall like we saw during the floods in Pakistan, America, Australia, and Europe this year can cause substantial damage to both physical structures and impact on human life.

(26)

38 of 80

Our trajectory from data and models

https://raw.githubusercontent.com/BrodiePearson/IPCC_AR6_Chapter9_Figures/main/Plotting_code_and_data/Fig9_03_SST/Fig9_03_SST.png

3153196

This is a very interesting figure that has a lot in it.

Focus on the center figure.

Now the black line is an average of all the climate models, there's 40 of them, and they have various scenarios to the future and that's why you see the spread on the right hand side of the center figure. the model data, the orange line are the observational reanalysis. These are climate data sets. These data sets, match the models, quite well for the entire period that we have observations.

And you can see the positive trend that starts around 1970 and the increase in positive ocean temperatures that continues through the 2000s to 2020 when the observational record ends, the models continue to match it and then they continue increasing

our mean is shifting, what we consider normal is shifting. (26:30)

39 of 80

Extremes are the new normal

3153196

40 of 80

Extremes are the new normal

A decade ago - scientists would argue - we can’t attribute any single weather event to climate.

In the last decade, we have all experienced major shifts in our climate through changes in our local weather and scientists have figured out ‘climate event attribution’

They look at the probability of the occurrence of an event (eg. a temperature extreme) in models run without human-influence and then compare it to the probability in models run with human-influence.

How?

https://www.nature.com/articles/d41586-021-01142-4

https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2021GL092765

3153196

41 of 80

Extremes are the new normal

The probability of an event changes

https://www.nature.com/articles/s41467-019-09729-2/figures/1

Model runs with all forcings

Model runs with only natural

How likely is it that there will be a 2 degree anomaly?

3153196

Here is the temperature anomaly, which is the temperature minus the mean value, so how is it warming or cooling. Ths is the temperature anomaly from a climate model HADGEM3 from 1960 to 2010. What you see is there is decadal variability. The red lines are model run with all forcings, including anthropogenic. The blue are models run with only natural, not human induced forcing. The blue anomalies are lower. So if you take these data and plot the probability density, you can see how likely it is to get a temperature anomaly. For only natural forcing, it is really rare to get a 1 deg warm or cold anomaly. When you include human induced warming, you see that the mean value is just over 1 deg, and warm anomalies over 2 deg which were never seen in the other runs, are fairly common.

(28:30)

42 of 80

Positive Feedbacks in the Climate System

3153196

43 of 80

Goal: Positive feedbacks in the climate system

The Earth is a system - a complicated game of dominos

As air temperature increases, sea ice concentration decreases…..

3153196

44 of 80

Goal: Arctic Amplification: a positive feedback

A positive feedback amplifies the initial perturbation

Albedo feedback:

The albedo of sea ice is ~0.5-0.7. Most sunlight is reflected back to space.
The albedo of the ocean is 0.04. Most sunlight is absorbed and warms the seawater.
A warmer ocean melts more sea ice.

reflectivity!

3153196

There are two different feedbacks going on and scientists aren’t sure which is the larger contributor… The first is the albedo feedback --- remember (click) albedo is reflectivity.

The albedo is on global average about 0.3 and we talked about that earlier so that's the amount of radiation reflected back into space.

Now the albedo of ocean water, 0.06 the ocean just absorbs all the radiation. Sea ice has an albedo of 0.5-0.7 ---

That's why it's hard to look at snow and ice, it just reflects it all back up to the sky.

Now, as ice melts more radiation is absorbed, because the ocean water absorbs more and it changes the Earth's albedo. More ocean == lower albedo. As we saw earlier, lower albedo, more global warming. So, you get warmer temperatures, it melts more sea ice, You have more ocean, the ocean absorbs more radiation, which increases the temperatures, those increased temperatures melt more sea ice, which results in more ocean, and this is a positive feedback in the climate system. (31:00)

45 of 80

Goal: Arctic Amplification: a positive feedback

A positive feedback amplifies the initial perturbation

Cloud feedback:

As sea ice melts, more ocean is exposed and more moisture is absorbed by the atmosphere
More moisture == more clouds
Clouds trap longwave radiation, warming the Arctic, melting more sea ice.

3153196

46 of 80

Can we look at this ourselves? --- this is global data --- 3D data

….. Latitude, longitude, and time…. How do we handle that?

https://www.nature.com/articles/s41467-019-09729-2/figures/1

Model runs with all forcings

Model runs with only natural

How likely is it that there will be a 2 degree anomaly?

3153196

47 of 80

Learning objective: climate data analysis

3D data (time,latitude,longitude)
Introduction to Xarray python library
Xarray Probability density functions
Xarray linear regression: calculating mean trends
Xarray global analysis of trends

3153196

48 of 80

ERA5 - 5th gen ECMWF atmospheric global climate ReAnalyses

ERA5 combines vast amounts of historical observations into global estimates using advanced modelling and data assimilation systems.
From 1979 - 2019, hourly estimates of atmospheric, land and oceanic climate variables.
30 km global grid, with 137 levels from the surface up to a height of 80 km.

https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5

An ECMWF snow depth analysis for Scandinavia using ERA5 data shows the highest levels in a decade.

https://medium.com/planet-os/era5-reanalysis-data-on-s3-cee2f22422ae

3153196

ERA 5 is the 5th generation European Centre for Medium-Range Weather Forecasts atmospheric global climate re-analysis.

What that means is they take VAST amounts of historical observations, put them in a model, and create 30 years of hourly files, globally at 30 km, with 137 vertical levels.

That is some BIG data. This is one of the foundational datasets for scientists - whether you are working on weather or climate. Here is a paper that used ERA5 data looked at snow depth in scandinavia found the highest levels in a decade. A link to the ERA5 dataset and its details. And a link to the ERA5 hosted for free access on AWS. This is one of the most popular datasets to explore our weather and climate and improve our understanding of what is happening, why it is happening, and improve our understanding of the Earth system.

(33:40)

49 of 80

Vast amount of data

The number of observations assimilated in ERA5 has increased from approximately 0.75 million per day on average in 1979 to around 24 million per day by the end of 2018
~14,000 GB (14 TB)
A key dataset used for understanding our weather and climate, but inaccessible to all but a few privileged institutions

3153196

50 of 80

Data Frames are not enough: not all data is tabular

3153196

51 of 80

Xarray

Xarray introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like multidimensional arrays, which allows for a more intuitive, more concise, and less error-prone developer experience.

‘data variables’ : temperature and precipitation “data variables”

‘coordinate variables’: they label the points along the dimensions.

3153196

Let’s think about 3D data. Tables aren’t really efficient representation of big 3D data. You have blocks of data, maybe even multiple variables like temperature and precipitation. When you watch a weather forecast and it shows the weather model, you are looking at a spatial map, in 2D, that moves in time, and that is the 3rd dimension. In the Xarray world, your different observations are called data variables. Then you have coordinates, that label what the points along each dimensions mean. Now, this is where the magic starts --- just like you can use column names to select panda columuar data, you can use the coordinates to select Xarray datasets. This makes for a intuitive, more concise, less error-prone code --- for example you can just ask for the latitude and longitude you want. (37)

52 of 80

Xarray: Dataset

ERA5

Xarray Datasets are essentially groups of DataArrays.

This is really valuable when you are looking at datasets that have multidimensional groups of data, for example, temperature, precipitation, cloud cover.

Some Xarray methods can be applied to all that Dataset contains.

For example, you can subset a Dataset and the subset, interpolate, calculate a mean, and it will do this across all the DataArrays the Dataset contains.

3153196

(slow down) students don’t really have 3d data yet. So introduce this carefully and slowly.

Now, Xarrays go a little crazy. Xarray Datasets are groups of DataArrays. What in world does that mean? A dataarray is a single variable. For example temperature or precipitation. Datasets are groups of these variables, so you can have a dataset which is a group of dataarrays that all share the coordinates and dimensions. The subset of ERA5 data we will be working with has 15 variables, like air-pressure, air-tempreautre. Each of these is a data array. But they are grouped into a dataset. Xarray does this because you can apply methods to the entire dataset. You can subset, calculate a mean, or interpolate across all the data variables at once, and this is really powerful.

If you look to the right, I’ve printed out the ERA5 dataset. First you see the dimensions, this tells you how many points are in each direction. There are 504 values in time, 90 in latitude, 180 in longitude. If you look right below that in the coordinates ou can see what those points are. Dimensions tell you the points. Coordinates tell you the values of those points. If you look at the coordinate for time, you can see it goes from 1979 to 2020. It is a datetime64 which is a core data type that makes doing time really easy and great. Latitude goes from -89 to 89, and longitude goes from 0 to 359. Lat and lon are both floating point numbers. So that tells you this is a global dataset from 1979-2020. So you know what it covers, but what data is in there? If you look at data variables, you can see a list of names, their coordinates, so they go time, lat, lon. These are 15 variables all global, 40 year datasets. Finally at the bottom are attributes which tell you about the dataset. These are called global attributes and are different than attributes for individual data variables. This is another feather of xarray, attributes tell you about the data. Global attribute tell you about the entier dataset. Attributes are for everything and now I’ll show you how to understand your data. (39)

53 of 80

Xarray read in a Dataset

3D data
Dimensions: 504x90x180
Coordinates: time,latitude,longitude
15 data variables (in DataArrays)
Attributes

import xarray as xr

ds = xr.open_dataset('./../data/era5_monthly_2deg_aws_v20210920.nc')

ds

3153196

54 of 80

Explore the data

Xarray allows you easily explore the data

Look at data attributes

3153196

Now we want to explore this dataset, and X ray allows you to easily explore the data when you put d s as the last part of your executable cell, it prints out the dataset. And if you click on the little file marker so it says look at data attributes there. If you click on that, it expands it just like it showed on the right, and it shows you all the attributes for that data array. In this case we're looking at air temperature, and that tells you the attributes for air temperature now, air temperature is actually the two meter, temperature, and it shows you the different names for that, and what units it said, which is in Kelvin, which is very important, of course, so you can actually expand all the different data arrays, or the different variables of data that you have to understand what each of them represents. (40:30)

55 of 80

Coordinates versus dimensions

DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system.
Coordinates can also have any number of dimensions but denote constant/independent quantities, unlike the varying/dependent quantities that belong in data
A dimension is just a name of an axis, like ‘time’

Text from Xarray docs http://xarray.pydata.org/en/stable/user-guide/data-structures.html

3153196

56 of 80

DataArray are data variables in a Dataset

A DataArray holds a multi-dimensional information
DataArray objects inside a Dataset may have any number of dimensions but are presumed to share a common coordinate system.
You can explore the data easily using either syntax

ds["air_temperature_at_2_metres"]

ds.air_temperature_at_2_metres

Text from Xarray docs http://xarray.pydata.org/en/stable/user-guide/data-structures.html

3153196

57 of 80

Review: DataFrame access: [], loc, iloc

iloc: integer/positional

loc: Labels

[]: flexible, confusing?

Strings, integers - row/column labels
Lists - similar, but always return dataframes
Slices of labels: end-point inclusive!
Boolean arrays: “mask” selection.

Always 0-based, for rows and columns.
Slices as usual, end-point exclusive.
Use carefully (error prone).

[]

List

[]

Numeric Slice

[]

Name

DataFrame

Series

Single Column Selection

Multiple Column Selection

(Multiple) Row Selection

3153196

58 of 80

New: DataArray access: [], sel, isel

isel: integer/positional

sel: coordinates

[]: flexible, confusing?

Strings, integers - coordinates*
Slices: end-point inclusive!

Always 0-based
Slices as usual, end-point exclusive.
Use carefully (error prone).

Only for DataArrays

point = ds.isel(time=0,

latitude=26,

longitude=119)

point = ds.air_temperature_at_2_metres[0,26,119]

point = ds.sel(time="1979-01",

latitude=37.125,

longitude=238.875)

3153196

Let’s look at those methods again in xarray world.

Ilon,loc become isel& sel.

Here I've selected air temperature at two meters, and then you just put points in.

You have no idea where those points are what they mean, but you can do that, and it's an easy way to print out dataset really easily. I so it's the same thing, I sell you producing positional selecting so you don't know what times zero is. You don't know whether latitude 26 is minus 90 or plus 90, and you don't know what the longitude is also I'll point out unless you notice that it's ISO, you can actually look at that and get really confused thinking that you are selecting latitude 26 and longitude and 119 now so you use a point. Now here you're actually selecting the values, and it's really easy to read and understand exactly what you're looking at.

(42:30)

59 of 80

Find data at a point or in a region:

sel: coordinates

Strings, integers - coordinates*
Slices of coordinates: end-point inclusive!

point = ds.sel(time="1979-01",

latitude=37.125,

longitude=238.875)

region = ds.sel(time="1979-01",

latitude=slice(30,40),

longitude=slice(230,250))

Xarray helps you understand your code.

3153196

60 of 80

Xarray has all sort of high-level cool tricks built in

Select data and plot data in one line (using matplotlib)

ds.air_temperature_at_2_metres.sel(time="1979-01").plot()

ds.air_temperature_at_2_metres.sel(

latitude=37.125,

longitude=238.875).plot()

Select a variable

Select coordinate

Apply a method()

Select a variable

Select coordinate

Apply a method()

3153196

61 of 80

Methods can be called across a DataArray or a Dataset -- LAZY

Select data and plot data in one line (using matplotlib)

Same thing, but across all variables

ds.air_temperature_at_2_metres.mean("time").plot()

mean_map = ds.mean("time")

mean_map.air_temperature_at_2_metres.plot()

Select a variable

Apply a method() across a coordinate

Apply a method()

Lazy

3153196

Now one of the really good neat things about Xarray is that methods can be called across a data array. So a single variable, or a data set. And it's done in a lazy way which means it doesn't actually execute it until it has to. So the top is example of selecting a variable, applying a method across it, and then applying another method so here we're taking the mean across all of the time, so this is 40 years. And we're plotting the value so here you see a map of, you know, average air temperatures over the last 40 years. Now, down below, we do the same thing, we calculate mean map as data set dot mean and time. But it's lazy it runs, it just shows that really fast, it doesn't actually calculate that until you ask it to plot it. So then when we call mean map dot air temperature dot plot, it actually calculates the mean across the air temperature and then plots it. These both yield the same values. One is just doing it for an individual data ray and the others doing it for the data set.

(44:30)

62 of 80

Plot the global trend in a variable

Calculate a time series
Take out the annual cycle
Plot the trend

3153196

63 of 80

Goal: Calculate time series

Xarray has high level methods like .mean(), .std(), etc.

ave = ds.mean("time")

ave.air_temperature_at_2_metres.plot()

Take the mean across all time

ave = ds.mean(("latitude", "longitude"))

ave.air_temperature_at_2_metres.plot()

Take the mean across all locations

Does that look right?

(279 K ~ 6 ºC ~ 43 ºF)

3153196

64 of 80

The map is flat - but the Earth is not - Gaussian grid

Programs aren’t smart - you are - so what went wrong?

Gridded data is nice to work with but what does it represent?

How many grid points are at 90N (the North Pole)?

How many grid points are at the Equator?

3153196

What's going on well, the map is flat, but the Earth is not, and programs aren't smart you are. So what went wrong here.

This is one of the really key lessons about working with data, you have to spend some time to understand what you're looking at, if you want to analyze it correctly gridded data is really nice and easy to work with but what does it represent. So think about how many grid points are at 90 North, the North Pole where everything collapses down to a single point.

And how many grid points are at the equator.

We're working on a grid and you can see in the dimensions and coordinates that they're still 180 points at a latitude of 90. And there's 180 points at a latitude of zero at the equator, but each point at the equator, represents a lot more space than the points at the pool.

(49)

65 of 80

Weigh your data

Xarray provides the ability to weigh your data

weights = np.cos(np.deg2rad(ds.latitude))

weights.name = "weights"

weights.plot()

3153196

66 of 80

Goal: Examine average values - weighted version

Xarray methods like .weighted() can be combined with .mean()

ds_weighted = ds.weighted(weights)

weighted_mean = ds_weighted.mean()

weighted_mean

Does that look right?

3153196

67 of 80

Goal: Weighted global time series data

You can create means across coordinates: eg. latitude and longitude

ds_weighted = ds.weighted(weights)

weighted_mean = ds_weighted.mean(("latitude", "longitude"))

weighted_mean.air_temperature_at_2_metres.plot()

288 K ~ 15 ºC ~ 59 ºF. 👍

3153196

68 of 80

Take out the annual cycle using .groupby()

Use .groupby on a coordinate

3153196

69 of 80

Goal: Calculate annual cycle

Can use .groupby & .mean

annual_cycle = weighted_mean.groupby("time.month").mean()

annual_cycle.air_temperature_at_2_metres.plot()

3153196

70 of 80

Put it all together and plot the trend

Can use .groupby & .mean

weighted_mean = ds_weighted.mean(("latitude", "longitude")) #weighted mean time series

annual_cycle = weighted_mean.groupby("time.month").mean() #calculate annual cycle

annual_cycle += annual_cycle.mean() #add back in the mean value

weighted_trend = weighted_mean.groupby("time.month") - annual_cycle

weighted_mean.air_temperature_at_2_metres.plot()

weighted_trend.air_temperature_at_2_metres.plot()

3153196

Now let’s put this all together. Calculate your weighted mean byb taking teh mean in latitude and longitude.

Fing yoru annual cycle using groupby and mean.

Calculate yoru weighted trend, group by a month, and subtract that annual cycle.

Now I'm going to add the mean of that back in, because the annual cycle doesn't have any mean it's just the annual cycle so you subtract that you're down to zero. So we're going to add back the mean so it's gonna plot correctly. And then we calculate the time series the time series is just the data set, the mean and latitude and longitude and the trend is then this d s trend in the mean and latitude and longitude and you can plot that right there just to look at it. know you've seen this data, we've calculated a time series, and a trend.

And now we want to talk about how things are changing with these climate extremes.

(61:00)

71 of 80

Goal: Are extremes more likely? PDF analysis.

xr.plot.hist(darray,

bins=bin_array,

density=True,

alpha=.9,

color="b",

)

3153196

72 of 80

ERA5 temperature PDFs

bins = np.arange(284, 291)

xr.plot.hist(

weighted_mean.air_temperature_at_2_metres.sel(time=slice("1980","2000")),

bins=bins,

density=True,

alpha=.9,

color="b",

)

xr.plot.hist(

weighted_mean.air_temperature_at_2_metres.sel(time=slice("2000","2020")),

bins=bins,

density=True,

alpha=.85,

color="r",

)

plt.ylabel("Probability Density (/K)")

3153196

So here, I've taken our weighted time series that we created for the air temperature two meters, and in blue. I've taken the first 20 years of our data set. And I'm doing plot hist, and now we're an x ray, so we're doing XR dot plot hist which links back to that earlier NumPy histogram routine. We've given it the bins. We've said density is true because we want the probability density function not the histogram. Now, alpha what alpha does is alpha sets the transparency of the data so that we can see one histogram in front of the other. And so the first one is I'm plotting the first 20 years in blue. Right below that now I'm going to xr plot hist and I'm plotting the next 20 years, and their probability density function, and I'm plotting that in red. And what you can see really clearly in that histogram is a dramatic shift to the right of the air temperatures at two meters from the first 20 years to the second 20 years.

(65)

73 of 80

Goal: Can we plot the trend with our ERA5 data?

pfit = ds.air_temperature_at_2_metres.polyfit("time", 1)

pfit.polyfit_coefficients[0] *= 3.154000000101e+16

pfit.polyfit_coefficients[0].plot(cbar_kwargs={"label":"trend deg/yr"})

3153196

74 of 80

Goal: How about all fancy on a globe?

import cartopy.crs as ccrs

p = pfit.polyfit_coefficients[0].plot(

subplot_kws=dict(projection=ccrs.Orthographic(0, 55), facecolor="gray"),

transform=ccrs.PlateCarree(central_longitude=0),

cbar_kwargs={"label": "trend deg/yr"},

)

p.axes.coastlines()

3153196

75 of 80

Goal: Positive feedbacks in the climate system

The Earth is a system - a complicated game of dominos

As air temperature increases, sea ice concentration decreases…..

3153196

76 of 80

Models and data tell us about our weather and climate

Xarray is a powerful tool to analyze climate data.

I’ve just given some simple examples here, but really, we need more eyes on the data, more eyes on the models.

The models use ‘parameterizations’ which are approximations for all sorts of different physical processes. Each parameterization uses coefficients derived fro data science - but the experiments may be limited or imperfect. Often parameters are adjusted to compensate for an error, but then end up causing other issues, and we are all still working on this.

We need more of you, more data scientists working with climate and weather scientists to look at this data, helping to find new discoveries, amplify messages about changes to our climate and their impacts, and build machine learning models to replace old parameterizations.

3153196

Xarray is a powerful tool to analyze climate data and I've just given some examples here but really we need more eyes on a data more eyes on the model.

There's a lot still to be discovered. These models use parameterizations which are approximations for all sorts of different physical processes, and each parameterization use coefficients derived from data science, but these experiments may be limited or imperfect.

Often the parameters are adjusted to compensate for an error but then end up causing other issues and we're still working on this, we need more of you more data scientists working with climate and weather scientists to look at this data to find new discoveries and amplify messages about our climate and their impacts, and build machine learning models to replace old parameterizations.

(71)

77 of 80

(Channeling Dr. Gentemann) I’m just an oceanographer

I started off like you, I’m just an oceanographer trying to find my way through the data science world. You are all data scientists maybe trying to find something interesting to work on.

Well, we all need your help. We need minds like yours to help solve the problems our and previous generations have caused. We need your voices, backed by solid data science, to mitigate what is going on, to change our trajectory.

I’ve shown you how to take pandas and leap to another level library, Xarray, you can do tutorials, leap to Scipy, or other libraries. I hope you enjoyed this small tour of Xarray and climate science.

3153196

I started off like you, I’m just an oceanographer trying to find my way through the data science world. You are all data scientists maybe trying to find something interesting to work on.

Well, we all need your help. We need minds like your to help solve the problems our and previous generations have caused. We need your voices, backed by solid data science, to mitigate what is going on.

I’ve shown you how to take pandas and leap to another level library, Xarray, you can do tutorials, leap to Scipy, or other libraries. I hope you enjoyed this small tour of Xarray and climate science.

There will be several NASA funded Summer hackweeks in 2022 - details are still being worked out but I’ll post on twitter and forward to Dr. Perez. Please consider applying.

Are there any questions?

(73)

78 of 80

79 of 80

https://science.nasa.gov/open-science/

https://github.com/nasa/Transform-to-Open-Science

to change everything, we need everyone

News post and video

80 of 80

2023

NASA’s Year of

Open Science

80

NASA Transform to Open Science Mission

Dr. Chelle Gentemann, Science Lead

Yvonne Ivey, Equity Lead

Cyndi Hall, Community Coordinator

Isabella Martinez, Content Coordinator

Dr. Yaitza Luna-Cruz, TOPS Program Officer

Dr. Paige Martin, TOPS Program Officer

Kevin Murphy, Chief Science Data Officer SMD

Katie Baynes, Deputy Chief Science Data Officer SMD

Dr. Steve Crawford, Science Policy Officer SMD

Andy Mitchell,

Dr. Elena Steponaitis, SMD Development Program Executive

Amy (Uyen) Truong, Chief Science Data Office Coordinator

Dr. Rachel Paseka, OSSI Program Officer

Dr. J.L. Galache, OSSI Program Officer

Dr. Demitri Muna, OSSI Program Officer

Molly Adams, OSSI Coordinator

TOPS Email List

TOPS Website

SCIENCE MISSION DIRECTORATE