1 of 78

trustedci.org

2 of 78

Trusted CI:

The NSF Cybersecurity Center of Excellence

Jupyter Security

​

Rick Wagner

rick@globus.org

Matthias Bussonnier� bussonniermatthias@gmail.com

Ishan Abhinit

iabhinit@iu.edu

Mark Krenz

mkrenz@iu.edu

​

3 of 78

Introduction: Agenda, etc.

4 of 78

Trusted CI:

The NSF Cybersecurity Center of Excellence

Our mission: to provide the NSF community a coherent understanding of cybersecurity’s role in producing trustworthy science and the information and know-how required to achieve and maintain effective cybersecurity programs.

https://trustedci.org/

5 of 78

What is Cyberinfrastructure (CI)?

“The comprehensive infrastructure needed to capitalize on dramatic advances in information technology has been termed cyberinfrastructure (CI). Cyberinfrastructure integrates hardware for computing, data and networks, digitally-enabled sensors, observatories and experimental facilities, and an interoperable suite of software and middleware services and tools. “

-NSF Cyberinfrastructure Vision for 21st Century Discovery

Image credit: NSF

6 of 78

Today’s Agenda

30 minutes:

  • Taking questions: What do you want to know about Jupyter & Jupyter security?
  • What are the gaps in the current documentation and understanding?
  • We’ll try to answer some of these questions at the end.

30 minutes:

  • What is Jupyter?
  • The landscape: Project Jupyter; notebooks; notebook server; IPython; JupyterHub; etc.
  • Architecture, where things run and how they communicate.

90 minutes, with a 30 minute break at 4:00 p.m:

  • Running Jupyter notebooks.
  • Viewing un-encrypted traffic.
  • Configure and run JupyterHub.
  • Configure security options & recheck traffic.
  • Set up external authentication

60 minutes:

  • Answer questions from the beginning
  • Discussion of Jupyter use cases and needs around security best practices.

​

7 of 78

Taking questions

  • Who are you?

​

  • What is your experience with Jupyter?

​

  • What are you hoping to get out of this workshop/which questions do you have?

​

8 of 78

Introduction to Jupyter

&

Jupyter Security

9 of 78

Where to ask for help & contribute

  • Jupyter.org/community
  • Mailing list and security repositories.
    • security@ipython.org (Google Group, You can request access)
    • More concern contact core dev of relevant projects or steering council
  • https://discourse.jupyter.org/
  • GitHub
  • gitter.im
  • Jupyter at Research Facilities google groups

​

10 of 78

Overview of Jupyter architecture & nomenclature

11 of 78

Jupyter Notebook

  • Notebook documents are produced by Jupyter Notebook App which contain both computer code and rich text elements.

​

  • The jupyter Notebook app is a server-client application that allows editing and running notebook documents via web browser.

​

  • Notebook can be executed on a local desktop requiring no internet access or can be installed on a remote server and accessed through the internet.

​

  • You will notice the Jupyter and Notebook terms are highly overloaded.

12 of 78

Running a Notebook Server

  • By default, a notebook server runs locally at 127.0.0.1:8888 & accessible only from localhost
  • By default use unique token authentication on filesystem to protect from other processes from other users accessing the API
  • Understanding Single-User threat model is critical to understand multi-user deployment threat-model.

​

  • Multi-user deployment should use JupyterHub

​

13 of 78

Jupyter Architecture

Browser

1 Server / User

​

Python Kernel

R Kernel

C++ Kernel

CHP/HUB

​

~ login shell

~ sshd

~ HPC job

workers ?

ZMQ, Signed – Not Encrypted

REST

& websocket

​

encrypted

Server+pyplugins, serves webapp + js-plugins

Server: high level API (File handling, shell)

Kernels: Computation (and local to them File IO)

​

14 of 78

Name / Nomenclature

  • IPython : Python package that handle executing the User-defined Python code. Historically all the Jupyter-(.+) where into IPython repo and named IPython-\1. Now only the CLI to execute Python
  • IPykernel: Small Wrapper around IPython that talk the Jupyter ZMQ Protocol.
  • Kernel: A process that execute user code.
  • Notebook Server: Single user web server that serve the html/js for the browser app handles files and some plugins. Websocket/ZMQ bridge.
  • Notebook App: JS app running in user’s browser
  • Notebook document/file: the actual json file on disk, or as loaded in the above JS APP
  • JupyterHub:
    • Set of processes, plugin and configuration to do Multi-User deployment
    • Hub itself (jupyter aware) + CHP(generic software defined networking)
  • Binder: Authentication-less, w/ ephemeral users JupyterHub.
  • Zero-To-JupyterHub: Hands on Deploy your own JupyterHub

​

Note: Many of these are defaults there are a number of alternatives implementations.

15 of 78

Threat Model vs Functionality

Avoid: Arbitrary Code Execution

Allow: Arbitrary Code Execution

Venn Diagram

16 of 78

Threat Model

  • Kernel: Ensure request come from identified source (signature). Beyond that arbitrary code execution is the goal. Even modifying kernel itself. Use other mechanisms to prevent issues: Quotas, Docker, Cgroup...

​

  • Server: Run as the unprivileged user. Gateway to all the functionality. Equivalent to shell. Largest attack surface. Most API need authentication (cookie). Target of enumeration attacks; bruteforce… Rarely interprets any user supplied data – though plugins can.

​

  • CHP/HUB: Proxy credentials; delegate some services; allow impersonation; privileged user. Smallest attack surface; ONLY PRIVILEGED PROCESSES

​

  • Browser: Arbitrary JS extension; interprete user input; opened files; Display embedded HTML/JS.

17 of 78

Threat Model Example of Attacks

  • Kernel:
    • Malicious Library (pip install panda <- no s, and import panda)
    • Sniff ZMQ request; user code has sensitive input (password in clear ?)

​

  • Server:
    • Rogue Plugin can access all files / executions.
    • Token with in World-Writable location, impersonate user.

​

  • CHP/HUB:
    • MITM Admin user

​

  • Browser:
    • XSS, Phishing incite user to “trust”

18 of 78

Browser

1 Server / User

​

Python Kernel

R Kernel

C++ Kernel

.ipynb

.js

.py

19 of 78

Jupyter core security model

From https://jupyter-notebook.readthedocs.io/en/latest/security.html

Our security model

  • Untrusted HTML is always sanitized
  • Untrusted Javascript is never executed
  • HTML and Javascript in Markdown cells are never trusted
  • Outputs generated by the user are trusted
  • Any other HTML or Javascript (in Markdown cells, output generated by others) is never trusted
  • The central question of trust is “Did the current user do this?”

​

20 of 78

Trusted JS/HTML

HTML which is or has been explicitly requested by the user

Example:

  • Shift Enter
  • Run All
  • Trust This notebook
    • Once trusted, hash of the file stored on disk in LRU cache with random eviction.

21 of 78

The rest of the security is pluggable

Example:

  • Authentication PAM/LDAP
  • Direct CHP or NGINX
  • Kubernetes Spawner/Slurm Spawner/SUDO spawner.
  • Resource limitation by cgroup.
  • Single managed environment or per-user environment.

22 of 78

Securing an Installation

STEP -1

​

  • If you don’t provide system wide install user will use ssh-tunnel and old versions. You want control and monitoring of how users use Jupyter.

​

  • I’m assuming your users are not actively hostile toward each other

​

23 of 78

Securing an Installation

STEP 0

​

  • You should use JupyterHub, and google for JupyterHub for any security related questions in multi-user scenario

​

“Secure Config Jupyter == Secure Config Bash”

“Secure Config JupyterHub == Secure Config sshd”

​

24 of 78

Securing an Installation

STEP 1

​

  • Mailing list and security repositories.
    • security@ipython.org (Google Group, You can request access)
    • More concern contact core dev of relevant projects or steering council
  • https://discourse.jupyter.org/

​

25 of 78

Securing an Installation

STEP 2

​

  • Securing Privileged processes. CHP/HUB
    • Run behind Nginx/Apache
    • Run as unprivileged (opt-in users)
      • Sudo Spawner (limit process privileges)
      • Slurm, Kubernetes Spawner (limit/isolate servers)

​

Cmnd_Alias JUPYTER_CMD = /home/jupyterhub/miniconda3/bin/sudospawner

​

jupyterhub ALL=(%jupyter) NOPASSWD:JUPYTER_CMD

​

26 of 78

Securing an Installation

STEP 2

​

  • None of Jupyter(Hub) processes are privileged
  • Known SSL termination/vetted endpoint.
  • per-user server isolated potentially even more.

​

  • ADMIN/Sudoers weak link to elevated privileges
  • Can run them in container/VM if you wish.

27 of 78

Securing an Installation

STEP 3

​

  • Use OAUTH/LDAP authenticator/2FA

​

28 of 78

Securing an Installation

STEP 3

​

  • Reduce User friction and possibility for Phishing.
  • Reduce possibility to use social engineering to get access.
  • Directly hook into monitoring, fail2ban...

​

29 of 78

Securing an Installation

STEP 3

​

  • SSL all the way (CHP to single-user servers)
  • and/or , Spawn in containers...

​

30 of 78

Securing an Installation

STEP 3

​

  • Reduce possibility for attacker that got access to 1 account to potentially see traffic between CHP/Single-users

​

​

31 of 78

Not worse than SSHD at that point

At that point we are about as secure as we can be than SSH.

Admin users still sensitive targets, but Oauth/LDAP passwd != System password (hopefully).

Admin could still impersonate and access some services other can’t.

32 of 78

Securing an Installation

STEP 4

​

  • Restrict and Configure Jupyter itself.

​

  • By default we assume users are not malicious once logged-in but you might want to control environment.

​

​

33 of 78

STEP N+

​

  • Spawn kernels on remote machine (Enterprise Gateway)
  • Use non-file based storage backend (can control “trusted” state of files).
  • Monitor/Cull idle kernels, Servers.
  • …

​

At that point Hub is more restrictive than user install + ssh tunnel

  • Control/Limit plugin installation
  • Monitor
  • (Auto) start and stop user’s servers

​

34 of 78

Risk awareness

  • Notebooks allow arbitrary code to be run by those you provide access to.
  • Notebooks are often shared between colleagues.
  • It is important to restrict access to the notebook server.
  • Jupyter 4.3+ has token based auth enabled by default. Token passed in URL, Header or via login form.
  • Projects are setting up the Jupyter server to be run in novel ways, such as on a supercomputer cluster.
  • ​

35 of 78

Hands-On Exercises

36 of 78

Jupyter Security Boundaries

Notebook Server

Local

Remote

Local

Remote

Local

Remote

37 of 78

Jupyter Security Boundaries

JupyterHub

Local

Remote

38 of 78

Hands on Exercises

Where to run�$ commands

Hosts:

  • Go to spreadsheet and pick a hostname and IP, put your name down
  • Ubuntu 18.04 LTS
  • Hostnames are trusted-ci[1-N].globus-training.net

User accounts:

  • "campusadmin" (passwordless sudo permissions) is for configuring Jupyter
  • "researcher" (unprivileged) is intended to demonstrate user access
  • Password will be on the whiteboard

Local

Remote

39 of 78

Driving Security Motivation: Calling APIs

  • Using usernames & passwords, tokens, keys in notebooks to call out
  • Cliens in kernels should use SSL out
  • The secrets should not be stored in notebooks, either

https://jupyter.demo.globus.org/

40 of 78

Single User Notebook Server

trusted-ci<N>.globus-training.net

Browser

1 Server / User

​

SSH:22

HTTP:8889

GET /tree

tcpflow

41 of 78

Access Remote Notebook Server

Start a single user Jupyter notebook server

local:~$ ssh campusadmin@trusted-ci<N>.globus-training.net

SSH to the server

campusadmin@trusted-ci<N>:~$ jupyter notebook --no-browser --port=8889

42 of 78

Run tcpflow

“tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis and debugging.”

It’s like tcpdump with readable text output. --Rick

Basic command

campusadmin@trusted-ci<N>:~$ sudo tcpflow -p -c -i lo port <port>

I don’t recommend leaving this running

​

​

campusadmin@trusted-ci<N>:~$ sudo tcpflow -p -c -i lo port 8889 | grep ‘password=’

Let’s run this on the HTTP port

https://github.com/simsong/tcpflow

43 of 78

Access Remote Notebook Server

local:~$ ssh -N -f -L localhost:8888:localhost:8889 \

campusadmin@trusted-ci<n>.globus-training.net

Port forward from your laptop the server

Change 8889 to 8888 & put that line in a browser

Look for this line in your Jupyter server output:

http://localhost:8889/?token=a0fac14de491e7eb80ba8ab0e6e1ee16ad6ef4532d1316ca

​

44 of 78

tcpflow HTTP Capture

campusadmin@trusted-ci1:~$ sudo tcpflow -p -c -i lo port 8889

tcpflow: listening on lo

127.000.000.001.43882-127.000.000.001.08889: GET /?token=1297a2b4dc6e3d7ec8a95f6443acff930f1f1920f3172ac8 HTTP/1.1

Host: localhost:8888

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Upgrade-Insecure-Requests: 1

Cookie: username-localhost-8888="2|1:0|10:1571099165|23:username-localhost-8888|44:MzlkZTk1NjNhYjBmNDU2MWEyYmVmOWE5MjhlNzlmOTE=|8a9b04bf9af42ef7d84da55c9f3cc06ac1b91a330608ca2378a1743d7ed617cb"; _xsrf=2|25101594|37f123c20fef78ecf3cb7bb75bb7b5f1|1569445998

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.2 Safari/605.1.15

Accept-Language: en-us

Accept-Encoding: gzip, deflate

Connection: keep-alive

​

​

127.000.000.001.08889-127.000.000.001.43882: HTTP/1.1 302 Found

Server: TornadoServer/6.0.3

Content-Type: text/html; charset=UTF-8

Date: Tue, 15 Oct 2019 05:04:56 GMT

Location: /tree?token=1297a2b4dc6e3d7ec8a95f6443acff930f1f1920f3172ac8

Content-Length: 0

​

​

45 of 78

Notebook Server <-> Kernel Traffic

trusted-ci<N>.globus-training.net

Browser

1 Server / User

​

SSH:22

ZMQ:<port>

tcpflow

Python Kernel

HTTP:8889

{“code”:

46 of 78

Start a Kernel & Notebook�Find its Ports

Connection information is in the notebook

47 of 78

Start a Kernel & Notebook�Find its Ports

On host, connect info also in file.

Keep file ACLs limited to user

​

If “you” get connect info and

key

“you” can connect and execute

code as user

​

campusadmin@trusted-ci1:~$ cd .local/share/jupyter/runtime/

campusadmin@trusted-ci1:~$ ls -1tr

notebook_cookie_secret

nbserver-8485.json

nbserver-8485-open.html

nbserver-9738.json

nbserver-9738-open.html

kernel-e6757c17-0699-4e15-8ac7-4596c7178844.json

campusadmin@trusted-ci1:~$ cat kernel-*

{

"shell_port": 33617,

"iopub_port": 56051,

"stdin_port": 52759,

"control_port": 55021,

"hb_port": 50873,

"ip": "127.0.0.1",

"key": "1a917676-c05a83835fc282317c41975d",

"transport": "tcp",

"signature_scheme": "hmac-sha256",

"kernel_name": ""

}

campusadmin@trusted-ci1:~$

​

Connection information is also in the user home directory

48 of 78

Listen to Kernel Input and Output

$ sudo tcpflow -p -c -i lo port 56051

tcpflow: listening on lo

​

127.000.000.001.56051-127.000.000.001.43522:�... .snip. ...�{"code":"import os\nprint('because no one ever stores secrets in the environment')\nfor k in ('USER', 'PWD', 'HOME'):\n print('{}: {}'.format(k, os.environ[k]))","execution_count":10}

... .snip. ...

{"name":"stdout","text":"because no one ever stores secrets in the environment\nUSER: campusadmin\nPWD: /home/campusadmin\nHOME: /home/campusadmin\n"}

​

​

​

49 of 78

JupyterHub Starting Point

trusted-ci<N>.globus-training.net

JupyterHub proxy

Browser

HTTPS:443

Apache

HTTP:8000

JupyterHub�hub

Notebook�Server

HTTP:<>

ZMQ:<>

Python�Kernel

<> = “random” ports

50 of 78

JupyterHub HPC/Cluster Model

10.0.0.0/16

c-N-M.local

jupyter.example.edu

JupyterHub proxy

Browser

Apache

JupyterHub�hub

Notebook�Server

Python�Kernel

SSH:22

compute�node

private

network

51 of 78

Setting Up JupyterHub

Going to set up JupyterHub available at�� https://trusted-ci<N>.globus-training.net/jhub/

These hosts have Let’s Encrypt certs enabled

Apache config /etc/apache2/sites-enabled/000-default-le-ssl.conf

52 of 78

Apache Rewrites & Proxies

RewriteEngine On

RewriteCond %{HTTP:Connection} Upgrade [NC]

RewriteCond %{HTTP:Upgrade} websocket [NC]

​

RewriteRule /jhub/(.*) ws://127.0.0.1:8000/jhub/$1 [P,L]

RewriteRule /jhub/(.*) http://127.0.0.1:8000/jhub/$1 [P,L]

<Location "/jhub/">

# preserve Host header to avoid cross-origin problems

ProxyPreserveHost on

# proxy to JupyterHub

ProxyPass http://127.0.0.1:8000/jhub/

ProxyPassReverse http://127.0.0.1:8000/jhub/

</Location>

​

Add to /etc/apache2/sites-enabled/000-default-le-ssl.conf�Add to

Bounce Apache

sudo systemctl restart apache2.service

53 of 78

JupyterHub Config & Start

$ cd /etc/jupyterhub

$ sudo jupyterhub --generate-config -f \� /etc/jupyterhub/jupyterhub_config.py

$ sudo jupyterhub --ip 127.0.0.1 --port 8000 --base-url \

'/jhub/' --config=/etc/jupyterhub/jupyterhub_config.py

local:~$ ssh campusadmin@trusted-ci<N>.globus-training.net

SSH to the server

Create default JupyterHub config and start JupyterHub

  • We’re going to be changing the JupyterHub config a lot.
  • Should go into a service script.

54 of 78

Login As Researcher

55 of 78

Login As Researcher

56 of 78

JupyterHub Internal Traffic

trusted-ci<N>.globus-training.net

JupyterHub proxy

Browser

HTTPS:443

Apache

JupyterHub�hub

Notebook�Server

<> = “random” ports

tcpflow

GET /jhub/user

57 of 78

Find the Proxy Port

Listen...

[I 2019-10-15 01:49:24.982 JupyterHub proxy:261] Adding user researcher to proxy /jhub/user/researcher/ => http://127.0.0.1:41009

01:49:24.984 [ConfigProxy] info: Adding route /jhub/user/researcher -> http://127.0.0.1:41009

01:49:24.985 [ConfigProxy] info: Route added /jhub/user/researcher -> http://127.0.0.1:41009

​

campusadmin@trusted-ci2:~$ sudo tcpflow -p -c -i lo port 41009

tcpflow: listening on lo

​

127.000.000.001.50114-127.000.000.001.41009: GET /jhub/user/researcher/api/sessions?_=1571122288275 HTTP/1.1

x-forwarded-proto: http

x-forwarded-port: 80

​

​

58 of 78

JupyterHub Internal SSL

trusted-ci<N>.globus-training.net

JupyterHub proxy

Browser

HTTPS:443

Apache

JupyterHub�hub

Notebook�Server

<> = “random” ports

tcpflow

+A94cf23e99

59 of 78

Update Config and Create Certs

In /etc/jupyterhub/

$ sudo jupyterhub --ip 127.0.0.1 --port 8000 --base-url '/jhub/' \� --config=/etc/jupyterhub/jupyterhub_config.py --generate-certs

[I 2019-10-15 02:16:07.663 JupyterHub app:1363] Adding CA for hub-internal: IP:127.0.0.1;DNS:localhost;DNS:ec2-54-67-48-138.us-west-1.compute.amazonaws.com;DNS:trusted-c2.globus-training.net;DNS:trusted-ci2

[I 2019-10-15 02:16:07.889 JupyterHub app:1383] Generating signed pair for proxy-api: IP:127.0.0.1;DNS:localhost;DNS:trusted-c2.globus-training.net;DNS:trusted-ci2

[I 2019-10-15 02:16:07.978 JupyterHub app:1383] Generating signed pair for proxy-client: IP:127.0.0.1;DNS:localhost;DNS:trusted-c2.globus-training.net;DNS:trusted-ci2

[I 2019-10-15 02:16:08.143 JupyterHub app:2301] Certificates written to directory `/etc/jupyterhub/internal-ssl`

​

​

c.JupyterHub.trusted_alt_names = ['DNS:trusted-c<N>.globus-training.net', 'DNS:trusted-ci<N>']

c.JupyterHub.internal_certs_location = '/etc/jupyterhub/internal-ssl'

c.JupyterHub.internal_ssl = True

​

​

Add to /etc/jupyterhub/jupyterhub_config.py

60 of 78

Update Config and Create Certs

$ cd /etc/jupyterhub

$ sudo jupyterhub --ip 127.0.0.1 --port 8000 --base-url \

'/jhub/' --config=/etc/jupyterhub/jupyterhub_config.py

https://trusted-ci<N>.globus-training.net/jhub/

Browse to

Start JupyterHub

61 of 78

Me or JH Bug? User Cert Dir Ownership

$ sudo ls -al /home/researcher/

total 40

drwxr-xr-x 7 researcher researcher 4096 Oct 15 02:24 .

drwxr-xr-x 6 root root 4096 Oct 13 22:45 ..

-rw-r--r-- 1 researcher researcher 220 Oct 13 22:45 .bash_logout

-rw-r--r-- 1 researcher researcher 3771 Oct 13 22:45 .bashrc

drwx------ 2 researcher researcher 4096 Oct 15 01:49 .cache

drwx------ 3 root root 4096 Oct 15 02:27 .jupyterhub

drwx------ 3 researcher researcher 4096 Oct 15 01:49 .local

-rw-r--r-- 1 researcher researcher 807 Oct 13 22:45 .profile

​

​

​

$ sudo chown researcher:researcher /home/researcher/.jupyterhub

​

​

'certfile "%s" does not exist' % self.ssl_options["certfile"]

ValueError: certfile "/home/researcher/.jupyterhub/jupyterhub-certs/user-researcher.crt" does not exist

​

https://github.com/jupyterhub/jupyterhub/pull/2785

FIXED!

62 of 78

Find the Proxy Port

Listen...

[I 2019-10-15 02:31:20.224 JupyterHub proxy:261] Adding user researcher to proxy /jhub/user/researcher/ => https://127.0.0.1:36193

02:31:20.231 [ConfigProxy] info: Adding route /jhub/user/researcher -> https://127.0.0.1:36193

02:31:20.231 [ConfigProxy] info: Route added /jhub/user/researcher -> https://127.0.0.1:36193

​

$ sudo tcpflow -p -c -i lo port 36193

tcpflow: listening on lo

​

​

63 of 78

Success!

$ sudo tcpflow -p -c -i lo port 36193

tcpflow: listening on lo

127.000.000.001.47846-127.000.000.001.36193: ]!='3|DqyUn/+0,'g(k$

jih98762.*&=5#@?>32101-)%</i$"trusted-ci2.globus-training.net

3t

127.000.000.001.36193-127.000.000.001.47846: =9w*=b;g

0H1013073119Z010Uuser-researcher0"0

vWF%i<

EVK-z$FN}

-/qB/)~F5/|~S@|[HbUxRd`zj694cQUf'L|=Z}]#g/C3I|4FRL@<o*fTUj=P]_[O`UsHMgx4}4v\Ky?of%g]Ne?n0l0 U%0++0HUA0?localhosttrusted-ci2.globus-training.net:0%Ied`0V,;?.i^akNLp[?B.$rh$7IWc4Sus~=BOaMIAV74mdb=e>4W|Ai^^=8;q1KFztPiPOk>{r:[<^3;5b5e8M9k%

6@.o:g[wGa-3*#mLkYc#pecT>`qQ#E;x9rhq_Z*!.#l>x.O3),N[Z=BZi])7DFm$;9{`6%W2G3"$T8C-A=`#C!Go:

​

0H1013072429Z010Uproxy-client0"00.001.36193: #000

B#\ <=LWBX3?lyW=v}jT-C@[&SxUnFl+zk\zw|q[PFlkMk$Xvk=#Th=-s_?d{8w0w^}$k@ck0W?]h-"mT4.SX`+S#wQ9C

​

64 of 78

JupyterHub Internal SSL

trusted-ci<N>.globus-training.net

JupyterHub proxy

Browser

HTTPS:443

Apache

JupyterHub�hub

Notebook�Server

<> = “random” ports

tcpflow

{“code”:

ZMQ:<>

Python�Kernel

65 of 78

Start a Kernel & Notebook�Find its Ports

$ sudo ls -1tr /home/researcher/.local/share/jupyter/runtime/

nbserver-8485.json

nbserver-8485-open.html

nbserver-9738.json

nbserver-9738-open.html

kernel-e6757c17-0699-4e15-8ac7-4596c7178844.json

$ sudo cat \

/home/researcher/.local/share/jupyter/runtime/kernel-...

{

"shell_port": 45317,

"iopub_port": 54279,

"stdin_port": 47461,

"control_port": 58485,

"hb_port": 46187,

"ip": "127.0.0.1",

"key": "92b993fc-759de490904af0c952884cbd",

"transport": "tcp",

"signature_scheme": "hmac-sha256",

"kernel_name": ""

}

{“code”:

66 of 78

Change Kernel Communication to IPC

  • IPC uses file descriptors to pass data
  • Hard to intercept traffic
  • Does not work between hosts
  • ZeroMQ config for TLS available, in Jupyter soon
  • Also good for single user notebook server

c.Spawner.args = ['--transport="ipc"']

​

Add to /etc/jupyterhub/jupyterhub_config.py

67 of 78

Start a Kernel & Notebook�Find its Ports

$ sudo ls -1tr /home/researcher/.local/share/jupyter/runtime/

nbserver-11237.json

nbserver-11237-open.html

kernel-0746a014-a85a-4959-8572-637bfcaef64d-ipc-4

kernel-0746a014-a85a-4959-8572-637bfcaef64d-ipc-3

kernel-0746a014-a85a-4959-8572-637bfcaef64d-ipc-2

kernel-0746a014-a85a-4959-8572-637bfcaef64d-ipc-1

kernel-0746a014-a85a-4959-8572-637bfcaef64d.json

kernel-0746a014-a85a-4959-8572-637bfcaef64d-ipc-5

$ sudo cat /home/researcher/.local/share/jupyter/runtime/kernel-...

{

"shell_port": 1,

"iopub_port": 2,

"stdin_port": 3,

"control_port": 4,

"hb_port": 5,

"ip": "/home/researcher/.local/share/jupyter/runtime/kernel-0746a014-a85a-4959-8572-637bfcaef64d-ipc",

"key": "1bc70c05-808f5bec9272a64f66967ac3",

"transport": "ipc",

"signature_scheme": "hmac-sha256",

"kernel_name": ""

}

​

​

{“code”:

{“code”:

68 of 78

JupyterHub Internal SSL

trusted-ci<N>.globus-training.net

JupyterHub proxy

Browser

HTTPS:443

Apache

JupyterHub�hub

Notebook�Server

ZMQ:<>

Python�Kernel

HTTPS:443

HTTPS:443

69 of 78

Homework: Apache <-> JupyterHub

trusted-ci<N>.globus-training.net

JupyterHub proxy

Browser

Apache

JupyterHub�hub

Notebook�Server

Python�Kernel

HTTP:8000

Apache config

  • Configure ProxyPass & Rewrites to use HTTPS
  • Add JupyterHub proxy certs to Apache

70 of 78

JupyterHub HPC/Cluster Model

10.0.0.0/16

c-N-M.local

jupyter.example.edu

JupyterHub proxy

Browser

Apache

JupyterHub�hub

Notebook�Server

Python�Kernel

meh

JupyterHub-only server

71 of 78

JupyterHub

Authenticators

Default is PAM (system username & pass)

Outsource authentication to OAuth

  • Auth0
  • Bitbucket
  • CILogon
  • GitHub
  • GitLab
  • Globus
  • Google
  • MediaWiki
  • Okpy
  • OpenShift
  • Generic (campus)

​

​

72 of 78

JupyterHub Globus OAuthenticator

  • Steps are very similar for all�OAuth providers:
  • Register app, get client ID and secret

For this demo:

  • Go to � https://developers.globus.org
  • Set name, callback URL
  • Get client ID, client secret

73 of 78

  • Login
  • Create a project and add an app

74 of 78

Create app

Make sure this matches your host

Copy this

Do this & copy

75 of 78

JupyterHub Globus OAuthenticator

sudo JUPYTERHUB_CRYPT_KEY=$(cat /etc/jupyterhub/crypt.key) jupyterhub \

--ip 127.0.0.1 --port 8000 --base-url '/jhub/' \

--config=/etc/jupyterhub/jupyterhub_config.py

​

New start command to pass key for user secrets

Add to /etc/jupyterhub/jupyterhub_config.py

from oauthenticator.globus import LocalGlobusOAuthenticator

c.JupyterHub.authenticator_class = LocalGlobusOAuthenticator

c.LocalGlobusOAuthenticator.enable_auth_state = True

c.LocalGlobusOAuthenticator.oauth_callback_url = \� 'https://trusted-ci<N>.globus-training.net/jhub/hub/oauth_callback'

c.LocalGlobusOAuthenticator.client_id = '<>'

c.LocalGlobusOAuthenticator.client_secret = '<>'

c.LocalGlobusOAuthenticator.create_system_users = True

​

​

We’re going to create a new system account for each user

Can also explicitly map users to system accounts

c.Authenticator.username_map = {

'service-name': 'localname'

}

JupyterHub database will get an encrypted area

76 of 78

https://trusted-ci<N>.globus-training.net/jhub/

77 of 78

Wrap Up:

Answers to Earlier Questions

Defining Jupyter Security Best Practices

​

​

​

78 of 78

What can be done to improve

security in the Jupyter Community?

  • Who are the stakeholders?
  • What are the use cases?
  • Let’s start with you…
    • Why are you running Jupyter?
    • How do you run Jupyter?
    • What are the challenges?