February 12-16, 2024

WELCOME TO

DEPENDENCY MANAGEMENT 101

THE WEBINAR WILL BEGIN SHORTLY

February 12-16, 2024

DEPENDENCY MANAGEMENT 101

​

​

​

​

Renata Curty

Research Data Services UCSB Library

rds@library.ucsb.edu

• Discuss the significance of enhancing reproducibility in computational research
• Present the root causes of "dependency hell"
• Outline strategies to manage project dependencies more effectively

​

We will…

Scope

Reproducibility

Same data and code, different researchers, rendering same results

Image by: Candace Savonen (2022).

Computational Research

• Not always “plug-and-play”
• Many dependencies and nuanced interconnections
• Runtime dependencies for executing code and reproduce analyses

​

​

​

What is a dependency?

A directional connection between two or more elements, indicating a logical or sequential relationship among them.

If any of these elements fail, it poses a risk to both the efficiency of the process and the intended outcome.

Dependency Hell

​

​

​

​

Over time, libraries, packages, software and your computational environment evolve, introducing new versions and additional dependencies or even becoming outdated and unsupported.

This can lead to script failures and irreproducible research outcomes.

Image by: Savonen (2022).

I swear it worked on my machine…

Disclaimer: it may also happen to your future self

Source: https://www.reddit.com/r/statistics/comments/sqp4lk/q_results_changed_after_updating_r_version_what/

Another Example

​

Default values changing from TRUE to FALSE

Dependencies - Multiple Layers

• System-level libraries and packages
• System configurations
• Operating system

​

Packages & Libraries

R: Packages are collections of R functions, data, and compiled code in a well-defined format, created to add specific functionality. The directories in R where the packages are stored are called libraries.

Python: A library is a collection of code that makes everyday tasks more efficient.

Declare your dependencies!

Always describe the computing environment and all the required packages and libraries along with their specific versions

Recommendations

• Document your computing environment (i.e., operating system, software, and versions)
• Ensure to record all packages and libraries along with their versions
• Use dependency management systems and virtual environments to automate and streamline the steps above
• Consider using containers to capture dependencies and offer an environment runtime

​

Consider it as an spectrum

Static

​

Manual

​

Less Robust

​

Local

Environment

Executable

​

Automated

​

More Robust

​

Independent Environment

Dependency management approaches vary and might complement one another to achieve better outcomes.

How?

• Document your computing environment (i.e., operating system, software, and versions), packages and libraries

README

Usage Notes

Documentation

Where?

Documentation

Documentation

We can do better than that…

Documentation

A better example

How?

• Include in your project folder a file that others can readily use to install required dependencies.

requirements.txt

Install.R

sessionInfo()

​

More Detailed Documentation

environment.yml

environment.yml

Sliding a bit more to the right…

isolated | portable | reproducible

Renv

Venv

You can turn it on and off as will work in multiple projects without affecting on another!

Virtual Environments

Renv & Venv

• Isolation: enables adding or updating packages and libraries in one project without impacting other projects.
• Portability: simplifies moving your projects across computers and platforms, streamlining required installations.
• Reproducibility: pins precise package versions you rely on, ensuring consistent installations wherever you work.

​

​

​

​

bit.ly/r-venv

Why Renv?

There are other packages to help you manage dependencies in R such as Groundhog, RANG, MiniCRAN, Require and Checkpoint

Renv is more widely adopted, but also has the advantage of being integrated with RStudio.

​

Why Venv?

​

The venv is a Python built-in module (part of the standard library) which offers a more simple solution to manage dependencies and covers all the basics for virtualenv which requires a separate installation.

Are there any limitations of using renv and venv?

Binder

Bundle your project and all its dependencies and make it available via a web browser.

Containers

Image by: Taka & Thiéry. (2018)

Binder-ize your project!

• If on a repo (e.g., GitHub, Fighshare, Zenodo, Dataverse)
• Let others and your future self avoid the “dependency hell”
• Allow easier access and others to interact with your project with no installations needed

How?

Add two files to your repo (root directory):

​

1. “runtime.txt” - enter the R version of Python version in use

​

r-4.3.2-2023-03-15

python-3.10

​

• install.R (R) or requirements.txt (Python) file with packages/libraries info

​

• Access: https://mybinder.org

​

See examples: https://github.com/binder-examples

1. Select the appropriate repository and enter its URL or DOI
2. Click launch
3. Wait patiently wait for the environment to build.
4. Share the link to your “binder-ized” repository, or copy the supplied code snippet to your repository README to a badge so others can launch your project with a single click.

Other Recommendations

• Ensure you have updated all your packages before running your script locally
• Add to the README file and usage notes when your analysis was last performed
• Be patient while your Binder builds! It might take a couple of minutes

​

​

​

​

​

Binder Limitations

• Not for heavy computing (<10’ and <10 MB)
• The repository should be public
• The repository should not require any personal or sensitive information (such as passwords)

​

​

​

​

Static

​

Manual

​

Less Robust

​

Local

Environment

Executable

​

Automated

​

More Robust

​

Independent Environment

In a Nutshell

README

& Usage notes

​

Virtual Environments

(e.g., Renv & Venv)

Containers

(e.g., Binder)

Supply Install files

Caution: there is more to it!

• Watch out for paths (always relative!)
• Comment your code so others know the “whys” behind it
• Data documentation remains key! Explain all your variables, follow a file naming convention and intuitive project structure.

​

​

​

Learn More

Data Literacy Series Handouts: https://rcd.ucsb.edu/data-literacy-series

​

Other Resources:

​

Renv documentation

https://rstudio.github.io/renv/articles/renv.html

​

Venv documentation:

​

https://docs.python.org/3/library/venv.html

​

The Binder project:

​

https://jupyter.org/binder

​

​

​

Q&A

​

rds@library.ucsb.edu