ActivitySim Auto-Calibrate Task Scope and Design Proposal

Zephyr ActivitySim Phase 11C

Last Updated: January 2023

April 28, 2026

With:

​

Driftless Labs, LLC

WSP

Today’ Presentation

• Auto-calibrate scope of work
• Design considerations
• Implementation ideas

​

Motivation

• Currently, model calibration relies on ad-hoc procedures
• Some manual, some automated
• Each agency\consultant has their own approach
• A lot of reinventing the wheel
• Downsides
• ActivitySim must be re-load skims and start over each iteration
• Lots of repeat structure across ad-hoc procedures to update coefficients, rerun ActivitySim, etc.

​

• The goal is to have a built-in procedure in ActivitySim that everyone can use which is efficient, well-structured, and documented.

Scope of work

• Phase 1: Design
• In progress. Design document underway.
• Will check back with consortium once we have a more complete prototype
• Phase 2: Implement and test
• Three models: auto ownership, work location choice, work tour mode choice.
• Test on SANDAG model system

​

Task 1: Design

ActivitySim Auto-Calibrate Task Scope and Design Proposal

Task 1: Design considerations

• Flexibility in specifying which parameters to calibrate
• Different methods for adjusting parameters
• User settings to control calibration process
• Observed data
• Reports provided during calibration process
• Bundling components for calibration
• Software implementation

​

Task 1: Key Features

• Ability to iterate over one or more model components and adjust the coefficients.
• Built-in process for summarizing results consistent with target data
• Write output summaries at each iteration so that calibration process can be monitored
• Model results versus targets for current iteration
• Coefficient adjustments and the updated coefficient values for next iteration
• Plots of coefficient adjustments over multiple iterations to show and test for convergence.
• Automated calibration with limited to no user intervention beyond setup and monitoring

​

User Expectations

• Up to user to prepare calibration targets. Know your data.
• Up to user to specify constraints on the calibration process and targets for convergence.
• Up to the user to ensure that the coefficients to be calibrated do not result in an over-specified model.
• Up to the user to pay close attention to the sign and magnitude of calibrated coefficients and ensure that they are reasonable.

​

We will provide examples for each of the above points for the model components and targets covered in this task but we cannot cover every instance of model parameters that could be adjusted using automated procedures.

Methods for Adjusting Coefficients: Binary

• Most common coefficients to adjust in calibration are binary (0,1)
• Alternative-specific constants
• Could be differentiated by socio-economic market (0-autos for example)
• Geographic constants
• Binary coefficients typically adjusted using simple formula:

Constant_adjustment = ln(observed_share/estimated_share) * damping_factor

• Sometimes adjusted using a more complex formula (odds ratio):

Constant_adjustment = ln(((obs_share*est_share)-obs_share) / ((obs_share*est_share)-est_share))) * damping_factor

• We will give the user the option to use either formula for each coefficient to be calibrated

​

Methods for Adjusting Coefficients : Non-Linear

• Non-linear functions are more complicated to calibrate

​

​

​

​

• One method is to adjust trip length using constants by distance bin using binary variables

​

​

SANDAG Workplace Location Choice Distance Terms

Methods for Adjusting Coefficients : Non-Linear

• Another option would be to calibrate step-wise coefficients

​

​

​

• Or use regression:

ln(observed_share_bin/estimated_share_bin) = coef_dist * distance_bin

+ coef_dist_sqrt * sqrt(distance_bin)

+ coef_dist_sqrd * distance_bin²

+ coef_dist_cubed* distance_bin³

These terms then get added to the original terms in the next iteration

The initial version of the implementation will support binary adjustments; we may implement non-linear adjustments depending upon available budget

​

​

Other considerations

• ActivitySim must be told how to calculate estimated share to compare to target
• These calculations should have access to all data available to ActivitySim: Households, persons, vehicles, tours, and trips
• Also skims and land-use data (aka the share of transit trips with one or more transfers)
• ActivitySim should write output file with all coefficients for use in full model run
• Ideally, damping factor would be adjusted automatically if software observes oscillations
• Software should report problems reaching convergence
• Other thoughts?

Task 2: Implement and Test

ActivitySim Auto-Calibrate Task Scope and Design Proposal

Implementation Requirements

Requirements:

• Calibrate:
• Summarize data
• calculate new coefficients, and
• Update coefficient values in-place
• Summarize:
• Plots
• Summary tables
• Want the interface to be easily transferrable yet optionally very flexible
• Building a framework with examples

Implementation Design

A new calibration run mode analogous to estimation

Settings:

• Primary calibration settings file specifying calibration options
• Component-level calibration config files
• Calibration spec file is very similar to our current config files

​

calibration.yaml prototype

Calibration Example Spec – Auto Ownership

User responsible for:

• Coefficient name matches model spec
• User responsible for setting up what they want calibrated

auto_ownership_calibration.csv

auto_ownership.csv

1 auto is base alternative

Calibration Example Spec – Auto Ownership

• Bespoke helper functions can be used to summarize the data
• Allows:
• segmentation (e.g. zero-autos by geographic region) and
• summarizing survey data within the ActivitySim model (e.g. scaling of survey households to match model households)
• Whatever you want – its python code!

Generic (default) Summary Reports

Output table with the following columns:

• Iteration, description, coefficient, target_value, model_value, difference, hold_fast, coef_before, coef_change, coef_after, converged, hit_min, hit_max

Plots derived from this table:

• Bar plots between target_value and model_value
• Coefficient changes by iteration

Summary Statistics by iteration:

• Max_difference, mean_difference, max_coef_change, num_clipped, num_hold_fast, num_converged, num_not_converged

Other useful summaries?

​

Bespoke Summary Reporting – User specified plots

• User can specify a reporting function and put in whatever code they want to be run
• Useful for comparing model data and survey data, handling weights, summarizing by segments, etc.
• Runs during each iteration

Questions\Discussion

20

Thank you

Joel Freedman

NOTE

PRINCIPAL

Joel.Freedman@rsginc.com

David Hensle

SENIOR CONSULTANT

David.Hensle@rsginc.com