Team FICO Final Report

Cal Poly Data Science Capstone 2018

Written by 

Aleksander Braksator, Tommy White, and Owais Safaraz

Advised by

Dr. Alex Dekhtyar, and Dr. Dennis Sun

FICO Representative: Steve Lakowske

Introduction

In this report we describe work done under supervision from FICO in attempting to qualify and quantify safe driving behavior. FICO has developed a computational model to generate a “Safe Driving Score” using smartphone-based data (primarily from events). FICO aims to utilize continuous time-series data in addition to this event data to help generate these scores. These scores could then be used by self-driving car companies, insurers, fleets, and others to assess crash risk. Our FICO representative, Steve Lakowske, guided us to see if we could construct a model which could identify driving behavior conducive to being involved in an accident.

We spent hours watching videos, developing novel algorithms, and clustering on fourier transforms of time series data trying to define safe driving behavior. In the process, we generated a dataset that could be useful to researchers conducting further work in this field. We will present not just our results, but also our process to provide readers with as much insight as possible into our work.

Data Description

We are using data from the Second Strategic Highway Research Program Naturalistic Driving Study (SHRP2 NDS) as provided by the Virginia Tech Transportation Institute, available at https://insight.shrp2nds.us/. This study was originally conducted “to address the role of driver performance and behavior in traffic safety” (Campbell). We are limited to using datasets with few accidents and behaviors simulated by researchers due to the lengthy approval process involved in obtaining real driver data.

The datasets we have access to consist of video data and sensor readings from automobile journeys conducted by researchers from VTTI. During these drives, researchers simulated a variety of maneuvers, including hard braking, parking lot driving, hard cornering, and more. For each trip, we have a data table containing all of the sensor readings taken during the trip and four video files (front-facing, rear-facing, one of the face, and one of the hands).

The time series data gathers data from the car itself and built in sensors which are updated once every 100 ms. GPS data is also included with 1 Hz frequency, and these variables track up to eight other vehicles that potentially could surround the Virginia Tech vehicle. The GPS data also has its own speed measurement, in addition to the speed data coming from the vehicle itself. There has been some preliminary work done with light-level sensors. This work may be expanded upon in future iterations once we have the ability to analyze video data. Presently, we are not pursuing a thorough and in-depth analysis of these sensor variables due to the fact that the variable nature of this data as well as the specific contexts in which these data are collected make them difficult to compare.

The video data is a set of four synchronized recordings of the front view and rear view from the car and the face and hands of the driver. This video is approximately 356x480 in resolution at 15 FPS. The length of the video can vary from five minutes to an hour. There are many ways we can use this data in the future including light level analysis and using CNNs for object detection (e.g. stop signs and red lights).

Encoding the Data

Our first instinct after receiving the data was to try and find apparent “abnormalities” that could identify good and bad driving behavior. These abnormalities included high speeds, hard braking, sharp turns, and other potentially dangerous maneuvers. However, we soon realized that whether a maneuver should be considered “dangerous” or not is highly dependent on the context in which the maneuver took place.

For example, a sharp turn of a steering wheel may be dangerous at high speeds on a highway. On the other hand, sometimes it is perfectly safe and necessary to turn the steering wheel sharply in order to turn into a parking space in a parking lot.

To identify the circumstances in which a maneuver was occuring, we wrote code with the purpose of “chunking” our data into segments. These segments would identify the context in which a maneuver would occur. For our purposes we identified three major maneuvers: cruising, stopping, and turning. However, we found that our methodology often led to many mislabeled, inaccurate, and missing chunks. Much of this was due to the variety of ways maneuvers were executed as well as situations in which multiple maneuvers were occuring at once.

To rectify this issue we decided to physically inspect the video data and manually label when these events of interest occurred, following a carefully constructed procedure^[1]. We watched at least 2 hours of video from each unique driver/vehicle combination and documented every instance of turning, curving roads, cruising, stopping, and lane changes. In addition to these maneuvers, we also noted environmental factors such as stoplights, stop signs, and whether the driver was in a parking lot^[2].

The product of this process was over 800 recorded maneuvers with the starting timestamp, ending timestamp, and notes about abnormalities or circumstances that could be of interest. These new manually labeled chunks provided us with ground truth upon which we could build and compare our models.

Process Overview

The final implementation of our pipeline solution works in three phases:

1. Time-series telematics data from a road journey is read into a “chunking algorithm” which extracts start and end times for all maneuvers throughout the journey.
2. A time-series dataframe of certain measurements for each event is extracted, zero-padded, and Fourier Transformed. A distance between each resulting dataframe (complex values transformed to magnitudes) is calculated and saved to a distance matrix.
3. K Means clustering algorithm is iterated on dataframes to separate maneuvers into different clusters. Analysis is done on the resulting clusters, including watching video segments from each cluster to visually identify what distinguishes the groupings, Silhouette coefficient minimization to find ideal parameter values, and ANOVA to identify variables (or frequencies) that distinguish from the clusters.

This process allows construction of a pipeline that could take in telematics data from a trip, identify maneuvers taken during the trip, and classify these maneuvers as safe or unsafe based on previous data. Each of the phases in our process will be described in detail in the following sections.

Chunking

Chunking, in our project, is the process of identifying segments of the time series data that correspond to the driver performing a maneuver. The maneuvers we decided to target in our analysis were cruising, turns, curves, and stops^[3]. We developed our own algorithms to identify these maneuvers, and tuned the parameters of our chunking algorithms using a Python package called Hyperopt and our manually labeled data (Hyperopt).

We have included information about the final chunking algorithms we used in identifying cruising, turn, curve, and stopping chunks below, in addition to a description of the optimization process and results.

Cruising

In order to detect cruising, we designed an algorithm to identify when the driver maintained a steady speed. In order to reduce noise due to the high frequency of the sensor’s velocity data, we used a windowed mean calculated as the mean of the previous n observations. We used this windowed mean velocity to identify segments where each observation in the segment has a windowed mean velocity within a margin of the mean velocity over the entire segment.

The parameters to be tuned were the minimum duration of a cruising chunk, the maximum tolerance when comparing a segment mean speed to a windowed mean speed, and the window size to use when computing the windowed mean speed.

An example of a four minute and 31 second cruising chunk identified by our algorithm can be seen in the graph above. Note that the y axis only spans from 100 to 108 kph, and the speed only varies between 103 kph and 105 kph during this segment.

Turns/Curves

To detect turns and curves, our final algorithm focused on the GPS-provided heading of the vehicle. In order to use this data, however, we had to convert the XYZ measurement provided into a continuous scale. To do so, we first converted the XYZ measurement into radians. Next, we unrolled this angular data so that the range could be infinite rather than bounded between 0 and 2ℼ. After this conversion, turns were identified by looking at periods where the derivative of this unrolled heading data was non-zero.

The only parameter to be varied during optimization of this algorithm was the threshold for the magnitude of the derivative of the heading. The graph below shows the derivative of the heading observations for a curve chunk identified by our process.

Stops

In identifying stops, we decided to analyze jerk, as recommended by our FICO representative. Jerk is the rate of change of acceleration, and periods of high jerk correspond to quick changes acceleration, such as hard braking or acceleration. We identified segments as starting when the value of jerk surpassed a provided threshold and ending when the value passed the threshold a second time.

The parameters to be manipulated during tuning of this algorithm were the jerk threshold, the number of frames to include from before the jerk passes the threshold, and the number of frames to include from after the steering wheel position passes the threshold again. The graph below shows the jerk observations for a stop chunk identified by our method.

Optimization

We use a Python package called Hyperopt in chunking to tune our algorithm parameters. Hyperopt requires the user to specify the statistical models from which to sample for each of the parameters of a model, and requires the user to specify the loss function to minimize (Hyperopt).

We originally decided to optimize on the F-score^[4], as it provides insight into the algorithm’s performance not just in correctly classifying segments, but also in identifying all segments. Later on, we realized that using precision serves our purpose better, as during later analysis we want to be as certain as possible that the segments we are using are correctly labeled.

All of our segmentation methods have been optimized using a precision metric derived from Gensler’s paper on performance metrics for time series segmentation (Gensler). This metric is calculated as the amount of frames that were correctly identified divided by the amount of frames that were identified by the algorithm. In discussion of results, we will also consider recall, which is calculated as the amount of frames that were correctly identified divided by the amount of frames that were hand-labeled as being in a chunk. Precision gives us a sense of how confident we can be in labels from our algorithms, and recall gives a sense of how many maneuvers of interest our algorithms are failing to pick up on.

Results

Cruise Results

For cruising, the final parameter values were a tolerance of 4.92 km/h, a minimum segment length of 2950 frames, and a window length of 52 frames. The final algorithm had a high training accuracy, but performed significantly worse in validation. We believe these results could potentially be explained by overfitting caused by the high number of trials and relatively small amount of data. Another potential issue is the discrepancy between a computer’s ability to notice changes in speed and that of a human. The hand-labeled cruising segments are much longer and have much more variation in speed than the computer-identified ones. To improve the algorithm, we could enforce a ‘memory” of sorts by increasing the influence of more recent observations when calculating the current average segment speed.

Turn Results

In identifying turning, the optimal parameter values were a dHeading threshold of 0.193, a cutoff of 18 frames before the threshold is crossed, and a cutoff of 48 frames after the threshold is crossed again. The optimized algorithm was able to achieve very high accuracy in both training and validation. We feared that the algorithm may have become overly selective (thereby missing many turns), as it only identified one or two turns per time series. To test this hypothesis, we checked the recall metric for our validation set, only to find that it had successfully identified 88.8% of all frames that were hand-labeled as turns.

Curve Results

When detecting curves, the optimized parameter values were a dHeading threshold of 0.0356, a cutoff of 142 frames before the maneuver, and a cutoff of 13 frames after the maneuver. Despite the fact that we use the same algorithm as we use when identifying turns, we were not as successful in identifying curves. The final algorithm achieved a significantly better validation accuracy (89%) than training accuracy (77%), as well as a low recall score of 33%. One potential source of inaccuracy currently is that there is no upper threshold on the curve identification, giving the algorithm no way to distinguish sharp curves from normal turns. To improve this algorithm, it should be further combined with turn identifying so that the combined algorithm identifies and distinguishes between turns and curves. This improved method would incorporate information about how sharply the turn started as well as distance covered during the maneuver to determine whether it is a turn or a curve.

Stop Results

Our final parameters when identifying stops were a jerk threshold of 4.7, a cutoff of 21 frames before the maneuver, and a cutoff of 19 frames after the maneuver. The algorithm based on a threshold on the jerk value proved ineffective in identifying stops, even with the help of Hyperopt. The final algorithm only correctly identified 24% of frames as stops in both training and validation, but it performed surprisingly well in another aspect. It achieved a relatively high recall rate, successfully identifying 78.5% of all frames that were hand-labeled as stops. The high misclassification rate, however, means that this algorithm is still not viable for producing chunks.

Clustering

The clustering stage itself comes in three phases. Some hyperparameters to consider for a given journey include the types of events to record (any combination of turns, curves, lanes, stops), the telematic measurements to be recorded, the size of the dataframes after zero-padding, the type of distance metric to be used, the number of iterations for the k means algorithm, and the number of final clusters. Here we explain the three stages of our clustering algorithm and where these hyperparameters specifically come into play.

Clustering Pre-Processing

The output of the chunking process requires pre-processing even before the Fourier transformation is applied. First, the time series for events of interest are extracted from the results of the chunking algorithm. This can include any subset of turns, curves, lane changes, and stop. Cruising chunks have been omitted from this study because they contain too many other events to be clearly clustered. The event labels of the chunking algorithm are saved in a list with corresponding indices to the list of time series The maximum number of data points among all these time series is saved as a variable called max_len.

Once the time series data frames have been found for all maneuvers of interest, we extract telematic measurements of interest, such as the gyro forces (pitch, roll and yaw) in the x, y, and z directions. Then we zero-pad each data frame of measurements so that the length of each is the smallest power of two greater than max_len. This step improves efficiency of the fourier transform algorithm without sacrificing information. If a power of two less than max_len is specified as a hyperparameter, then the dataframes are downsampled to be the length of that power of two.

The zero-padded time-domain dataframes are finally transformed into the frequency domain with a numpy fast fourier multivariate transform. The pre-processing is complete once the magnitudes of the complex-values of the fourier data frames are computed.

Distance Matrix Calculation

The distances between all n fourier dataframes are then computed in an nxn distance matrix to improve efficiency of the clustering algorithm. The two distance metric options are euclidean distance and a “lambda” distance metric. Once this step is completed, all three data structures necessary for clustering are ready: the list of fourier transformed data frames, the list of corresponding labels, and the distance matrix.

To compute the euclidean distance between two dataframes with three columns downsampled to 1024 rows, find the square root of the sum of squared differences between all 3*1024 corresponding values between the two dataframes.

Our lambda distance metric is based on a paper by Bagnall, Janacek, and Powell which details how to compute a log likelihood ratio between two fourier transforms (Bagnall et al.). We implemented this algorithm into our clustering, but failed to get good results as we could not determine a proper way to minimize distances in the log likelihood space. This inability to minimize inter-observation distance caused the clustering algorithm we used to be unable to identify clusters with high purity^[5].

K Means Clustering

At this stage, we have a list of N frequency-domain data frames, a list of N labels determined by the intelligent chunking algorithm, and an NxN distance matrix. The implementation of our K means algorithm begins by choosing K random data frames to be the initial centroids for the clusters.  Then, the distance matrix is used to assign the rest of the dataframes to the cluster with the closest centroid. The centroids are then recalculated and all points are reassigned. The number of iterations of this step is given by the program user along with K.

The clustering output shows the number of each maneuvers within each cluster, the purity of each clusters, and the labels for all points in each cluster. There is also information on which events were closest and furthest to their centroids so that they could be rewatched to gain intuition on the contents of each cluster.  

Results

In this section, we will present the results of our clustering analysis. Clustering was performed on the entire set of maneuvers to see if the gyro forces (x, y, and z directions) can help identify which maneuver is being performed in a given segment. Then, we will cluster on each category of maneuvers (e.g. only turns) to see if there are clusters that help distinguish between safe and dangerous driving maneuvers. The analysis of the clustering performed on all maneuvers should help us identify variables to monitor in future work in chunking the time series data. The single category analysis will provide insight into generating a safety score for drivers.

First Round

The first round of results come from clustering with just turn and curve maneuvers. We used K  = 6 clusters with 25 iterations of the K means algorithm, calculating euclidean distances between all fourier data frames representing turn and curved maneuvers, with length 2014 (210) on three columns representing the FFT transformation of the three gyro force variables. Before interpreting the contents of each cluster, we visualized the purity of results for each cluster to demonstrate how well the algorithm is able to distinguish turns from curves:

In cluster 0, we see 98% purity with turns. Taking a closer look at the videos associated with this cluster, we see these to be standard right angle intersection turns.

Cluster 1 contains 100% curves which were at constant speed, usually on a banked highway with medium angular velocity.

Cluster 2 contains a 50/50 split of maneuvers that all have significant steering wheel movement (changing gyro z variable) with some jerky movements (such as deceleration or shaky wheel movements) that disturb the change in forces in the x and y directions. These jerky disturbances can occur among both turns and curves, hence why we see both in this cluster.

Cluster 3 contains steering maneuvers on loopy on ramps and off ramps. Since on ramps and off ramps are combinations of turns and curves, we see low purity here based on the discrepancy between data encoders on whether to classify these events as “turns” or “curves”. Regardless, they all share the common trait that these maneuvers take place on a loopy freeway on ramp or off ramp.

Cluster 4 contains very long freeway curves, with constant, gentle angle. This cluster also was comprised of 100% curves.

Lastly, cluster 5 consisted of 97% curves that were so gentle and light on steering that they were difficult to identify without looking at the curved road ahead.

Overall, it is clear that a K means algorithm on the fourier representations of the pitch, roll, and yaw forces is able to provide meaningful distinctions between different types of curves. It even finds commonalities among maneuvers labeled differently such as Cluster 2.

Second Round

Next, clustering with K = 5 was performed on only lane changes. Aside from the 4 clusters of generic lane changes, lane change going with and against the curve, there was a fifth cluster with only 4 observations which was of interest. Upon close examination of these four “outlier” events, they were all lane changes made from a stop and go situation. For example, a driver was blocked by a stopped truck in the middle of a small road, stopped behind the truck, and made a lane change from a full stop to maneuver around the obstacle. It is up to the situation and the eye of the beholder to determine whether these maneuvers are risky or not. The algorithm’s performance to group these lane changes together, however, is promising to distinguish potentially more dangerous, last second lane changes from regular lane changes. Our data, however, does not contain very many dangerous lane changes.

When we add lane change maneuvers to a K = 6 clustering along with curves and turns, we see a few of the same clusters remain mostly unchanged such as the on ramp and off ramp cluster. In the bar graph below, it is clear that a lot of the lane changes end up grouped in the cluster of very slight curves (Cluster 5). This is because a lot of lane changes are essentially forced slight curves that draw a similar path to a slight curvy road.

Final Round

The last round of clustering to analyze is for all events concerning stops, curves, lanes, and turns. For this test, we doubled our data frame size to include 2048 rows, found the ideal K to be 7 based on the sum of distances of each point to its centroid, and kept using Euclidean distance, 25 iterations, and the same gyro force variables. Here is what the distribution of ground truth labels looks like within each cluster:

It is unfortunate that we were unable to achieve any cluster with a majority of events being labeled as “stops”. Instead, “stops” seem to get grouped in all the clusters where lane changes and curves seem to be present. A possible future adjustment for this would be to introduce new variables into the pre-fourier transformed time series and to adjust the distance metric to account for the new variables if need be.

Cluster 0 is mainly the lane change cluster, with some curves that resemble lane change maneuvers and some stops that seem out of place. Upon closer examination of some of the stops, they are typical stopping maneuvers one would make at a red light. This is interesting because when we increased K, the stops were still being clustered into clusters with lanes and curves instead of forming their own cluster. In fact, Cluster 0 has one of the smallest average distances of each point to its centroid. This is more reason to add more telematic variables to the algorithm to help distinguish stops better into their own clusters.

Cluster 1 contains all the maneuvers that contain jerky behavior. This is why it contains a little of each category, and stops are not out of place here since they are sometimes what constitute the jerky behavior during another maneuver.  

Cluster 2 is mostly a curve cluster once again with quite a few lanes and stops.The point, closest to the centroid, was actually a stop.

Cluster 3 is another cluster which contains events that exhibit somewhat “jerky” behavior. These were caused by events such as a driver swerving during a curve, or many times by the car going up or down a slope such as a freeway exit.

Cluster 4 is a cluster containing long freeway banked curves that would require moderate steering wheel adjustment.

Cluster 5 is the same cluster as in round 1 containing right angle turns.

Cluster 6 is the same cluster as in round 1 containing lef

To conclude this round of clustering, adding the “stop” events to our clusters did not add any insight to the data. According to Euclidean distances, “stop” events were seen as more similar to lanes and curves than they were to each other. This caused events labeled as stops to be distributed among 4 different clusters where they did not really belong. Some clusters remained unchanged from before, which further reinforces the success of using the three gyro force variables to label different kinds of turns and curves.

Insights

It is still difficult to predict how well the K means clustering will do when distinguishing actual dangerous events. If a corpus of relatively safe events for a driver were collected and clustered by K-Means, then ideally we would hope that more “dangerous” maneuvers get clustered to a centroid that they are farther away from than the rest of the points, based on the distance metric being used. Then, based on the average distances of points of interest to the centroids of already clustered events, a standardized “Driving Score” could begin to take shape. This, however, was beyond the scope of this project.

Conclusion

Every year, there are approximately 6 million car crashes in the USA. Every month, approximately 250,000 people are injured in these accidents. Every day, about 90 people die by them. Insurance companies are tasked with the duty of protecting drivers so that when they are in an accident, they have the peace of mind provided by knowing that they are financially covered. However, insurance companies like FICO may struggle to identify dangerous incidents appropriately, making it difficult to identify high-risk drivers that are responsible for most of these accidents. Our team has worked tirelessly on helping FICO start the process of identifying what makes someone a high-risk driver.

Over the course of this project, our team begun the implementation of algorithms aimed at identifying different driving maneuvers and clustering them based on context. While we were unable to create a foolproof 100% accurate methodology, our results helped deliver insight to the understanding of what constitutes safe driving and its different classes of maneuvers. In addition, we were able to deliver a hand-coded dataset of over 800 recorded events from 6 hours of front camera video footage.

We would like to thank FICO for the opportunity to work on this project. It is our hope that they can take what we have done and use it to help improve their models and make the roads a safer place for all drivers.

References

Andre Gensler, Bernhard Sick, “Novel Criteria to Measure Performance of Time Series Segmentation Techniques”.

Campbell, Kenneth L. “The SHRP 2 Naturalistic Driving Study: Addressing Driver Performance and Behavior in Traffic Safety.” SHRP2 NDS, Insight, Oct. 2012, https://insight.shrp2nds.us/documents/shrp2_background.pdf.

“Hyperopt.” Hyperopt by Hyperopt, hyperopt.github.io/hyperopt/.

Janacek, Gareth J., Anthony J. Bagnall, and Michael Powell. "A likelihood ratio distance measure for the similarity between the fourier transform of time series." Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, Berlin, Heidelberg, 2005.

Appendix A: Concessions

As we continued to work on the project there were many options and leads we wanted to explore. However, due to a variety of circumstances (such as the nature of the data, team capabilities, and time pressure) we could not pursue every idea. This section will outline some of the  various methods of analysis we considered and why we chose not to pursue them.

Maneuver Selection

In our chunking process, we chose not to target lane changes and any behavior that occured in a parking lot. Lane changes were not considered because turn signal information was left out of all of the datasets we had access to, making detection significantly harder. Parking lot driving was ignored because the researchers would often use parking lots to simulate abnormal behavior. We believe that the limited set of maneuvers we targeted during this analysis provide adequate insight into driver behavior.

OpenCV

We had a set of videos, all at the same resolution, synchronized to a csv of sensor data. Thus, computer vision seemed like a natural fit as a method of analysis for our video data. In order to explore computer vision as a method of analysis we used the library OpenCV to examine the video data. In an ideal case we could have used OpenCV to identify important features of interest such as what sort of road a driver was driving on, circumstances of stopping, and how much of the road was clearly visible to the driver. However upon conducting exploratory analysis of the video data using OpenCV we found that these goals were more easily said than done. For one, we found that much of our video data took place in weather that hindered visibility to the point of it being near impossible to find any interesting features. Moreover, aligning the video data to the data present in our CSV files would be very difficult. Our CSV files had a frequency of 100 milliseconds. Each frame of the videos corresponded to approximately 66 milliseconds. Working around these problems would have used too much time. As such we chose to set aside OpenCV as an option for analysis.

Stoplights and Stop Signs

In our manual chunking process we made sure to note when stoplights and stop signs would appear in our data. We had hoped to use these to identify the circumstances under which stops would occur. We had to eliminate stop signs from consideration immediately as they did not occur enough in the data we looked at (<10 observations) for us to get statistically significant insight. We then looked into using OpenCV to identify stoplights from the video data, but, due to time restrictions and other issues discussed above (under OpenCV), this goal also had to be abandoned. Consequently, we did not end up using information recorded about stoplights or stop signs in any significant way in the final version of this project.

Appendix B: Hand Coding Process Description

Goal

Identify every incidence of each of the following maneuvers in our dataset:

• Turning
• Navigating a curvy road
• Cruising
• Stops
• Lane changes

In addition to the following environmental factors:

• Stoplights
• Stop signs
• Parking lots

We will be tagging these for roughly two hours of each vehicle-driver combination. There are three vehicles and three drivers, leading to nine possible vehicle-driver combinations and nine spreadsheets of tagged data.

Criteria

Turning :: turn

        Identify when a driver turns at an intersection. Note the direction (right/left).

Curvy Roads :: curve

        Identify when a driver navigates a curve in a road (not turning). Curves will be defined as points in the road where the driver must adjust the steering wheel to stay on the road. Note the direction (right/left).

Cruising :: cruise

         Record start when the driver seems to begin maintaining a steady speed. Record end when the driver breaks from this pattern. Note any errant actions taken during the middle of the cruising on their own row with the ‘middle’ tag.

Lane Changing :: lane

        Record start when the driver begins to leave their lane. Record end when the driver is fully in control in the other lane.

Stops :: stop

        Record start when the driver begins noticeably decelerating with intent of stopping. Record end when the driver begins accelerating again. Only mark the stop if it is reasonably complete (where the driver ends up going <~5mph as a result of the maneuver). Make a note if the stop is a “rolling stop”.

Stoplights :: stoplight

        Record start as soon as a red light is visible (whether by coming into view or turning red). Record end when the red light turns green. For yellow lights, if the driver slows down to a stop, record a stop entry and note that they stopped because of the light. If the driver runs the yellow, record it as continued cruising (assuming they were already cruising) and note that they ran the yellow.

Stop signs :: stopsign

        Record start as soon as the stop sign is visible and record end when the driver passes it.

Parking lots :: park

        This one is fairly self-explanatory, simply identify when they are in parking lots. Record start when they enter the lot and end when they leave. Do not record stops/turns in this.

Product

This process will result in nine spreadsheets. Our spreadsheets will follow the following format: