1 of 23

Tests from Traces: Automated Unit Test Extraction for

Filip Křikava, Czech Technical University

Jan Vitek, Northeastern University

2 of 23

Motivation - Why R?

Popular programing language for data analysis and statistics
CRAN - Comprehensive R Archive Network

curated repository of 12,000+ packages
on avg 8 new packages/day¹

2

Uwe Ligges, 20 Years of CRAN, keynote at UseR'17!

3 of 23

Motivation - Test code coverage for 1,500 packages

3

Corpus

TOP 100 most downloaded CRAN packages
plus random 1,000 CRAN packages
plus their dependencies

1.7M lines of R code

On average 19% code coverage

4 of 23

Motivation - Testing R package

4

Rvmmin/

├── DESCRIPTION

├── inst/

├── man/

├── MD5

├── NAMESPACE

├── NEWS

├── R/

├── tests/

│ └── BTbad.R

└── vignettes/

No asserts or checks - just an ordinary R program

# BTbad.R -- inadmissible bounds to�# see if Rvmmin with keepinputpar = TRUE stops.�#####################�… … … �… … … �n<-10�xx <- rep(0,n)�lower <- rep(1,n)�upper <- rep(3,n)�bdmsk <- rep(1,n) # all free parameters�ansbt <- try(

Rvmmin(xx, bt.f, bt.g, lower, upper, bdmsk,� control=list(trace=1, keepinputpar=TRUE))

)

�if (class(ansbt) == "try-error") {� cat("Successful stop when out of bounds\n")�} else {

print(ansbt)

}

5 of 23

Motivation - Testing R package

R programs that are checked if they ran without exceptions

R programs whose textual output is compared

producing warnings in mismatch�

Unit testing framework that checks expectations

~ 3,000 / ~ 12,000 use some unit testing framework
testthat is by far the most popular

5

6 of 23

Motivation - R package examples

6

Rvmmin/

├── DESCRIPTION

├── inst/

├── man/

│ ├── Rvmminb.Rd

│ ├── Rvmmin.Rd

│ └── Rvmminu.Rd

├── MD5

├── NAMESPACE

├── NEWS

├── R/

├── tests/

│ └── BTbad.R

└── vignettes/

└── Rvmmin.Rmd

\name{Rvmmin}�\title{Variable metric nonlinear function minimization, driver.}�\description{A driver to call the unconstrained and bounds ....}�\usage{� Rvmmin(par,fn,gr,lower,upper,bdmsk,control=list(),\dots)�}�\arguments{� \item{par}{A numeric vector of starting estimates.}

… … …

}

\examples{

� fr <- function(x) {� x1 <- x[1]� x2 <- x[2]� 100 * (x2 - x1 * x1)^2 + (1 - x1)^2� }�� ansrosenbrock <- Rvmmin(fn=fr,gr="grfwd", par=c(1,2))� print(ansrosenbrock)

… … … �}

7 of 23

Motivation - R package vignettes

7

Rvmmin/

├── DESCRIPTION

├── inst/

├── man/

│ ├── Rvmminb.Rd

│ ├── Rvmmin.Rd

│ └── Rvmminu.Rd

├── MD5

├── NAMESPACE

├── NEWS

├── R/

├── tests/

│ └── BTbad.R

└── vignettes/

└── Rvmmin.Rmd

… … …

8 of 23

8

Can unit tests be extracted?

Writing and maintaining unit tests is time consuming and boring
Yet, there is runnable code
Would it be possible to extract tests from the existing code?

Runnable code in R packages

On avg 86 lines

Reverse dependencies of R packages

9 of 23

9

genthat - R package for automated unit test generation from traces

10 of 23

Tracing example

10

filter <- function(xs, p) xs[sapply(xs, p)]

Target mypkg package

> is_odd <- function(x) x %% 2L != 0L

> m <- 1�> filter(floor(runif(10, 1, 10)) + m, is_odd)�[1] 5 9 9 7 5 3

Client code (eg., example, vignette)

$ :List of 6� ..$ fun : chr "filter"� ..$ pkg : chr "mypkg"� ..$ args : List of 2� .. ..$ xs: language floor(runif(10, 1, 10)) + m� .. ..$ p : symbol is_odd� ..$ globals:List of 2� .. ..$ floor : language base::floor � .. ..$ runif : language stats::runif � .. ..$ is_odd : function (x) x %% 2L != 0L � .. ..$ m : num 1� ..$ seed : int [1:626] 403 624 507561766 ...� ..$ retv : int [1:6] 5 9 9 7 5 3

Trace

filter <- function(xs, p) {� `__captured_seed` <- get(".Random.seed", envir=globalenv())� on.exit(with_paused_tracing({� retv <- returnValue(default=deflt_retv)� if (!normal_retv(retv)) {� record_trace(

name="filter", pkg="mypkg",

args=as.list(match.call())[-1], retv=retv,

seed=`__captured_seed`, env=parent.frame()

)� }� })

� xs[sapply(xs, p)]�}

Instrumented target mypkg package

11 of 23

Generating unit tests

11

$ :List of 6� ..$ fun : chr "filter"� ..$ pkg : chr "mypkg"� ..$ args : List of 2� .. ..$ xs: language floor(runif(10, 1, 10)) + m� .. ..$ p : symbol is_odd� ..$ globals:List of 2� .. ..$ floor : language base::floor � .. ..$ runif : language stats::runif � .. ..$ is_odd : function (x) x %% 2L != 0L � .. ..$ m : num 1� ..$ seed : int [1:626] 403 624 507561766 ...� ..$ retv : int [1:6] 5 9 9 7 5 3

Trace

.trace

.ext

seed : int [1:626] 403 624 507561766 ...

External variables�(binary)

.R

library(testthat)

library(mypkg)��.Random.seed <<- .ext.seed��test_that("filter", {� is_odd <- genthat::with_env(

function(x) x %% 2L != 0L

)� m <- 1�

expect_equal(� filter(

xs = base::floor(stats::runif(10, 1, 10)+m),

p = is_odd

), � c(5L, 9L, 9L, 7L, 5L, 3L)� )

})

Unit test (testthat format)

> is_odd <- function(x) x %% 2L != 0L

> m <- 1�> filter(floor(runif(10, 1, 10)) + m, is_odd)�[1] 5 9 9 7 5 3

Client code (eg., example, vignette)

12 of 23

12

How well can automated trace-based unit test extraction actually work in practice for R?

Research question

13 of 23

R Language

No type annotations or static types
Redefinable symbols inc. if, +, (, …
Full reflection (introspection, intercession)
By-need evaluation
Mostly vectors or lists values
Multiple object systems
Copy-on-write semantics
Functions as the main abstraction

13

https://en.wikipedia.org/wiki/R_(programming_language)

14 of 23

CRAN Experiment - Setup

14

1,726 total packages

TOP 100 most downloaded CRAN packages + dependencies (from cranslogs)
1000 random CRAN packages + dependencies�

1,544 ran packages

179 failed (timeout or runtime error)�

1.7M lines of R code

157K examples (avg 102 per package)
32K vignettes (avg 21 per package)
163K tests (avg 105 per package)

15 of 23

CRAN Experiment - Scale

15

	Overall	Average per package
Total number of calls	5,277,897	3,411 (s=13,366, m=141)
Traced unique calls	1,617,842	1,045 (s=3,142, m=82)
Generated tests	1,515,246	979 (s=3,065, m=68)
Passing tests	1,306,045	844 (s=2,821, m=51)
Non-redundant tests	26,967	17 (s=33, m=9)
Ratio of reproduced tests	0.8	0.75 (s=0.31, m=0.9)

Scale of the experiment

16 of 23

CRAN Experiment - Errors

1.6M unique traces
1.3M passing tests
300K errors

16

	Overall	%
Tracing errors� - skipped traces (size > 512kB)	70,020�60,360	4.32%
Test generation errors� - environments with cycles	32,576�24,200	2.01%
Replaying test errors� - incorrect test (value mismatch)� - invalid test (execute with error)	209,201 77,850 131,351	12.93%

Errors breakdown

17 of 23

CRAN Experiment - Code coverage

Tests avg 19%

17

18 of 23

CRAN Experiment - Code coverage

Tests avg 19%�
Tests with genthat avg 53%�

18

19 of 23

Reverse dependencies experiment

19

65 packages

random with at least 20 dependencies�

42/65 increased code coverage�
Increase avg 52% - 60%

20 of 23

Limitations, shortcomings and future work

External pointers are not supported
Stateless recording
Non-standard evaluation does not always work
Other object systems than S3 are only supported in binary format (.ext file)�
Large tests - (avg 18kB, median 490B, max 1.5MB)
Non-determinism other than random values
Over specific testing oracle
Brittleness (false-positive, false-negative)
Tracing time (avg 21x slower, median 5x), 75% of tracing finished < 4 min

20

21 of 23

Limitations, shortcomings and future work

External pointers are not supported → record the expression creating ext pointers / R altrep
Stateless recording → increase the amount of recorded state
Non-standard evaluation does not always work
Other object systems than S3 are only supported in binary format (.ext file)�
Large tests - (avg 18kB, median 490B, max 1.5MB) → test minimization
Non-determinism other than random values
Over specific testing oracle → randomize input
Brittleness (false-positive, false-negative)
Tracing time (avg 21x slower, median 5x), 75% of tracing finished < 4 min

21

22 of 23

22

Conclusion

How well can automated trace-based unit test extraction actually work in practice for R?

Works surprisingly well

80% of traced calls can be reproduced in passing tests
34% increase of code coverage
despite severe limitations (no global state, no external pointers)�

It seems that it works because of functional nature of R

mutation is rarely used�

Helps to bootstrap unit tests
Regression testing

23 of 23

Tests from Traces: Automated Unit Test Extraction for

https://github.com/fikovnik/ISSTA18-artifact

Filip Křikava, Jan Vitek

34% increase of code coverage

in 1,500+ CRAN packages

genthat package

https://github.com/PRL-PRG/genthat