1 of 14

VMAF contribution guide

Zhi Li, Kyle Swanson

Netflix

2 of 14

Agenda

  • VMAF license
  • Overview of VMAF code structure
  • Python library core classes
  • Algorithmic contribution guideline
    • Creating a new feature extractor
    • Create a new train/test model class
    • Calling run_vmaf_training script to train a new model

​

3 of 14

VMAF license BSD+Patent

4 of 14

VMAF repo

code structure

libvmaf (C)

​

feature

feature

extractor

feature

collector

VIF

motion

adm

ssim*

ms_ssim*

PSNR*

third-party

pdjson (unlicense)

libsvm (libsvm license)

IQA*

(BSD)

y4m_input (daala license)

model prediction

Python

​

Matlab

​

spEED^

STMAD^

STRRED^

MatlabPyrTools (MIT)

feature

extractor

model training / prediction

other

housekeeping tools

*Not part of the VMAF algorithm

^Not part of the VMAF algorithm;

research code, doesn’t have a

explicit license

...

...

Brisque^

no-ref

fex

Niqe^

SI-TI*

iCID (BSD)

5 of 14

Python library

core classes

  • FeatureExtractor
    • “Aggregate” feature
    • “Atom” feature
  • FeatureAssembler
  • TrainTestModel
    • Train
    • Predict
  • QualityRunner

​

inheritance

association

6 of 14

Ways to contribute

  • Submitting a bugfix
  • Improving documentation
  • Making the code run on a new platform
  • Robustifying the build system
  • Improving the CI loop
  • Improving code coverage by adding tests
  • Optimizing the speed
  • Implementing a well-known quality metric
  • Implementing a custom VMAF model for a specific use case

7 of 14

Algorithmic contribution

  • Case I: implement a well-known quality metric in the literature

​

​

  • Case II: implement a custom VMAF model
    • Create new elementary features
    • Create a new machine learning regression model
    • Train on a specific subjective dataset

Create a new

FeatureExtractor

Create a thin QualityRunner wrapper

Create a new

FeatureExtractor*

Create a new

TrainTestModel*

Call run_vmaf_training script

Create a new

FeatureExtractor*

Create a new

FeatureExtractor*

*optional

8 of 14

Create a new FeatureExtractor

  • Way #1: New FeatureExtractor in native Python code
    • Example: PypsnrFeatureExtractor (“Py-PSNR”)

​

  • Way #2: New FeatureExtractor via Python calling libvmaf in C
    • Example: PsnrhvsFeatureExtractor (PSNR-HVS)

​

  • Way #3: New FeatureExtractor via Python calling Matlab
    • Example: StrredFeatureExtractor (ST-RRED)

9 of 14

Create a new TrainTestModel

  • Default model for VMAF training:
    • LibsvmNusvrTrainTestModel (libsvm NU-SVR model)

​

  • Another example by Cosmin
    • Logistic5PLRegressionTrainTestModel (logistic regression with 5 parameters)

10 of 14

Call run_vmaf_training script

example_dataset.py:

11 of 14

Call run_vmaf_training script (Cont’d)

vmaf_feature_v6.py:

“Aggregate” feature (TYPE of a FeatureExtractor)

“Atom” feature (in ATOM_FEATURES or DERIVED_ATOM_FEATURES of a FeatureExtractor)

12 of 14

Call run_vmaf_training script (Cont’d)

libsvmnusvr_v2.py:

13 of 14

Validate the new model on a different dataset

14 of 14

Reference