Mocking library selection for the OWTF Unit Testing Framework

By Alessandro Fanio González


Mocking libraries comparison

First of all, I have found this comparison between mocking libraries in python, from the official wiki. The PyDoubles framework (which I mentioned in the project proposal) is not shown in the listing. However, it seems that the official PyDoubles page [1] is offline right now, so I have contacted with one of the developers in order to know if the project is still maintained. Because of this, PyDoubles is not a choice to consider any more.

From this comparison, I have selected some of the proposed libraries taking into account the capabilities that are offered in their descriptions and the opinion of other developers on the Internet.

Mocking libraries to analyze

Comparison

Ludibrio [2] is a test doubles framework that has support for mock and stub objects. In addition it allows mocking or stubbing any external module, so that when you import it, you run a test double and not the real implementation. It also supports dummy classes and has a proxy feature. I can’t find so much documentation regarding some of the features, such as the module mocking or the proxy/spy functionality.

Mock [3] is a library for testing, that has joined to the Python standard library in Python 3.3, as unittest.mock. The main functionalities are the creation of mock and stub objects, and the ability to patch module and class level attributes within the scope of a test. In the documentation page we can review a comparison of this framework with other testing libraries.

Mocker [4] is another platform for test doubles in Python, and is inspired in other mocking libraries such as pmock, PyMock and EasyMock. It has support for mocks, stubs, fakes, dummies and proxies.The page says that the framework integrates with unittest.TestCase and adds some new assertion types. One good thing of this library is that the developers have used TDD and the tool has a 100% code coverage.

Mox [5] is a mock object framework based on EasyMock, a Java mock object framework. It allows the creation of mocks in replay mode. It provides also some comparators with a syntax similar to Hamcrest [6].

Mockito for Python [7] is a Test Spy Framework based on the Java Mockito library. It allows the creation of mocks, stubs, and provides a lot of flexibility with the assertions via argument matchers, also similar to the Hamcrest matchers.

        

Flexmock [8] is a testing library for python inspired in a Ruby library of the same name, and its main goal is to make the creation of fake object as unobtrusive as possible. The authors say that the API has been designed to be highly expressive, so as to be read more like English sentences than API calls. It has support for various Python implementation and it integrates with all major test runners, including unittest and nose (which are used in this Unit Testing Framework.) Finally, it allows the creation of mocks, stubs, spies and fake objects.

Features comparison table

Ludibrio

Mock

Mocker

Mox

Mockito for Python

Flexmock

Mocks

YES

YES

YES

YES

YES

YES

Stubs

YES

YES

YES

YES

YES

YES

Spy/proxies

YES

YES

YES

NO

YES

YES

Dummy/fake objects

YES

NO

YES

NO

NO

YES

Monkey patching support

NO

YES

YES

NO

NO

YES*

Date of last official release

26/11/2011

05/11/2012

18/09/2010

04/05/2010

04/08/2010

30/03/2013

* This feature is called partial mocking or method replacing.

Other aspects to take into account

Conclusions

As we can see in the comparison table, some of the projects have not been updated for a long time, so, in my opinion, we can discard Mocker, Mox and Mockito for Python. Despite of Mocker seems to be the more complete library, the documentation says that only the versions 2.4, 2.5 and 2.6 of Python are supported.

As exposed before, I have not been able to find some more documentation about Ludibrio, so some of its functionalities are not clearly explained. In contrast, both Mock and Flexmock have extensive documentation and examples, which is better and faster for developers to learn how the library works.

The main difference between Mock and Flexmock resides in the creation of fake/dummy objects. On the other hand, the Flexmock documentation says that it is compatible with pypy, jython and python from version 2.4 to 3.3, and it integrates, with unittest and nose, the technologies selected for the OWTF Unit Testing Framework implementation. Finally, another aspect is the verbosity level of the libraries. Using the previous comparisons, it seems that Flexmock is more expressive than any other framework, and less code is necessary for implementing test doubles.

To sum up, Flexmock has been selected as the mocking library for the framework, because of its capabilities, expressiveness, the support of various Python versions, the integration with other tools and because the last release is dated in 2013.

References

[1] Official PyDoubles website: http://www.pydoubles.org/ 

[2] Ludibrio project page: https://github.com/nsigustavo/ludibrio/ 

[3] Mock documentation page: http://www.voidspace.org.uk/python/mock/ 

[4] Mocker official page: http://labix.org/mocker 

[5] Mox project website: http://code.google.com/p/pymox/ 

[6] PyHamcrest project repository: https://github.com/hamcrest/PyHamcrest 

[7] Mockito for python webpage: http://code.google.com/p/mockito-python/ 

[8] Flexmock documentation page: http://has207.github.io/flexmock/ 

[9] Comparison of mocking libraries (I): http://www.voidspace.org.uk/python/mock/compare.html 

[10] Comparison of mocking libraries (II): http://has207.github.io/flexmock/compare.html