Beating legacy code into shape with the help of Selenium

By Sasha Pachev

Goals

• Provide insights on the nuts and bolts of UI testing of a legacy web application.
• Share some Selenium tricks
• Convert those who are not yet converted to the true religion of automated UI testing and strengthen the faith of the converts for the battles ahead

How it all started

Challenge: To be able to make significant code changes in a legacy mission-critical CRM web app written in PHP with MySQL backend, and then have some degree of assurance that it still works as expected.

Mode of testing at the start of the project: Manual. Time consuming. Heavily reliant on the application knowledge of the testers.

Arguments For Automated Testing

• A competent developer not familiar with the code base can make a change and reliably validate it independent of other engineers or QA
• Be able to detect introduced bugs in minutes instead of months
• Test corner cases that manual test cannot predictably hit
• Avoid human error inherent in manual testing
• Speed up development cycle by performing the tedious setup steps automatically when debugging

Common Objections To Automated Testing

• It takes too long to automate the tests
• The automated test suite takes too long to run
• The tests have high maintenance overhead
• The tests produce false positives
• The tests miss critical bugs

Addressing The Common Objections

• Invest the time and effort to become proficient with the automation tools and techniques or hire somebody who already is
• Develop a good test driver abstracting common testing operations
• Have a one basic test suite for the critical functionality, and others for less critical. Develop a reasonable policy when to run which one.
• Think of a test as a performance critical piece and develop both the test and the application accordingly
• Properly abstract the data. E.g. read the price of the product out the database instead of hard-coding it in a test.

Addressing The Common Objections - cont.

• Understand what your product is supposed to do before writing the test. Make sure the test is logically consistent with the expectations.
• Any time you discover a critical bug write a test before you fix it. Use the test as you work on the fix. Include the test in the test suite.
• Overall, follow solid software engineering principles. Automated testing harmonizes with good software engineering.

Our Solution to Automated Testing

Selenium!

Pipe dream

• Record a test case with Selenium IDE browser plugin
• Add it to the test suite

This does not work, because:

• Selenium IDE has no clue about timings
• Input and output need to be dynamic
• Validation involves checking the database

Actual solution

• Custom testing framework written in Python
• Abstract all UI operations so the test never has issue direct calls to the Selenium driver
• Abstract as much application specific logic as possible. E.g. one API call for log in, one API call to fill out a signup form, one API call to bring up a customer record, etc.
• Abstract database validation

Challenges

• Headless testing
• Optional remote visual display
• Properly handling timing dependencies
• Working around Selenium/Firefox bugs/limitations
• Interacting with AngularJS
• Ability to do interactive test runs
• Extracting configuration from the PHP application code into the Python test driver

Solutions - Headless testing

• Use Linux
• apt-get install xvfb
• export SEL_DISPLAY=:99
• Xvfb $SEL_DISPLAY -ac # disable access control
• os.environ["DISPLAY"] = os.environ.get("SEL_DISPLAY",":99")
• self.driver = webdriver.Firefox(capabilities=d)

Firefox started by Selenium will now display its output on the dummy X display :99

Solutions - Optional remote visual display

Naive: install X on the remote machine, set DISPLAY inside the test driver to point to that X server

Problem with naive: every X event gets forwarded over the network, runs slow

Better: Run Selenium on the remote, then instantiate the driver with:

self.driver = webdriver.Remote(command_executor=self.remote,desired_capabilities=d)

self.remote would be something like http://mydesktophost:4444/wd/hub

​

Solutions - Properly Handling Timing Dependencies

• Properly identify and test for pre-conditions for proceeding. E.g. make sure the page finished loading before looking for expected output in it, wait for checkbox to become visible before you try to click, etc.
• If the pre-condition is not clear, but with 99.999% certainty it is fulfilled in T < 5 seconds, we use our API hack_sleep(T) call that swears and then sleeps as the last resort.

Solutions: Working around Selenium/Firefox bugs/limitations

• Make sure the Firefox version is compatible with Selenium
• For flaky UI actions, verify that the input actually registered, and retry in a loop until timeout if necessary.
• driver.execute_script() can work miracles when all else fails
• Sometimes creativity is required - e.g. one of our tests clicks on a checkbox, waits a second, checks if it is still checked, if not, keeps repeating the clicks until timeout is reached, in which case it declares it possessed.

​

Solutions: Interacting with Angular JS

• def call_angular(self,el_id,func_with_args):� self.exec_js('var scope = angular.element(document.getElementById("' + el_id + '")).scope();' +� ' scope.' + func_with_args + '; scope.$apply();')�
• def set_angular_var(self,el_id,name,val):� self.exec_js('var scope = angular.element(document.getElementById("' + el_id + '")).scope();' +� ' scope.' + name + ' = "' + str(val) + '"' + '; scope.$apply();')�

Solutions: Interactive Test Runs

• Instrument the code with self made breakpoints
• Can be implemented as a call to your own API that will wait for you to hit the key on the console if enabled
• Then run the test in the visual mode with appropriate breakpoints enabled via arguments to the test script
• This allows pleasant debugging of cases that require a good amount of tedious input on a few pages followed by the actual debugging where you need to enter input manually

Solutions: Extracting PHP configuration files data into Python

• Black magic:

cmd = "(cat " + git_root + "/includes/global/constants.php; " + \� "echo '$a=get_defined_constants(); echo join(\"\\n\",array_map(function ($k,$v) { $v=trim($v);" + "return \"$k=\\\"$v\\\"\";}," + \� " array_keys($a),array_values($a)));') | php”�exec subprocess.check_output(cmd, shell=True)�

Conclusion

Have fun testing!