Double Negative

Software, code and things.

Testing with Swift and XCode 6

Over the past month I have been developing an iOS app for one of the websites that I own and operate. I wanted to share a few considerations that I encountered when investigating testing and iOS.

Objective C has been around and developing as a language for an extremely long time. Given that, there are a lot of tools out there that have been written to aid developers in testing their apps. For example: the Specta testing framework, Expecta an assertion library, and OCMock for mocking.

With the release of Swift, things have changed somewhat. Even though Swift and Objective C integrate seemlessly, I would much prefer to write all of my code in Swift. In my opinion Swift is a much cleaner language - it is easier to read and write, and feels similar to the languages I regularly use to write web apps. PHP for example.

When it comes to legacy testing tools, Swift has some problems. For example, this page outlines exactly why OCMock won't work well with Swift. That said you can still achieve everything you could possibly want/need with Swift. You might just have to approach it in a different way.

XCode 6 also throws some problems of its own into the mix. To test your codebase it needs to be accessible by your testing target. I found that the easiest way to achieve this is to toggle "Allow testing host application APIs" under the "General" section of your testing targets settings. This alone however will not work because of Swifts new access control considerations.

For a class to be accessible within the testing target, it needs to be defined as being public. This makes me cry a little inside - I really hope Apple fix this very soon. None the less.. for now.. once this is done you can import your app into your tests by simply using import AppName

Even then I encountered some intriguing issues - I could not for example instantiate a class without getting errors (Use of unresolved identifier). It seems to be the case that your testing target won't import your app target correctly if there are any issues with your app which will prevent it from building. This is somewhat annoying but it is essentially another type of test :)

My Setup

As you may well know, I am a big fan of functional testing. That is why I wrote Phantom Mochachino.

The guys over at Square developed a great functional testing tool in KIF - I highly reccomend it.

When I am testing, i like to know that things work. KIF allows me to test that my app flows how I expect it to. Some people advocate mocking out HTTP requests when using KIF, but I see no reason too. I use KIF to test that my app works end to end. If I mock out my HTTP requests I simply verify that my app works if my HTTP requests work. That in my opinion is pointless.

KIF integrates seemlessly with Swift. As outlined by Brad Heintz. All you need to do to utilize KIF with Swift syntax is create a class containing the following code

class SwiftKIFTestCase: KIFTestCase {  
    func tester(_ file : String = __FILE__, _ line : Int = __LINE__) -> KIFUITestActor {
        return KIFUITestActor(inFile: file, atLine: line, delegate: self)
    }
}

You then simply extend this class when writing your test classes and call the KIF test methods on the return value of a call to tester().

One thing that I felt was lacking from KIF was the ability to reset your app. I implemented a code snippet to allow this in a similar way to which I implemented testing alternate paths in Phantom Mochachino.

To achieve this, I added the following code to my App Delegate:

    public func resApp() -> () {

        _initialStoryboard = window!.rootViewController!.storyboard;

        for view in self.window!.subviews {
            view.removeFromSuperview()
        }

        let initialScene:UIViewController = _initialStoryboard!.instantiateInitialViewController() as UINavigationController
        self.window!.rootViewController = initialScene;
    }

Calling this method will reset your app to the initial view controller.

I then extended KIFTestCase and added a class method to make calling this method from the testing target easy.

extension KIFTestCase {  
    class func reset() -> () {
        var delegate = UIApplication.sharedApplication().delegate as AppDelegate
        delegate.resApp()
    }
}

To reset your app within a test class, you just need to call KIFTestCase.reset(). Simple.

Unit Testing

In addition to using KIF I have written some very simple unit tests with XCTest. My app is relatively simple and as such XCTest provides all the power I need to unit test my apps.

There is an extremely insightful article over at objc.io that discusses using XCTest for unit testing.

The only other consideration that I have encountered in my iOS testing journey (thus far) is mocking. Mattt has written an interesting piece on testing which outlines why mocking is a non-problem with Swift.

Conclusion

I get the impression that testing iOS apps has not always been particularly easy. Then as iOS developed things got better. Now we have Swift one may have been concerned that things could take a step backwards.

In my opinion things have not. Yes, it may take a while for there to be as much 'documentation' on testing with Swift. Likewise it may take a while until we see reliable, well used, and well maintained Swift based testing libraries. Still, I came to testing with Swift with little to no experience and i found it to be relatively painless.

Finally.. if you were worried, I do unit test my HTTP Requests.

Functional testing with Phantom Mochachino

I previously wrote about the internals of Mocha. The reason that I was intrigued by the inner workings of Mocha was because I was in the process of building a functional testing tool - Phantom Mochachino.

Phantom Mochachino is an extension of Mocha which works with the PhantomJS headless web browser to offer an end to end functional testing utility. A detailled explanation of how you can get started using Phantom Mochachino is found at the link above.

I am a big fan of PHP, and the site that I built Phantom Mocachino for is written in PHP. As such I wrote a PHP script to run my functional tests with.

I am sharing my implementation to demonstrate how you can use Phantom Mochachino to make functional testing fun.

<?php

$testRunner = "FunctionalTestRunner.js";

$testFiles = array(
    'RegisterTest.js' => array(
        'paths' => array(
            '/account/register'
        ),
        'useCookies' => true,
        'argumentsVar' => 'loginArguments'
    ),
    'LoginTest.js' => array(
        'paths' => array(
            '/account/login'
        ),
        'useCookies' => true,
        'argumentsVar' => 'loginArguments'
    ),
    'DoActionsTest.js' => array(
        'paths' => array(
            'do/actions'
        ),
        'useCookies' => true
    ),
);

while (true) {

    $loginArguments = array(
        'username' => substr(md5(rand()), 0, 7), //random username
        'password' => 'password'
    );

    foreach ($testFiles as $testFile => $dataArray) {

        echo "\n";
        echo ">> PHP TEST RUNNER - NEW FILE \n";
        echo ">> RUNNING " . $testFile . "\n" ;
        echo "\n";

        $testPaths = $dataArray['paths'];

        foreach ($testPaths as $path) {

            echo "\n";
            echo ">> Against " . $path . "\n";
            echo "\n";

            $commandParts = array();
            $commandParts[] = "phantomjs";

            if ($dataArray["useCookies"]) {
                $commandParts[] = "--cookies-file=cookies.txt";
            }

            $commandParts[] = $testRunner;

            $commandParts[] = $testFile;
            $commandParts[] = $path;

            if (isset($dataArray['argumentsVar'])) {
                $arguments = ${$dataArray['argumentsVar']};

                if (count($arguments) > 0) {
                    foreach ($arguments as $argument) {
                        $commandParts[] = $argument;
                    }
                }
            }

            $command = join(" ", $commandParts);

            passthru($command, $response);

        }
    }

    echo "\n";
    echo ">> Sleeping for 10 seconds";
    echo "\n";

    sleep(10);
}

The internals of the Mocha Javascript testing framework

In an attempt to extend the functionality of Mocha for a specific use case I was required to look into its source code.

I wanted to write the flow down so it was clearer to myself. Maybe someone else will find this useful.

Mocha

Firstly you have the Mocha 'object' which in many respects is a container for all the other constituent parts (outlined below).

An instance of Mocha is exposed as mocha. The mocha instance has some helper methods for setting up Mocha with the correct options. For example the interface with which you will write your tests (BDD/TDD/qUnit).

One method on the Mocha prototype is run. Calling this instantiates a Reporter and a Runner (amongst other things).

Runner

A Runner object runs yours tests. Mocha by default creates a base level suite which the runner is instantiated with.

Each suite requires a Context. This context references a Runnable, and provides various prototype methods to set context specific settings. Each Test is a Runnable, as is each Hook.

When you call mocha.run, it creates this runner, sets various things up and calls its run prototype method.

Reporter

A reporter programatically specifies how the test output will be shown. A suprising amount of Mocha's codebase is different types of reporter. You can output your results in 'Nyan Cat' format if you so wish :)

On creation the Mocha object loads a reporter based on the passed options. When you run mocha the Reporter instance is instantiated, passing the Runner in.

Hooks

Hooks are hooks. They are pieces of code that can be hooked in at different stages of the process. Mocha has hooks such as before(), after(), beforeEach(), and afterEach().

Behind the scenes, they are set up in exactly the same way as Tests.

Running your tests

Mocha relies heavily on nodes EventEmitter to pass messages around. For example it is used to tell the reporter that the test suite has started running. When we run our Runner it emits a message 'start'. I wont explicitly mention these messages after this point - essentially at each stage of the execution an appropriate message is sent/received/acted upon.

When we call mocha.run a UI is setup. The default is the BDD (behaviour driven development) UI. This essentially defines a number of methods.. namely describe(), it(), and the various hook methods. These are what you use to write your tests.

describe

When you call describe a new suite is created. What Mocha does here (if I have understood correctly) to allow for nested suites is very very clever.

Javascript executes syncronously, line after line. Within describe, a suite is created using the suite at the front of the locally available suites array as its parent. Once a suite has been created it is added to the start of this array.

It then executes the passed in function body of the describe call. If it has nested describe calls, these now use the new suites[0] as their parent when the respective suite is created.

After the function has completed executing, the suite is removed from the suites array.

Another interesting tidbit is that suites inherit their parents context (ctx).

  var context = function() {};
  context.prototype = parentContext;
  this.ctx = new context();

This allows them to inherit timeout settings amongst other things.

it

When you call it, a new Test is created and this is added to the suite. it() is executed in the context of its suite by using call() from within describe().

The Runner has a runSuite method which is called with Mocha's initial base suite. This method runs the appropriate 'suite level' Hooks, and then runs any tests (runTests). runTests calls the beforeEach hooks, which on completion runs the test, which on completion runs the afterEach hooks, which on completion runs the next test. Once all the tests are completed the callback from the level above is executed which will recursively execute runSuite on any nested suites.

The runTests method does similar in the sense that it runs the respective hooks at the right times (beforeEach, and afterEach). afterEach is passed a callback which on completing the hook is executed. This callback runs the next test allowing for test recursion.

The running of a test involves calling the run method on the test object.

The test object inherits this method from Runnable. Runnable has the method run which sets the test as the runnable within the Context object (initially set at the very lowest level in the instantiation of Mocha).

Now.. one of the great features of Mocha is that you can have tests running asynchronous code. The way this works is simple. I initially suspected that this would be done with timers and regular polling. It is in fact a lot cleaner and simpler than that.

The execution of sequential tests as mentioned above is controlled by its previous test in that within the call Test.run(fn), fn is a callback which executes the following test. As such the Runnable run method simply does not execute this callback until it is complete (you have triggered done()) or until the test times out.

Reporting

So we have written our tests and can run them. At each step of the way we emit messages using nodes EventEmitter.

A reporter, in extremely simple terms, listens to these messages and produces an output.

In addition to that, the reporters format the output so that it is useful. Mocha comes with jsDiff so you can see visual diffs where appropriate. They also control important things like the colours of your output, and printing out cat faces..

Further reading

Mocha is quite complex under the hood, and makes use of a number of complex yet clever design patterns. This is a general more in depth overview of its internals. The next step is to read the source code - explaining all the fine details in written text would be more challenging than the actual concepts.

If anything is not clear please do let me know.

PHPUnit, DBUnit and their quirks

I utilize PHPUnit for my backend testing and have noticed a number of things whilst using it. I have outlined these below - hopefully they will help someone.

DBUnit, composite datasets and foreign keys

There is a bug in the PHPUnit source code which means that composite datasets are truncated in the same order that they are created. If there are foreign key constraints between these tables, you will encounter a number of errors.

I have fixed this issue, and created a pull request here.

Properly tearing down database tests

Make sure you correctly tear down your database connections otherwise you may encounter various errors. I overload tearDown() to do this. Make sure that you call the parent method such that any operation you define within in getTearDownOperation() is called appropriately.

    public function tearDown() {
        parent::tearDown();
        $this->dbh = null;
    }

Open Files

If you have a lot of tests you may encounter the "Too many open files" error.
You can fix this by changing the number of files a process can open using ulimit -n 5000

Test annotations

An annotation block fora PHPUnit test is as follows:

    /**
     * @test
     */

It is not simply a comment block - the first line must have an extra asterisk. This is the same as for docblocks in phpDocumentor.

Test Size Annotations

PHPUnit provides the annotations @medium and @large which indicate the 'size' of a test. Unannotated tests are considered small.

You can configure the test runner to timeout if these tests take longer than expected.

<phpunit  
    beStrictAboutTestSize="true"
    timeoutForSmallTests="1"
    timeoutForMediumTests="50"
    timeoutForLargeTests="50">

The curveball in the mix is that as of version 4.2, all database tests are hard coded as large tests within the source code. I personally think that this should be changed such that database tests default to being 'large' but can still have their size overridden.

For now you'll either have to edit the source code yourself or run database testing independently of any other test setup where 'large' tests have a different nature.


If anyone else is aware of any quirks that it would be worth bringing to the attention of others, please let me know.