Our products

We are passionate about building niche multi-platform products that provide real utility to our customers.

Anfield
Keep up to date with Liverpool FC news from across the web.
Domain Bites
Domain name industry news and resources.
Pub Reviews
Find the best pubs near you. Share your thoughts and photos.
Mmmm
Capture your taste and share it with the world.

Blog

We like to blog about software. Our stack, engineering problems, open source developments.. you will find it all discussed here.

Writing a good API, and being a good client

When rebuilding the Mmmm.com search functionality I managed to quite effortlessly completely break the mobile applications client integrations with the API.

The main issue was that I was entering some unchartered territory, and trying to do some pretty complex things. I was in a programming 'zone' and all I wanted to do was write the new functionality. I managed to do this without issue. The problem was that having written the new endpoints I was aware that I had almost certainly changed something that would cause 'less than positive user experiences' for users of older app versions.

At least I was aware.. right? ;)

On the API end I had made a two changes which I suspected would cause issues.

  • I had changed the name of the GET parameter for our search endpoint from name to searchTerm (for clarity).
  • I had implemented some complex logic such that location search results were returned from the Google Places API (they had not been previously).

The former (in principle) has a simple fix - the API endpoint should look for a GET property named searchTerm, and if it cannot find one it should search for a property named name.

Whilst this is in fact the resolution I went for, careful consideration should be given. I absolutely do not want my API codebase to become unmaintainable and complex as a result of lots of 'fixes' over time. On careful consideration I decided that this was a one off, and that in future I would pay more attention to choosing semantically accurate property names from the get go.

This did however make me think about alternatives, one of which was an appropriate API versioning methodology.

The problem with clients

The problem with API clients is that as an API provider, you cannot be certain as to if or when your clients will update their implementations. If one could guarantee instant updates to clients when breaking changes were pushed then there would be no issue.

Whilst the Mmmm.com API is not currently used by external clients, it is used by our own Android and iOS apps.

The iOS review process means that it is essentially impossible to have an anywhere near instant update to the client integration. Even if it were, this does not actually guarantee that users of the application would actually update it.

As such the API needs to support all historical client integrations.

Initial testing

Out of intrigue I loaded up a previous version of the iOS application and had a play. I executed a search from the main screen of the application, and I was unfortunately met with a crash. Fortunately this was the only crash that I encountered during my hands-on integration tests and as such my fear of breaking changes was limited to a singular breaking change.

Having investigated the Crashlytics report it became apparent that I had been a fool. It was not an API breaking change that was the issue, but rather the client wasn't prepared for potential changes to the API - the client was badly written.

The cause of the crash was a simple one. The search callback was trying to handle a result type of google-location - a type of which it was not aware (it was not defined in the enumeration).

After resolving the issues (see below) I went ahead and refactored the codebase such that it gracefully handles any unexpected situations like this that result from further development of the API. I.E I wrote the client how I should have initially.

API versioning

Given the nature of the issue, API versioning was now a requirement. My client would not work 100% with the new version of the API and given that an instant client update was not possible I needed to implement versioning such that 'old' versions of the apps would not receive results with type google-location. It was not however simply the case that I didn't want to return results of type google-location, but rather that I wanted to return the same API response as when the client version had been designed.

As I was implementing API versioning retrospectively (less than ideal) it was necessary for any API request without a specified version to be considered to be using API version 0 (the legacy API). Any updated app versions would need to explicitly specify the API version that they wanted to use.

This differs from 'normal' implementations where a lack of specificity is indicative of wanting to use the most recent version. It is my intention that in time (once it is apparent that no-one is using the 'legacy' versions) we will transition to the 'normal' approach.

Implementation

Here, there are a number of fantastic (and in depth) answers to best practices when versioning an API.

Having read this, I decided to opt for passing version and response format information through an "Accept" header sent with each request.

The API (which is built on top of the PhalconPHP framwork) reads the header information, extracts the important information (using a regex - see below) and makes it available to API methods by proxy of a static helper class.

  • #application/vnd.mmmm(?:\.v(?<version>[0-9]*))?\+(?<format>json|xml)#

So now the API knows which response version is expected from a particular endpoint, but we still need to implement the logic which guarantees that the correct response is returned.

I implemented this in an interesting way utilising PHPs magic methods (specifically __call).

  • The basic logic is that a versioned API method is renamed. You can use any format you want, but I chose to simply prepend an underscore. As such the method signature search($inputModel) became _search($inputModel).

  • I then implemented the magic method __call as follows:

    public function __call($method, $arguments) {

        $realMethodName = "_" . $method;

        if (method_exists($this, $realMethodName)) {

            $response = $this->callOldVersion($arguments);

            if ($response) {

                return $response;
            }

            return call_user_func_array(array($this, $realMethodName), $arguments); 
        }
    }
  • The endpoint attempts to execute the search method. As this method does not exist, __call is triggered (this is important - __call is only called for unknown methods). The __call function then searches for a method called _search, and if it exists (it does) it executes a method called callOldVersion.

  • The logic within callOldVersion checks as to whether the requested version of the API differs to the current version of the API. If it does it attempts to call the legacy implementation of the method.

    public function callOldVersion($argumentsList) {

        //If they have requested a NON current API version
        if (APIHelpers::$API_VERSION != APIHelpers::CURRENT_API_VERSION) {

            //get the name of the method that they have called
            $actionName = $this->dispatcher->getActionName();

            //build the name of the legacy action method
            $legacyActionName = $actionName . APIHelpers::$API_VERSION;

            //if there is a legacy version..
            if (method_exists($this->oldApiService, $legacyActionName)) {

                //iterate through previous versions, and convert the arguments
                for ($toVersion = APIHelpers::$API_VERSION;  $toVersion < APIHelpers::CURRENT_API_VERSION; $toVersion++) {

                    $fromVersion = $toVersion + 1;
                    //echo "Convert from " . $fromVersion . " to " . $toVersion;

                    //build the convertor method name
                    $convertorName = $actionName . $fromVersion . "to" . $toVersion;

                    //if a convertor with that name exists
                    if (method_exists("ArgumentConvertors", $convertorName)) {

                        //convert the arguments
                        $argumentsList = call_user_func_array(array("ArgumentConvertors", $convertorName), $argumentsList);
                    }

                    //otherwise the legacy version must take the same parameters
                }

                $response = call_user_func_array(array($this->oldApiService, $legacyActionName), $argumentsList);

                return $response;
            }

            //indicate that there was no legacy handler so we should just call the normal method
            return false;
        }
    }

Notes

  • The ArgumentConvertors helper class allows for any arguments to the current endpoint method to be converted to the required arguments for the legacy endpoint method

  • If there is no legacy method implementation the fallback is to simply call the current version.

Bloated codebase

My main concern with such an implementation (and perhaps yours?) was that if (over time) lots of endpoint implementations are being changed then oldApiService might become bloated with old implementations of legacy endpoints.

Sadly this is exactly the case.. but logically to return different responses for different version of the API.. you need different code.

In reality however this is not a significant issue. The Mmmm.com codebase is fully covered with unit tests. As such, when a current endpoint method becomes a legacy endpoint method we simply transfer the unit tests into oldApiServiceTests, and write new ones for the new implementation.
Complete test coverage guarantees that legacy and current API method calls return the expected responses.. indefinitely.

Furthermore (as alluded to previously), by keeping track of what versions of your API are being utilised by clients, and by implementing staged phase-outs (deprecation) of historical API versions you can keep your oldApiService pruned.

Conclusion

This post outlines a number of interesting things.

  • You should be vigilant when building API consuming clients. Make sure you make them safe.
  • APIs need to support returning appropriate responses to clients for as long as they are requested.
  • Retrospective API versioning can be done, but it is advisable to 'do it right' prior to an initial release.

As a result of this 'learning experience' the codebase for the Mmmm.com applications have been improved significantly. The apps are also now more 'future proof' - they are resistant to issues associated with external change.

If you would like further clarification on any of this, do not hesitate to ask - I would be happy to help.

PHP Regular Expressions 101

Regular expressions are one of those things that I never really took the time to learn. Rather, whenever a situation arose I would discern and 'learn' the specific thing that I needed to resolve my problem.

Having built Whois Browser as well as many other internal tools which heavily depend on regular expressions this is somewhat of a bemusing approach.

If you have a similar approach, this post seeks to serve as a basic overview of what you need to know to utilise regular expressions within PHP (at the most basic level).

For advanced level regular expressions check out http://www.regular-expressions.info/ - it is an awesome resource. It is well worth setting aside an afternoon and reading it all.

preg functions

As outlined in the PHP manual there are a number of preg functions contained within the PHP language. These functions all 'perform (a search for) a regular expression' and then do something.

You can for example use:

  • preg_replace to find any numerical values prepended with a £ sign and replace them with something else.

  • preg_match_all to find all phone numbers in a given text block

These methods serve as your main interface for utilising regular expressions within PHP. They are incredibly powerful.

Perl Compatible Regular Expressions (PCRE)

PHP utilises Perl Compatible Regular Expressions (PCRE). This is a format for writing regular expressions - the things that you are searching for.

The PHP manual has its own page on pattern syntax.

The basics

A basic code snippet outlining usage of the preg_match function would be as follows:

$regex = "#cat#";

preg_match($regex, $contentToSearch, $outputArray);  

This would search your $contentToSearch for the word cat and populate $outputArray with any instances of cat that it found (this is a completely pointless example).

The # symbol delimits the start and the end of the expression. You can utilise any "non-alphanumeric, non-backslash, non-whitespace character" as a delimiter but it is advisable to use something that is obvious, and will not be utilised within the expression itself (otherwise you'll have to escape characters).

Modifiers

After the closing delimiter you can add any number of modifiers. These modify the expression so for example you can:

  • make your expression case independent
  • make your expression search across multiple lines

If my regex above were $regex = "#cat#i"; it would match cat, CAT, Cat etc

Captures

Any sub-expression contained within brackets (( and)) is captured. That is to say it is appended to your $outputArray.

$contentToSearch = "123-ABC-456";

$regex = "#([0-9]*)-[A-Z]*-([0-9]*)#";

preg_match($regex, $contentToSearch, $outputArray);  

In this example the contents of $outputArray would be:

Array  
(
    [0] => 123-ABC-456
    [1] => 123
    [2] => 456
)
  • The first value is the whole expression match.
  • The second and third values are the values of the two capturing blocks.

Matches

[ and ] enclose acceptable sets of characters.

  • [0-9] allows any character from 0-9
  • [A9] allows the characters A or 9

How many

  • [A-Z] will match one character from A-Z.
  • [A-Z]* will match any number of characters from A-Z
  • [A-Z]{2,} will match two or more characters from A-Z

Greediness

$contentToSearch = "abc123 welcome back";

$regex = "#.* #";

preg_match($regex, $contentToSearch, $outputArray);


$regexTwo = "#.* #U";

preg_match($regexTwo, $contentToSearch, $outputArrayTwo);  

In this example $outputArray would contain the following:

Array  
(
    [0] => abc123 welcome
)

The . character matches any character. So this expressions searches for any number of any characters followed by a space.

The match is by default greedy so it wants the largest possible match. abc123 welcome is the largest set of any number of any characters followed by a space.

You can use the U modifier to make the regex ungreedy by default.

$outputArrayTwo would contain:

Array  
(
    [0] => abc123
)

There you go

Whilst I am happy to answer any questions that you may have, there are no shortage of questions (and answers), tutorials, and websites outlining the usage of regular expressions.

This post is a bare minimum introduction to regular expressions. That is to say with that outlined above you can probably achieve most simple things.

Whilst the actual expressions often times look ugly, they are in fact incredibly simple. Don't be put off.

Take the time now to get a grasp on regular expressions and you will be thanking yourself the next time you reach for a strpos and str_replace combo and realise that it is a hell of a lot easier with preg_replace ;)

References in Java. Considerations in Android.

References are handled differently across different programming languages. Given that I work with a number of different languages it is important that I have a clear understanding of how they work.

References in Java have caused me a few issues - issues that are mainly a result of their utilisation within the context of the Android framework.

This post seeks to clarify how references work in Java whilst also outlining some considerations when working with Android.

Jon Skeet outlines the most important points quite definitively:

  • "Everything in Java is passed by value
  • "The values of variables are always primitives or references, never objects."
int x = 0;

//We pass the value 0 to foo - 0 is an int (primitive)
foo(x);

String y = "This is a string";

//We pass a reference (by value) to the String object containing "This is a string"
bar(y);  

Another useful explanatory quote that I found here is:

"Java modifies the meaning of 'pass by reference' from languages like C/C++. In C/C++, the term 'pass by reference' means passing the value of the variable's address (this is what a pointer is!)"

There is an analogy pertaining to balloons which states that passing an object variable to a function is akin to attaching another string to a balloon.

If in the bar function mentioned above one were to state y = new Thing() you would essentially be reattaching the string to a new balloon.

Significance in Android

The significance of this within Android is that often you might have a Thing instance in your activity, a property of which you would like to update.

You want to also have access to this Thing in a displayed Fragment, and you want it to be the case that updating the variable in your Activity will make the changes in your Fragment too.

Example

In my Activity I have an instance property var thing = new Thing();.

Within my onCreate method I create a Fragment using a static newInstance method (see here) and add it to my view. What exactly have I passed to my Fragments initializer?

When I pass the variable thing to my Fragments initialiser I am attaching another string to the same instance of Thing in the heap.

If I now change a property of the Thing within my Fragment e.g thing.key = "newKey";, then that change will be reflected if I access it within the Activity because both 'strings' point to the same object.

The problem

The problem that arises can be demonstrated in this simple demo that I have put together.

When the initial Activity is created we instantiate a new Thing instance. The constructor for Thing sets the value of a key property to the current timestamp.

We pass this Thing when instantiating our Fragment and display the values of the key properties.

At this point we have two 'strings' pointing to the same balloon. The value outputted is the same because the 'strings' point to the same Thing object in the heap.

If I now click the Button, we change the value of the key property of the Thing referenced by the Fragment. This updates the key property of the Thing referenced by the activity as well because they both reference the same object.

If however we now rotate our screen, the Activity is recreated by the Android framework. The Fragment however has setRetainInstance(true) specified and as such the Android framework maintains the Fragment instance.

At this point the values displayed for the key property of the Activity and Fragment differ. The Activity has been recreated. A new balloon has been created, and a new string attached to it. The Fragment however still has a 'string' pointing to the original Thing object on the heap.

If now we press the Button, the key property of the Thing referenced in the Fragment increments. The key property of the Thing referenced by the Activity does not - the respective 'strings' now point to different balloons.

The third line (output on screen) pertains to a second fragment that I have created for demonstrations purposes - a 'Data Fragment'. This is a viewless Fragment which simply holds and retains data. You will notice that when following the steps above the value for the key property of the MainFragment and the DataFragment stays in sync. This indicates that when retaining Fragments (on orientation change for example), the Android framework is not doing anything untoward with our variables - the 'strings' in both point to the same balloon.

savedInstanceState

As a matter of intrigue, I also wanted to investigate the situation whereby a Fragment is completely destroyed, its state saved in savedInstanceState and then restored.

The Fragment documentation clearly states that in some situations the Fragment instance may still be completely destroyed:

"There are many situations where a fragment may be mostly torn down (such as when placed on the back stack with no UI showing), but its state will not be saved until its owning activity actually needs to save its state."

I was concerned that perhaps in the process of parcelling a Parcelable object and then restoring it from a Bundle our 'strings' might be manipulated by the Android framework. This turns out not to be the case ! Wahoo !

In the demo (linked above), I have saved the Thing instance of our MainFragment and restored it in onViewStateRestored (e.g after an orientation change). The fact that pressing the Button after an orientation change updates the key value for both MainFragment and DataFragment indicates that everything is how we would like.

Extras

At this point I was becoming increasingly intrigued, and as such wanted to see how 'extras' work in respect of the subject matter. Sadly in this case the results were less positive :(

You may have been wondering as to why there is a 'Start new activity' button in my demo. Well, the reason is so that we can see how the Android framework handles passing variables to new Activities by proxy of 'extras'.

When we click this button, we pass an extra containing our Thing instance from MainFragment.
If you look at the logcat output of the new Activity you will see that the key value of the Thing passed to the new Activity is one and the same as that from the MainFragment (to be expected).

What i have then done is implemented an update to the Thing key in MainFragment. This is done on a 5 second delay I.E After opening the second Activity, 5 seconds later we update thing in MainFragment.

In the onCreate method of ExtraActivity we have set a 6 second timer after which we will check the key property of the Thing contained within the ExtraActivity.

We would hope (in an ideal world) that these values would be one and the same, and that the strings are pointing to the same balloons. Sadly they are not.

I have had a look through the Android source code to try and gain an insight into why this is the case. On the face of it, both saving instance state and passing extras transmit data by proxy of a Bundle. I will of course follow up if I gain any further insight on this. If you have any ideas, perhaps you could fill me in in the comments?

Implications

The implications of this are pretty obvious - namely that when working with data that needs to be utilised and manipulated in multiple locations, you need to be very careful about where your variables point.

If you are not, you may end up with data integrity issues across areas of your application.

Hopefully this has provided a little clarity on references within Java, and provides insight into the appropriate patterns to utilise when developing Android applications.

If you have any questions or comments, please feel free to post them. I am always happy to help.

Android foolishness - things not to do

In the process of building the MMMM.com Android app I encountered a number of small and foolish issues which I thought that I would outline such that you don't make the same mistakes.

I probably shouldn't be admitting to making these kinds of mistakes, but.. well.. I am :)

Match you lifecycle calls

I utilise greenrobot's EventBus library to send various notifications across the application.

I could not for the life of me see as to why a subscriber was not picking up an event that I was posting.

The problem was that I was subscribing to the event in my Activity's onCreate but unsubscribing in onStop.
onStop is called when you open another Activity (even when the initial activity is not destroyed).

As such, when I went back to the initial activity, it was no longer subscribed to the event. Because it wasn't being recreated it was not being resubscribed.

The resolution is to unsubscribe in the appropriate lifecycle pairs:

  • onCreate and onDestroy
  • onStart and onStop
  • onResume and onPause

Learn more about the activity lifecycle here.

The other consideration here is that if you are going to have a global subscriber (e.g in a generalised activity superclass), don't use the onCreate onDestroy pairing. Just because an activity is not on the screen it does not necessarily mean that it has been destroyed. You do not want 'old' activities to be receiving events not destined for them.

Use the code analyzer provided by AndroidStudio

Having released the latest version of the Mmmm application, I noticed that the APK was somewhat bigger than one might expect (~17mb). AndroidStudio has a little known treasure - its code analyzer. If you select 'Analyze > Inspect code', you can have your code analysed (at which point you can optimise).

In my case, I had left a number of drawables in the APK which I had not ended up using (silly me). Removing them reduced my APK size significantly.

Use SparseArray instead of HashMap

This one is a little tidbit. It will likely make an insignificant difference, but hey.. every little helps.

As outlined in the documentation for SparseArray,

"SparseArrays map integers to Objects. Unlike a normal array of Objects, there can be gaps in the indices. It is intended to be more memory efficient than using a HashMap to map Integers to Objects, both because it avoids auto-boxing keys and its data structure doesn't rely on an extra entry object for each mapping."

So.. replace those instances of HashMap<Integer, Object> with SparseArray<Object> - it is more efficient.

That is all

That is all for now. If you have encountered any similarly foolish issues in your app development journey, please do share them in the comments. Save us all some time and self loathing ;)