Design-first APIs: let user stories drive what your API looks like

Public APIs are truly here to stay - they are just everywhere. Arguably we had APIs since the beginning of IT - whenever two components communicated they did this through an API; it may just not have been called that, and sometimes they weren't even manifested as one.

With public APIs though - and I'll include internal APIs used within an enterprise or larger software system - the requirements for stableness have gone up dramatically: now we have a potentially unlimited number of consumers, all doing something different, expecting their version of the API will continue to work.

We can't just change an API - it needs to be well thought-through, adaptable to change, and needs to be backwards-compatible. The logical conclusion? Let's get our smartest people together in a group to design the best API possible.

Well, we have seen this before, haven't we? Architecture groups, database committees, SOA service review boards - and likely something similar before that ensured mainframes only run "the right" software. This is like putting on a giant handbrake into a market that is driven by speed, and the ability to out-learn your competition.

To make matters even worse, the API is now becoming a product by itself - after all, it has its own group to look after it, right? This has become an industry trend in its own right, of course proudly supported by vendors of API management products (just google). Of course, selling something to make a "product" is much easier than a "reverse proxy".

An API is not a product

This should be obvious, yet API management in many enterprises is treated as its own unit, with its own deliverables.

You never sell anything because you have an API. You also never actually sell anything with a mobile app (except the guys from Angry Birds), or a website. You are selling a service. There is no "build an API, and they will come" - it doesn't work like this.

Twitter is not selling an API - it is selling a service that allows people to send short messages to each other. The API is just a way to use their service.

You do sell though if you have a compelling service - that has an API. The API is an essential feature of your service; if you don't have it, then your chances of people buying into your service are considerably less.

An API is not an enterprise IT strategy

Thomas H. Davenport and Bala Iyer wrote in "Move Beyond Enterprise IT to an API Strategy" that "You need to start building an ecosystem, and APIs are the way to do it.", and argues that "Salesforce.com, Facebook, Twitter, Google, and eBay [...] have seen fantastic growth" - and they have APIs. So you should do it too?

This conclusion is terribly flawed. Salesforce, Facebook and others have fantastic services - that's why they saw growth.

Enterprises must focus on offering compelling services, being it public or private, free or paid-for. These are questions for the service, not the API. An API adds a new channel to provide this service - and the team providing the service needs to take it on board.

APIs need to be designed

Well, I agree with this statement ... but differently to what many people would expect.

Generally, APIs need to be designed, because

  • They should work intuitively
  • They should be usable for a variety of use-cases
  • They should work consistently

(Apigee has a nice in-depth guide)

Note that these reasons have user experience in common. So doesn't that need to be "designed"? The rest of the world uses style guides for consistent design and user experience, so why are we hell-bent on designing each and every API?

And, given that the internet is full of style guides for APIs: do we even need to create our own? And employ an "API consultant", who at best copies one from the internet, and at worst spends a lot of money re-working all your APIs to that standard?

Consumer-driven API design

Coming back to the "traditional" API design process - it looks something like this:

Traditional API Design

(If you read my earlier blog post "Agile Architecture: Dependency Management" you'll recognise a lot of dependencies and latency)

It all starts with a user story - so far so good. But then something else happens: To come up with the perfect API, designers try to anticipate how future users are going to use this, and in the process abstract and alter the requirements. I have seen APIs that look nothing like the original story, just to be future-proof. And it never works ...

And of course, now that the API is de-coupled from the story its usage and intentions need to be documented and handed over in an almighty thud. Ah, those poor trees ...

What if we turned this around, and made the consumer the driver?

Consumer Driven API Design

Oh, I can already hear you objecting ... shouldn't the API reflect what the user wants? Read what we want them to want? We can't just have it coupled with the service. Coupling is bad.

Let's return to the statements above: The service matters. If your service is correct, then the API, as it evolves with it, must also be correct, right? But what if you want to expose the data differently, say a subset, or, more sinister, fit it into an enterprise process or data model?

Your service implements a model within a certain context - domain driven design calls this a bounded context. If your service doesn't expose the correct model, then this means your service isn't correct. And if your model doesn't fit into an enterprise data model, then either the model is incorrect, or your enterprise data model doesn't recognise domains. None of these problems should be solved through up-front API design.

And regarding coupling: This is the place where you want to couple - how could you want to decouple the definition of a service from what the service does? This is different from the implementation, which is about how a service does its thing.

But what if the service changes? That though means that the API should change. The concern of supporting clients using the previous version is a different concern, one that shouldn't be mixed in. You can solve this by leaving the previous version deployed, and versioning. And if the core of your service changes too often: Perhaps you should have a look at the quality of your stories, and if the domain is properly understood?

Going one step further: If you follow an API-first approach, then likely your first consumer will be your website. We can even go one step further, and integrate the feedback channel between the UX designer and your development team:

Specification Design

How often do we hear: "I know it when I see it"? Users are usually much better at evaluating whether a service suits their needs if it is presented visually through a UI (not JSON snippets or XML spy screens). When form follows function, then a user interface is a good point to start when designing your APIs. The UI will tell you the model, and the interactions (assuming it is "user experience" designing this, not "just" a UI designer).

Using service mocks (for example Apiary) allows for quick UI iterations - and the API specification is generated automatically. How does that sound?

Conclusion

If you want to sell your services, and grow like Facebook: Well, focus on your services. Listen to your users, internal and external, how do they want to use your service? Let user stories drive what the API should look like - and "govern" it through style guides.

Creating an API shouldn't be an enterprise concern - and it definitely is not a product to sell. At best it might be a part of your web strategy, but that is already pushing it. It is a concern for the service owner, and we should treat it that way.