Dynamic APIs: Negotiating Change

As originally posted September 21, 2015  by Dataversity – http://www.dataversity.net/dynamic-apis-negotiating-change/

By Dave Duggal

IT departments expose functionality via software interfaces to a wide variety of applications, micro-services, systems, databases, devices and machines. Such interfaces are generally referred to as APIs (Application Programming Interfaces).

The use of APIs has exploded with the growth of distributed computing driven by the popularity of the Web, Cloud and now Internet of Things (IoT). It logically follows that the more we want to interact with a distributed software-defined world, the more interfaces we need.

So what are APIs?

According to Wikipedia – (accessed September 11th, 2015)

An API expresses a software component in terms of its operations, inputs, outputs, and underlying types. An API defines functionalities that are independent of their respective implementations, which allows definitions and implementations to vary without compromising the interface. A good API makes it easier to develop a program by providing all the building blocks. A programmer then puts the blocks together.

While the entry provides a concise encapsulation of the nature of an API and the objective of facilitating composability of software development “building blocks”, most of us recognize that the real-world is not quite as plug-and-play as the definition suggests.

APIs: the wild-wild West

There is no singular API standard. The designs of interfaces vary based on how they are specified, communicated, and what they expose and how they expose it. While REST APIs have eclipsed more formal SOAP/WSDL based interfaces (which supplanted CORBA), REST APIs are far from formalism. Developers have progressively traded up for lighter-weight, less formal protocols for convenience, but the net result is that every API is a snowflake. Each implementation of an interface is unique – APIs are as diverse as the functions they expose.

Wonderful, but how do developers cope if everything that is connected is a “one-off” manual integration. Moreover, how do the resulting composite applications constructed from these building blocks deal with change?

Since all businesses are now Digital Businesses, we need to advance interface design in order to realize the vision of Service Oriented architecture and achieve the truly composable enterprise solutions. We need a pattern for Dynamic APIs.

APIs: a business perspective

At its essence an interface is implemented as a contract between two systems that defines the agreement or “shared-understanding”, established up front between the provider of the API and its consumers. The API describes concepts and policies, much like a legal contract provides terms and conditions. The provider’s technical implementation details (e.g. underlying data and code objects) are generally hidden; which is why APIs and Services are often referred to as black-boxes.

The API only works as long as the shared-understanding is maintained, otherwise the contract is broken. Generally no one willfully seeks to break an API that is in use. However, the technical implementation details naturally evolve over time (i.e. data and code objects are updated, upgraded and replaced), which requires changes to the agreed Interface concepts and policies.

If we step back for a second and consider contracts generally, we know that agreements between organizations are routinely amended. Standard legal terms generally anticipate change by specifying a means for amendment. No business person wants to re-negotiate a contract from scratch for every change if it can be avoided – it would require lawyers and be unnecessarily expensive and time consuming – instead the parties focus on the specific change at hand.

However, with APIs there are no such standard mechanisms to accommodate change. All concepts and policies are set in advance (static schema) and the back-end technical details are hard wired (static implementation), which leads to tightly coupling, and thus a brittle realization of a contract. As a result, when the provider of an API makes a modification the agreement is abruptly broken, even if the change is relatively immaterial to the shared-understanding. The business is interrupted, and now IT is required to manually re-integrate.

APIs: an embarrassment of riches

As organizations become ever more distributed, dynamic and diverse their business processes are becoming increasingly complex composites, but IT simply can’t keep up with the demand for manual API integration (and re-integration), it simply does not scale. This problem is at the heart of the tension between the business and IT.

Services and APIs are only loosely coupled in conceptual architecture, but not in the real world where people still have to manually discover and integrate them at the Application and Process layers. This is the reason why Service Orientation has yet to have delivered “Business Agility”.

From that high-level Enterprise Architecture perspective (as opposed to the bottom up IT view) it’s not useful or scalable to be stuck manually considering individual APIs. It would be better to have an abstraction, a model for looking at all of these endpoints as objects that expose functionality, but hide their complexity and automate management of the end-to-end API contract lifecycle in order to advance our Service Orientation architectures.

If legal contracts can anticipate change, when it isn’t even a routine occurrence, we can agree that IT, which is highly dynamic, should be able to anticipate and adapt to new requirements. APIs should be ‘living documents’ so they can safely evolve over time.

Dynamic APIs: a recommendation

Below is a matrix identifying EnterpriseWeb’s maturity model for Dynamic APIs (Level 0-3). Level 0 represents the mainstream status quo (fully-manual / disruptive). Levels 1-3 allow for an API to advertise new/modified features in its payload (the body of its messages) so that a consuming system client can be made notified of the changes and react to them.

Note that the API itself is not dynamic. Dynamism is represented by actual behavior, a property of an Actor (human or system). What makes a Dynamic API “dynamic” is expanding the scope of its contract to define an interaction-model between the systems that is designed to cope with change (e.g. “on update you can expect this behavior from my system”). This definition should be semantically understood by both human and system clients. In effect, it supports the ability for the Actors to ‘negotiate’ change during the lifecycle of the agreement.

Level 1 provides for simple notification for a human intervention (semi-automatic / disruptive). Level 2 provides notification and documentation to improve human intervention (semi-automatic / disruptive). Level 3 adds constraints to dynamically maintain synchronization between systems (fully automatic / non-disruptive).

Dynamic APIs: Low cost, High Return

While EnterpriseWeb supports the declarative modeling of Dynamic APIs and a runtime for using them, Dynamic APIs are not a product; it is a pattern that decouples interfaces from implementations to support resilient applications and processes.

This separation-of-concerns is very powerful. Instead of encoding all intelligence into a static interface at design-time, which makes it static, non-responsive and brittle, the interface can now describe its change semantics and this allows for dynamic, flexible and adaptable implementations. Interfaces are transformed into living documents; we refer to as “Active Contracts”.

Conceptually, the Active Contract is a promise of a capability, an “intent”, with related Service Level Agreements and policies rather than a single explicit/static representation.

We are currently working with standards bodies, which see value in this approach and hopefully, in the not too distant future, there will be refined and well-documented models that will be supported across domains, providing much needed consistency. While implementation would add some formalism, the approach itself would add little cost, impose no specific obligation (you can stay fully manual!), but offer great benefit.

We believe that widespread adoption of the Dynamic APIs pattern would be general good; helping everyone to facilitate non-disruptive API evolution, regardless of technology, and providing for a loosely-coupled form of interoperability.

Copyright 2015, EnterpriseWeb LLC