Introduction

In this chapter, we’re going to take a higher-level look at Web APIs and their place in the larger ecosystem. Web APIs have become a ubiquitous part of technology today, and it is very likely that most developers will be tasked with either writing their own API or using another API at some point in their career. Therefore, a larger understanding of Web APIs is a very useful skill to build.

We’ll look at some of the other aspects of Web APIs beyond just the RESTful architectural style, including how to handle authentication, documentation, and more.

Some of the key concepts and terms that will be introduced in this chapter are:

• Web API
• Simple Object Access Protocol (SOAP)
• Endpoint
• XML Schema

Web APIs

Video Materials

As the name implies, a web API is simply an interface for accessing and modifying resources stored on a web server. So, from a certain point of view, we could think of the basic HTTP itself as a web API. However, traditionally web APIs are meant to be built on top of HTTP itself – HTTP defines how web servers and web clients can communicate in general, but a web API uses additional information in the structure of the request, such parameters included as part of the URL or the body of the request, to specify exactly what resources should be affected and the action to be performed on those resources.

Web APIs are popular because they decouple the resources stored on the server from the client-side application that is designed to only interact with the web API. So, if an organization has some data or resources they’d like to make available, they can create a web API to make those resources available, and then other developers can build tools that interface with that web API to use those resources in some unique way.

¹

Example - Communication Platforms

A great example of this can be found by looking at online tools such as Twilio and Discord. Both of these tools are communication platforms – Twilio focuses on communication between a company and its customers and clients, while Discord is more focused on providing a chat and discussion platform for social groups.

What makes these companies similar is that they both provide a very robust web API for interacting with their platforms. In the case of Twilio, the web API is the only way to really use their product, which is primarily targeted at developers themselves. For Discord, they provide the core application for interacting with their platform that is used by most users, but their web API allows developers to make use of their platform in a variety of unique ways. Of course, these are just two examples from a very large number of web APIs available on the internet today, and that number continues to grow.

Twilio - Sending Text Messages

Let’s look at a quick example from the Twilio API Documentation, sending an SMS, or Short Message Service, message to a particular phone number. Many users would commonly refer to these as “text messages.”

First, let’s look at how to send this message using curl - a Linux terminal tool for making raw HTTP requests to web servers and web APIs:

EXCLAMATION_MARK='!'
curl -X POST https://api.twilio.com/2010-04-01/Accounts/<TWILIO_ACCOUNT_SID>/Messages.json \
--data-urlencode "Body=Hi there" \
--data-urlencode "From=+15017122661" \
--data-urlencode "To=+15558675310" \
-u <TWILIO_ACCOUNT_SID>:<TWILIO_AUTH_TOKEN>

Even without knowing exactly how curl works, we should be able to learn quite a bit about how this API works just by examining this command. First, we can guess that it is using an HTTP POST request, based on the -X POST portion of the command. Following that, we see the URL https://api.twilio.com/2010-04-01/Accounts/<TWILIO_ACCOUNT_SID>/Messages.json, which gives us the endpoint for this command. Just like programming APIs include classes and functions that we can call and get return values from through message passing, web APIs have endpoints which we can send requests to and receive responses. In fact, web APIs really reinforce the concept that “message passing” and calling a function are similar.

Below that, we see three lines of data prefixed by --data-urlencode. We can guess that these three lines construct the data that will be sent as the payload of the HTTP POST request. In fact, this data is structured nearly identically to the data that is generated when an HTML form is submitted using a POST request and the application/x-www-form-urlencoded encoding method.

Finally, the last portion -u <TWILIO_ACCOUNT_SID>:<TWILIO_AUTH_TOKEN> provides the user authentication information for this request. The first part before the colon : is the username, and the second part is the password. So, when this information is sent, it will also include a username and password to authenticate the request. That way, Twilio will know exactly which user is sending the request, and it prevents unauthorized users from sending spam text messages through their system.

Finally, notice that many parts of this command are enclosed by angle brackets <>. This simply means that those are meant to be variables, so it is up to the developer to replace those variables with the correct values, either by setting them as shown on the first line, or by some other means.

So, it looks like this curl command is just sending an HTTP POST request to a specific endpoint in the Twilio API. It will include three data elements, as well as some authentication information.

Twilio - Response

When we send that request to Twilio, their documentations says we should expect a response that looks like the following:

{
  "account_sid": "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
  "api_version": "2010-04-01",
  "body": "Hi there",
  "date_created": "Thu, 30 Jul 2015 20:12:31 +0000",
  "date_sent": "Thu, 30 Jul 2015 20:12:33 +0000",
  "date_updated": "Thu, 30 Jul 2015 20:12:33 +0000",
  "direction": "outbound-api",
  "error_code": null,
  "error_message": null,
  "from": "+14155552345",
  "messaging_service_sid": null,
  "num_media": "0",
  "num_segments": "1",
  "price": null,
  "price_unit": null,
  "sid": "SMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
  "status": "sent",
  "subresource_uris": {
    "media": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Messages/SMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Media.json"
  },
  "to": "+14155552345",
  "uri": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Messages/SMXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.json"
}

We won’t dig too deeply into this response, but we can easily see that it includes lots of useful information about the request itself. We can see when it was sent, what it contained, any if it caused any errors, all directly from the response. The Twilio API Documentation describes each of these in detail.

REST

Thus far, we’ve mainly discussed the REST architectural style for web APIs, since it has become commonly used on the internet today. However, let’s briefly look at one other architectural style for web APIs and see how it compares to REST.

¹

SOAP

The Simple Object Access Protocol, or SOAP, is a standardized protocol for exchanging information between web servers and clients that was first developed in 1998 (a few years before REST was first written about). So, unlike REST, which is simply an architectural style without an underlying standard, SOAP was designed to have a specific standard and implementation.

SOAP was designed to use XML to transfer data between the client and the server, and includes a specific three-part message structure consisting of an envelope, a set of encoding rules, and a way to represent the actual endpoint requests (function calls) and responses. It uses a specific XML Schema to define the structure of those messages.

Wikipedia includes a short example showing how to use SOAP to request the stock price for a stock symbol:

POST /InStock HTTP/1.1
Host: www.example.org
Content-Type: application/soap+xml; charset=utf-8
Content-Length: 299
SOAPAction: "http://www.w3.org/2003/05/soap-envelope"

<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:m="http://www.example.org">
  <soap:Header>
  </soap:Header>
  <soap:Body>
    <m:GetStockPrice>
      <m:StockName>T</m:StockName>
    </m:GetStockPrice>
  </soap:Body>
</soap:Envelope>

Notice that it includes a SOAPAction header in the HTTP request, as well as lots of XML in the body of the request. The server would respond with a similarly structured message containing the current stock price of the stock.

Comparison to REST

While SOAP does have several advantages, such as defining a standardized protocol that can be used with any web service that properly implements it, it also has several disadvantages. Most notably, the data must be encoded into XML, and the resulting XML must be parsed before it can be used. While XML and JSON are both well structured representations of data, the verbosity of XML compared to JSON makes it slower to send, receive and parse.

Likewise, since SOAP requires additional structure and rules to be added to the HTTP request itself, it makes it more difficult to work with than it seems. Unlike REST, SOAP itself doesn’t specify the structure of the endpoints and how to manage state - each application implementing SOAP may use an entirely different structure for the various endpoints used to access, update, and delete resources. REST, on the other hand, doesn’t specify exactly how it must be done, but it does lead toward a simple, more usable model for interacting with the server.

Because of this, most web APIs today follow a RESTful architectural style built directly upon HTTP, instead of using SOAP, even though SOAP is an officially supported web protocol.

Documenting Web APIs

Web APIs are very useful parts of the internet today, but their usefulness can be limited if they aren’t properly documented. Thankfully, there are many standards available for documenting how to use and interact with RESTful web APIs.

RESTful API Description Languages

Wikipedia has a list of RESTful API Description Lanuages, or DLs, that are meant to provide a formal way for documenting the structure and usage of a web API. This is very similar to the standard format we use for documentation comments in our code - if they are structured correctly, we can use other tools such as javadoc or pdoc to generate additional resources for us, such as developer documentation.

These DLs follow a similar concept - by standardizing the structure of the documentation for the web API, we can build additional tools that use that information for a variety of different uses. For example, it could generate a website containing all of the documentation required to interact with the library, just like we are able to do with our existing source code comments.

However, another great use of these tools would be to build software libraries that can be used to interface directly with the web API itself. The library would include functions that match each API endpoint, including the expected parameters and values. Then, when we call the functions in our code, the library would handle constructing the request, sending it to the API endpoint, and receiving and even parsing the response for us. This would allow us to even quickly develop libraries that can interact with our API in a variety of programming languages.

OpenAPI

One of the most common RESTful API DLs used today is the OpenAPI Specification. It is supported by a large number of both open-source and enterprise tools for constructing the document itself, as well code generators for a variety of languages.

Let’s look at the structure of a single endpoint to see what it looks like in the OpenAPI Specification. This example comes from their Getting Started document.

¹

This diagram shows how the OpenAPI Specification follows a hierarchical structure for each API endpoint, called a “path” in OpenAPI, is documented. Each path can have multiple operations defined, such as GET, PUT, POST, DELETE, which easily correspond to various HTTP methods. Each of those operations contains additional information about the data expected in the request and possible HTTP responses that could be returned.

For example, here is a short snippet of an OpenAPI Specification document for a web API for playing the classic Tic Tac Toe game:

openapi: 3.1.0
info:
  title: Tic Tac Toe
  description: |
    This API allows writing down marks on a Tic Tac Toe board
    and requesting the state of the board or of individual squares.    
  version: 1.0.0
paths:
  # Whole board operations
  /board:
    get:
      summary: Get the whole board
      description: Retrieves the current state of the board and the winner.
      responses:
        "200":
          description: "OK"
          content:
            $ref: "#/components/schemas/status"

So, this API contains an endpoint with the URL ending in /board, which can be used to get the whole board. Further in the document, it describes what that status message could contain:

  schemas:
    ...
    status:
      type: object
      properties:
        winner:
          $ref: "#/components/schemas/winner"
        board:
          $ref: "#/components/schemas/board"

So, we know that the status message would contain the winner of the game, if any, as well as the board. Those two objects are described as:

  schemas:
    ...
    board:
      type: array
      maxItems: 3
      minItems: 3
      items:
        type: array
        maxItems: 3
        minItems: 3
        items:
          $ref: "#/components/schemas/mark"
    winner:
      type: string
      enum: [".", "X", "O"]
      description: Winner of the game. `.` means nobody has won yet.
      example: "."

So, the board is just a 3 by 3 array, and the winner message is the character of the winning player.

As we can see, this document clearly describes everything a developer would need to know about the /board enpoint, including how to use it and what type of response would be returned.

The full Tic Tac Toe example is full of additional information about the entire API itself.

Handling Authentication

Another important concept related to web APIs is handing authentication. First, let’s review a bit about what authentication is and why it is important.

Authentication vs. Authorization

In computer security, we commonly use two related terms to describe limits placed on access to a particular resource.

• Authentication refers to providing information that confirms the user’s identity. This could be through the use of a password or some other secure token that is only known to the user, or through some other means.
• Authorization refers to determining if the user has access to a particular resource.

So, a user must first be authenticated to determine their identity. Then, the application must determine if that user is authorized to use the resource requested.

In this discussion, we are only concerned with authentication.

HTTP Authentication

One simple form of authentication that can be used for a web API is already built in to HTTP itself. The HTTP standard allows the webserver to ask for authentication credentials when accessing a given URL. So, the client can provide those credentials within the HTTP headers of a request, and the server will confirm that they are correct before providing the response.

This method is simple and easy to use, and we saw it earlier in this chapter already. However, it does have one major caveat - these authentication schemes require placing the secure information directly in the HTTP headers of an HTTP request. Since HTTP is a text-based protocol, anyone who sees that request (such as an internet service provider or a malicious user performing a man in the middle attack) can obtain the authentication information from it and then use it themselves.

So, HTTP authentication should only be used when combined with another encryption method, such as the use of HTTPS to create a secure connection between the client and the server.

API Keys

Due to the limitations of HTTP authentication, many web APIs, especially RESTful APIs, use an authentication method known as API keys. In this method, a user registers with the provider of the API, and along with their user account is given a special key, called an API key, to identify themselves. This key is usually a very long string of alphanumeric data, and should be protected just like any password.

When making a request to the API, the user should include the API key along with the request. The server will then check that the API is key valid before returning the response.

Unfortunately, as with HTTP authentication, API keys are also included directly in the HTTP request, so they should be combined with encryption such as HTTPS to prevent them from being compromised.

Other Methods

Finally, many APIs today use a variety of other methods for authentication. One popular choice is the OAuth, which is a way for users of a web API to request authentication through a 3rd-party service, and then pass the results of that authentication request to the web API.

Many users on the internet today are familiar with websites that present the option to log in using a different service, such as Facebook or Google, instead of registering an account directly with the site itself. This authentication method similar to OAuth, and sometimes is actually implemented using OAuth, such as OpenID.

Applications that use a web API can follow a similar process. The application first requests authentication via the 3rd-party service by submitting information such as a password or API key, and then it will receive a response. That response is their “ticket” to access other resources. So, when the application sends a request to the web API, it sends along the “ticket” to prove its identity.

Using a Web API

Video Materials

Finally, now that we’ve covered all of the aspects related to web APIs and how they are implemented, let’s look at a quick example for how we can use a web API to interact with resources stored on the web.

Finding an API

There are many great resources that can be used to locate a particular web API on the internet. Using search engines such as Google is one option, but there have also been many attempts to generate a directory of the most used and most useful web APIs such as the Public APIs project on GitHub.

There are also many scientific and research organizations that make their data available publicly via a web API. One of the most well-known is NASA, which provides several APIs for developers to use. This includes everything from the Astronomy Picture of the Day to information about the current weather conditions on Mars.

For this example, we’ll use the Astronomy Picture of the Day API. It is a very simple API - all we have to do is send a request to https://api.nasa.gov/planetary/apod and provide an API key. For testing, NASA provides the API key DEMO_KEY that can be used up to 30 times per hour and 50 times per day per IP address to test the APIs and explore what types of data they provide. So, we can use that.

Making a Request

To make a request to the API, simply open a Linux terminal, such as the one in Codio, and enter the following command:

curl -X GET https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY

Alternatively, in most web browsers you can simply visit the url https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY to view the response in a browser. Since we are using a GET request, it is really simple to access.

Viewing the Response

If done correctly, the API should send back a response formatted in JSON. The response itself may be just a blob of text, but if we reformat it a bit we can see that it has a simple structure:

{
   "date":"2021-04-19",
   "explanation":"What does the center of our galaxy look like?  In visible light, the Milky Way's center is hidden by clouds of obscuring dust and gas. But in this stunning vista, the Spitzer Space Telescope's infrared cameras, penetrate much of the dust revealing the stars of the crowded galactic center region. A mosaic of many smaller snapshots, the detailed, false-color image shows older, cool stars in bluish hues. Red and brown glowing dust clouds are associated with young, hot stars in stellar nurseries. The very center of the Milky Way has recently been found capable of forming newborn stars. The galactic center lies some 26,700 light-years away, toward the constellation Sagittarius. At that distance, this picture spans about 900 light-years.",
   "hdurl":"https://apod.nasa.gov/apod/image/2104/GalacticCore_SpitzerSchmidt_6143.jpg",
   "media_type":"image",
   "service_version":"v1",
   "title":"The Galactic Center in Infrared",
   "url":"https://apod.nasa.gov/apod/image/2104/GalacticCore_SpitzerSchmidt_960.jpg"
}

As we can see, the API returned the picture for April 19th, 2021, along with the title, URL, and description of the image. So, if we make this request in an application, we can use the JSON response to download the image and display it, along with the title and description. That image is shown below.

¹

It’s really that simple! For more advanced APIs, we may have to include additional information in our request, as seen in the Twilio example earlier in this chapter, but, in general, using a RESTful web API is meant to be a simple and powerful way to interact with resources on the web.

Summary

In this chapter, we covered some more information about web APIs. We discovered how they are structured and how we can interact with them in our applications. We even learned a bit about how they handle documentation and authentication.

Of course, this merely scratches the surface of information related to web APIs. A later course in the Computational Core curriculum, CC 515, covers web application development and goes in-depth about how to build and use web APIs using a RESTful architecture.

For this course, we’ll simply focus on making a small REST API for a portion of our ongoing project. This allows us to learn a bit about how we could construct our own web APIs and make them available for others.

Review Quiz

Check your understanding of the new content introduced in this chapter below - this quiz is not graded and you can retake it as many times as you want.