Introduction

Not all web applications are built to be viewed in a browser. Many are built to be used by other programs. We call these web applications Web APIs (Application Programming Interfaces). These also make HTTP or HTTPS requests against our apps, but usually instead of serving HTML, we serve some form of serialized data instead - most commonly XML or JSON.

Making Requests

Making a HTTP request is a multi-step process. First you must establish a connection with the server, then create the request data, then you must stream that data to your server through the connection. Once the server has received and processed your request data, it should stream a response back to you.

You can write code to handle each step, but most programming languages provide one or more libraries that provide a level of abstraction to this process. The C# language actually offers several options in its system libraries, and there are multiple open-source options as well.

WebRequest

The simplest of these is the WebRequest object. It represents and carries out a single HTTP request and provides the response. Let’s take a look at an example, which retrieves a “Joke of the Day” from a web API at https://jokes.one:

WebRequest request = WebRequest.Create("http://api.jokes.one/jod");

This one line of code creates the WebRequest object. Notice that we are not using a constructor. Instead, we invoke a Create() method. This is an example of the Factory Method Pattern, which you will learn more about in CIS 501. But to briefly introduce the concept, the WebRequest class is actually a base class for a multiple different classes, each representing a specific kind of web request (i.e. using HTTP, HTTPS, FTP and so on). Based on the URI supplied to WebRequest.Create(Uri uri), the method will determine the appropriate kind of request to make, and create and return the corresponding object.

Now that we have our request, we can send it and obtain a response with:

WebResponse response = request.GetResponse();

This opens the connection to the server, streams the request to it, and then captures the sever’s response. We can access this response as a stream (similar to how we would read a file):

using Stream responseStream = response.GetStream() 
{
  StreamReader reader = new StreamReader(responseStream);
  string responseText= reader.ReadToEnd();
  Console.WriteLine(responseText);
}

You likely are wondering what the using and curly braces {} are doing in this code. They are there because the Stream object implements the IDisposable interface. We’ll discuss this in detail in the next section. But for now, let’s focus on how we use the stream. First we create a StreamReader to read it:

  StreamReader reader = new StreamReader(responseStream);

Then read to the end of the stream:

  string responseFromServer = reader.ReadToEnd();

And write the response’s text to the console:

  Console.WriteLine(responseText);

Finally, we must close the WebResponse object’s connection to the server once we are done:

  response.Close();

This last step is important, as the open connection is actually managed by our operating system, and unless we close it, our system resources will be tied up, making our computer slower and potentially unable to make web requests for other programs (including your browser)!

Working With Disposables

In the previous section, we looked at a line of code that included the keyword using in a way you haven’t probably seen it before:

using Stream responseStream = response.GetStream() 
{
  // TODO: Use the responseStream
}

Let’s examine this statement in more detail. This use of using is a using statement, not to be confused with a using directive.

When you put a statement like using System.Text, you are using the using directive, which instructs the compiler to use types in the corresponding namespace without needing to provide the fully qualified namespace. You’ve been using this technique for some time, so it should be familiar to you.

In contrast, the using statement is used in the body of your code in conjunction with an object implementing the IDisposable interface. Objects that implement this interface have a Dispose() method, which needs to be called when you are done with them. These kinds of objects typically access some resource from outside of the program, which needs to be released when you are done with it.

Managed vs. Unmanaged Resources

To understand this better, let’s talk about managed vs. unmanaged resources. We say a resource is managed when obtaining and releasing it is handled by the language. Memory is a great example of this. In C#, we are using managed memory. When we invoke a constructor or declare an array, the interpreter automatically creates the memory we need to hold them.

In contrast, C uses unmanaged memory. When we want to create an array, we must allocate that memory with alloc(), calloc(), or malloc() function call.

This might not seem very different, until we are done with the array. In C#, we can simply let it fall out of scope, knowing the garbage collector should eventually free that memory. But in a C program, we must manually free the memory with a call to free().

Sometimes in C#, we need to access some resource in a way that is unmanaged - in which case, we must be sure to free the resource when we are done with it.

IDisposable

The IDisposable() interface provides a standard way of handling this kind of situation. It requires any class implementing it to define a Dispose() method that frees any unmanaged resources. A stream (the data being read in from a file, the network, or a terminal) is a good example of an unmanaged resource - the stream is actually created by the operating system, and the Stream object (a FileStream, BufferedStream, etc) is a C# object providing access to it.

Let’s focus on a FileStream for a moment. One is created every time you ask the operating system to open a file, i.e.:

FileStream fs = File.OpenRead("somefile.txt");

The File.OpenRead() method asks the operating system to provide a stream to the file named "somefile.txt". We can then read that stream until we reach the end of file marker (indicating we’ve read the entire file):

byte data = fs.ReadByte();
// Invariant: while there are bytes in the file to read
while(data != -1)
{
  // Write the current byte to the console
  System.Out.Write(data);
  // Read the next byte
  data = fs.ReadByte();
}

Once we’ve finished reading the file, we need to call Dispose() on the stream to tell the operating system that we’re done with it:

fs.Dispose();

If we don’t, then the operating system will assume we’re still working with the file, and refuse to let any other program read it. Including our own program, if we were to run it again.

But what happens if an error occurs while reading the file? We’ll never reach the call to Dispose(), so we’ll never free the file! In order to access it, we’d have to restart the computer. Not great.

We could manage this with a try/catch/finally, i.e.:

try 
{
  FileStream fs = File.OpenRead("somefile.txt");
  byte data = fs.ReadByte();
  // Invariant: while there are bytes in the file to read
  while(data != -1)
  {
    // Write the current byte to the console
    System.Out.Write(data);
    // Read the next byte
    data = fs.ReadByte();
  }
  fs.Dispose();
}
catch(Exception e) 
{
  // Do something with e
}
finally
{
  fs.Dispose();
}

But you have to catch all exceptions.

Using Statement

A using statement operates similarly, but takes far less typing:

using FileStream fs = File.OpenRead("somefile.txt")
{
  byte data = fs.ReadByte();
  // Invariant: while there are bytes in the file to read
  while(data != -1)
  {
    // Write the current byte to the console
    System.Out.Write(data);
    // Read the next byte
    data = fs.ReadByte();
  }
}

It also comes with some benefits. One, it creates a new scope (within the {} following the using statement). If for some reason the stream can’t be opened, this scope is skipped over. Similarly it jumps execution to the end of the scope if an error occurs. Finally, it automatically calls Dispose() when the scope ends.

Syntax Shorthand

As of C# 8.0, a shorthand for the using statement that omits the scope markers (the {}) is available. In this case, the scope is from the start of the using statement to the end of its containing scope (usually a method):

using FileStream fs = File.OpenRead("somefile.txt");
byte data = fs.ReadByte();
// Invariant: while there are bytes in the file to read
while(data != -1)
{
  // Write the current byte to the console
  System.Out.Write(data);
  // Read the next byte
  data = fs.ReadByte();
}

This format can be nice when you need to nest multiple using statements, but I would suggest sticking with the scoped version until you are comfortable with the concepts involved.

Parsing API Data

Web APIs typically provide their data in a structured format, i.e. XML or JSON. To use this within a C# program you’ll need to either parse it or convert it into an object or objects.

The Joke of the Day API can provide either - we just need to specify our preference with a Accept header in our HTTP request. This header lets the server know what format(s) of data we are ready to process. XML is signified by the MIME type application/xml and JSON by application/json.

To set this (or any other header) in our WebRequest object, we use the Header property’s Add() method:

WebRequest request = WebRequest.Create("http://api.jokes.one/jod");
request.Headers.Add("Accept", "application/json");

For JSON, or:

WebRequest request = WebRequest.Create("http://api.jokes.one/jod");
request.Headers.Add("Accept", "application/xml");

For XML.

Parsing XML

Let’s start by examining the older format, XML. Assuming you have set the Accept header as discussed above, you will receive a response similar to (but with a different joke):

<response>
  <success>
    <total>1</total>
  </success>
  <contents>
    <jokes>
      <description>Joke of the day </description>
      <language>en</language>
      <background/>
      <category>jod</category>
      <date>2021-11-29</date>
      <joke>
        <title>Signs for every job</title>
        <lang>en</lang>
        <length>1749</length>
        <clean>0</clean>
        <racial>0</racial>
        <date>2021-11-29</date>
        <id>HqJ1i9L1ujVCcZmS5C4nhAeF</id>
        <text>
        In the front yard of a funeral home, "Drive carefully, we'll wait." On an electrician's truck, "Let us remove your shorts." Outside a radiator repair shop, "Best place in town to take a leak." In a non-smoking area, "If we see you smoking, we will assume you are on fire and take appropriate action." On a maternity room door, "Push, Push, Push." On a front door, "Everyone on the premises is a vegetarian except the dog." At an optometrist's office, "If you don't see what you're looking for, you've come to the right place." On a taxidermist's window, "We really know our stuff." On a butcher's window, "Let me meat your needs." On a butcher's window, "You can beat our prices, but you can't beat our meat." On a fence, "Salesmen welcome. Dog food is expensive." At a car dealership, "The best way to get back on your feet - miss a car payment." Outside a muffler shop, "No appointment necessary. We'll hear you coming." In a dry cleaner's emporium, "Drop your pants here." On a desk in a reception room, "We shoot every 3rd salesman, and the 2nd one just left." In a veterinarian's waiting room, "Be back in 5 minutes. Sit! Stay!" At the electric company, "We would be delighted if you send in your bill. However, if you don't, you will be." In a Beauty Shop, "Dye now!" In a Beauty Shop, "We curl up and Dye for you." On the side of a garbage truck, "We've got what it takes to take what you've got." (Burglars please copy.) In a restaurant window, "Don't stand there and be hungry, come in and get fed up." Inside a bowling alley, "Please be quiet. We need to hear a pin drop." In a cafeteria, "Shoes are required to eat in the cafeteria. Socks can eat any place they want."
        </text>
      </joke>
    </jokes>
    <copyright>2019-20 https://jokes.one</copyright>
  </contents>
</response>

We can parse this response with C#’s XmlDocument Class from the System.Xml namespace. First, we create an instance of the class, using our response text. We can use one of the XmlDocument.Load() overrides, which takes a stream, to process our response stream directly:

using Stream responseStream = response.GetStream() 
{
  XmlDocument xDoc = new XmlDocument();
  xDoc.Load(responseStream);
  // TODO: get our joke!
}

Then we can query the XmlDocument for the tag we care about, i.e. response > contents > jokes > joke > text (the text of the joke). We use XPath syntax for this:

  var node = xDoc.SelectSingleNode("/response/contents/jokes/joke/text");

XPath is a query language, much like CSS selectors, which allow you to navigate a XML document in a lot of different ways. In this case, we are just finding the exact element based on its path. Then we can pull its value, and do something with it (such as logging it to the console):

  Console.WriteLine(node.InnerText);

Parsing JSON

JavaScript Object Notation (JSON) has become a popular format for web APIs, as it usually requires less characters than the corresponding XML, and is natively serializable from JavaScript making it extremely compatible with client-side web applications.

Assuming you have set the Accept header as discussed above, you will receive a response similar to (but with a different joke):

{
  "success":{
    "total":1
  },
  "contents":{
    "jokes":[
      {
        "description":"Joke of the day ",
        "language":"en",
        "background":"",
        "category":"jod",
        "date":"2021-11-30",
        "joke":{
          "title":"Class With Claus",
          "lang":"en",
          "length":"78",
          "clean":null,
          "racial":null,
          "date":"2021-11-30",
          "id":"LuVeRJsEIzCzvTnRmBTHXweF",
          "text":"Q: What do you say to Santa when he's taking attendance at school?\nA: Present."
        }
      }
    ],
    "copyright":"2019-20 https:\/\/jokes.one"
  }
}

The C# system libraries provide JSON support in the System.Text.Json namespace using the JsonSerializer class. The default behavior of the deserializer is to deserialize into a JsonDocument composed of nested JsonElement objects - essentially, dictionaries of dictionaries. As with the XDocument, we can deserialize JSON directly from a Stream:

using Stream responseStream = response.GetStream() 
{
  JsonDocument jDoc = JsonSerializer.Deserialize(responseStream);
  // TODO: get our joke!
}

Then we can navigate from the root element (a JsonElement instance) down the nested path of key/value pairs, by calling GetProperty() to access each successive property, and then print the joke text to the console:

  var contents = jDoc.RootElement.GetProperty("contents");
  var jokes = contents.GetProperty("jokes");
  var jokeData = jokes[0];
  var joke = jokeData.GetProperty("joke");
  var text = joke.GetProperty("text");
  Console.WriteLine(text);

Asynchronous Requests

Now that we’re more comfortable with using statements, let’s return to our request-making code:

WebRequest request = WebRequest.Create("http://api.jokes.one/jod");
using Stream responseStream = response.GetStream() 
{
  StreamReader reader = new StreamReader(responseStream);
  string responseText= reader.ReadToEnd();
  Console.WriteLine(responseText);
}
response.Close();

The response.GetStream() triggers the http request, which hits the API and returns its result. Remember a HTTP request is streamed across the internet, then processed by the server, and the response streamed back. That can take some time (at least to a computer). While the program waits on it to finish, it cannot do anything else. For some programs, like one that only displays jokes, this is fine. But what if our program needs to also be responding to the user’s events - like typing or moving the mouse? While the program is waiting, it is effectively paused, and nothing the user does will cause the program to change.

Asynchronous Methods

This is where asynchronous methods come in. An asynchronous method operates on a separate thread, allowing execution of the program to continue.

let’s revisit our WebRequest example:

WebRequest request = WebRequest.Create("http://api.jokes.one/jod");

We can then make the request asynchronously by calling the asynchronous version of GetResponse() - GetResponseAsync():

WebResponse response = await request.GetResponseAsync();

The await keyword effectively pauses this thread of execution until the response is received. Effectively, the subsequent code is set aside to be processed when the asynchronous method finishes or encounters an error. This allows the main thread of the program to continue responding to user input and other events. The rest of the process is handled exactly as before:

using Stream responseStream = response.GetStream() 
{
  StreamReader reader = new StreamReader(responseStream);
  string responseText= reader.ReadToEnd();
  Console.WriteLine(responseText);
}

Writing Asynchronous Methods

Normally we would wrap the asynchronous method calls within our own asynchronous method. Thus, we might define a method, GetJoke():

public string async GetJoke()
{
  WebRequest request = WebRequest.Create("http://api.jokes.one/jod");
  WebResponse response = await request.GetResponseAsync();
  using Stream responseStream = response.GetStream() 
  {
    XmlDocument xDoc = new XmlDocument();
    xDoc.Load(responseStream);
    var node = xDoc.SelectSingleNode("/response/contents/jokes/joke/text");
    return node.InnerText;
  }
  return "";
}

Asynchronous ASP Request/Response Methods

ASP.Net includes built-in support for asynchronous request handling. You just need to add the async keyword to your OnGet() or OnPost() method, and the ASP.NET server will process it asynchronously.

For example, we could invoke our GetJoke() method in a OnGet():

public class JokeModel : PageModel 
{
    public async IActionResult OnGet()
    {
        var joke = await GetJoke();
        return Content(joke);
    }
}

This will cause the text of the joke to be sent as the response, and allow other pages to be served while this one request is awaiting a response from the Joke API.

RESTful Routes

Many web applications deal with some kind of resource, i.e. people, widgets, records. Much like in object-orientation we have organized the program around objects, many web applications are organized around resources. And as we have specialized ways to construct, access, and destroy objects, web applications need to create, read, update, and destroy resource records (we call these CRUD operations).

In his 2000 PhD. dissertation, Roy Fielding defined Representational State Transfer (REST), a way of mapping HTTP routes to the CRUD operations for a specific resource. This practice came to be known as RESTful routing, and has become one common strategy for structuring a web application’s routes. Consider the case where we have an online directory of students. The students would be our resource, and we would define routes to create, read, update and destroy them by a combination of HTTP action and route:

Here the [ID] is a unique identifier for the individual student. Note too that we have two routes for reading - one for getting a list of all students, and one for getting the details of an individual student.

REST is a remarkably straightforward implementation of very common functionality, no doubt driving its wide adoption. Razor pages supports REST implicitly - the RESTful resource is the PageModel object. This is the reason model binding does not bind properties on GET requests by default - because a GET request is used to retrieve, not update a resource. However, properties derived from the route (such as :ID in our example, will be bound and available on GET requests).

With Razor Pages, you can specify a route parameter after the @page directive. I.e. to add our ID parameter, we would use:

@page "{ID?}"

We can use any parameter name; it doesn’t have to be ID (though this is common). The ? indicates that the ID parameter is optional - so we can still visit /students. Without it, we only can visit specific students, i.e. /students/2.

Route parameters can be bound using any of the binding methods we discussed previously - parameter binding, model binding, or be accessed directly from the @RouteData.Values dictionary. When we are using these parameters to create or update new resources, we often want to take an additional step - validating the supplied data.

Summary

In this chapter we explored using web APIs to retrieve data from remote servers. We saw how to use the WebRequest object to make this task approachable. We also revisited ideas you’ve seen in prior courses like the IDisposable interface and using statements to work with unmanaged objects. We saw how to consume XML data we receive as a response from a web API.

We also discussed using async methods to allow our programs to continue to respond to user input and incoming web requests while processing long-running tasks in parallel. Finally, we discussed RESTful routes, a standardized way of determining the routes for your web API.