Web server from scratch with NodeJS Streams

Intro

A quick look at Node's built-in HTTP module

NodeJS comes with a simple HTTP server built-in. This server allows us to listen on an arbitrary port and provide a callback function that will be invoked on every incoming request. The callback will receive two arguments: a Request object and a response object. The request object will be filed with useful properties about the request, and the response object will be used to send a response to the client.

This will start a web server listening on port 3000. When a request comes in, a message will be logged in the console with the current date followed by the request's method and URL. This data comes from the parameter req, which is the Request object created by Node. This request object also has a property called .socket, which is the lower-level TCP socket. In the code above, the fact that we have access to a nice object with "req.method” and "req.url " means that someone other than us went through the trouble of parsing the text of the request and making it into a nice object.

In the next section, we'll look at the structure of an HTTP request to get hints on how to parse it.

Dissecting HTTP

Receiving and parsing an HTPP request

NodeJS provides the built-in net module to create a streaming TCP server. "Streaming” refers to the fact that the server can send and receive data over time, using the node's stream API. Where the HTTP server emits a request event with request and response objects, the TCP server will emit a connection event with a socket object. Socked objects are pure duplex streams. In short, this means that they can both be read from and written to. If we make an HTTP request to our TCP server, reading from the socket will give us the text of the request. There are two ways to read from a readable stream: subscribing to its data event or calling its .read() method. Let's look at the first one:

Put this code in a file called server.js and execute it from your command line with node server.js. In another terminal, use cURL to make a simple request to your server:

You should see a verbose output from cURL, showing you the request and response header. In the Node terminal, you should see something like this:

Let's make a post request with some data:

This should give you the following output in your terminal:

If you were to send a longer request body, you would receive it in multiple chinks. However, a web server doesn't always need to process a request body. Often, the header part of a request is sufficient to start processing the request. Since we are writing a generic web server, we have to find a way to stop receiving request data from the socket as soon as we reach the string \r\n\r\n in the input. This is more easily done by using the .read() method of the socket since we can control when to stop pulling data from the stream. Here's some code to do this:

The code is a bit longer because we need some logic to decide when to stop reading from the stream. This allows us to separate the header from the. body and let the developer who will use our web server determine what to do with the body if anything. The key part here is the socket.unshift line, which "puts back” any extra data we read back into the readable stream. This will allow us to pass this socket along to our users in case they need to read from it. And here's the full code of our basic web serve, putting together everything we have seen so far. Our server exposes a function createWebSERVer(requestHandler). This function accepts a handler of the form (req, res) = void, just like the node's basic web server. The comments in the code explain what each step is doing.

By using some lower-level building blocks such as net, stream, and buffer, we were able to create. a rudimentary HTTP server based on a TCP server and some parsing logic. In the process, we learned a bit about how HTTP requests and responses work, as well as the basics of readable and writable streams and buffers.