HTTP

The Hypertext Transfer Protocol (HTTP) is the protocol behind transferring web pages. When a client (like your web browser) wants to request a page from a server, it sends that request in the format specified by HTTP. This is called an HTTP request. The server returns the page you asked for along with some other details about the page. This is called an HTTP response.

Here is the simplest possible HTTP request that works on most modern servers:

GET / HTTP/1.1
Host: example.com

You can try this out yourself. If you're on Mac or Linux, at the command line type nc example.com 80 and then paste in the exact request above with a blank line after it. If it works, you'll get a response back from the server containing the contents of the example.com homepage along with some other information. It's pretty neat to be able to download a web page by making a direct connection yourself without having to rely on a web browser.**On Windows, probably the simplest way to do this is to install the program Putty. You'll need to select “raw” for the connection type and tell it not to close the window on exit.

The three most important versions of HTTP are 1.1, 2.0, and 3.0. All are in common use. We will focus initially only on version 1.1 because it is the basis for the later versions, and it is also text-based, whereas the later versions are in a binary format.

In general, HTTP 1.1 requests follow a format like above. The first line is in the form below:

   HTTP/1.1

The most common method types are GET and POST. More on them later. The resource location is the name of the page you want on the server. Putting a single slash asks for the default homepage. If you wanted a file passwords.html in the directory admin, then you would use /admin/passwords.html. The last part of the first line indicates the version of HTTP being used.

After the first line of the HTTP request there are usually several header lines. These are ways of sending additional info about your request to the server. The one in the example above is called Host, and it's required in HTTP version 1.1. The reason for it is that some sites host multiple websites at the same IP address and they use the Host header to tell which site is being asked for.

The server will respond with a status code, which tells to what degree the server was able to (or not) serve the request. After that there are response headers that give information about the page being requested. After the headers usually comes the actual contents of the page.

To see the contents of an HTTP request, there are several ways. Probably the simplest is to use the developer tools of your web browser. To do this, open up the developer tools before loading the page. Then go to the network tab and load the page. You will be able to see all of the request and response headers, as well as the status code. We will look at other ways to do this later.

Headers

There are dozens of headers that can be set in HTTP requests. Usually your web browser is responsible for setting them. They can specify things to do with language, caching, asking for specific bytes of a file, and more. A few of the most important request headers are listed below.

• Host — As mentioned earlier, often multiple sites are hosted at the same IP address; this is used to tell the server which one you want.
• Referer — Note that it's actually misspelled like this in the HTTP spec (proper spelling is “referrer”). This header is used for your browser to indicate what site caused the request. For instance, if you search Google for Mount St.\ Mary's and click on the link to the Mount's homepage, the referer header in the request to msmary.edu is set to google.com. The referer header is one way for the operator of a website to know how people are getting to their site. If a page contains images or scripts, the HTTP requests to load those images and scripts will have the referer header set to the page.
• User-Agent — This is a string that usually gives information about the web browser making the request. The string is long and convoluted for historical reasons, but it usually contains the browser type and operating system info of the client. The server can use the user agent string to decide what to send back based on the browser type. Sometimes servers will ignore requests if the user agent string does not match something that looks like a real browser.

Besides these, another important header is the cookie header, which we will look at a bit later. The request Accept tells the server what document formats the client understands. The related headers Accept-Encoding and Accept-Language tell the server what types of encodings and what languages the client understands. Browsers also typically now send various headers whose names start with Sec-Fetch that are designed to give the server information about where the request originated to help avoid various cross-site attacks.

There are also a few dozen response headers. Some have to do with caching, some tell about the file type and encoding. One header identifies the type of web server. There is also a response header for cookies.

Status codes

Status codes are three-digit numbers that indicate what happened with the request. The most well known code is 404, for page not found, which we've all seen many times. Status codes follow this scheme:

• 100s — Informational message (not used very often)
• 200s — Success (to say that the page was successfully sent)
• 300s — Redirect (to indicate page has moved or is located elsewhere)
• 400s — Client error (the person downloading the page has done something wrong)
• 500s — Server error (something went wrong on the server)

There are several dozen status codes in total. Here are a few of the more common ones:

• 200 OK — This is what you usually get if everything went okay.
• 301 Moved Permanently — This is used to tell the user/web browser that the page is no longer at this location and to give its new location.
• 304 Not Modified — Used for caching. Specifically, the server can send this status code to let the client know that there is not a newer version of the resource than what the client has in their cache.
• 400 Bad Request — You'll see this if you're experimenting with manual HTTP requests and you do something wrong.
• 403 Forbidden — If you are not allowed to access a resource, you may get this message. Sometimes you'll see this if you try to look at a directory listing, like when editing a URL.
• 404 Not Found — The page you're looking for either doesn't exist, or maybe it did and was deleted and now you're out of luck.
• 500 Internal Server Error — Generic error message for a problem on the server.
• 503 Service Unavailable — Sometimes you'll get this if a site is overloaded with traffic.

URLs

A URL is a uniform resource locator. Examples are http://www.example.com and https://brianheinold.net/python/python_book.html. Most URLs follow the description below. If you want to see all the intricate details, see the Wikipedia page on URLs.

1. URLs start with a scheme, which is usually http or https, though there are other possibilities, like file or ftp.

2. Next comes the host name, which is usually a domain name, such as www.example.com. After the host name you can optionally put a colon and a port number, like example.com:8080. This is used if you want to access a page at a nonstandard port. Usually it's left out, but sometimes people put their sites at unusual port numbers. This can be to hide the site or because something else is running at the standard port. Programs running on a local machine, such as a web framework for developing websites, will often use a special port like 5000 or 8000.

3. Next comes a path indicating directories and the file you want. In https://brianheinold.net/python/python_book.html, the path is /python/python_book.html, indicating the directory and file name of the desired resource. It's often possible to leave off the file name. This will either allow you to view the directory contents or go to a default file in that directory.

4. Next comes the query string. This is optional. It starts with a question mark and contains name/value pairs separated by ampersands. This is used to send info to the server. A typical example might look like this:

   http://example.com/form.php?name=steve&age=27&search=chocolate
   

   Query strings are used as a way to send data to a server in the URL. Many forms use this. For example if you do a search at DuckDuckGo for “wikipedia”, the URL generated is the following:

   https://duckduckgo.com/?q=wikipedia&t=hk&ia=web
   

   The q=wikipedia is generated based on what we put into the form at DuckDuckGo. If you change the URL in the URL bar to q=computers, it will take you to the search results for computers. (The other two things in the query string are things DuckDuckGo adds for reasons we won't worry about here.)

5. Finally comes the fragment portion. It is an optional link to a specific portion of a page. It starts with a # symbol. For instance, https://en.wikipedia.org/wiki/Url#History links directly to the history part of the Wikipedia article on URLs.

URLs are a special case of something called URIs (uniform resource identifier). Some pedantic people get angry if you mix them up, but many people use the terms interchangeably.

Certain characters in URLs, like the question mark, have special meanings. If we need a question mark at a certain place in our URL, we have to escape it. This is done by using the % symbol followed by the symbol's ASCII character code in hex. The code of the question mark is 3F in hex, so it would be encoded as %3F. If you go to DuckDuckGo and search “what is http?”, the query string it generates will look like this:

https://duckduckgo.com/?q=what+is+http%3F&t=hk&ia=web

We see that the question mark in our query is replaced with %3F. Notice also that the spaces of our search are replaced with plus signs. This is because spaces are not allowed in query strings, and the rule is to replace them with plus signs.

As another example, if we enter abc123!@#$%^ into a form, it will be transformed into abc123!%40%23%24%25^. Notice that some of the special characters are encoded, while others aren't. There are rules specifying which ones must be encoded, but the details aren't important here. Note also that any character can be URL-encoded. For instance, %61 is the encoding for a lowercase a. Sometimes people use this trick to get around security precautions. If a site prevents you from entering the word cab into a form, it's possible you could get around that by putting %63%61%62 into the query string.

GET, POST, and other HTTP Methods

In HTTP requests, the two most common methods are GET and POST. Both can be used to request a web page from a server, and both also can be used to send data to the server in the request. The difference is that GET requests send the data in the URL query string, while POST requests send the data inside the body of the HTTP request.

Since GET requests store the data in the URL, they are useful for things that you would want to bookmark. For instance, the search terms from a Google search are sent in a GET request. This allows you to bookmark the page of results. On the other hand, when you order something online, the data is usually sent in a POST request. The idea here is that paying for an order is not something you want to repeat, unless you enjoy paying twice for the same thing. It happens once and then it's done. In short, sending data in a GET request is best if the exact request is repeatable, and POST is best if it's not something where you would want to repeat the exact request. When a GET request is sent, the idea is that nothing on the server should be changed by that request, while POST data can affect things.

Let's look at an actual GET HTTP request versus a POST request. Suppose we are submitting data to a form at example.com/form.php. There are two fields, name and age, that we are setting to the values steve and 27. Here's what a GET request for this looks like. Notice that the data is sent in the URL.

GET /form.php?name=steve&age=27 HTTP/1.1
Host: example.com

Here is what the equivalent POST request looks like:

POST /form.php HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
Content-Length: 17

name=steve&age=27

Notice that the data is sent in the body of the HTTP request. In order to use POST, a few other headers need to be set to tell the server what format the data is in (URL-encoded in this case) and how long the data is (17 bytes here). If you visit a page and want to see what data your browser is sending via GET or POST, you can do it with the developer tools under the network tab. Click on a request, and look for “query-string parameters” or “payload”.

There are several other HTTP commands. One useful one is HEAD, which returns header information about the page without returning the page itself. This is useful before you download a page if you want to know info like how large the page is. Two other fairly well-known methods are PUT and DELETE. They were originally designed for uploading and deleting files on the server. They aren't used for that purpose much anymore. However, many websites provide various services over APIs (application programming interfaces). For instance, there are APIs where you can send the server a food type, and it will return nutrition info, not in a full web page, but just the info you want, formatted in a convenient format, like JSON. GET and POST are by far the most common methods for APIs, but some also use other methods.

Cookies

Cookies are a way for a web server to store a small amount of data on a client's device.

The way cookies work is as follows: When a client first visits a website, that website sends back a Set-Cookie response header which contains the name and value of the cookie. Along with those there is often also an expiration date for the cookie. The web browser stores all that info. Then every time a client visits the site again (until the cookie expires or is deleted), their browser will send back the cookie name and value to the server in the Cookie request header. The key point to remember is that the cookie is sent back to the server with every request it applies to (though they can be limited under certain conditions, especially for security reasons).

Here are some common applications of cookies:

1. A simple application of cookies is for a web page to remember user settings. Whenever the user visits the site, their browser will send a cookie containing the user's settings. The web page can then send back appropriate data based on those settings.

2. One of the most common applications is for logging onto a page. When you first log on with a username and password, the server sends back a session ID. This is a long, random string that is unique to you. Every time you make a request to that page, your browser sends the server a cookie containing the session ID. That's how the server recognizes it's you. It's important to make sure no one else obtains that session ID because they could use it to impersonate you.

3. An unfortunate use of cookies is for tracking which websites people visit. This is done usually through third-party cookies. When you load a webpage, that page often makes requests to a variety of different domains. Some of those domains host resources, like images, that the page needs. Others host JavaScript files used by the page. Each time you load one of those resources, your browser may send any cookies set by the domains of those resources to those domains.

   Here then is how the tracking happens: A tracking network has to find a way to get one of their resources on a variety of different sites. Sometimes they have agreements with those sites. They could also buy ad space on those sites. Often these resources are scripts or tiny 1-pixel images that you would never notice. The tracking network makes sure they can tie the resource to the site, maybe with the referer header, query strings, or the name of the resource. Then when you visit the site, your browser will load the tracking network's resource and send any cookies for the tracking network's domain back to that domain. The tracking networks make sure the values of those cookies are identifiers that are unique to each client. Combining those identifiers with identifiers of which page the resource is on allows the tracking network to build a dossier of which sites you visit. They could use this to show targeted ads to you. They could also sell that dossier of sites you visit to various unsavory parties.

   These tracking cookies are third-party cookies since they are not cookies for the domain of the main page being visited. Several major browsers now restrict third-party cookies, which makes tracking this way harder, though there are other ways sites track users, such as via browser fingerprinting.

You can use your browser to view all the cookies it stores for a particular site. On Chrome, you can do this by clicking on the icon to the left of the domain name in the URL bar. In Firefox, they are in the web developer tools under the storage tab. You can also use your browser to delete cookies.

Caching

Recall that caching is a computer science term for storing data for quick access. In HTTP, caching frequently accessed resources saves the time and network usage of constantly downloading things fromthe server. Web browsers typically cache pages. Sometimes you might have to clear out your web browser's cache if you need to be sure you are getting the newest version of a page, though there are mechanisms we will see that mean you rarely have to do this. Some organizations have servers that act as caches for many devices on their network (see the section on proxy servers below). At a higher level, many large internet companies have content delivery networks (CDNs). CDN servers are often located closer to people than remote websites, and those CDN servers will have cached copies of pages for faster access.

So how does the client know if a cached copy of a page is stale or not? This is done with a variety of HTTP headers. One of the simplest is the Cache-Control response header. A typical value might be max-age=3600. This tells the client that the page will be fresh for the next 3600 seconds (1 hour). So anytime someone needs the page for the next hour, they can use the saved one on the client without having to contact the server at all.

Once that expires, if we want to download the page again, we still don't necessarily have to do a full download of the page. This is because of another response header, ETag, which is an identifier for the specific version of the page that was sent. If the ETag was set to abc123, the client can send an HTTP request with the If-None-Match header set to that same value. The server can then check if the current version of the page matches that tag. If so, it will send back a response with an HTTP 304 Not Modified status code, telling the client that the version they have is still current. If it doesn't match, only then will the server send a new version of the page.

The Cache-Control header can be used for other purposes, too. For instance, if the server sets it to no-store, that tells the client not to cache it at all. If it's set to private that means it can be cached on a web browser, but not at a proxy or CDN.

Proxy servers

In English, the word proxy means someone that takes someone else's place. For instance, a person who is supposed to attend a meeting but can't make it might send a proxy in their place. In the internet, proxy servers stand between the client and the web server. There are proxies that stand in for clients and there are proxies that stand in for web servers.

Client-side proxies have a few different roles. Any request you send will pass through that proxy before it goes out to the internet. That proxy could be used to block certain requests, possibly to stop people on a network from accessing certain pages. That proxy can sometimes be used to view encrypted HTTPS traffic. Usually, they let that traffic pass through, but there is something called TLS interception where instead of the client having an encrypted connection to the remote server, the proxy inserts itself in between and acts as a middleman, maintaining separate encrypted connections with the client and the server. So what the client sends is decrypted by the proxy, read, and then reencrypted when sent out to the remote server.

Client-side proxies are also used by people to hide their IP addresses from web servers. In order to make an HTTP request, you need an IP address, and web servers usually log that. If you want to protect your privacy or if you want to get around IP address restrictions, you could use a proxy. Note, however, that the proxy itself will see your IP address and the destinations you are requesting, so you have to trust it.

Finally, client proxies can also be used as caches. If there are several clients behind the proxy that all want the same page, the proxy can download it just once itself and serve cached copies of it to all the clients, saving some network usage.

Server-side proxies (often called reverse proxies) sit in front of a web server. One use is for security, adding an additional layer between the internet and the web server. The proxy could be used to filter out certain types of requests to prevent them from getting to the web server. A common use is for DDoS protection. Server-side proxies can also be used for load-balancing. Big sites tend to have multiple web servers. A proxy could be used to spread out the requests evenly among the web servers.

HTTP/2 and HTTP/3

There is still a weakness to this approach, which has to do with the fact that the connection is a TCP connection. TCP is a protocol we will cover a bit later. The main thing to know about it for the moment is that if a network problem causes some of the packets to be lost, then the whole connection will essentially pause until those packets are resent and received. Even if the packets were only part of one particular HTTP request, this will pause transmission on the entire connection for all the requests. The HTTP/3 standard, addresses this by replacing TCP with a new protocol called QUIC that essentially keeps the HTTP requests separate on that connection so that if one of them has a problem, it doesn't block the others.

Some other differences of HTTP/2 and HTTP/3 include the fact that they are binary protocols as opposed to the plain text HTTP/1.1. They are almost always used with HTTPS (see below), not plain HTTP, whereas HTTP/1.1 is used for both plain HTTP and HTTPS. HTTP/2 performs what is called header compression as a way to not have to send entire headers with every request like HTTP/1.1 does. When HTTP/1.1 was first standardized, web pages usually involved downloading an HTML file, a CSS file, and maybe a couple of images. In the current internet, loading a page involves retrieving many dozens of resources, and sending the same headers over and over with each of them wastes time and network bandwidth.

HTTPS

The two most common protocols you see in web browsers are HTTP and HTTPS. The S in HTTPS stands for “secure”. A big problem with plain HTTP is that it is not encrypted, so usernames, passwords, and everything else are visible to anyone intercepting HTTP traffic.

The security in HTTPS is provided by a protocol called TLS (Transport Layer Security). People sometimes use the term SSL (Secure Sockets Layer) interchangeably with TLS. SSL came first, and TLS was to a large degree based on SSL version 3, so some people still use the name SSL.

TLS provides three things: encryption, integrity, and authentication. Encryption is where the contents of a message are scrambled so that they are unreadable to anyone but the intended recipient. Integrity is where the recipient can be sure that the message they receive was not tampered with. Even if a message is encrypted, someone intercepting a message can remove or modify the some of the encrypted text, and those modifications can in turn affect what it decrypts back to. Integrity provides a way to determine if a modification has happened. Authentication is the process where a client can be sure that the server really is who it claims to be and not someone impersonating it.

A TLS session starts with the client contacting a server and indicating what types of encryption it supports. The server picks one from the client's list that it also supports. The server also sends a copy of its certificate for the client to verify that the server is who it claims to be. The client and server also create keys to use for encryption. This whole process is called a TLS handshake. Once this is established, then ordinary HTTP information can be sent back and forth, and now it will be secured.

TLS has been very successful. The majority of web traffic is now HTTPS, especially once web browsers started showing warning messages when accessing pages over plain HTTP. TLS is also used to secure other things, such as email and DNS queries.

Tools

For more advanced work, there are tools like Burp Suite that act as client-side proxies and allow you to edit and view all sorts of parts of HTTP requests.

The curl tool or its underlying library is used by many systems, such as those in cars and gaming consoles, to make API calls, download software, and send data to remote servers.

import requests

page = requests.get('https://www.example.com')
print(page.text)

It's also easy to use requests to view the headers and status code, as well as send data via POST and edit the request headers to be sent with the request.