Enindu Alahapperuma
Software Engineer
Sri Lanka
I'm a back-end engineer, system administrator, and website security expert.
There are countless articles and videos on the internet explaining what HTTP is. If I were to write this post in the same way, it would just be another one among many. Instead, I'll be breaking down HTTP from the perspective of a back-end engineer, highlighting common issues developers face with it. Let's dive in.
HTTP is the backbone of web development. While many developers believe that knowing a CMS like WordPress or tweaking some code is enough to make them proficient, a solid grasp of the fundamentals is essential. Don't get me wrong—I'm not referring to all web developers, but specifically to those who assume they can handle web development projects by learning just a bit of coding and relying on drag-and-drop tools, especially full-stack developers.
HTTP stands for Hypertext Transfer Protocol. But what exactly does that mean? First, let's talk about protocols. Simply put, a protocol is a set of rules that dictate how certain tasks should be performed. Since we're discussing the web, the task in question is web communication. The web is a vast network of interconnected servers, and we use clients (Such as web browsers) to connect to that network. So, HTTP is essentially a set of rules that govern how data is transmitted between clients and servers on the web.
How does data get transmitted between a client and a server? To understand that, you need a basic grasp of the OSI model. The OSI model consists of seven layers, but for now, we only need to focus on three key layers.
Let's walk through an example. Imagine you open your favorite web browser, type "enindu.com" into the address bar, and hit enter.
In the OSI model, your computer will initiate an HTTP request, meaning a set of data will be generated by the client, following the rules defined in HTTP. This process happens at the application layer of the OSI model. Simply put, the web browser itself is considered the application layer. Before HTTP/2, this data was created in plain text. A basic HTTP request would look like this:
GET / HTTP/1.1
Host: enindu.com
Connection: keep-aliveWith all the knowledge we've covered so far, you now understand that the browser creates the request in this way because HTTP dictates it. It's a rule defined in the protocol, and that's how it works—nothing more, nothing less.
Let's move on to the next layer of the OSI model, the transport layer. This is where we determine how data is transmitted between clients and servers. Several protocols operate in this layer, such as TCP, UDP, UNIX, and others. Each has its own advantages and disadvantages, which I won't go into in this article—perhaps another time. Once again, HTTP has a rule specifying which protocol must be used for transportation, and that's TCP. As I mentioned earlier, TCP is just another protocol, like HTTP, but it resides in the transport layer rather than the application layer. This makes TCP a lower-level protocol than HTTP. I'll cover TCP in a separate article since the focus here is on HTTP. In addition to the protocol, the transport layer also determines the port to use for the connection. For websites, this is typically port 80 or 443.
Now that we know the HTTP request will be sent via TCP to the server, let's move on to the final layer we need to understand: the network layer. While we create the HTTP request and intend to send it via TCP, how do we determine where that request should go? This is where the network layer comes into play. As stated in the HTTP ruleset, we must use IP, which is also a protocol like HTTP and TCP. IP is another set of rules that defines how data should be routed. Using these rules, the computer creates data packets and sends them to the correct destination. I'll go into more detail about IP in a separate article. In reality, the transmission happens in the lowest layer of the OSI model, which is the hardware layer.
Now that you understand how data transmission works on the client side, you're equipped to create your own HTTP client. All you need to do is follow the HTTP rules. With the right knowledge of these rules and programming, you could even create a brand new web browser at this point, using HTML, CSS, and JavaScript for rendering.
Let me clarify this quickly: There is no strict rule that HTML, CSS, and JavaScript must be used on the front-end. We use them because the founder of the web said so, and the W3C, a foundation established by the web's creator, confirmed it by standardizing these technologies. Web browsers also support them. However, we could technically use any technology on the front-end, as long as we create a client to support it and a server to generate the appropriate responses. It would still follow the HTTP architecture because the use of specific front-end technologies is not a rule of HTTP.
Imagine you create an HTTP-like client using UDP instead of TCP for the sake of speed. While it might not technically be an HTTP client because it doesn't follow the HTTP rules, it is still very possible. In that sense, you could even build a completely different architecture similar to HTTP. In fact, there are already many architectures like HTTP on the internet, such as SMTP, POP3, IMAP, FTP, SSH, and others. Protocols are always fun to experiment with.
Now, let's shift our focus to how servers handle the response in this process.
After receiving the request, the network layer will verify the destination IP address, ensuring that the data packets are delivered to the correct interface.
In the transport layer, the connection should be established and stabilized since it's using TCP, ensuring it connects to the correct port.
In the application layer, the server will read the request and generate a specific response to send back to the client. Sending responses in the OSI model follows the same process as the client-side, which I mentioned earlier. A simple HTTP/1.1 response would look like this:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 100
<!doctype html>
<!-- Code goes here -->Just like with a custom HTTP client, now you know how to create a custom HTTP server. All you need to do is follow the rules.
As I mentioned earlier, playing with protocols is fun because once you understand the bounds of a protocol, you can do whatever you like. You can even invent an entirely new protocol. It will truly refine your understanding—trust me.
Now that you know what HTTP is and how it works, let's discuss what this has to do with web developers and what problems they often face.
Almost all web developers today rely on pre-built HTTP servers or language- or framework-specific methods to handle HTTP requests. This used to be a core responsibility of developers, but nowadays, many web developers don't even know how to manage such low-level functionalities. This has led to a lack of understanding of pure web development practices.
First, you need to understand that we are creating web applications for the server, not the client. Even if you're only working on the front-end of a website, you're still creating it for the server. The web application you create resides on the server, and it's the HTTP server's responsibility to serve it to clients. This means the HTTP server also resides on the server. In reality, the front-end code won't be executed server-side, but on the client-side. However, the server still needs to send it to the client via HTTP. If a page has dynamic content, the server will generate the front-end code. Otherwise, it will simply send the content of the page to the client. This concept can be a bit tricky, and it's something many average web developers aren't fully aware of.
Now that you understand how the OSI model works on both the client and server sides, let's dive deeper into the application layer, focusing specifically on how it works on the server-side.
When someone sends an HTTP request to the server, the HTTP server will first validate the request. It will read the request, gathering the HTTP start line, headers, and the body (If any). The start line contains several instructions, such as the request method (GET, POST, etc.) and the requested path. The headers carry important information and configurations related to the request, like the host and connection headers. Based on this information, the HTTP server will route the request to the appropriate interface. All of this is handled by the HTTP server, whether it's a pre-built one or a language-specific server. If you were to create a custom HTTP server, you would need to handle these tasks manually.
Only after the HTTP server processes the request does the program or website created by the developer come into play. More specifically, the business logic is executed only after the HTTP server handles the request. Once the execution is complete, the HTTP server will then serve the response to the client. However, most average developers only have knowledge in this area. While it's an essential part of the web development process, it's just a small piece of a much larger architecture. Nowadays, most web developers primarily focus on business logic and building features, but they often overlook the underlying infrastructure and protocols that make everything work seamlessly. Understanding the full stack, from network protocols to server-side handling, is crucial for creating robust and efficient web applications.
With all the knowledge we've covered, we now have a solid understanding of what HTTP is and how it works. From the OSI model and the various layers involved to how HTTP requests and responses are processed by servers and clients, we've gained insight into both the fundamentals and the practical applications. This understanding is crucial for anyone working with web technologies, as it lays the groundwork for building more efficient and effective web applications.
In most cases, understanding low-level concepts and network-related programming is the responsibility of back-end engineers and system administrators, and an average developer doesn't need to be concerned with these details as long as they can do their job well. I totally agree with that. However, my argument is that a typical average developer cannot fully handle an entire web development project on their own. This is because an average web developer or even a full-stack developer typically has only a bit of knowledge across a vast architecture, as I mentioned earlier. While they might excel at specific tasks, like handling front-end design or implementing business logic, they might lack a deeper understanding of the underlying infrastructure and protocols that make everything work together. Without this holistic view, they may struggle to address performance, security, and scalability challenges, which are critical to building a successful, fully functioning web application.
In my final thoughts, without a solid understanding of the fundamentals of web development, like HTTP, developers will never be able to create a truly robust web application project. Here's why:
Without a deep understanding of these concepts, even the most talented developers will be limited in their ability to create well-rounded, high-quality web applications.
So, I would say that they will be able to create static websites or even simple dynamic websites without much hassle. However, they will never be able to create advanced, scalable websites or applications like CRMs, ERPs, marketing platforms, APIs, or even some e-commerce platforms. When it comes to deployment and maintenance, even simple websites will pose challenges for them due to a lack of expertise in pure web development practices.