How to make your own Protocol (like HTTP) in 10 minutes
Before you get a panic attack or start questioning my sanity let me tell you that a protocol is nothing but fundamentally just an agreement. An agreement where two progr…
Before you get a panic attack or start questioning my sanity let me tell you that a protocol is nothing but fundamentally just an agreement.
An agreement where two programs agree:
“When I send bytes in this format, you will interpret them in this way.”
That’s pretty much it.
So let’s first think about a name… For giving up myself a reward to come up with the idea to write an article about it I will call it “ATTP” which means Abhay Tiwari Transfer Protocol :) and because this is going to be the first version of this protocol the full name of this version would be ATTP/1.0
For clarification we will be building a tiny application-layer protocol in about 10 minutes.
It will support requests like:
WANT /hello (similar to GET in HTTP)
and responses like:
OK 17
Hello From ATTP:)
The entire purpose to this exercise is to make protocols much simpler to wrap your mind around.
I am not going to write 100 pages of theory on need of a protocol and what HTTP or other protocols are. I expect you to have atleast a little bit knowledge of them.
So let’s get our hands dirty and start design of our ATTP.
Designing ATTP
I decied that our protocol will run on top on TCP.
TCP handles things such as:
delivering bytes
preserving their order
retransmitting lost packets
maintaining a connection
Our protocol will decide what those bytes mean.
Let’s define a request:
METHOD PATH_
example:
_WANT /hello
Our response will look like this:
STATUS LENGTH
BODY_
example:
_OK 17
Hello From ATTP:)
note: the blank line seperates metadata from the response body.
So our protocol specification is already:
Request
<Method> <Path>\n
**Response
<Status> <Body_Length>\n
\n
That small definition is our protocol.
Build the Server
We’ll use Go because (it’s my project I can do whatevery the hell I want :P jk) it’s networking API makes TCP programming pleasantly small.
- Create server.go then write
package main
import (
"buffio"
"fmt"
"net"
"strings"
)
func main(){
listener, err:= net.Listen("tcp", ":9000")
if err!=nil{
panic(err)
}
fmt.Println("ATTP server is running on :9000" )
for {
conn, err:=listener.Accept()
if err!=nil{
continue
}
go handleConnection(conn)
}
}
func handleConnection(conn net.Conn){
defer conn.Close()
reader:=bufio.NewReader(conn)
request, err:= reader.ReadString('\n')
if err!=nil{
return
}
request = strings.TrimSpace(request)
parts := strings.Split(request, " ")
if len(parts) != 2{
sendResponse(conn, "Error", "Invalid request")
return
}
method := parts[0]
path := parts[1]
if method == "WANT" && path == "/hello" {
sendResponse(conn, "OK", "Hello From ATTP:)")
return
}
sendResponse(conn, "NOT_FOUND", "Resource not found")
}
func sendResponse(conn net.Conn, status string, body string){
response := fmt.Sprintf(
"%s %d\n\n%s",
status,
len(body),
body,
)
conn.Write([]byte(response))
}
Run it:
go run server.go
You should see:
ATTP server is running on :9000
Our server is now waiting for TCP connections
Talk to it Without writing a client
Similar to above server we can write a client as well (that I leaves on you to try out and post it in comments if you are able to write one yourself). But for the simplicity of this excercise we will interact with the protocol using Netcat.
Open another terminal (keep running the server.go ) and run following command:
nc localhost 9000
then type:
WANT /hello
the server should respond:
OK 17
Hello From ATTP:)
Congratulations!!
you have just created your own application-layer protocol.
Yeahhh…. I know it is tiny and only understand one method and one route, but the basic idea is exactly the same.
- A client sends bytes.
- Both sides agree on the format of those bytes.
- The server parses the message according to that agreement.
- The server sends back another message following another agreed format.
That agreement itself is the protocol.
An important detail I want you to understand.
- TCP itself does not understand messages.
- TCP only gives us a stream of bytes.
For example, TCP does not know that this:
OK 17
Hello From ATTP:)
is one complete ATTP response.
That meaning comes entirely from the rules we designed.
that 17 is especially useful because it tells a client exactly how many bytes it should read for the body.
Without some kind of framing mechanism like:
- a body length,
- a delimiter,
- or closing the connection,
the client would not necessarily know where one message ends.
So… the question is did we really create a protocol?
Yes.
But at the same time ATTP/1.0 is nowhere close to HTTP.
HTTP supports things like:
- headers
- many request methods
- status codes
- caching
- content types
- authentication
- persistent connections
- proxies
- compression
- cookies
- and a ridiculous number of edge cases accumulated over decades
But HTTP started from the same fundamental idea:
Programs agree on how message should be formatted and interpreted.
Our tiny protocol already defines:
Transport: TCP
Port: 9000
**Message format:
**Request: <Method> <Path>\n
_Response:
<Status> <Body Length>\n
\n
Methods:
WANT
Statuses:
OK
NOT_FOUND
ERROR
And one route:
/hello
That is enough to call ATTP/1.0 a protocol.
But keep in mind that there is no committee, no RFC, no 500-page specification.
There are just two programs following the same rules we defined in the starting.
The Problem
Our current server works perfectly for this example:
WANT /hello
But what about the following cases?
- What if I want to send multiple requests over the same TCP connection?
- What if the body containes multiple lines?
- What if the request itself needs a body?
- What if I want to send JSON?
- What if two versions of ATTP behave differently (breaking changes)?
- What if somebody sends: GIVE_ME_PIZZA /hello ?
- What if the client sends only half of the message and the remaining bytes arrive slightly later?
Welcome to protocol design :)
So what I want you to know and learn is that basic idea behind a protocol is simple: Two programs agree on how bytes should be formatted and interpreted.
But in real world these protocols are not limited to the above idea only. For example for a good protocol you need to think about message boundaries, errors, versioning, connection handling, security, timeouts, encoding, and backward compatibility, etc.
And this is exactly why I am intentionally stopping this exercise here.
The goal was not to build the next HTTP.
The goal was to understand what a protocol fundamentally is. And with just a TCP server, a few formatting rules we can create our own protocol.
Maybe someday in another exercise one of you might build ATTP/2.0 with headers, request bodies, persistent connections, version negotiation, and all the other things that make protocol designers question their life choices.
But for not:
WANT /hello
is enough :)
