Soweli Lukin

Protocol pondering intensifies, Pt II

In the previous post in this series 1 I compared request formats for gopher and HTTP and thought a bit about what a good anonymous document system actually needs. I ended up deciding that the answer was nothing more than gopher already provides. In this post I'll continue that discussion, focussing instead on the response format.

Recall that a gopher server's response to a request consists of nothing more than the content. What does HTTP look like? Here's a quite light real world example, obtained by requesting the /index.html path from

--------- HTTP/1.0 200 OK Server
nginx Date
Fri, 14 Jun 2019 19:16:18 GMT Content-Type
text/html Content-Length
45 Last-Modified
Sat, 21 Apr 2018 12:23:32 GMT Connection
close ETag
"5adb2d44-2d" Accept-Ranges

It works!

The very first part, HTTP/1.0, is of course the protocol version. Notice that this was the last component of the request, but it's the first component of the response. What's all that about? Anyhow, in general I think it makes a heck of a lot of a sense for a response to a request to use the same protocol version as the request, which the client of course is already aware of, so this is dead weight. The next part, "200", is a status code, indicating whether or not the request was successful or triggered an error. It's followed by a human-friendly version of the machine-friendly status code, in this case simply "OK". There are lots of lots of status codes in HTTP 2!

Then we have a bunch of headers, which look just like the request headers from last post. There's a lot of dead weight in here, for simple purposes. The "Date:" is in there for cache-related reasons. In a protocol without caching, this is useless. Specifying the "Server:" software and version serves no useful purposes, and many webserver admins actually disable this feature to avoid giving away hints about which vulnerabilities might be applicable to their server. The "Content-Length:" is useful for when a single TCP/IP connection is used for multiple request/response pairs. There's some overhead involved in setting up and tearing down these connections, and as webpages started to trigger more and more requests - to fetch stylesheets, and scripts, and images - this overhead added up to a non-trivial part of total time for a website to render. Re-using connections is one solution to this, and it means that the client needs to know when the server is done responding, and it can do this by counting bytes until the entire "Content-Length" has been received. A better solution to this problem is to Stop Making So Many Damn Requests, which means the server can signal the end of the content by just closing the connection, rendering this header useless.

Is there anything of value in here? I think so! The status code is interesting. Did you know that gopher has no real way to signal an error? You might be thinking "Hey, what about item type 3?", but the thing about item type 3 is, well, it's an item type. When do we see item types? In gopher menus, and in gopher menus only. If a gopher client sends a request for what it thinks should be a text file, but it's followed a misspelled selector and the file doesn't exist, the client isn't going to try to parse the response as a menu, so it's not going to have any way to recognise the error item type. Indeed, if you request a non-existent selector from a gopher server, it'll say something to you like "Error: File or directory not found!" (this is what Gophernicus will say), but it's only you, as a human, who hopefully reads English, who can recognise this as an error. A simple script has no way to distinguish this situation from a totally successful transaction. Because of this, it's e.g. impossible to write a script to crawl a gopherhole looking for broken links. Well, maybe not impossible, but certainly non-trivial: you could figure out the particular server's idiosyncratic choice of error message by requesting a couple of randomly-generated long selectors which are highly unlikely to be in actual use, and use the most common response as the "404 equivalent string". Needless to say, this is not exactly simple. It's perhaps not the biggest problem in the world, but it's certainly a shortcoming of gopher which could very easily be avoided.

But much more interesting and important is the "Content-Type" header. Gopher, frankly, sucks at signalling content type. If you've arrived at a document via a gopher menu, then you know its item type. What if you want to request a document directly, not by following an item in a menu? Maybe your friend has told you the selector in an email or via XMPP. Maybe you bookmarked it last month. You can request that document by just sending the path and a , but how do you know what kind of content you're getting back? If you don't somehow know it in advance, you need to figure it out for yourself by looking at it hard ("you" here are a gopher client, not an end user). This is the reason that gopher has its very own unique URL scheme with its own RFC (RFC4266), where the itemtype is introduced as an extra component of the path. You need to write gopher:// instead of just gopher:// because with the later option your client would have no idea whether or not it should try to parse what comes back as a menu, display it as text or save it as a binary file. This problem is also the reason that if you write a gopher client with bookmark support, you need to store the item type along with host, port and selector. Neither of these things are terribly hard, but they are examples of small, inelegant extra hoops which have to be jumped through because gopher, in this respect, is too simple. It's too simple to straightforwardly handle a perfectly reasonable situation like "I'd like to fetch this document from this server but I've never seen it appear in a menu because my friend just emailed me the link". To me it makes a lot of sense that the only piece of information you should need to request and then make use of a resource is that resource's path. That seems, well, simple.

This problem in gopher is more widespread than just not knowing what item type a document is. Even if you know that a path points to an item type 0 text file, you can have problems. One of the earliest bug reports I got after releasing VF-1 turned out to be the result of using iso-8859-1 text encoding to support accented characters in some of their content. VF-1 had just assumed that everything on gopher was ASCII, which turned out to be very wrong. There are a lot of encodings out in the wild on gopher. Standard gopher has no way of telling you what they are. The only way to write a client which can Just Go Anyway is to user some kind of third party party library to try to "sniff" the encoding (VF-1 uses Chardet 3 for this). That's a hard problem, which is never guaranteed to be solvable, and is only possible using a big slab of natural language corpus statistics. This requirement massively flies in the fact of the RFC1436-enshrined philosophy that "intelligence is held by the server". When all a protocol does is shovel a bunch of bytes down your throat and say "you figure out what this is and what to do with it!", you need a very intelligent client for it to really work out in all conditions. I don't think it makes much sense to have every client repeat exactly the same set of expensive computations after requesting a document in order to figure out information that the server already knows, but didn't share.

There's a saner alternative to this, and it's for the server to tell the client, succinctly, what it's actually getting. This can be implemented with a very small increase in protocol complexity, which can result in a very large decrease in client complexity.

Consider the following as a response format, in a hypothetical protocol which retains gopher's bare bones request format:

A concrete example:

200 text/plain  utf-8
Hello, world!

The text encoding could be optional for non-text MIME types. We could get away from having to specify an encoding at all if this protocol specified "Thou shalt use UTF-8 and no other encoding shalt thou use", saving us ~5 bytes, but I dunno if that's too authoritarian. Yes, you can represent any language you like in UTF-8, but some languages can be represented more compactly in other encodings, and it seems like a good thing to provide the ability to minimise the number of bytes sent over the network. Isn't that also part of the spirit of a minimalist protocol? A compromise: if you use UTF-8, it's valid to leave off the third component of the response header. UTF-8 is the implicit default, but other encodings are possible for a tiny extra cost.

For the sake of fully specifying a system, including a navigation solution, without any further discussion or design, let's keep gopher's menu system as is, and introduce a new pseudo-MIME type for it, like text/menu or something. I'm not saying this is a great idea, it just provides a complete concrete example to talk about for the rest of this post.

If we give gopher a complexity score of 1 and full-blown HTTP a complexity score of 100, I don't see how this new protocol can be reasonably scored higher than 10. It's still absolutely trivial to write a client for this protocol, a nice little weekend project. You can memorise the protocol easily so you don't need to look up a complicated RFC to remind yourself of some detail while coding. You can still cobble together a client out of standard unix utilities: the response header is guaranteed to be one line long, so you can just pipe what you get from the network through tail -n +2 to cut it off. I'm not sure if that would work for binary files, admittedly, but for something vaguely gopher-like that's an edge case anyway. You could even still use telnet as a client for this protocol if you wanted to. Yes, you would see one short line of noise at the top of each file, but that's a heck of a lot better than seeing a full set of HTTP headers and I guarantee you'd get used to it and stop even consciously seeing them after a day of practice. None of the extra information in this header represents any threat to a user's privacy. The network overhead is around 20 bytes per request, which is less than 1% of the size of a typical phlog post.

Compared to gopher, this protocol can:

This protocol is not as simple as gopher, but I would argue its power to weight ratio is substantially greater. It's still very simple, and its still totally harmless. Crucially, it's non-extensible: the response header is not open ended, like HTTP's is, so people can't just add in whatever extra junk they like. I don't want to say that extensibility is a bad thing, it's often a very smart engineering solution to some particular problem, but I think I do want to say that extensibility is the enemy of intentionally brutal simplicity. Optional extra cruft will inevitably accumulate and then become a de facto requirement.

In the third and final post in this series, I'll address possible solutions to the problem of navigation.

Soweli LukinIssuesAlex Schroeder