After talking to Alex Schroeder about an upload protocol for Gemini I have realized, that his version of the protocol is not quite what I envisioned, and that my version of the protocol is not clearly defined, this document is an attempt to define my protocol:
My goals for this protocol are:
This protocol is intended to be used on the same port as spec compliant Gemini server as a companion protocol to allow for uploading files, nothing less and nothing more.
Uploading a file is divided into several stages: (Get it? Gemini? Multistage rocket to get something into space?)
The client opens a connection and writes the following line:
\t meaning an ASCII horizontal tab caracter
To which it expects a response in the format of a regular Gemini response header except, that it also accepts non numeric response codes. The responses can be sorted into three categorys. Any non success response code will cause the upload to fail, which MUST be displayed to the user!
If the response code is a “WR” the client knows, that it can carry on to the next stage, the
<META> part MUST be ignored by the client and MUST be left blank by the server it MAY NOT even be there.
In case of the “WR” response code the server MUST not drop any data, that was sent in the time between the client finishing the request and the response being send. This means that a simple client (like the commandline example) does not have to wait for the WR response, but can start writing immediately after sending the request, the worst that will happen is that the server will ignore what the client says if an error comes up.
The upload protocol defines a few other response codes:
ES - The file will not be accepted because it is too large, the
<META> part of the response MUST contain the maximum allowed filesize.
EM - The file will not be accepted because of it’s MIME-type
<META> contains a message which should be displayed to the user.
E_ - The file will not be accepted
<META> contains a message which should be displayed to the user.
A smarter client could show the user the error and let the user retry the upload with a different file. A less smart client can get away with looking at the first character of the response and tell the user that the file was rejected. After sending an Ex code the server MUST skip to the third stage and ignore any data the client sends, the server MAY close the receiving side of its socket, if the underlying network permits that. When receiving an Ex code the client MUST skip the second stage, or MAY close the connection.
A range of official response codes MAY be used to indicate various errors at this stage, all other codes should be treated like unknown Gemini response codes:
3x - the client SHOULD ask the user if redirecting to the same host, it MUST ask if the upload will go to a different host, if the redirect leads to a gemini+upload: uri the client MAY ask the user if they want to retry uploading the file
4x - the codes mean exactly the same as defined in the gemini spec, a client should offer the option to retry the upload
41 - The sever SHOULD use this code for indicating a temporary lack of enough storage (a full disk) as it fits within the definition of an overload, and tells the client, that it can retry in a few hours
5x - the codes mean exactly the same as defined in the gemini spec, the resource being the ability to upload a file
53 - the server will use this one or 59 to indicate, that it does not support this protocol at all, bots should stop trying to upload to the server
59 - the server will use this one or 53 to indicate, that it does not support this protocol at all, bots should stop trying to upload to the server
6x - the codes mean exactly the same as defined in the Gemini spec, a client should offer the option to retry the upload after asking for a certificate
NOTE: The reason for not including the
1x range in the list of valid response codes is to satisfy the non-interactive upload requirement, if you want to ask the user something do it using normal Gemini and give them the upload link after they respond (and if you really want to, you can point to such an input with a redirect back to a
gemini:// URI) This is intended to make it inconvenient to use this protocol for delivering content or asking the user for things which is what the Gemini protocol is for.
The server MUST close the connection after sending a response using a official code.
2x codes MUST be treated like unknown codes in this context! A client should close the connection after receiving an unknown code!
In the second stage the client simply sends the file to the server, by simply throwing it down the pipe. When all bytes are received, the server has two options of replying to the client with a line that again looks like a Gemini response:
The two possible status codes are:
OK - The file was uploaded successfully,
<META> SHOULD contain the URI to use for downloading the file again, it MAY NOT contain anything else, at this point the client can consider the upload a success
EC - The file was rejected because of its content,
<META> MAY contain a message that a client SHOULD display to the user
NOTE: The EC code was previously an E_ code, but if a client decides to use the statuscodes directly, it would have had no way of telling it from an E_ general error and would either have to guess or store the fact, that it uploaded the file (this is only a minor detail, but it should make it easier to implement a very simple client and reusing statuscodes is not friendly to people who try to debug software)
These codes will tell any client if the upload was a success independent of possible content that the server sends for a human reader in stage 3. If the client receives an invalid response code it MUST assume, that the upload failed, close the connection and tell the user.
After indicating that an upload was successful a server immediately sends another Gemini header line. This one does not use custom codes and only allows a limited rand of the official ones.
The allowed ones are:
1x - handle as specified by the Gemini specification
2x - handle as specified by the Gemini specification
3x - handle as specified by the Gemini specification
40 - indicates that the server won’t send a response, the
<META> section should be empty and be ignored by the client.
All other response codes, have to be treated as unknown which in this case means that no relevant information will follow and BOTH server and client MUST close the connection.
NOTE: When sending a
2x response please keep in mind, that some clients will only have a cached copy of the file they currently display, and that this will be lost as soon as the user leaves the page (reloading is not practical as the client would have to reupload the file to get the page again)
NOTE: When using
1x codes to authenticate a file upload after it finished you are doing it wrong! These are intended for asking the user about a filename, tags or something similar.
Most readers will have noticed by now, that this protocol reserved no space for authentication, that is, because Gemini already provides the option to have a client certificate, which is supported by this protocol by making the
6x range of response codes a valid return codes. An alternative would be to use “public” endpoints with an access-token embedded in the uri.
The thought behind leaving out a dedicated authentication field is, that Gemini already supports ways to tell the server for example a username and or password by entering them into Gemini input fields (response code
NOTE: A server MAY use any of the above authentication schemes in any combination (including none at all or both)
NOTE: A format that utilizes this protocol to upload text only forms would also be great to have, as such a thing allows the user to know all the questions, take their time to answer them and use one request to tell the server all the answers (similar to the
gopher+write protocol specified in 2017-12-30 Gopher Wiki or what gopher+ specifies).
To avoid a collision with Alex’ proposal I suggest using the following scheme for this protocol
How the path and query are used is left to the imagination of the reader.
How exactly a Gemini response header looks like is defined by the Gemini specification in section 3.1. At the time of writing (2020-06-05) the Gemini specification had the following to say about response headers:
Gemini response headers look like this:
<STATUS>does not belong to the “SUCCESS” range of codes, then the server MUST close the connection after sending the header and MUST NOT send a response body.
If a server sends a
<STATUS>which is not a two-digit number or a
<META>which exceeds 1024 bytes in length, the client SHOULD close the connection and disregard the response header, informing the user of an error.
Stage 1 C: opens a TLS connection with a valid client certificate to port 1965 of example.org C: gemini+upload://drop.example.org/upload 1035 text/plain<CR><LF> S: WR<CR><LF> Stage 2 C: *sends 1035 bytes of plain text* S: OK gemini://drop.example.org/files/2020-06/9a8d4186-90c8-4d2b-a296-b4990973892f<CR><LF> Stage 3 S: 20 text/gemini<CR><LF> S: #Your upload was successful <LF> S: The sha256 hash of your file is: 183f4e791e9d4f69d01c0fa67e9a7f6fa2cef48663555dc27114d144307b1f24 <LF> S: => gemini://drop.example.org/files/2020-06/9a8d4186-90c8-4d2b-a296-b4990973892f You can find your file here<LF> S: => gemini://drop.example.org/files/2020-06/9a8d4186-90c8-4d2b-a296-b4990973892f/delete?cd76f8a7-6ae5-45cd-9fe4-80cd6af0735d<LF> S: *closes the connection*
Note, that the client only sends the URI with the metadata and the file-contents for which it doesn’t even have to wait for a response from the server.
Stage 1 C: opens a TLS connection without a client certificate to port 1965 of example.org C: gemini+upload://drop.example.org/upload?uplo-ad_t-oken 1035 text/plain<CR><LF> S: WR<CR><LF> Stage 2 C: *sends 1035 bytes of plain text* S: OK gemini://drop.example.org/files/2020-06/9a8d4186-90c8-4d2b-a296-b4990973892f<CR><LF> Stage 3 S: 20 text/gemini<CR><LF> S: *sends response body* S: *closes the connection*
NOTE: sending a redirect on a successful upload is possible, just currently in none of the examples
Stage 1 C: opens a TLS connection without a client certificate to port 1965 of example.org C: gemini+upload://drop.example.org/upload?uplo-ad_t-oken 999999999 image/png<CR><LF> S: ES 50000000<CR><LF> Stage 2 gets skipped because of the Ex statuscode Stage 3 S: 30 gemini://drop.example.org/uploadguide<CR><LF> S: *closes the connection*
NOTE: The maximum size is inclusive, so attempting to upload exactly 50000000 Byte (50 MB) would have been okay
Stage 1 C: opens a TLS connection without a client certificate to port 1965 of example.org C: gemini+upload://drop.example.org/upload?uplo-ad_t-oken 34782 application/vnd.microsoft.portable-executable<CR><LF> S: EM Get out, who do you think you are? Trying to upload a microsoft executable ... rude ...<CR><LF> Stage 2 gets skipped because of the Ex statuscode Stage 3 S: 20 gemini://drop.example.org/uploadguide<CR><LF> S: To prevent the spread of malware to unsuspecting users, we have banned the direct uploading executables S: If you really have to share an executable, precompiled library or similar try packaging them in an archive of some kind. S: Uploading the sourcecode instead of the application is encouraged S: => gemini://drop.example.org/uploadguide You can find our upload guide here S: *closes the connection*
C: opens a TLS connection without a client certificate to port 1965 of example.org C: gemini+upload://image.example.org/upload?uplo-ad_t-oken 34782 image/png<CR><LF> S: WR<CR><LF> Stage 2 C: *uploads an image with a broken header* S: EC Unfortunately, the file you uploaded is not an image Stage 3 S: 20 gemini://drop.example.org/uploadguide<CR><LF> S: The file you uploaded did not look like an image. S: Either the images format did not match the mimetype "text/png" or the file was damaged S: Please verify, that the image is in the right format, repair it S: Hint: Opening it in an image editing program and saving it again can help in a lot of cases S: => gemini://image.example.org/uploadguide You can find our upload guide here S: *closes the connection*
This example demonstrates how a server can use existing gemini mechanisms for allowing the user to upload a file after answering a captcha question
This example will use a randomly generated token in the uri as a kind of very limited cookie (more like an on the fly generated API key)
A page containing a link that will lead to the upload as an entry point
URI: gemini://drop.example.org 20 text/gemini *blah blah blah* => gemini://drop.example.org/upload Upload a file *blah blah blah*
A redirect, to a page that will ask the user the captcha question. This redirect should just generate a random token and not set any state in the server
URI: gemini://drop.example.org/upload 30 gemini://drop.example.org/validate/rand-dom_-code
The user gets asked a captcha question (the server should at this point link the token to the correct answer and an expiry date) giving a correct answer will validate the code for uploading a file
URI: gemini://drop.example.org/validate/rand-dom_-code 10 What is bigger on the inside than on the outside?
NOTE: At this point the server could ask for a username or other information
For a correct answer the user will get redirected to an upload uri already containing the now valid token ...
URI: gemini://drop.example.org/validate/rand-dom_-code?tardis 30 gemini+upload://drop.example.org/upload?rand-dom_-code
..... which can be used to upload a file.
URI: gemini+upload://drop.example.org/upload?rand-dom_-code *a file upload as described above*
For an incorrect answer the server can serve an error page, and delete the token and the associated information.
URI: gemini://drop.example.org/validate/rand-dom_-code?no%20idea 20 text/gemini Wrong answer! => gemini://drop.example.org/validate/rand-dom_-code Try again!
NOTE: The validation thingy should always ask a random question, however the server may ask the same question for the same token until it gets validated and used or it expires
NOTE: since these codes are temporary then could be kept in ram by the server, keep in mind that take space and creating a lot of them without ever validating them is a possible DOS attack (to prevent crawlers from getting trapped in anything consider blacklisting the entire validate, upload construct in you servers robots.txt)
This one uploads the file
test.txt to the URI
gemini+upload://localhost/Test using an unencrypted connection (replace
gnutls-cli --insecure to encrypt it).
(sleep 1 \ && echo -e "gemini+upload://localhost/Test\t"$(wc -c < test.txt)"\t"$(file --brief --mime-type test.txt) \ && sleep .01 \ && cat test.txt) \ | nc localhost 1965
Stage 1 ======================================================================= ┏━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Client opens connection ┃ ┗━┯━━━━━━━━━━━━━━━━━━━━━━━┛ │ ┏━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Client sends Uri, filesize and mimetype ┃ ┗━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ │ ┏━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Server sends response ┃ ┠────┬────────────┬────┬─────────┨ ┃ WR │ official │ Ex │ invalid ┃ ┃ │ error code │ │ code ┃ ┗━┯━━┷━━━━┯━━━━━━━┷━┯━━┷━━━━┯━━━━┛ │ │ │ └──────────────┐ │ │ └──────────────────┐ │ │ │ │ │ │ ┏━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │ │ │ ┃ Client shows an error message ┃ │ │ │ ┗━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━┛ │ │ │ │ │ │ │ ┏━━━━┷━━━━━━━━━━━━━┓ │ │ │ ┃ Cient and Server ┃ │ │ │ ┃ close connection ┃ │ │ │ ┗━━━━━━━━━━━━━━━━━━┛ │ │ Stage 2 ==│====================================│===│=========================== │ │ │ ┏━┷━━━━━━━━━━━━━━━━━┓ │ │ ┃ Client sends file ┃ │ │ ┗━┯━━━━━━━━━━━━━━━━━┛ │ │ │ │ │ ┏━┷━━━━━━━━━━━━━━━━━━━━━━┓ │ │ ┃ Server sends response ┃ │ │ ┠────┬────┬──────────────┨ │ │ ┃ OK │ EC │ invalid code ┃ │ │ ┗━┯━━┷━━┯━┷━━━━━━━━━━━━┯━┛ │ │ │ │ └───────────────│─. │ │ │ │ \│ │ │ │ │ ==│=====│==============================│===│=========================== │ │ │ │ │ ┏━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━┓ │ │ ┃ Client shows an error message ┃ │ │ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ │ │ │ │ Stage 3 ==│=====│==================================│=========================== │ │ │ ┏━┷━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┓ │ ┃ Server sends response ┃ │ ┠────┬──────────┬──────────────┨ │ ┃ 2x │ 1x 3x 40 │ invalid code ┃ │ ┗━┯━━┷━━━━┯━━━━━┷━━━━━━━━━━━━┯━┛ │ │ │ └───────────. │ │ └───────────┐ \│ │ │ │ ┏━┷━━━━━━━━━━━━━━━━━┓ │ │ ┃ Server sends file ┃ │ │ ┗━┯━━━━━━━━━━━━━━━━━┛ │ │ │ │ │ ┏━┷━━━━━━━━━━━━━━━━━━━┷━━━━┓ │ ┃ Client displays response ┃ │ ┗━┯━━━━━━━━━━━━━━━━━━━━━━━━┛ │ │ │ ==│========================================│=========================== │ │ │ ┏━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━┓ │ ┃ Client shows an error message ┃ │ ┗━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━┛ │ │ ┏━┷━━━━━━━━━━━━━━━━━━━━━━━━┓ ┏━━━━━┷━━━━━━━━━━━━━━━━━━━━┓ ┃ Server closes connection ┃ ┃ Client closes connection ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━┛