2020-06-04 Gemini Upload

Update: this proposal has been superceded by Titan. There’s a short demo available.


I’m talking with @baschdel on Mastodon about a possible extension to Gemini that would allow us to return to the read-write web. So how about a protocol variant via the scheme?

gemini+write://alexschroeder.ch/Test → an attempt to write the resource that would be served by requesting gemini://alexschroeder.ch/Test – and we’d like to indicate a MIME type of the data and length (important because the server can’t tell when an upload is finished unless we mandate a timeout). How about simply sending that along on separate lines?

(And, as suggested by deshipu in the comments below, an authorization token!)

My goal, as always, is to make this work from the command line first. Here’s how it might work:

(sleep 1 \
 && echo "gemini+write://alexschroeder.ch/Test" \
 && echo "hallo" \
 && file --brief --mime-type test.txt \
 && wc --bytes < test.txt \
 && cat test.txt) \
 | nc alexschroeder.ch 1965

If the server doesn’t accept uploads, it can close the connection after discovering the scheme “gemini+write”, or when it discovers an unsupported MIME-type, or when it discovers that the size of the upload exceeds its own restrictions.

The status code 59 seems to be appropriate for any such refusals. A super well-behaved client might then peek at the socket and see whether any error message is forthcoming before sending the whole file... And if they they’re not waiting like in my command-line example above, then no harm done: the client is simply going to get a write error on the socket as the server drops the connection after sending the error message.

At first I thought that if the server accepts the upload and saves the data, a new OK status code would be nice to have. Something like 29 CREATED (the equivalent of the HTTP status code 201, perhaps). But when I implemented a client for Gemini+Write, I realized that a simple redirect on a successful save is good enough. There is no need for a “created” status 🙂

As you can see in the comments, Baschdel has different suggestion. His most important objection, as far as I can tell, are the missing codes. He would like the server to indicate whether it allows uploads or not, whether the MIME type is supported or not, whether the size exceeds some limit or not, and so on. There’s a lot more info on the page Baschdels spin on Gemini uploading.

Tags:

Comments

How about some kind of optional auth? A lot of wikis had to add this to avoid spam, maybe it would make sense to leave a provision for that from the start? If you leave room for some kind of auth token, you can leave the details of obtaining the token to the implementation (might be a simple password, or may be some kind of one-time token).

deshipu 2020-06-04 15:46 UTC


When I was working on a Gopher front-end, I had decided to use a fenced metadata block, like this:

```
username: Alex
summary: a different Haiku
```
Cold and dark outside
Remember all the sick beds
Good riddance old year
.

Right now I’m thinking that Recent changes with my colour coded IP number will work just as well: translate IP numbers edits into a colour code, using a hash function to turn the IP number into a 32bit number, convert it to octal, take the first four digits and show one of eight colours for each one. As many of the Campaign Wiki users don’t provide a username, you can see the effect on Recent Changes for Campaign Wiki. What do you think?

As for secure authentication, Gemini says you should be using client certificates, I think.

– Alex Schroeder 2020-06-04 18:21 UTC


Actually, now that I have working code, I realize that my wiki does not allow me to save because I haven’t answered the security question which is exactly what you said: an authorization token. 😅

– Alex Schroeder 2020-06-04 19:03 UTC


Right, the issue is not recognizing who is who, the issue is limiting who can post. Certificates don’t strike me as a particularly good or simple solution to that problem.

deshipu 2020-06-04 19:09 UTC


On the mailing-list, Sean Conner wrote:

unless you think you would be using the query portion of the URL for something else, why not indicate meta information there?

gemini+write://alexschroeder.ch/Test?size=5612&mime=text/markdown&authtoken=foo

Or even better (as it too, is part of the spec)—use the ’;’ to designate the information on the last part of the path (the ’;’ is classified as a sub-delimeter and in the earlier URL RFCs was specifically called out as such for the path portion):

gemini+write://alexschroeder.ch/Test;size=5612;mime=text/markdown;authtoken=foo

That way, you still have the query portion for other uses.

I’m not sure about making the authtoken as easy to use. Somebody could write a crawler that would work for every instance of my software. Then again, who am I kidding. It’s probably going to remain the only instance. But still. I guess I’m envisioning a client where people have to provide a little codeword or password and that allows them to use the site and it’s not clear where they get it from. It could be on the top page, it could be something they need to get from the author. Oddmuse passwords work the same way. There are no usernames, just one or more passwords that you can hand out to people you trust.

I’m also not sure about adding the MIME type to the URL because that would only work for uploads of a known type. On this wiki, however, a page can hold any MIME type, as long as I have it on my whitelist. That’s why a page may contain either text or an image, for example.

At least Sean likes that I picked a different URL scheme. That was actually @baschdel’s idea. 🙂

Anyway, I think I’m going to be working on this for a little bit more.

– Alex Schroeder 2020-06-04 22:35 UTC


I was able to edit a page on the wiki using the command line and the gemini+write protocol. 😀

– Alex Schroeder 2020-06-04 22:49 UTC


As a client developer, I really like the idea of putting the information about the file itself into (a) separate line(s) because that way way it can be easily appended by the code that handles the uploads. But I’m strongly opposed to putting an auth-token into these lines, as the token is not something, that can be derived from the file, but is input needed from the user. As the user already supplies the URI as the upload destination it is (for me at least) is the natural place to put such a thing, the main advantage being, that the uploader does not even have to be interactive when you give it a file and an URI.

A simple way of getting the token into the URI without having to ask the user to edit the URI itself would be to chaining up a Gemini input (status code 10) with a redirect to the upload-URI with valid token, if the correct information is given. For anything non temporary or more secure Gemini already uses client certificates.

Using this approach also makes the protocol re-useable and people can start using it for file uploads to their gemlogs or a Gemini-board or whatever they can come up with. (I would use it if I had a server)

I’ll follow up with a file on how I’d implement such a thing

PS. Writing this in the browser without the clutter of a webpage actually feels quite comfortable

– baschdel 2020-06-05 14:02 UTC


Here is the promised proposal: Baschdels spin on Gemini uploading. The main differences to this one is, that it does not have an authentication field (authentication either happens trough client certificates or trough a token included in the upload URI), puts all the metadata on the same line as the URI, and the server will send a lot of metadata about the (possible) success of the upload

– baschdel 2020-06-05 21:07 UTC


Looking at this again, and Sean’s idea of putting information after a semicolon, ... If we turn that around I can still have my only single URI and a file needed for uploading thing without modifying the protocol itself. At least that would solve the “I have to ask the user for an answer without giving them a question” part (when combining it with inputs and redirects).

gemini+write://<host>:<port>/<path>?<query>;<token>

Or butcher the index as it doesn’t really make sense when uploading

gemini+write://<host>:<port>/<path>?<query>#<token>

Just throwing ideas around (I’ll probably end up implementing gemini+write and gemini+upload into my client while trying to keep both of them first class citizens, but I’ll have to turn uploads into first class citizens, before I can do that, so I won’t be able to show code too soon)

However I’d really like some kind of response from the server, just so that I don’t have to make assumptions about why the upload was rejected (Having to say upload success = unknown feels wrong and telling the user that something went wrong for an unknown reason, or worse not even knowing that something went wrong is never good design and will annoy the user)

– baschdel 2020-06-10 01:32 UTC


Thanks for linking to my proposal, I updated the page of mine to link back to yours too.

I have also implemented both protocols into my gtk based client wich currently goes by the codename dragonstone. You can find its sourcecode here.

– baschdel 2020-06-13 16:33 UTC


I just realized, that I forgot to mention how to get dragonstone to use a non-empty token:

simply use the following uri scheme: gemini+write://<authority>/<path>?<token>

<token> should be the uri encoded token (sould it also be transmitted uri encoded or more like a gopher selector?)

It currently gets transmitted uri encoded because nothing was specified, the only usecase out there doesn’t care (meaning: there is nothing that should/must be encoded in the token)

– baschdel 2020-06-14 01:16 UTC


I think it must be URI encoded, because it’s part of the URI. Or to look at it another way: a space cannot be part of the URI therefore the space must be encoded.

– Alex Schroeder 2020-06-14 07:43 UTC


Just saw Sean’s Titan proposal on the mailing list (the name was decided upon in a later email). It’s an interesting thread.

– Alex Schroeder 2020-06-14 08:38 UTC


I read so much I disagree with on the mailing list. 😢

Solderpunk on facilitating posting for everybody:

I’m wary that facilitating “user friendly” Gemini hosts which anybody can post to using a client carries a very real risk of fostering a culture of dependency where people don’t know how to do anything not offered as a feature by their provider, who has the ability to change, remove, or charge for those featurs, and also of pushing us toward a more centralised Geminispace where large numbers of users congregate on a small number of servers which make themselves especially appealing to non-technical users. These servers could easily become little semi-closed gardens with internal-only comment notification facilities and other such niceties which work much more smoothly than the various decntralised solutions people have already been talking about. They might not, but the risk is there: we might just end up creating new versions of LiveJournal, Wordpress, Blogspot, etc, etc.

Solderpunk on embracing clunkiness:

It wouldn’t necessarily be the smoothest experience in the world, but if most content could be written in a single request and 99% with one or two “append” requests, I think it would be usable enough. Heck, this is the “slow internet”, right? A little bit of inconvenience as part of a careful and deliberate process should not scare us away.

Solderpunk on non-technical authors:

It should also be very easy to write an app targetted at “non-technical” authors which lets them submit chunks of writing up to 2 KiB or so, with an “append” link at the submission confirmation page to submit a follow up chunk. It wouldn’t necessarily be the smoothest experience in the world, but if most content could be written in a single request and 99% with one or two “append” requests, I think it would be usable enough.

Solderpunk on this being a text-first protocol:

I realise this works a heck of a lot better for textual content than base64-encoded binary content, where uploading, say, a JPG photograph would require many, many chunks. I think I’m okay with this. Gemini is deliberately and unapologetically a text-first protocol.

Here’s what I really liked, however! 😃

Sean Conner on the original idea of the read & write web:

One of Sir Tim Berners-Lee’s initial ideas for HTTP was to allow the user to edit the page in the browser and have that update the file on the server. HTTP didn’t get this capability until May of 1996 [1] where various methods were defined that included PUT and DELETE. It’s sad to think that no browsers at that time even attempted to do that [2]. Instead, we kept on using FTP, later sftp and maybe even rsync for the technically minded.

Sean Conner explaining the problems with letting everybody self-host:

As someone who has worked for various ISPs and webhosting companies for most of my career, I think this slamming of IPSs is unwaranted. And as someone who runs both a public server and a few services on my home network there are some things you need to consider. [...]

Sean Conner on the Titan protocol:

Upon reflection, given the semantic meanings involved, I can cut the number of new schemes down to just one: “titan:”. Here are the three examples again:

titan://example.com/post-handler/endpoint?size=1234&mime=text/plain
titan://example.com/path/to/new/resource;size=1234&mime=text/plain
titan://example.com/path/to/remove;size=0

The logic goes something like this:

  • if the request has a query, it’s an upload of data—accept data
  • if the request has no query, and the path parameter (marked by ’;’) doesn’t exist—error
  • if the request has no query, and the path parameter exists:
    • if size==0, delete the resource
    • if size>0, accept data and make the resource available

Sean Conner providing references for Titan:

This proposal is mostly based upon a new URL scheme, and I’ve spent the day diving deep into RFC-7595 (URI Scheme Guidelines June 2015, current standard).

Thank you for all the hard work, Sean! 🙂

– Alex Schroeder 2020-06-14 18:42 UTC


I’ve switched my server and client over to Titan, by the way. The only addition I use is a token parameter, which is where clients must provide an answer to a security question before they are allowed to edit.

– Alex Schroeder 2020-06-14 18:43 UTC


I just catched up with the mailing list on gemini uploading and there are a few thoughs I have to put out there:

Seans titan protocol doesn’t look bad and looks like something I’d invest some time implementinhg it when there is a single file that I can refer to. I’m not really a friend of letting the client modify the uri itself (thats the reason I choose tabs) but with some infrastructure for uriparsing this turns into a few ifs and an append. (If youre reading this Sean, I’d be interested in how you modified GLV-1 to provie upload functionality, I thought of forking it to try out my own protocol but havn’t gottn arounfd to do it yet)

But then I saw a whole lot of upload protocols, that seemed like a very complicated way to negotiate the equivalent of $cat file | nc host 1234 without machine and or user readable feedback or feedback that didn’t make sense to me. So let me put that out there (not the first and not the last time): I want to know if my upload succeeded and if not I want to know that and why it failed and user readable feedback only seems enough until someone wants to write a dedicated client.

NOTE: When you only provide human readable feedback sooner or later someone will build something, that channels machine readable feedback on top of it and that will not only look ugly but will also require special sofware that knows the services conventions or a very flexible piece of software, that requires a lot of work to builin order to take advantge of the machine readable feedback (or “as machine readable as an html file”).

I also realized, that this brings us one step closer to the ... thing that we call the modern web, but I also think that as long as we have clearly defined standards and no servive specific clients (read: apps think: android/ios apps) or client side scripting we should be pretty safe. (Yes I know what that means, I thought the same same about the weather about 24 hours ago and got wet feet about six hours later) What I really want to avoid is every service having its own protocol (even if it just changes an existing standard a littbit by modifying a well-known magic value or similar) because everyone who wants that: Here’s a blank http header printed out on paper and a pen and now get out!

What I like about them though is that they all just require an uri a file and optionally a client cert for uploading.

That’s all for now, good night.

Ps. Using already existing primitives is a valid pint that solderpunk has, but chopping up the data and uploading it using a mechanism, that is based on convrntion using multiple connections with all the overhead and state maintining sounds like a nightmare on the server AND client side.

– baschdel 2020-06-15 00:30 UTC


PPS.: I though of making something for deleting files too (writing 0 bytes should produce an empty file). Seans proposal of using the 52 GONE as a success message looks okay to me, however that should forbid the other statuscodes that make no sense 1× 2× 3× so that when deleting clients treat the like unknown codes or clearly define what they mean. (has anyone thaought of a 20 no_content yet?)

– baschdel 2020-06-15 00:38 UTC


I’ve been trying to figure out how we could extend error messages for Titan and other upload proposals. You know that I suggested using 59 everywhere. You say you want to be more specific. I want to avoid clients that cannot handle unknown errors but if the error codes returned only occur in the context of Titan or an upload protocol, there’s no reason we can’t use new ones.

Looking at my implementation and your proposal, here are things I could do on my end:

  • use 53 to reject uploads for URLs that cannot be edited
  • use EM to reject uploads because of unsupported MIME types
  • use ES to reject uploads because the size exceeds the limit
  • use EC to reject uploads because of their content (spam, etc)

Did I miss anything?

How about “use ET to reject uploads because of an invalid token”?

– Alex Schroeder 2020-06-15 14:33 UTC


Then again, if Solderpunk is calling for a permanent freeze I guess that gives us leeway to put the error codes in the 50-59 range; and if somebody wants to get technical, we will say that this short exchange is for the Titan protocol and not the Gemini protocol, and thus the similarity is just incidental:

  • use 54 to reject uploads for URLs that cannot be edited
  • use 55 to reject uploads because of unsupported MIME types
  • use 56 to reject uploads because the size exceeds the limit
  • use 57 to reject uploads because of their content (spam, etc)

I also keep wondering about the 40–49 vs. 50–59 ranges.

  • The nature of the failure is temporary, i.e. an identical request MAY succeed in the future.
  • The nature of the failure is permanent, i.e. identical future requests will reliably fail for the same reason.

I mean, configurations can change... 🤔

– Alex Schroeder 2020-06-15 19:39 UTC


baschdel, for my “proof-of-concept” I did not modify GLV-1.12556, but instead wrote a new server that only supported gemini+put: and gemini+del: schemes, and ran it on another port. I don’t support proxying in GLV-1.12556, so I found it easier to just write a separate server.

As far as using 52 Gone to mark a successful deletion, I figure any other error code in the 4× or 5× range to indicate that the deletion was not successful (say, the resource doesn’t exist, or there’s an actual file system permission problem).

Back when solderpunk and I were hashing out the status codes, I was pushing for status codes based on HTTP (3-digit—GLV-1.12556 at the time even used HTTP status codes) and solderpunk was pushing 1 digit codes. He finally relented on using two digit codes [1] and for some reason changed the server error to the 4× range (temporary, since a server will be fixed) and the client errors to the 5× range (permanent errors). I find it kind of silly myself, but hey, progress was made.

[1] When I pointed out that what he was actually defining was a single-character status code, and people would use other characters, thus a potential 95 codes. A two digit code gives 100, but a way to segregate similar codes into a range.

Sean Conner 2020-06-15 23:42 UTC


Hah. I also remember suggesting HTTP status codes.

– Alex Schroeder 2020-06-16 08:27 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Just say HELLO