2018-09-12 OAuth and Mastodon

This page is mostly about trying to debug OAuth for Mastodon. Here’s the thing: I have a web application called Trunk. When you use it, you can mass-follow a number of accounts.

Sadly, I see OAuth issues piling up. I get the impression that my app mostly works, but every day I see reports by users getting “Client authentication failed due to unknown client, no client authentication included, or unsupported authentication method”. The 3 instances I looked at reported their version to be 2.5.0 according to /api/v1/instances. What do you think I should do? How to debug this?

This is why I started writing this page. I’m hoping that some Mastodon client developer can look at the flow of things and tell me where I’m doing it wrong. Or perhaps they are running into similar issues and found a way around it? Perhaps register the app every single time? Who knows. All I know is that I’m seeing multiple reports.

If you want to look at the source code while you read this, you can find it in my repository.

The basic overview of the OAuth flow is this:

  1. Create your app in the instance and make sure you use the same scopes and redirect URL later. This gives you the client credentials client_id and client_secret.
  2. Redirect your user to the instance using the same scopes, redirect URL, and client credentials.
  3. If all went well, you’ll your redirection URL gets visited with a code parameter. Transform it into an access token ASAP (using the same parameters as before).
  4. Use the token as bearer token for each call to the mastodon API.

Here’s what this looks like in my app.

You start out with /follow_confirm. The web site says:

Please confirm
We’re going to use kensanata@octodon.social to follow all the accounts in Test.
Let’s do this
If you’d like to help us find the authorisation problems we’ve been having, please use the following link instead:
Let’s log this

When you click the link, you’re sent to the /auth route. There, we’re going to get ourselves a client. We check if the client is already registered. We do this by checking our file with all the client credentials and look for the octodon.social credentials. If we don’t find them, we’re going to register the client. in this case this is not necessary.

The /auth route now has a client. We’re going to authorise it. This means storing a bunch of cookies in the browser and then redirecting to the oauth URL of octodon.social. If the browser has the octodon.social cookie set, then no action is required. Otherwise, I need to click the big Authorize button. octodon.social then redirects back to our application, i.e. the / route, with an authorisation code.

I think this is where people are getting the error: “Client authentication failed due to unknown client, no client authentication included, or unsupported authentication method”. They get the error from the instance, not from my web application, after all.

Screenshot of the error message

Anyway, assuming everything worked as intended... We’re back at / and we need to figure out what to do. If we are being called without a code, then we’re just going to show our main page. But in this case, we did get a code. So now we look at our cookies in order to discover what it was I was trying to accomplish. Ah! I was trying to follow the Test list, right. So we’re going to redirect to /do/follow.

The /do/follow route looks at the cookie and the parameters, again. Yep, got an account, got a list name, got a code... So now we’re going to get ourselves a client! You know how it is. The HTTP protocol is stateless. So we have to redo it all: get the client credentials from the file. This time we have a code, which is cool, so authorise the client again. Now we get the real deal: the authorisation token. Well, in my case the Mastodon::Client library handles all the painful details. But here we are, now. We have a client with the token. We’re good to go!

Now the application starts remote-following the accounts in the list. We end our journey on the /do/follow route. The URL actually contains the code but not the token. I still replaced the client_id and the authorisation code in the log with XXX.

Here’s what the log says:

[Wed Sep 12 22:58:24 2018] [debug] GET "/follow_confirm" (4fa2b1fd)
[Wed Sep 12 22:58:24 2018] [debug] Routing to a callback
[Wed Sep 12 22:58:24 2018] [debug] Rendering cached template "follow_confirm.html.ep" from DATA section
[Wed Sep 12 22:58:24 2018] [debug] Rendering cached template "layouts/default.html.ep" from DATA section
[Wed Sep 12 22:58:24 2018] [debug] 200 OK (0.003705s, 269.906/s)
[Wed Sep 12 22:58:32 2018] [debug] GET "/auth" (9f1d8f22)
[Wed Sep 12 22:58:32 2018] [debug] Routing to a callback
[Wed Sep 12 22:58:32 2018] [debug] kensanata@octodon.social wants to authorize the 'follow' action for the Test list
[Wed Sep 12 22:58:32 2018] [debug] kensanata@octodon.social found client credentials for octodon.social: id yes, secret yes
[Wed Sep 12 22:58:32 2018] [debug] kensanata@octodon.social has a client: yes
[Wed Sep 12 22:58:32 2018] [debug] kensanata@octodon.social is being redirected to https://octodon.social/oauth/authorize?redirect_uri=http%3A%2F%2Flocalhost%3A3000&response_type=code&client_id=XXX&scope=follow+read+write
[Wed Sep 12 22:58:32 2018] [debug] 302 Found (0.024886s, 40.183/s)
[Wed Sep 12 22:58:32 2018] [debug] GET "/" (8a14f8db)
[Wed Sep 12 22:58:32 2018] [debug] Routing to a callback
[Wed Sep 12 22:58:32 2018] [debug] kensanata@octodon.social is authorized for the 'follow' action using the Test list, redirecting to /do/follow?code=XXX&account=kensanata%40octodon.social&name=Test&logging=ok
[Wed Sep 12 22:58:32 2018] [debug] 302 Found (0.000799s, 1251.564/s)
[Wed Sep 12 22:58:32 2018] [debug] GET "/do/follow" (b79fbbdb)
[Wed Sep 12 22:58:32 2018] [debug] Routing to a callback
[Wed Sep 12 22:58:32 2018] [debug] kensanata@octodon.social found client credentials for octodon.social: id yes, secret yes
[Wed Sep 12 22:58:32 2018] [debug] kensanata@octodon.social has a client: yes
[Wed Sep 12 22:58:32 2018] [debug] kensanata@octodon.social authorizing using a code: yes
[Wed Sep 12 22:58:33 2018] [debug] kensanata@octodon.social begins to follow 2 accounts from list Test
[Wed Sep 12 22:58:33 2018] [debug] kensanata@octodon.social followed 2 accounts from list Test
[Wed Sep 12 22:58:33 2018] [debug] Rendering template "follow_done.html.ep" from DATA section
[Wed Sep 12 22:58:33 2018] [debug] Rendering cached template "layouts/default.html.ep" from DATA section
[Wed Sep 12 22:58:33 2018] [debug] 200 OK (0.762821s, 1.311/s)

Tags:

Comments

Did you try with scope written with %20 instead of +?

"&scope=follow+read+write"

Other than that this call fail:

redirect_uri=https%3A%2F%2Fcommunitywiki.org%2Ftrunk&response_type=code&client_id=xxxxxxxxxxxxxxxxxxx&scope=follow+read+write

If it’s not the scope, your client_id is no longer known by the instance. Did you try to register the app each time to see if the problem comes from your way of storing app clients?

https://github.com/tootsuite/documentation/blob/master/Using-the-API/API.md#apps

– tom79 2018-09-13 05:41 UTC


It also fails with a fresh client id due to a server side error on this callback for code:

https://communitywiki.org/trunk?code=XXXXXXXXXXXXXXXXXXXXXXX

Header:

Location: /trunk/do/follow?code=XXXXXXXXXXXXXXXXXXXXXXX&account=tom79%40ins.mastalab.app&name=Biology&logging=

– tom79 2018-09-13 05:54 UTC


Hm, I need better logging at that point: I see in my logs that you got a 500 error:

[Thu Sep 13 07:51:40 2018] [info] 178.209.50.237 GET http://communitywiki.org:8080/trunk/do/follow?code=XXXXXXXXXXXXXXXXXXXXXXX&account=tom79%40ins.mastalab.app&name=Biology&logging= 500 0.1443s

But that’s it. As far as I can tell from here, the flow looks correct: you did get authorised or the instance would not have sent you back to the redirect URL with a code. But it did, so that worked. So now the code is about to do this:

  1. get the authorisation token using the code it just got
  2. use the token to create the list, follow people and add these people to the list

– Alex Schroeder 2018-09-13 07:03 UTC


OK, I found a problem! I currently have 159 client credentials but when I look for duplicates, there’s a problem:

qoto.org 4
mastodon.social 23
ins.mastalab.app 2
mastodon.art 3
mstdn.io 2

This is obviously a problem because only one of them is going to be the correct client_secret.

Well, at least here’s a clue!

– Alex Schroeder 2018-09-13 07:16 UTC


OK, here’s something else: I while ago I tried to follow the Test list using kensanata@octodon.social and got the error “Authorisation failed. Did you try to reload the page? This will not work since we’re not saving the access token.” And just now I tried again and it worked. Clearly not the same error (since the authorisation seems to work) but disconcerting none the less.

– Alex Schroeder 2018-09-13 07:50 UTC


Improved logging, tried following the Test list using my account on humanities.one.

“Client authentication failed due to unknown client, no client authentication included, or unsupported authentication method”

The log:

[Thu Sep 13 10:12:49 2018] [info] 178.209.50.237 GET http://communitywiki.org:8080/trunk/follow_confirm?account=kensanata%40humanities.one&name=Test 200 0.0223s
...
[Thu Sep 13 10:12:54 2018] [debug] GET "/trunk/auth" (8d6b9c08)
[Thu Sep 13 10:12:54 2018] [debug] Routing to a callback
[Thu Sep 13 10:12:54 2018] [debug] kensanata@humanities.one wants to authorize the 'follow' action for the Test list
[Thu Sep 13 10:12:54 2018] [debug] kensanata@humanities.one found client credentials for humanities.one: id yes, secret yes
[Thu Sep 13 10:12:54 2018] [debug] kensanata@humanities.one has a client: yes
[Thu Sep 13 10:12:54 2018] [debug] kensanata@humanities.one is being redirected to https://humanities.one/oauth/authorize?redirect_uri=https%3A%2F%2Fcommunitywiki.org%2Ftrunk&response_type=code&client_id=XXX&scope=follow+read+write
...
[Thu Sep 13 10:12:54 2018] [info] 178.209.50.237 GET http://communitywiki.org:8080/trunk/auth?account=kensanata%40humanities.one&action=follow&logging=ok&name=Test 302 0.0516s
...

And that’s the end of it! OK, let’s compare this with the flow I described: We have a client and we want to authorize it. We’re going to redirect the user to their instance. If they have the instance’s cookie set, they’re authorized immediately and should get back to our redirect URL (the / route).

But this is not what happens. We’re not being sent back to the redirect URL and the error is shown instead.

– Alex Schroeder 2018-09-13 08:27 UTC


I’m looking at some info @NicolasConstant kindly sent me and the URLs he uses are:

create new app:
    FORM POST
    https://[instance]/api/v1/apps

    form data: 

    client_name
    redirect_uris
    scopes
    website


redirect sign-in url:        
     https://[instance]/oauth/authorize?scope=[scopes]&response_type=code&redirect_uri=[redirect_uri]&client_id=[client_id]


access_token retrieval 
    POST
    https://[instance]/oauth/token?client_id=[client_id]&client_secret=[client_secret]&grant_type=authorization_code&code=[code]&redirect_uri=[redirect_uri]

API access : 
    In the headers:
    'Authorization': 'Bearer [access_token]'

In this case, we’re not yet at the access_token retrieval stage. We’re at the “redirect sign-in URL” stage: https://humanities.one/oauth/authorize?redirect_uri=https%3A%2F%2Fcommunitywiki.org%2Ftrunk&response_type=code&client_id=XXX&scope=follow+read+write matches https://[instance]/oauth/authorize?scope=[scopes]&response_type=code&redirect_uri=[redirect_uri]&client_id=[client_id].

So why is the server rejecting it?

Let’s try the following:

Just do it again. A temporary fluke? → fail

[Thu Sep 13 11:06:46 2018] [debug] Routing to a callback
[Thu Sep 13 11:06:46 2018] [debug] kensanata@humanities.one wants to authorize the 'follow' action for the Test list
[Thu Sep 13 11:06:46 2018] [debug] kensanata@humanities.one found client credentials for humanities.one: id yes, secret yes
[Thu Sep 13 11:06:46 2018] [debug] kensanata@humanities.one has a client: yes
[Thu Sep 13 11:06:46 2018] [debug] kensanata@humanities.one is being redirected to https://humanities.one/oauth/authorize?redirect_uri=https%3A%2F%2Fcommunitywiki.org%2Ftrunk&response_type=co
de&client_id=XXX&scope=follow+read+write
[Thu Sep 13 11:06:46 2018] [debug] 302 Found (0.04456s, 22.442/s)
[Thu Sep 13 11:06:46 2018] [info] 178.209.50.237 GET http://communitywiki.org:8080/trunk/auth?account=kensanata%40humanities.one&action=follow&logging=ok&name=Test 302 0.0488s

Delete the client credentials and do it again. Perhaps the client_id is no longer valid? → fail

[Thu Sep 13 11:07:21 2018] [debug] kensanata@humanities.one wants to authorize the 'follow' action for the Test list
[Thu Sep 13 11:07:21 2018] [debug] kensanata@humanities.one found no client credentials for YYY
[Thu Sep 13 11:07:22 2018] [debug] kensanata@humanities.one registered client and got new credentials for YYY: id yes, secret yes
[Thu Sep 13 11:07:22 2018] [debug] kensanata@humanities.one has a client: yes
[Thu Sep 13 11:07:22 2018] [debug] kensanata@humanities.one is being redirected to https://humanities.one/oauth/authorize?redirect_uri=https%3A%2F%2Fcommunitywiki.org%2Ftrunk&response_type=code&client_id=XXX&scope=follow+read+write
[Thu Sep 13 11:07:22 2018] [debug] 302 Found (0.182976s, 5.465/s)

Oops! What’s YYY doing here? I wonder. My log writing got confused. Will rewrite this section and try again.

– Alex Schroeder 2018-09-13 09:13 UTC


Still borked:

[Thu Sep 13 11:28:07 2018] [debug] GET "/trunk/auth" (a8dae4dc)
[Thu Sep 13 11:28:07 2018] [debug] Routing to a callback
[Thu Sep 13 11:28:07 2018] [debug] kensanata@humanities.one wants to authorize the 'follow' action for the Test list
[Thu Sep 13 11:28:07 2018] [debug] kensanata@humanities.one found client credentials for humanities.one: id yes, secret yes
[Thu Sep 13 11:28:07 2018] [debug] kensanata@humanities.one has a client: yes
[Thu Sep 13 11:28:07 2018] [debug] kensanata@humanities.one is being redirected to https://humanities.one/oauth/authorize?redirect_uri=https%3A%2F%2Fcommunitywiki.org%2Ftrunk&response_type=code&client_id=XXX&scope=follow+read+write
[Thu Sep 13 11:28:07 2018] [debug] 302 Found (0.043568s, 22.953/s)
[Thu Sep 13 11:28:07 2018] [info] 178.209.50.237 GET http://communitywiki.org:8080/trunk/auth?account=kensanata%40humanities.one&action=follow&logging=ok&name=Test 302 0.0520s

And after removing the credentials – it worked‽

[Thu Sep 13 11:29:38 2018] [debug] GET "/trunk/auth" (353e6e43)
[Thu Sep 13 11:29:38 2018] [debug] Routing to a callback
[Thu Sep 13 11:29:38 2018] [debug] kensanata@humanities.one wants to authorize the 'follow' action for the Test list
[Thu Sep 13 11:29:38 2018] [debug] kensanata@humanities.one found no client credentials for humanities.one
[Thu Sep 13 11:29:38 2018] [debug] kensanata@humanities.one registered client and got new credentials for humanities.one: id yes, secret yes
[Thu Sep 13 11:29:38 2018] [debug] kensanata@humanities.one has a client: yes
[Thu Sep 13 11:29:38 2018] [debug] kensanata@humanities.one is being redirected to https://humanities.one/oauth/authorize?redirect_uri=https%3A%2F%2Fcommunitywiki.org%2Ftrunk&response_type=co
de&client_id=XXX&scope=follow+read+write
[Thu Sep 13 11:29:38 2018] [debug] 302 Found (0.591784s, 1.690/s)
[Thu Sep 13 11:29:38 2018] [info] 178.209.50.237 GET http://communitywiki.org:8080/trunk/auth?account=kensanata%40humanities.one&action=follow&logging=ok&name=Test 302 0.6006s
[Thu Sep 13 11:29:42 2018] [debug] GET "/trunk" (2ca628ab)
[Thu Sep 13 11:29:42 2018] [debug] Routing to a callback
[Thu Sep 13 11:29:42 2018] [debug] kensanata@humanities.one is authorized for the 'follow' action using the Test list, redirecting to /trunk/do/follow?code=XXX&account=kensanata%40humanities.
one&name=Test&logging=ok
[Thu Sep 13 11:29:42 2018] [debug] 302 Found (0.001252s, 798.722/s)
[Thu Sep 13 11:29:42 2018] [info] 178.209.50.237 GET http://communitywiki.org:8080/trunk?code=XXX 302 0.0078s
[Thu Sep 13 11:29:42 2018] [debug] GET "/trunk/do/follow" (1d4bad6a)
[Thu Sep 13 11:29:42 2018] [debug] Routing to a callback
[Thu Sep 13 11:29:42 2018] [debug] kensanata@humanities.one found client credentials for humanities.one: id yes, secret yes
[Thu Sep 13 11:29:42 2018] [debug] kensanata@humanities.one has a client: yes
[Thu Sep 13 11:29:42 2018] [debug] kensanata@humanities.one authorizing using a code: yes
[Thu Sep 13 11:29:42 2018] [debug] kensanata@humanities.one begins to follow 4 accounts from list Test
[Thu Sep 13 11:29:45 2018] [debug] kensanata@humanities.one followed 4 accounts from list Test
[Thu Sep 13 11:29:45 2018] [debug] Rendering template "follow_done.html.ep" from DATA section
[Thu Sep 13 11:29:45 2018] [debug] Rendering cached template "layouts/default.html.ep" from DATA section
[Thu Sep 13 11:29:45 2018] [debug] 200 OK (3.888279s, 0.257/s)
[Thu Sep 13 11:29:45 2018] [info] 178.209.50.237 GET http://communitywiki.org:8080/trunk/do/follow?code=XXX&account=kensanata%40humanities.one&name=Test&logging=ok 200 3.8928s

Hm. I am convinced that my rewriting just changed the way things are being logged. Just to be sure, though, I will delete all the client credentials (except for the one I just saved) and we’ll see whether the situation improves.

– Alex Schroeder 2018-09-13 09:38 UTC


I tested and it now works as expected.

– tom79 2018-09-13 11:15 UTC


Amazing. 🤷‍♂️

– Alex Schroeder 2018-09-13 13:51 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.