Mastodon Archive

Releases are installed via pip. See Mastodon Archive. Documentation and source code are available via git.

2018-09-03 Unfollowing Lurkers

A direct message reminded me of another way you might want to trim the list of people you are following. Following too many people is a burden. When I see that it is taking too long for me to read all the new toots, it’s time to trim the list. I prefer to follow less people instead of not reading all the toots.

So I’ve added a following command to mastodon-archive. You can list the people you’re following who never mentioned you, and if you’re hardcore enough, you can unfollow them all.

This can probably use some fine-tuning:

  1. editing the list before submitting it?
  2. having exceptions for your favourite bots?

If you have strong opinions either way, let me know.

Tags:

Comments on 2018-09-03 Unfollowing Lurkers

Released v1.1.0 of Mastodon Archive which includes this feature.

– Alex Schroeder 2018-10-04 21:53 UTC

Add Comment

2018-08-17 No Repost

There is no way to migrate an account and take your old toots with you. You download those via the export function in Mastodon, or using Mastodon Archive, but you cannot import then back. The API does not allow it.

It also makes no sense to repost toots on your new account: you would be annoying everybody under the sun as the local timeline gets flooded and all the mentioned accounts get notified again. Even if you decided to post them all unlisted, notification would still be a problem. And all the timestamps would be wrong.

What currently works: account migration: Settings > Edit profile > Move to a different account.

  1. Your toots stay.
  2. Your followers get notified and might or might not follow your new account.
  3. You keep the people you follow so you won’t miss a thing.
  4. Their toots start populating the federated timeline of your new instance; this will help small instances get better connected to the rest of the Fediverse.

Tags:

Add Comment

2018-05-12 Mastodon OAuth2 Issues

I signed up for a new instance and noticed that Mastodon Archive doesn’t allow me to log in using OAuth2. What’s going on? Let’s look at the API again.

Register the application:

curl -X POST -d "client_name=mastodon-archive&redirect_uris=urn:ietf:wg:oauth:2.0:oob&scopes=read" -Ss https://tabletop.social/api/v1/apps

This gives me a JSON containing client_id and client_secret. Store these as environment variables of the same name.

Store the password in an environment variable, too.

export client_id=...
export client_secret=...
export password=...

Login to get the access token:

curl -X POST -d "client_id=$client_id&client_secret=$client_secret&grant_type=password&username=kensanata@gmail.com&password=$password" -Ss https://tabletop.social/oauth/token

Store it in an environment variable, of course.

export access=token=...

Test it:

curl --header "Authorization: Bearer $access_token" -sS https://tabletop.social/api/v1/timelines/home

This seems to work even though it doesn’t work for Mastodon Archive!

Let’s create the necessary files, manually:

echo $client_id > tabletop.social.client.secret
echo $client_secret >> tabletop.social.client.secret
echo $access_token > tabletop.social.user.kensanata.secret

And test it:

$ mastodon-archive archive kensanata@tabletop.social
Get user info
Get statuses (this may take a while)
Get favourites (this may take a while)
Skipping mentions
Skipping followers
Saving 33 statuses, 13 favourites, 0 mentions, and 0 followers

Thus, we now know that the problem is the app registration and obtaining the access token. Everything else works as expected.

Tags:

Comments on 2018-05-12 Mastodon OAuth2 Issues

Debugging what Mastodon.py does... Here is what my code does, essentially:

print("Registering app")
Mastodon.create_app(
    'mastodon-archive',
    api_base_url = url,
    to_file = client_secret)

print("Log in")
mastodon = Mastodon(
    client_id = client_secret,
    api_base_url = url,
    debug_requests = True)

url = mastodon.auth_request_url(
    client_id = client_secret,
    scopes=scopes)

print("Visit the following URL and authorize the app:")
print(url)

print("Then paste the access token here:")
token = sys.stdin.readline().rstrip()

mastodon.log_in(
    username = username,
    code = token,
    to_file = user_secret,
    scopes=scopes)

And I get the error in the last statement.

Registering app
Log in
Mastodon: Request to endpoint "/api/v1/instance/" using method "GET".
Parameters: {}
Headers: None
Files: {}
Mastodon: Response received with code 200.
response headers: {...}
Response text content: {...}
Visit the following URL and authorize the app:
https://social.nasqueron.org/oauth/authorize?client_id=...8aba&response_type=code&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=read
Then paste the access token here:
...033d
Mastodon: Request to endpoint "/oauth/token" using method "POST".
Parameters: {'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob', 'code': '...033d', 'grant_type': 'authorization_code', 'client_id': '...8aba', 'client_secret': '...b138', 'scope': 'read'}
Headers: None
Files: {}
Mastodon: Response received with code 401.
response headers: {... 'WWW-Authenticate': 'Bearer realm="Doorkeeper", error="invalid_grant", error_description="The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."' ...}
Response text content: {"error":"invalid_grant","error_description":"The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."}

I started wondering: is this because of the scope parameter? I was missing the scope parameter in my call to create_app. I made that change but saw no difference.

I then thought perhaps the problem is the missing redirect_uris in auth_request_url and redirect_uri in log_in. I added them both but saw no difference.

I then thought perhaps something about scopes was wrong so I removed scopes, redirect_uris and redirect_uri from the various calls but saw no difference.

Sadly, the Mastodon.py documentation and the Doorkeeper documentation it refers to is very sparse indeed. Mastodon.py doesn’t tell me how to really do OAuth and Doorkeeper only tells me how to do it with a password (Resource Owner Password Credentials) and not how to use Client Credentials.

It is all quite frustrating.

And most disturbing of all: it continues to work for octodon.social but doesn’t work for any of the other instances I have tried, e.g. social.nasqueron.org.

As I said, it is all quite frustrating.

As expected, switching the entire thing to password credentials “works”:

mastodon = Mastodon(
    client_id = client_secret,
    api_base_url = url)

print("We need your credentials just this once to log in.")
sys.stdout.write("Email: ")
sys.stdout.flush()
email = sys.stdin.readline().rstrip()
sys.stdout.write("Password: ")
sys.stdout.flush()
password = sys.stdin.readline().rstrip()

mastodon.log_in(
    username = email,
    password = password,
    to_file = user_secret,
    scopes=scopes)

The result:

$ mastodon-archive login kensanata@social.nasqueron.org
We need your credentials just this once to log in.
Email: kensanata+nasqueron@gmail.com
Password: *secret*
Get user info
Login OK

The word of the day is “frustrating”.

– Alex Schroeder 2018-05-13 19:23 UTC

Add Comment

2018-01-20 Mastodon Archive 0.0.5

I made a new Mastodon Archive release.

Changes from 0.0.4 to 0.0.5:

  • use a User-Agent of Mastodon-Archive/0.0 when downloading media files
  • warn if toot expiration is going to take a long time due to rate limiting
  • report on favourites
  • report default to a 12 week time window with an option to change it
  • report default to top 10 tags with an option to change it
  • add an option to include boosted toots in your hashtag report
  • when providing all as your user, then the subcommand will run for every archive in the current directory

Tags:

Add Comment

2017-04-27 Record Keeping

Remember Record Keeper, the Meatball Wiki page on people who keep records of our textual lives? Those who have records of our chat logs, email archives and social media posts have power over us. They take conversation more seriously than others. They prevent forgiving and forgetting.

This started a discussion on Mastodon, but all my toots on that topic have since expired.

What I hate about social media sites is that all the data is kept but I can’t find anything I need. The admins, or the new owners, however, can always comb through the data. Given that I can’t find shit, I’d be better off if old data got deleted. Delete inactive accounts after 90 days. Delete Toots after 90 days. Delete media after 30 days. Data is a liability. Make it easy for people to export threads (share a thread as HTML mail attachment, or save as HTML page for example).

Allowing me to download my own data is just part of the story. Luckily we can do that often enough: Google Takeout, Twitter Download, Facebook Download. But data is still a liability. If the bad guys take over, all your secrets are revealed, basically. Data is a liability because we cannot predict the future. And then there is the sheer scale of it. Once I realized that my GMail archive was more than 2G of mail, I decided to archive the file and delete the data on the servers because I don’t have the tools to work with the data offline. Better to delete it.

Now, this was relatively easy to do using a mail client and some free time. And yes, I can always go back and delete all the tweets, all the Facebook posts, all the Flickr photos, all the Instagram pics, but have you tried doing it? Deleting old stuff is a sad chore and nobody does it. Forgive and forget? You wish. The search engine indexes would forget, if only you’d delete, but you effectively can’t.

In the end, it was simply easier to delete my Facebook and Flickr accounts. And where as I did find a tool that allowed me to delete all my tweets on Twitter (Twitter Archive Eraser), I ended up deleting my Twitter accounts, too.

I was reminded of this talk by Maciej Cegłowski, Haunted By Data, where data is compared to nuclear waste. “In a world where everything is tracked and kept forever, like the world we’re for some reason building, you become hostage to the worst thing you’ve ever done.”

In that old thread on Mastodon, some people claimed that I was making it easy for evidence to disappear, for nazis to hide, for white supremacists to hide. But I disagree. If the solution to old Nazis hiding somewhere is setting up the Stasi, I’m not sure we’re getting the best deal that we could. I’m not saying all records should be destroyed, videos should self destruct, newspapers should disappears, libraries should burn down, or archives be dissolved. No! I’m saying that Facebook and Twitter and G+ and Mastodon should make it easy (or: the default behaviour) to forget stuff. It’s not the same thing. Facebook posts and Tweets are not history, books, archives and newspapers.

Some people claimed that using social media is like printing pamphlets. You’ll never get them back. The information will stay out there, forever. And this is true, in a way. There is no protection from snoops and crooks. But this is not an all or nothing decision. The inability to create a perfect solution does not prevent us from making baby steps in the right direction. Some ISPs must keep meta data around. Secret services keep our data. And still, I can prefer an admin who doesn’t keep my data around. I can prefer the Mastodon implementation that forgets by default.

Let’s not forget: software is the way it is because that’s how we decided to implement it. These days, forgetting is harder than keeping data forever. Software engineers coming from a background full of version control software have a particularly hard time grasping the importance of forgetting for our daily lives. But we don’t have to do it that way. We could also implement it like printing on very cheap newspaper. These pamphlets will fall apart sooner or later and special efforts might be required to preserve them.

My wiki replaces all IP numbers in the log file with Anonymous eventually and it has an option to delete old page revisions after a while. Yes, somebody can still set up a feed reader and keep a copy of all the stuff. But I am not keeping a copy of the stuff. And if you’re a nice admin, perhaps you’re also trying to limit the stuff you keep. Data is a liability. We need to design software to minimise the footprint.

I think we need a general change in attitude. Software needs to be built such that it will allow us to forget.

Sometimes people will mention archives. If we all expire our data, will future historians think of our times as the dark ages? But let’s not forget: real archives need curation. We can’t just keep the dregs of daily life forever. The future will drown in our micro blogs. Are you curating your micro blog? Neither am I. But this blog is somewhat curated. And I don’t expire pages from the blog without reason. We need this continuum of options.

There’s also the question of whether we consent to this future. Now that we are living in a world where forgetting keeps getting harder: is that something you agree with? If David Duke deleted all his posts, and provided no more clues to his white supremacist opinions, and we no longer remembered them, is this not the sort of forgiving and forgetting that makes public life bearable? We can set up monuments to remind us of war crimes, or crimes against humanity, of tragedies and mistakes, but do we really need remember all the names, link them, trace them?

Is it not ridiculous when presidents have to claim on TV that they did not inhale when confronted with old pictures? I think the only alternative is something straight our of David Brin’s The Transparent Society. Since we can’t beat the snoops, we should join them and the ensuing balance of terror will keep us all quiet. If everybody posts pictures of the misdeeds as a teenager on Facebook then there is no reason to attack others for them. We will have made us vulnerable to the communities we live in. And perhaps this kind of trust and openness is something that will grow over time.

These days we accept kids doing stupid things and say to ourselves, we were kids once, we did stupid things too. Let’s not call the cops.

I don’t know. I think as a society we have not learned to deal with this kind of openness. On the contrary. When people are released from prison, we accept them back into our midst and consider it unfortunate when they can’t find their way back into society. We want them to work and pay taxes, we want them to be good citizens again and we don’t want to remind them of their crimes whenever their name comes up.

As a society, in the non-digitised world, we have designed mechanisms to strike a balance between record keeping an forgetting. But you know Lawrence Lessig’s book, Code is Law, c.f. Code and Other Laws of Cyberspace. The software we build that never forgets is the software that disrupts our ability to forget. Right now, we no longer have the choice.

I want to make that choice. We are built to forget the information that is not relevant. Forgetting is important.

Fuck it The alternative, the thing that we’re building right now, the default future if you will, is the exact opposite. We’re building a Panopticon where the rich and powerful can keep watch on us, where surveillance capitalism reaps the fruit of our data and we can’t trust a single website.

That is why I might want to keep a copy of my toots on Mastodon, but I don’t think they have much value going back months and years. I never read through years of tweeting history! This only benefits your enemies, never your friends. I want to expire my toots. We can always write a blog post about the good stuff.

Tags:

Comments on 2017-04-27 Record Keeping

“...become hostage to the worst thing you’ve ever done” does hit the nail on the head. There has always been a tension between forgetting and remembering. We are already hostages to the worst things we’ve done, does it really matter if others also know the worst thing we’ve done?

You are right to say that we have not learned to deal with this kind of openness. However, younger folks who’ve grown up with Facebook and ubiquitousness of cameras and recording devices are working out their own evolutionary responses to this.

IMHO, in the final analysis, it appears that the benefits of remembering outweigh the harm it brings. Whether this opinion is my own or something that the powers-that-be have trained me to have is an open question.

AlokSingh 2017-05-10 07:48 UTC

Add Comment

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.