Jupiter

Planet Jupiter is a feed aggregator that creates a single, static web page, presenting the posts from all the sites it is following as a river of news – that is, there is no unread count, and thus no ever increasing pressure to “catch up”. You don’t catch up – you just dip your foot every now and then.

2020-12-31 My feed aggregators

@solderpunk recently announced an interesting change to CAPCOM:

“On the first day of each month, a cron job runs the … pipeline to randomly select 100 unique feeds … Those feeds are then aggregated by CAPCOM for the remainder of the month. This way the scope of the CAPCOM instance can grow without limit over time, but the actual amount of new content to be found each day should remain fairly constant over time.”

I like it!

As for my own feed agregators, I actually wrote two of them in 2020.

For the traditional web, I wrote a replacement for Planet Venus which was powering the RPG Planet because Planet Venus is written in Python 2 and I was unable to port it to Python 3. Writing my own was more fun!

It powers all three of the RPG Planets I run:

I also use it for a list of other blogs I’m following that aren’t directly RPG related:

As you can see I didn’t bother to change the theme, haha! 😄

Since I also started getting interested in Gemini, I rewrote moku pona, the Gopher “aggregator” I had. The reason I’m using quotes here is that moku pona 1 was just a URL watcher: given a list of URLs, it would check them for changes and move changed resources to the top of the list. When I added support for Gemini feeds, I realised that this was not very different from supporting all sorts of feeds, so now moku pona supports URL watching, as well as RSS and Atom feeds, via Gopher, Gemini, and HTTP. The fact that it works amazes me every day. 😆

The main difference in terms of user interface is the granularity. Jupiter a “traditional” aggregator that shows an excerpt of every post: site name, post title, post except. Mona pona on the other hand simply shows the date of the last change per site, with the nickname you provided.

Add Comment

2020-01-20 How to deal with weird stuff in feed aggregation

Planet Jupiter is entering the phase where all the features I want have been implemented and now I’m only dealing with weird stuff and workarounds. I’ve seen blog posts containing double escaped HTML. I unescape the first layer and turn the post to plain text (adding pilcrow signs for paragraph breaks) and so on. But if somebody adds double escaped HTML, that still leaves escaped HTML in the output. I’m guessing this is a bug in their Wordpress plugins responsible for indie web support or something. But what am I going to do: tell them to fix their setup? Or add a workaround that simply tries to unscape HTML a second time? Guess what I’m doing...

I had to consider other things as well: what about elements that aren’t being rendered by browser? They also shouldn’t be “rendered” by the feed aggregator. One feed, for example, contains a style element. The style element contains CSS instructions ­– this should not end up in the page excerpt. Anything else I need to remove from the excerpt?

Tags:

Comments on 2020-01-20 How to deal with weird stuff in feed aggregation

Blocking CSS (and more) is easy if you use CSP: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy

Ashwin 2020-04-21 00:40 UTC


Good point.

– Alex Schroeder 2020-04-21 06:05 UTC

Add Comment

2020-01-10 Planet Jupiter

OK, I’ve been using Planet Venus for the longest time. I use it to generate the the Planets on Campaign Wiki. The problem is that it is written in Python 2 and I was unable to quickly migrate it (and all its dependencies) to Python 3.

So here I am... writing Planet Jupiter. 🙄

Please report any issues you run into, or leave a comment.

Oh well, what could go wrong...

(I wonder whether we should use ♃ for Jupiter somewhere...)

Tags:

Comments on 2020-01-10 Planet Jupiter

Actually, it’s looking pretty good. I’ll see about making the switch tomorrow. Sure, it has much, much fewer functionality than Planet Venus but it has just the right kind of functionality I need.

I guess I should have picked a really small planet to name this project but it is what it is. I like Jupiter.

The big thing that remains to be done is caching. But it should be good enough already!

– Alex Schroeder 2020-01-10 19:58 UTC


The planets I run have been switched over. Let me know if there is anything wonky.

– Alex Schroeder 2020-01-11 13:55 UTC


Yikes, rewriting it to use XML::Feed made it a lot slower! 😭

– Alex Schroeder 2020-01-12 22:08 UTC


@PresGas noticed that the merged feed does not exist. Perhaps we can fix this using a template? XML::Feed also has a feature to add entries to a feed, so we could make our own.

Still getting a lot of null responses → I guess I need to be slower in accessing Blogspot?

– Alex Schroeder 2020-01-13 14:31 UTC


Let’s hope feeds work now! I’m generating feed.xml in RSS 2.0 format and redirection the old atom.xml to it...

– Alex Schroeder 2020-01-13 19:45 UTC


Sadly, there is an encoding bug for Atom feeds, e.g. a feed title such as “Sküll” gets double encoded. 😟

– Alex Schroeder 2020-01-15 12:56 UTC


For the moment I’m trying a rewrite without XML::Feed just because it seemed so much faster... The encoding problem still exists, however. Thus, it’s not introduced by XML::Feed.

– Alex Schroeder 2020-01-16 06:29 UTC


And... I think it’s done! 😁

– Alex Schroeder 2020-01-17 15:06 UTC

Add Comment

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Just say HELLO