Diary SiteMap RecentChanges About Contact Calendar

Search:

Matching Pages:

Journal

2010-05-31 Blognapping

For Blogger, using bash, perl, and curl. You need to replace XXX with the magic number you get when you look at the blog’s source. The HTML header will contain a line like the following: <link rel="service.post" type="application/atom+xml" title="..." href="http://www.blogger.com/feeds/XXX/posts/default" /> – this is where you get the number from.

Once you have it:

for i in `seq 40`; do
  start=$((($i-1)*25+1))
  curl -o foo-$i.atom "http://www.blogger.com/feeds/XXX/posts/default?start-index=$start&amp;max-results=25"
done

This should get you 40 files called foo-1.atom to foo.40.atom with 25 articles each in your current directory. Delete the ones that don’t contain any results or increase the number if you’re looking at a blog with more than 1000 posts that you’re interested in.

Next, how to extract the HTML links from these Atom feeds: save the following in a Perl script called url.pl.

#!/usr/bin/perl
use XML::LibXML;
undef $/;
$data = <STDIN>;
my $parser = XML::LibXML->new();
my $doc = $parser->parse_string($data);
die $@ if $@;
my $context = XML::LibXML::XPathContext->new($doc);
$context->registerNs('atom', 'http://www.w3.org/2005/Atom');
foreach ($context->findnodes('//atom:entry'
			     . '/atom:link[@rel="alternate"][@type="text/html"]'
			     . '/attribute::href')) {
  print $_->to_literal() . "\n";
}

Now you can extract all the URLs and fetch them:

for f in *.atom; do
    for url in `perl url.pl < $f`; do
        curl -O "$url";
    done;
done

The use of the -O option assumes that the file names given by the URL will be unique – this is not necessarily true as http://localhost/2010/05/test.html and http://localhost/2010/06/test.html will result in one overwriting the other.

You should end up with a ton of HTML files in your current directory.

This doesn’t get any required extra files like CSS or images, but it might be good enough for a blog backup.

For a Wordpress blog, we try to do the same thing. First, get the atom pages:

for i in `seq 100`; do
  curl -o foo-$i.atom "http://foo.wordpress.com/feed/atom/?paged=$i"
done

This should get you 100 files called foo-1.atom to foo.100.atom in your current directory. Delete the ones that don’t contain any results or increase the number if you’re looking at a blog with more posts.

Now, unless the author has disabled it somehow, the atom feeds already include the complete articles. It’s certainly possible to fetch them all again, but it’s not necessary. Save the following in a Perl script called extract.pl.

#!/usr/bin/perl
use strict;
use XML::LibXML;
undef $/;
my $data = <STDIN>;
my $parser = XML::LibXML->new();
my $doc = $parser->parse_string($data);
die $@ if $@;
my $encoding = $doc->actualEncoding();
my $context = XML::LibXML::XPathContext->new($doc);
$context->registerNs('atom', 'http://www.w3.org/2005/Atom');
foreach my $entry ($context->findnodes('//atom:entry')) {
  my $title = $entry->getChildrenByTagName('title')->[0]->to_literal;
  $title =~ s!/!_!gi;
  $title =~ s!&amp;!&!gi;
  $title =~ s!&#(\d+);!chr($1)!ge;
  my $content = $entry->getChildrenByTagName('content')->[0]->to_literal;
  open(F, ">" . $title . ".html") or die $! . ' ' . $title;
  print F <<EOT;
<html>
<head>
<meta content='text/html; charset=$encoding' http-equiv='Content-Type'/>
</head>
<body>
$content
</body>
</html>
EOT
  close F;
}

Run it on the Atom files:

for f in *.atom; do
    perl extract.pl < $f
done

You should end up with a ton of HTML files in your current directory.

Tags: RSS RSS

Comments on 2010-05-31 Blognapping

I’m surprised (and somewhat impressed) that it works for Wordpress blogs too.

Good work!

greywulf 2010-06-01 05:51 UTC


Now that the coding has been done, I need to do actual text assembly. Yikes! :)

AlexSchroeder 2010-06-01 17:52 UTC



AlexSchroeder
If you’re wondering how to do this… Assume you want to pull a copy of A Hamsterish Hoard of Dungeons and Dragons. Examine the source code and you’ll find a link to the atom feed within blogger. This is important, because it’ll provide us with the blog Id! In this case:

<link rel="alternate" type="application/atom+xml" title="A Hamsterish Hoard of Dungeons and Dragons - Atom" href="http://hamsterhoard.blogspot.com/feeds/posts/default" /> <link rel="alternate" type="application/rss+xml" title="A Hamsterish Hoard of Dungeons and Dragons - RSS" href="http://hamsterhoard.blogspot.com/feeds/posts/default?alt=rss" /> <link rel="service.post" type="application/atom+xml" title="A Hamsterish Hoard of Dungeons and Dragons - Atom" href="http://www.blogger.com/feeds/5373792969086619654/posts/default" /> ← that’s the one we’re looking for!

Start with a small set: the last 100 entries:

for i in `seq 4`; do
  start=$((($i-1)*25+1))
  curl -o taichara-$i.atom "http://www.blogger.com/feeds/5373792969086619654/posts/default?start-index=$start&amp;amp;max-results=25"
done

Save it in a script such as download-atom.sh and run it using bash download-atom.sh. You’ll end up with the files taichara-1.atom taichara-2.atom taichara-3.atom taichara-4.atom.

Now take the Perl script from the main page and save it as url.pl. It will extract the page URLs from the Atom files.

for f in *.atom; do
  for p in `perl url.pl &lt; $f`; do
    wget $p
  done
done

Once you’ve verified it, you can fetch more Atom pages.

AlexSchroeder 2011-11-16 19:53 UTC

Add Comment

2006-09-16 Atom Revisited

I went back to look at <span class="site">Atom</span><span class="separator">:</span><span class="page">:XML</span> and even wrote a mail to the maintainers, BenjaminTrott and Tatsuhiko Miyagawa, reporting on missing man pages and questions I had regarding the mode attribute of content elements.

But the most important piece I needed to know in order to write an Atom server inside Perl’s CGI.pm was knowing how to read the rest of the data from a POST request. The XML doesn’t come as part of the form!

I had to actually read the CGI.pm sources to figure it out. And the solution is very simple, except that it’s not documented in the man page:

  my $data = $q->param('POSTDATA');
  my $entry = XML::Atom::Entry->new(\$data);

Yes, an undocumented parameter. Argh!!

The system already knows how to handle POST (for new pages), GET (to read existing pages), there’s an Atom feed (but not yet as flexible as the existing RSS 2.0 feed). So, there’s PUT to implement for updating pages, and testing to do. I’m testing my extension using the XML::Atom::Client library. Having this kind of unit test really helps! Once you have the infrastructure set up, haha. I can’t believe it took me so long to figure out the POSTDATA thing. And looking back it seems incomprehensible to try and develop for Flock directly without the XML::Atom::Client library to write unit tests.

Update: Hah. Time passes. Little Alex wants to write the code that accepts PUT requests. And finds that no data is being read. How’s that? Correct, the stupid POSTDATA hack only works when looking at a POST request. D’oh!

So now I’m using the following:

sub AtomEntry { my $data = $q->param('POSTDATA'); if (not $data) { # CGI provides POSTDATA for POST requests, not for PUT requests local $/; # slurp $data = <STDIN>; } my $entry = XML::Atom::Entry->new(\$data); return $entry; }

Well, actually, since I was being paranoid, I looked at the CGI.pm source code again and rewrote it as follows:

sub AtomEntry { my $data = $q->param('POSTDATA'); if (not $data) { # CGI provides POSTDATA for POST requests, not for PUT requests. # The following code is based on the CGI->init code. my $content_length = defined($ENV{'CONTENT_LENGTH'}) ? $ENV{'CONTENT_LENGTH'} : 0; if ($content_length > 0 and $content_length < $MaxPost) { $q->read_from_client(\$data, $content_length, 0); } } my $entry = XML::Atom::Entry->new(\$data); return $entry; }

Tags:

Add Comment

2006-09-11 Looking at Flock

Have you tried Flock? I’m giving it a try at the moment. It says that it can integrate blogging using “Wordpress, Movable Type, Typepad, LiveJournal, Typo, and Drupal and integrates with the Blogger, Meta Weblog, Typepad, and Atom APIs.”

Maybe I should write a complete Atom API for Oddmuse. Right now the Oddmuse:Atom Extension is a proof-of-concept extension that’s not really as flexible as the existing Oddmuse:Rss Action.

At least exporting my subscriptions from BlogLines as an OPML feed, and subscribing to all of them in flock went smoothly. Yay for open standards! <3 <3 <3

Actually the Bloglines searches did not work out of the box, because I hadn’t subscribed to them as RSS feed. Redoing the searches allowed me to subscribe to the RSS feeds, so that went well. Since I’m reading news from multiple machines, however, moving away from Bloglines is not an option. I was just “checking”… ;) I see that Flock is planning to look into the issue, however. [1] I think they should just interface with Bloglines!!

Logging into LiveJournal as a test automatically notified Flock that I signed into a site that it supported for blogging, and offered to do all the configuration automatically. It felt eerie, but it worked. Except that I don’t want to blog on LJ. — Or do I?

Anyway, I started working on Atom.

I downloaded XML::Atom, but it seemed complex. So I started to read the draft [2] and started implementing: Return a simple “introspection document”.

Alpinobombus:~ alex$ curl http://localhost/cgi-bin/wiki/atom <?xml version="1.0" encoding='UTF-8'?> <service xmlns="http://purl.org/atom/app#"> <workspace title="Oddmuse" > <collection title="Wiki" href="http://localhost/cgi-bin/wiki/atom"> <accept>entry, image/jpeg, image/png</accept> </collection> </workspace> </service>

I’m using the correct MIME type (application/atomserv+xml), and yet Flock keeps complaining that something’s wrong about it.

I guess I didn’t neet to go very far to realize I wouldn’t get far. :(

The meat of my Oddmuse extension:

push(@MyInitVariables, \&AtomInit); sub AtomInit { SetParam('action', 'atom') if $q->path_info =~ m|/atom\b|; } $Action{atom} = \&DoAtom; sub DoAtom { my $id = shift; DoAtomIntrospection(); } # from http://www.ietf.org/internet-drafts/draft-ietf-atompub-protocol-09.txt sub DoAtomIntrospection { print GetHttpHeader('application/atomserv+xml'); my @types = ('entry', ); push(@types, @UploadTypes) if $UploadAllowed; my $upload = '<accept>' . join(', ', @types) . '</accept>'; print <<EOT; <?xml version="1.0" encoding='$HttpCharset'?> <service xmlns="http://purl.org/atom/app#"> <workspace title="Oddmuse" > <collection title="$SiteName" href="$ScriptName/atom/pub"> $upload </collection> </workspace> </service> EOT }

Too bad Flock didn’t like it!

I’ve also installed RPC::XML, considering the implementation of the MetaWeblog API. But that, too, looks complicated. I really, really like simple code. :)

So now I wrote a simple skeleton for the MetaWeblog API:

use RPC::XML::Server; require RPC::XML::Procedure; $srv = RPC::XML::Server->new(port => 9000); my $new = RPC::XML::Procedure->new({ name => 'metaWeblog.newPost', code => sub { OddmuseNew(@_) }, signature => [ 'string', 'string', 'string', 'struct', 'boolean' ] }); my $edit = RPC::XML::Procedure->new({ name => 'metaWeblog.editPost', code => sub { OddmuseEdit(@_) }, signature => [ 'string', 'string', 'string', 'struct', 'boolean' ] }); my $get = RPC::XML::Procedure->new({ name => 'metaWeblog.getPost', code => sub { OddmuseGet(@_) }, signature => [ 'string', 'string', 'string' ] }); $srv->add_method($new); $srv->add_method($edit); $srv->add_method($get); $srv->server_loop; # Never returns # metaWeblog.newPost (blogid, username, password, struct, publish) # returns string sub OddmusePost { my ($blogid, $username, $password, $struct, $publish) = @_; return "foo"; } # metaWeblog.editPost (postid, username, password, struct, publish) # returns true sub OddmuseEdit { my ($postid, $username, $password, $struct, $publish) = @_; return 1; } # metaWeblog.getPost (postid, username, password) returns struct sub OddmuseGet { my ($postid, $username, $password) = @_; }

As you can see, no real code there.

I then created a file containing my request:

<?xml version="1.0"?> <methodCall> <methodName>metaWeblog.newPost</methodName> <params> <param> <value><string>Wiki</string></value> <value><string>AlexSchroeder</string></value> <value><string>Hibiskus$$Fresser</string></value> <value><struct> <title>Testing Meta</title> <description>This is my text!</description> </struct></value> <value><boolean>1</boolean></value> </param> </params> </methodCall>

And ran the server and tested it using curl:

Alpinobombus:~ alex$ curl --data @~/oddmuse/meta-new.xml localhost:9000 <title>403 Forbidden</title> <h1>403 Forbidden</h1>

Didn’t get very far here, either!

Perhaps I should use Net::Blogger instead of Flock to test my implementation. ;)

Later, I returned to Atom! See 2006-09-16 Atom Revisited.

Tags:

Add Comment

Show Google +1

Define external redirect: :XML

EditNearLinks: LiveJournal