For Blogger, using bash, perl, and curl. You need to replace XXX with the magic number you get when you look at the blog’s source. The HTML header will contain a line like the following: <link rel="service.post" type="application/atom+xml" title="..." href="http://www.blogger.com/feeds/XXX/posts/default" /> – this is where you get the number from.
Once you have it:
for i in `seq 40`; do start=$((($i-1)*25+1)) curl -o foo-$i.atom "http://www.blogger.com/feeds/XXX/posts/default?start-index=$start&max-results=25" done
This should get you 40 files called foo-1.atom to foo.40.atom with 25 articles each in your current directory. Delete the ones that don’t contain any results or increase the number if you’re looking at a blog with more than 1000 posts that you’re interested in.
Next, how to extract the HTML links from these Atom feeds: save the following in a Perl script called url.pl.
#!/usr/bin/perl
use XML::LibXML;
undef $/;
$data = <STDIN>;
my $parser = XML::LibXML->new();
my $doc = $parser->parse_string($data);
die $@ if $@;
my $context = XML::LibXML::XPathContext->new($doc);
$context->registerNs('atom', 'http://www.w3.org/2005/Atom');
foreach ($context->findnodes('//atom:entry'
. '/atom:link[@rel="alternate"][@type="text/html"]'
. '/attribute::href')) {
print $_->to_literal() . "\n";
}Now you can extract all the URLs and fetch them:
for f in *.atom; do
for url in `perl url.pl < $f`; do
curl -O "$url";
done;
doneThe use of the -O option assumes that the file names given by the URL will be unique – this is not necessarily true as http://localhost/2010/05/test.html and http://localhost/2010/06/test.html will result in one overwriting the other.
You should end up with a ton of HTML files in your current directory.
This doesn’t get any required extra files like CSS or images, but it might be good enough for a blog backup.
For a Wordpress blog, we try to do the same thing. First, get the atom pages:
for i in `seq 100`; do curl -o foo-$i.atom "http://foo.wordpress.com/feed/atom/?paged=$i" done
This should get you 100 files called foo-1.atom to foo.100.atom in your current directory. Delete the ones that don’t contain any results or increase the number if you’re looking at a blog with more posts.
Now, unless the author has disabled it somehow, the atom feeds already include the complete articles. It’s certainly possible to fetch them all again, but it’s not necessary. Save the following in a Perl script called extract.pl.
#!/usr/bin/perl
use strict;
use XML::LibXML;
undef $/;
my $data = <STDIN>;
my $parser = XML::LibXML->new();
my $doc = $parser->parse_string($data);
die $@ if $@;
my $encoding = $doc->actualEncoding();
my $context = XML::LibXML::XPathContext->new($doc);
$context->registerNs('atom', 'http://www.w3.org/2005/Atom');
foreach my $entry ($context->findnodes('//atom:entry')) {
my $title = $entry->getChildrenByTagName('title')->[0]->to_literal;
$title =~ s!/!_!gi;
$title =~ s!&!&!gi;
$title =~ s!&#(\d+);!chr($1)!ge;
my $content = $entry->getChildrenByTagName('content')->[0]->to_literal;
open(F, ">" . $title . ".html") or die $! . ' ' . $title;
print F <<EOT;
<html>
<head>
<meta content='text/html; charset=$encoding' http-equiv='Content-Type'/>
</head>
<body>
$content
</body>
</html>
EOT
close F;
}Run it on the Atom files:
for f in *.atom; do
perl extract.pl < $f
doneYou should end up with a ton of HTML files in your current directory.