Comments on 2010-05-31 Blognapping

I’m surprised (and somewhat impressed) that it works for Wordpress blogs too.

Good work!

greywulf 2010-06-01 05:51 UTC

Now that the coding has been done, I need to do actual text assembly. Yikes! :)

AlexSchroeder 2010-06-01 17:52 UTC

If you’re wondering how to do this… Assume you want to pull a copy of A Hamsterish Hoard of Dungeons and Dragons. Examine the source code and you’ll find a link to the atom feed within blogger. This is important, because it’ll provide us with the blog Id! In this case:

<link rel=”alternate” type=”application/atom+xml” title=”A Hamsterish Hoard of Dungeons and Dragons - Atom” href=”>

<link rel=”alternate” type=”application/rss+xml” title=”A Hamsterish Hoard of Dungeons and Dragons - RSS” href=”> <link rel=”” type=”applicationatom+xml” title=”A Hamsterish Hoard of Dungeons and Dragons - Atom” href=”>## ← that’s the one we’re looking for!

Start with a small set: the last 100 entries:

for i in `seq 4`; do
  curl -o taichara-$i.atom "$start&amp;amp;max-results=25"

Save it in a script such as and run it using bash You’ll end up with the files taichara-1.atom taichara-2.atom taichara-3.atom taichara-4.atom.

Now take the Perl script from the main page and save it as It will extract the page URLs from the Atom files.

for f in *.atom; do
  for p in `perl &lt; $f`; do
    wget $p

Once you’ve verified it, you can fetch more Atom pages.

AlexSchroeder 2011-11-16 19:53 UTC

Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. See Info for text formatting rules. You can edit this page if you need to fix typos. You can subscribe to updates by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.