Download/backup BibTeX library from CiteULike, including all the attached PDF

CiteULike is a convenient solution for bibliographic management, but sometimes an off-line replica of the bibliography is also needed. This can be achieved by exporting the CiteULike library in BibTeX format, furthermore BibDesk (and probably other managers too) can be synchronized directly with the CiteULike server.

While it is easy to export a .bib file from CiteULike, downloading all the attached files and having the .bib linking to the downloaded files is a different story.

BibTeX and JSON exports of the CiteULike library can be retrieved from:

  http://www.citeulike.org/json/user/USERNAME

  http://citeulike.org/bibtex/user/USERNAME

The BibTeX export can be read directly into a bibliographic manager, but the JSON export contains more information than the .bib, in particular it contains the location of the attached PDFs. So the idea of the Python script linked below is to do the following: 

1. download the CiteULike library in BibTeX and JSON formats
2. parse the JSON export and download all the attachments
3. modify the .bib file to include links to the downloaded copies of the attachments

the links in the .bib file should work for BibDesk and JabRef.

Before running the script:

• setup CITEULIKE_USERNAME and CITEULIKE_PASSWORD variables in the script
• verify you have wget and pybtex installed

Download the Python script:  citeulike_backup.zip

Resources:

http://www.macresearch.org/how_do_you_organize_digital_article_collections

http://www.andrewberman.org/projects/sync/

http://www.academicproductivity.com/2009/citeulike-bibdesk-sync-your-references-and-live-smarter/ 

http://wiki.citeulike.org/index.php/Importing_and_Exporting#JSON

---

Gory details from http://wiki.citeulike.org/index.php/Importing_and_Exporting#JSON:
# save session cookies

> wget -O /dev/null --keep-session-cookies  --save-cookies cookies.txt --post-data="username=xxxx&password=yyyy&perm=1" http://www.citeulike.org/login.do

# download bibtex with private comments and download an attachment

> wget -O export.bib --load-cookies cookies.txt http://www.citeulike.org/bibtex/user/xxxx
> wget --load-cookies cookies.txt http://www.citeulike.org//pdf/user/xxxx/article/123456/891011/some_99_paper_123456.pdf

Data URL scheme, include files in-line in the web page

There is a restriction in the blogger.com sites: you can't upload files.
The data URL scheme provides a way to include data in-line in web pages as if they were external resources.

The format of data URL:

data:[<mediatype>][;base64],<data>

Some examples:

• Image:   (If you don't see an image your browser does not support data URL)

<IMG

SRC="data:image/gif;base64,R0lGODdhMAAwAPAAAAAAAP///ywAAAAAMAAw

AAAC8IyPqcvt3wCcDkiLc7C0qwyGHhSWpjQu5yqmCYsapyuvUUlvONmOZtfzgFz

ByTB10QgxOR0TqBQejhRNzOfkVJ+5YiUqrXF5Y5lKh/DeuNcP5yLWGsEbtLiOSp

a/TPg7JpJHxyendzWTBfX0cxOnKPjgBzi4diinWGdkF8kjdfnycQZXZeYGejmJl

ZeGl9i2icVqaNVailT6F5iJ90m6mvuTS4OK05M0vDk0Q4XUtwvKOzrcd3iq9uis

F81M1OIcR7lEewwcLp7tuNNkM3uNna3F2JQFo97Vriy/Xl4/f1cf5VWzXyym7PH

hhx4dbgYKAAA7"

ALT="Larry">

• Link to an image: file.dat

<A

HREF="data:image/gif;base64,R0lGODdhMAAwAPAAAAAAAP///ywAAAAAMAAw

AAAC8IyPqcvt3wCcDkiLc7C0qwyGHhSWpjQu5yqmCYsapyuvUUlvONmOZtfzgFz

ByTB10QgxOR0TqBQejhRNzOfkVJ+5YiUqrXF5Y5lKh/DeuNcP5yLWGsEbtLiOSp

a/TPg7JpJHxyendzWTBfX0cxOnKPjgBzi4diinWGdkF8kjdfnycQZXZeYGejmJl

ZeGl9i2icVqaNVailT6F5iJ90m6mvuTS4OK05M0vDk0Q4XUtwvKOzrcd3iq9uis

F81M1OIcR7lEewwcLp7tuNNkM3uNna3F2JQFo97Vriy/Xl4/f1cf5VWzXyym7PH

hhx4dbgYKAAA7">file.dat</A>

• Link to the same image declared with MIME type: application/octet-stream:  file.dat

• An example with a zip file (MIME type: application/zip):  data.zip

This site http://www.dopiaza.org/tools/datauri/ provides an easy way to convert a file to the data URL format. Otherwise one may need to uuencode the file.

Some resources:
http://en.wikipedia.org/wiki/Data_URI#cite_note-MSDN-3
http://tools.ietf.org/html/rfc2397
http://htmlcoderhelper.com/how-to-force-save-as-dialog-box-in-firefox-besides-changing-headers/