Showing posts with label code. Show all posts
Showing posts with label code. Show all posts

Sunday, November 20, 2022

Download files from google drive with python - no API key

There are several scripts to download files from google drive using a shareable link. Some require
authentication others don't work. Here we're interested in those that do not require authentication.

The gdrivedl.py script provided by ndrplz just works fine.

> gdrivedl.py file_id

Another notable, currently non-working, alternative is googledrivedownloader. It probably worked at some point in time but today it fails. There's however a simple fix proposed in this thread. The code is here: 

import requests
 
def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"
 
    session = requests.Session()
 
    response = session.get(URL, params={"id": id, "confirm": 1}, stream=True)
    token = get_confirm_token(response)
 
    if token:
        params = {"id": id, "confirm": token}
        response = session.get(URL, params=params, stream=True)
 
    save_response_content(response, destination)
 
def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith("download_warning"):
            return value
    return None
 
def save_response_content(response, destination):
    CHUNK_SIZE = 32768
 
    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk:  # filter out keep-alive new chunks
                f.write(chunk)
 
if __name__ == "__main__":
    file_id = "TAKE ID FROM SHAREABLE LINK"
    destination = "DESTINATION FILE ON YOUR DISK"
    download_file_from_google_drive(file_id, destination)

Sunday, November 13, 2022

Fix your matplotlib colorbars!

Matplotlib colorbars are a mess, especially if using subplots I often get tall bars like in this case

  

but instead I wanted

 


This great article by Joseph Long describes this issue in detail and provides some solutions. 
The code for generating the first figure is

import matplotlib.pyplot as plt
from skimage import img_as_float, data, color, transform
from scipy import ndimage, signal, fft, io
import numpy as np
 
img = color.rgb2gray(img_as_float(data.astronaut()))
I1 = np.abs(fft.fft2(img, norm='ortho')**2)
 
#plt.figure(figsize=(7.5,2.5))
 
subplotnum=1
 
plt.subplot(1,2,subplotnum); subplotnum+=1
plt.imshow(img)
plt.colorbar()
plt.title('image')
 
plt.subplot(1,2,subplotnum); subplotnum+=1
plt.imshow(np.log(fft.fftshift(I1)+1e-6))
plt.colorbar()
plt.yticks([])           # desable yticks in the second image
plt.title('log spectrum')
 
plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0)
plt.show()   

A solution provided in the article consists in using a custom colorbar, the important changes are highlighted

import matplotlib.pyplot as plt
from skimage import img_as_float, data, color, transform
from scipy import ndimage, signal, fft, io
import numpy as np
 
def colorbar(mappable):
    from mpl_toolkits.axes_grid1 import make_axes_locatable
    import matplotlib.pyplot as plt
    last_axes = plt.gca()
    ax = mappable.axes
    fig = ax.figure
    divider = make_axes_locatable(ax)
    cax = divider.append_axes("right", size="5%", pad=0.05)
    cbar = fig.colorbar(mappable, cax=cax)
    plt.sca(last_axes)
    return cbar
 
img = color.rgb2gray(img_as_float(data.astronaut()))
I1 = np.abs(fft.fft2(img, norm='ortho')**2)
  
subplotnum=1
 
aa = plt.subplot(1,2,subplotnum); subplotnum+=1
ax = plt.imshow(img)
colorbar(ax)
aa.set_title('image', size=14)
 
aa = plt.subplot(1,2,subplotnum); subplotnum+=1
ax = plt.imshow(np.log(fft.fftshift(I1)+1e-6))
colorbar(ax)
aa.get_yaxis().set_ticks([])  # desable yticks in the second image
aa.set_title('log spectrum', size=14)
 
plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0)
plt.show()    

Sunday, October 14, 2012

Download/backup BibTeX library from CiteULike, including all the attached PDF

CiteULike is a convenient solution for bibliographic management, but sometimes an off-line replica of the bibliography is also needed. This can be achieved by exporting the CiteULike library in BibTeX format, furthermore BibDesk (and probably other managers too) can be synchronized directly with the CiteULike server.

While it is easy to export a .bib file from CiteULike, downloading all the attached files and having the .bib linking to the downloaded files is a different story.

BibTeX and JSON exports of the CiteULike library can be retrieved from:
  http://www.citeulike.org/json/user/USERNAME
  http://citeulike.org/bibtex/user/USERNAME
The BibTeX export can be read directly into a bibliographic manager, but the JSON export contains more information than the .bib, in particular it contains the location of the attached PDFs. So the idea of the Python script linked below is to do the following
  1. download the CiteULike library in BibTeX and JSON formats
  2. parse the JSON export and download all the attachments
  3. modify the .bib file to include links to the downloaded copies of the attachments
the links in the .bib file should work for BibDesk and JabRef.

Before running the script:
  • setup CITEULIKE_USERNAME and CITEULIKE_PASSWORD variables in the script
  • verify you have wget and pybtex installed
Download the Python script:  citeulike_backup.zip

http://wiki.citeulike.org/index.php/Importing_and_Exporting#JSON


Gory details from http://wiki.citeulike.org/index.php/Importing_and_Exporting#JSON:
# save session cookies
> wget -O /dev/null --keep-session-cookies  --save-cookies cookies.txt --post-data="username=xxxx&password=yyyy&perm=1" http://www.citeulike.org/login.do
# download bibtex with private comments and download an attachment
> wget -O export.bib --load-cookies cookies.txt http://www.citeulike.org/bibtex/user/xxxx
> wget --load-cookies cookies.txt http://www.citeulike.org//pdf/user/xxxx/article/123456/891011/some_99_paper_123456.pdf

Saturday, August 25, 2012

Change text inside multiple files (command line)

I always forget these commands, and they are so useful.
The sed command to replace some text in a set of files:
     sed -e 's/old text/new text/g'  -i .trash  *.c 

The regular expression 's/some text/new text/g' is applied to each line of each file *.c.
And the -i flag saves a backup of each file.

The regular expression language is powerful, for instance allows to remember parts of the pattern in the substitution. An example: the following command puts quotation marks around everything that follows the first = sign in each line of script.sh.
  sed -e 's/=\(.*\)/="\1"/' script.sh

The pattern enclosed between \(   and   \) is stored in \1, and then used in the replaced text.

Source: http://www.labnol.org/internet/design/wordpress-unix-replace-text-multiple-files/1128/

Saturday, November 5, 2011

Minted: Highlighted source code in LATEX

Minted is a LaTeX package that facilitates expressive syntax highlighting using the powerful Pygments library. The package provides options to customize the highlighted source code output.
The example below shows how C code is highlighted,  and the output is shown at right:

\documentclass{article}
\usepackage{minted}

\begin{document}

\begin{minted}{c}
int main()
{
  printf("hello, world");
  return 0;
}
\end{minted}

\end{document}

Pygments also provides a command line interface pygmentize, among the possible output formats there are HTML, RTF, LaTeX and ANSI sequences.
For command line highlighting a solid alternative to Pygments is GNU Source-highlight.



Source: http://mirrors.ircam.fr/pub/CTAN/macros/latex/contrib/minted/minted.pdf
A couple of on-line highlighters: http://www.hilite.me/http://quickhighlighter.com/