Serving static file in nginx with a some check

Recently I developed  a small web site for my lab, and we are hosting it on Google App Engine (GAE), as Google provides quite enough resources for a site of several pages ;)

However we also need to provide download of huge files, which can not be store in GAE, thus we had to host it in one of our severs. Initially we do not want our content directly accessible, so we want to use dynamic URL but we don’t want to loose the performance of nginx.

My final solution is use rsa module by Sybren A. Stuvel (which is pure python) to encrypt the real url (with fixed private key) in GAE, and give the encrypted url to user. In the file host, I use nginx’s X-Accel-Redirect feature and a tiny cherrypy server to decrypt the url (with corresponding pub key), and simply set the X-Accel-Redirect header to the real internal path of nginx.

This solution works well in my case :)

GAE Hello world error

Google App Engine supports SSL now, however Python 2.5 does not come with SSL socket in stock :(

gae$ /usr/local/bin/dev_appserver.py \
~/projects/hello/
Traceback (most recent call last):
File "/usr/local/bin/dev_appserver.py", line 50, in <module>
execfile(script_path, globals())
File "/usr/local/google_appengine/google/appengine/tools/dev_appserver_main.py", line 338, in <module>
sys.exit(main(sys.argv))
File "/usr/local/google_appengine/google/appengine/tools/dev_appserver_main.py", line 297, in main
server = MakeRpcServer(option_dict)
File "/usr/local/google_appengine/google/appengine/tools/dev_appserver_main.py", line 259, in MakeRpcServer
host_override=option_dict[ARG_ADMIN_CONSOLE_HOST])
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py", line 114, in __init__
self.opener = self._GetOpener()
File "/usr/local/google_appengine/google/appengine/tools/appcfg.py", line 317, in _GetOpener
opener.add_handler(urllib2.HTTPSHandler())
AttributeError: 'module' object has no attribute 'HTTPSHandler'

To resovle the problem you got to recomile Python with SSL support as instructed by Patrick Altman and then follow a regular  configure – make – make install routine.

Then you need to Edit the Modules/Setup.dist to uncomment a couple of lines:

# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
SSL=/usr/local/ssl
_ssl _ssl.c 
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl 
-L$(SSL)/lib -lssl -lcrypto

Idea for a simple bib tool

I was using Zotero for my research readings, however, it is a bit too heavy for me, and I want a tool that can be portable across machines. So I am planning to write a bib tool for myself when I am a bit free sometime next spring.

The most important concept in this tool:

• Keep it Simple: there will NOT be fancy setup interface, but a config XML; there will not be many export styles for that is latex’s business.
• Depend less: there is already many advanced tools for bib organization, but they depend on this and that. I plan to implement all this tool with Python and sqlite3, so it can be put into U-key/Dropbox, and used anywhere.

The basci senarios is:

1. User start the tool, which set up a local web sever at a high port (eg: 8080), and open any browser to land on the interface.
2. To add a new paper, user have to
   1. upload a pdf file
   2. add bibtex info by
      1. upload a crospending a bibtex file
      2. fill a form to generate bibtex info
   3. add optional tags
   4. add optional note
3. To view paper:
   1. start a search (empty field means any)
   2. a list of reference is returned
   3. (optional) open pdf file
4. To export
   1. start a search
   2. select paper and add to “basket”
   3. go to export folder and get a sorted bib for all and BIB’s for each files and copies of PDF’s

Suggestions are always welcomed :)

How to process large text file efficiently in Python

Abstract

Three different ways of processing text file line by line are given in the order of increasing efficiency.

I have to handle a large text file of space-separated data in python, and the data goes like this:

tag1 tag2 tag3
12 34 12
123 345 12

the first line is tags for each column, and the rest lines hold data. Since the tags are fixed, I can code it directly in to my script, that is to say the first line should be skipped. My first script goes like this:

file = open('foo.txt', 'r')
for line in file.readlines()[1:]:
  #do something

This script requires a vast amount of RAM, since it has to store a list of all lines! So it is wise to use iterator:

file = open('foo.txt', 'r')
first = True
for line in file:
  if first:
    first = False
  else:
    #do something

The second script works much better than the first one, because the lines are read one by one from the file by using a iterator. However, the first flag is not a neat way to skip the first line for we have to test the flag many times, which makes no sense. And the problem is solved in the third script:

file = open('foo.txt', 'r')
file.readline()
for line in file:
  #do something

the 'fileread.line()' command will perfectly move the file position one line forward, and the iter will then start from the second line:)

a python script to generate cue file

Abstract

A python script to generate cue file from file names in a same folder is given.

Sometimes we have lessless music (ape, flac…) split in to different files, and a cue file may help orgnize them in some music jukebox application, i.e foobar2000. so I create a really silly script to help me…

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import os
import sys
import glob
from string import Template
from optparse import OptionParser

cue_head='''REM GENRE $GENRE
REM DATE $DATE
REM DISCID $DISCID
PERFORMER "$PERFORMER"
TITLE "$ALBUMTITLE"'''

cue_file='''
FILE "$FILE" WAVE
TRACK $TRACK AUDIO
TITLE "$TRACKTITLE"
INDEX 01 00:00:00'''

def cue_gen():
"""
generate cue file
"""
parser = OptionParser()
parser.add_option('-d', dest='dir',
type='string')
parser.add_option('-s', dest='sfx',
type='string', default='ape')
options, args = parser.parse_args()

if options.dir is None:
directory = os.getcwd()
else:
directory = options.dir
suffix = options.sfx
print directory, suffix

files = glob.glob(os.path.join(directory, '*.'+suffix))
files = [ff[1] for ff in [os.path.split(f) for f in files]]
files.sort()
for file in files:
print file
begin = int(raw_input('Title offset? '))

genre = raw_input('Genre? ')
date = raw_input('Date? ')
discId = raw_input('DiscId? ')
performer = raw_input('Performer? ')
albumTitle = raw_input('Title? ')
d1 = dict(GENRE = genre,
DATE = date,
DISCID = discId,
PERFORMER = performer,
ALBUMTITLE = albumTitle)
head = Template(cue_head).substitute(d1)
cuepath = os.path.join(directory, (albumTitle+'.cue'))

f = open(cuepath, 'w')
f.write(head)

track = 0
for file in files:
trackTitle = file[begin:-(len(suffix)+1)]
track = track + 1
d2 = dict(FILE = file,
TRACK = track,
TRACKTITLE = trackTitle)
body = Template(cue_file).substitute(d2)
f.write(body)

f.close()

if __name__ == '__main__':
cue_gen()



I know it’s quite rough and a lot of enhancement can be made… maybe it should search for the right file extension automatically or even search the genre for the internet, and that’s just why I put it here, hopefully some of you would help to improve it XD