Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

2013-01-14

staticgenerator

I am building a static website but want to use a generation tool to build all the pieces.  I am using Python and originally looked at Hyde.  But I had difficulites and want to learn the Django frameworks. So instead of using Hyde, I am going to try out creating sites using Django and then using staticgenerator to take copies. pystatic is a newer alternative.

The original documentation at http://superjared.com/projects/static-generator/ has gone but the project is here:

http://pypi.python.org/pypi/staticgenerator/1.4.1

and there is a clone here:

http://superjared.com/projects/static-generator/

I have copied the documentation here from a Wayback Machine archive of the original:

http://web.archive.org/web/20101221135325/http://superjared.com/projects/static-generator/

StaticGenerator for Django

StaticGenerator is on GitHub!

Introduction

How many CPU cycles do you suppose are wasted on blogs that are generated every request? Wouldn’t it make more sense to generate them only when they’re updated? StaticGenerator is a Python class for Django that makes it easy to create static files for lightning fast performance.

Download

You can get StaticGenerator using easy_install:

easy_install staticgenerator


Or download from the cheeseshop.



Usage



There are two ways to generate the static files. Both setups first require WEB_ROOT to be set in settings.py:



WEB_ROOT = '/var/www/example.com/public/'


Method 1 (preferred): Middleware


As of StaticGenerator 1.3, Middleware is available to generate the file only when the URL is requested. This solves the 404 Problem (see below).



First, add Regexes of URLs you want to cache to settings.py like so:



STATIC_GENERATOR_URLS = (
r'^/$',
r'^/(blog|about|projects)',
)


Second, add the Middleware to MIDDLEWARE_CLASSES:



MIDDLEWARE_CLASSES = (
...snip...
'staticgenerator.middleware.StaticGeneratorMiddleware',
'django.contrib.flatpages.middleware.FlatpageFallbackMiddleware',
...snip...
)


Note: You must place the StaticGeneratorMiddleware before FlatpageFallbackMiddleware if you use it.



When the pages are accessed for the first time, the body of the page is saved into a static file. This is completely transparent to the end-user. When the page or an associated object has changed, simply delete the cached file (See notes on Signals).



Method 2: Generate on Save


The second method works by saving the cache file on save. This method fakes a request to get the appropriate content. In this example we want to publish our home page, all live Posts and all FlatPages:



# Passing url, a QuerySet and Model
from staticgenerator import quick_publish
quick_publish('/', Post.objects.live(), FlatPage)


Deleting files and paths is just as easy:



from staticgenerator import quick_delete
quick_delete('/path-to-delete/')


Note: Directory deletion fails silently while failing to delete a file will raise an exception.



The “404 Problem”


The second method suffers from a problem herein called the “404 problem”. Say you have a blog post that is not yet to be published. When you save it, the file created is actually a 404 message since the blog post is not actually available to the public. Using the older method you’d have to re-save the object to generate the file again.



The new method solves this because it saves the file only when the URL is accessed successfully (read: only when the HTTP status is 200).



Using Signals


Integrating with existing models is easy using Django’s signal dispatcher. Simply create a function to delete your models, and connect to the dispatcher:



from django.contrib.flatpages.models import FlatPage
from blog.models import Post
from django.db.models import signals
from staticgenerator import quick_delete

def delete_index(sender, instance):
quick_delete(instance, '/')

signals.post_delete.connect(delete_index, sender=Post)
signals.post_delete.connect(delete_index, sender=FlatPage)


Every time you save a Post or FlatPage it deletes the static file (notice that I add ‘/’ so my homepage is deleted as well). What happens when a comment is added? Just delete the corresponding page:



from django.contrib.comments.models import Comment, FreeComment

def publish_comment(sender, instance):
quick_delete(instance.get_content_object())

signals.post_save.connect(publish_comment, sender=Comment)
signals.post_save.connect(publish_comment, sender=FreeComment)


Configure your front-end



Sample Nginx configuration


This configuration snippet shows how Nginx can automatically show the index.html page generated by StaticGenerator, and pass all Django requests to Apache.



# This example configuration only shows parts relevant to a Django app
http {

upstream django {
# Apache/mod_python running on port 7000
server example.com:7000;
}

server {
server_name example.com;
root /var/www/;

location / {
if (-f $request_filename/index.html) {
rewrite (.*) $1/index.html break;
}
if (!-f $request_filename) {
proxy_pass http://django;
break;
}
}

}

}


It’s not for Everything



The beauty of the generator is that you choose when and what urls are made into static files. Obviously a contact form or search form won’t work this way, so we just leave them as regular Django requests. In your front-end http server (you are using a front-end web server, right?) just set the URLs you want to be served as static and they’re already being served.

2013-01-04

Sample Python script to control Photoshop

I have been using Python to script Photoshop and came across something that wasn’t immediately obvious to me:

A sample section would be:

// Create a new 2x4 inch document and assign it to a variable
var docRef = app.documents.add( 250, 33 )

which I got to work as:

# Create a new 2x4 inch document and assign it to a variable.
docs = psApp.Documents
docRef = docs.Add(250,33)

you are not able to two level of indirection on COM objects but keep need to making Python objects of the references.

The complete sample code is:

import win32com.client

psApp = win32com.client.Dispatch("Photoshop.Application")

# Remember current unit settings and then set units to
# the value expected by this script
originalRulerUnits = psApp.Preferences.RulerUnits
psApp.Preferences.RulerUnits = 1 # 1= psPixesl, 2 = inches

# Create a new 2x4 inch document and assign it to a variable.
docs = psApp.Documents
docRef = docs.Add(250,33)

# Create a new art layer containing text
layers = docRef.artLayers
artLayerRef = layers.add

artLayerRef.kind = 2 #Text layer
# Set the contents of the text layer.
textItemRef = artLayerRef.TextItem
textItemRef.Contents = "Hello, Web!"

# Restore unit setting
psApp.Preferences.RulerUnits = originalRulerUnits

2012-03-16

Configuring Django on Vagrant not including chef

In my last post the formatting was going odd so have started again.  My next job is to get Django installed on the UAT rather than a plain Ubuntu box.  You will see at the end I deferred using chef and just got the machine up and running.

I am going to try and follow Simon William’s post. I have a working server.  I also looked at this video. git was not installed on my base machine so I logged in and installed it, made a new directory and got my test software:

sudo apt-get install git
git clone
https://github.com/simeonwillbanks/vagrant-up-django-app-server.git

The syntax of the command in the Vagrant file had changed from

config.vm.forward_port “http”, 80, 8080

to

config.vm.forward_port 80, 8080

Then I needed to install chef which I hadn’t done. So trying the bootstrap knife on my local machine, and having the usual problem about verifying hosts:

C:\Users\me\chef-repo\.chef>knife bootstrap 192.168.1.7 –x me -P mypassword --sudo --no-host-key-verify

It didn’t work.  Unsure why I just installed directly on my base station using the apt-get process which worked fine and I tested it had worked with:

chef-client –-version

However I had forgotten the /etc/chef/validation.pem file which I moved with a fabric script:

def base_key():
    put(r'c:\chef\validation.pem','/home/me/validation.pem')
    run('sudo mv /home/me/validation.pem /etc/chef/validation.pem')




Then chef-client worked (don’t forget the sudo)




sudo chef-client




I had to reedit the Vagrantfile.  Vagrant up now nearly worked.  However I would get the message “The file /etc/chef/validation.pem does not contain a correctly formatted private key.”  It looked fine but I must have edited it with notepad as it had CR (Ctrl-M) at the end of each line.



Now it booted ok but it complained that it had already been registered (which it had in the last debug cycle).  At the end of the cycle I just destroyed the machine leaving the node registered on the chef server.  I need to bring the machine offline gracefully.



This command is still failing:




chef-client -c /etc/chef/client.rb -j /etc/chef/dna.json




One of my problems mois that I am building Ubuntu lucid boxes which have chef 0.09 and I am running it from a base build of Ubuntu Oneiric 11.10 with chef 0.10.  So I am going to use veewee to make a new base box.  I install veewee:




sudo gem install veewee




It didn’t go well veewee installation didn’t complete and it corrupted some gemspec files that I had to hand edit to fix.  I rebooted and my test vagrant box is working again.  I learnt one useful thing in that the Vagrantfile directory (on my base machine ~/uatbox32) is mounted as /vagrant in the vagrant virtual machine.



I mounted the VBoxGuestAdditions.iso in the client box and that eliminated one problem.  I still need to work on the chef upgrade from 0.09 to 0.10 which I did on the virtual box with:




sudo gem install chef




I hadn’t realised that chef had both clients and nodes and I needed to delete both from the chef server page after a vagrant up.  On a reboot the guest additions reverted – I should have done halt rather than destroy.  Luckily I had packaged the build so can use that as a base.



I am going back to Fabric for deployment.  A  historical chanes that might catch you out (it caught me out).  In 0.9 and later versions of fabric (I am using 1.4) Fabric uses the ‘env’ environment dictionary which used to be called config.  So any examples with config.x = are no obsolete.



I have found some nice guidance on django best practices:



http://lincolnloop.com/django-best-practices/projects.html



After a lot of mucking about I now have a working site.  The next stage is to simplify and to roll out for the dev,test,uat and prod environments.



dev is on my development machine and might do anything.



test is on production type hardware and is for testing.



uat is on production type harward and a recent backup of the production database for final acceptance.



prod is then the delivered product.

2012-02-29

Using Boto and Python to connect to a specific region

Here is my slightly inelegant but working code to connect to specific region without setting up a configuration file or altering the default in the library (the access keys you can find under the tab security credentials on :

aaaa = your AWS Access key
sssss = your Secret Amazon key

from boto.ec2.connection import EC2Connection
default_conn = EC2Connection('aaaaa','sssss')

regions = default_conn.get_all_regions()

for r in regions:
    print r.name
    if r.name == 'eu-west-1':
       conn = EC2Connection('aaaaa',’sssss',region =r)

print regions

rs = conn.get_all_security_groups()
print rs

2012-02-15

Installing Python Pyjamas and Pyjamas desktop for Windows 7

I am testing out python pyjamas as a gui front end which I can scale from a desktop app to web app to cloud deployable scalable app with very little change.  I have tried a number of different GUI stacks, the last was pytgtk which only worked with 32 bit Python which I had to install on top of my 64 bit python.  To start I have gone back to 64 bit python.

Going to the pyjamas wiki getting started  I started using the windows help.  I noticed it was a little out of date so decided to write my experience here.  I have already got a working version of Python 2.72 64 bit on Windows.

Next I installed tortoiseGIT as I have used TortoiseSVN a lot.  I also needed to install the Windows GIT client which I should have done first.  Then I create a pyjamas directory in my download directory and use right click Git clone to create the directory.

gitClone

You need to change the URL to git://pyjs.org/git/pyjamas.git and press OK.

gitCloneDialogBox

The first time I did it it failed so I deleted everything and tried again and this time it worked.

I then installed comtypes which was version 0.6.2.  I had two versions of Python installed so I selected Python 2.7.  It then had an error at then end:

ComtypesError


Carrying on, assuming that GTK may not be needed as Pyjamas desktop is using MSHTML and for my desktop I am at the moment only worried about desktop Windows apps. (I really don’t want to go back to Python 32bit again) I copied the pyjamas directory to my C drive so that I had the following directory C:\pyjamas\pyjd

I updated the system path  ControlPanel->System->Advanced system settings->(Advanced Tab)Environment Variables->Edit path  I added C:\python27;C:\pyjamas\bin. Note the pyjamas bin directory is added in the next step:

Open up a command window and then run which will create the bin directory:

cd \pyjamas
python bootstrap.py

I then created a build.bat in the helloworld directory:

pyjsbuild.bat --print-statements Hello

and also a run file:

pyjd Hello.py

Pyjamas works by combining files from the public directory eg files and template html and your code.  In order to use either the output or the desktop you must run the build process first.  Then you can use the run file first the app just opened up  in a new windowSmile and worked:

helloworld

If I opened the browser with output\hello.html I just got the static text but need to change to the output directory and run the html file (so the default location of other files works) then I get the application in a browser window.  I was using by default IE and didn’t work on getting the others to work.  This from start to end took me about 3 hours including writing my notes as I went.

2011-05-16

Getting Sage VMWare image 4.6 to run

I have had trouble getting the SAGE VMWare image to talk to the external network but have found the solution:

sudo rm /etc/udev/rules.d/70-persistent-net.rules

I tried using VMware server, then Virtual Box and then VMWare player.  Vmware Player was painless although it uninstalled VMWare server.  Virtual box need to be adapted so as to do a new install then add the two hardisks and then it worked although I still had the networking problem.  I haven't tested this solution for Virtual Box - I deleted the modified virtual machine.

Solutions came from this thread who referenced Tim Child

2009-01-19

Getting Python to work

I have been investigating Python as I thought it might have a slightly simpler syntax from Ruby (I am finding it takes time to switch syntax from Mathematica, C#, VB etc) , be more better supported on .NET eg IronPython ahead of IronRuby and better scientific support and some work done on creating an interactive platform like Mathematica/Matlab. Enthought has a prepackaged Scientific distribution.

Anyway just to say not quite as good as I hoped in that Python is the middle of a big change in syntax from 2.x to 3.x, IronPython wont install on my server and it seems to work slightly differently on different computers when driving Excel.

After playing and buying a number of tools the free ActiveState Python distribution seems to work ok. The only snag I have found so far is that the PythonWin debugger really needs to be killed and restarted to work well if you are debugging.