Django query logging

Recently I had to optimise some piece of legacy code that worked slow. Among other refactoring procedures I wanted to check what queries gets invoked exactly. Having dug Django Docs and internets I came up with solution.

Using this, you'll get all executed queries in the console

For example:

2016-02-25 15:57:47 DEBUG    utils.py:89 (0.055) SET SQL_AUTO_IS_NULL = 0; args=None
2016-02-25 15:57:47 DEBUG    utils.py:89 (0.435) SELECT `table`.`id` FROM `table` ORDER BY `table`.`id` ASC LIMIT 10000; args=()

Add the code below to your local_settings.py (or whatever it's called in your project), make sure the include of this code is AFTER defining LOGGING variable.

DB_LOG = True

from settings import LOGGING
if DB_LOG:
    import logging, re

    class QueryFilter(logging.Filter):
        def filter(self, record):
            # exclude PyCharm debug window queries
            return 'LIMIT 21' not in getattr(record, 'sql', '')


    class SqlFormatter(logging.Formatter):
        def format(self, record):
            record.msg = re.sub(r"((?:u?')?\d+(?:u?')?)"
                                r"(?:,\s*(?:u?')?\d+(?:u?')?)+",
                                r'\1, ...', record.msg)
            return super(SqlFormatter, self).format(record)

    params = {
        'filters': {
            'db_query_filter': {
                '()': QueryFilter,
            },
        },
        'formatters': {
            'debugging': {
                '()': SqlFormatter,
                'format': '%(asctime)s %(levelname)-8s %(filename)s:%(lineno)d %(message)s',
                'datefmt' : '%Y-%m-%d %H:%M:%S',
            },
        },
        'handlers': {
            'db_console': {
                'level': 'DEBUG',
                'filters': ['require_debug_true', 'db_query_filter'],
                'class': 'logging.StreamHandler',
                'formatter': 'debugging',
            },
        },
        'loggers': {
            'django.db': {
                'level': 'DEBUG',
                'handlers': ['db_console'],
                'propagate': False,
            },
        }
    }
    for key, param in params.items():
        try:
            LOGGING[key].update(param)
        except KeyError:
            LOGGING[key] = params

Computed field on TabularInline (StackedInline)

Django admin doesn't provide list_display attribute on its inline admin classes. That's right, since the only supposed view for those classes is editable form. So, to display a result of custom function's call, it needs to add the name of the function (that can be declared either on the admin or model classed) to both fields and readonly_fields.

Example:

class StatsInline(admin.TabularInline):
    model = Stats
    fields = ('clicked', 'shown', 'avg')
    readonly_fields = ('avg',)
    verbose_name = 'Stats'
    verbose_name_plural = 'Stats'
    can_delete = False

    def avg(self, obj):
        return float(obj.clicked) / obj.shown if obj.shown else 0

P.S. float conversion is needed to avoid casting result to int (e.g. 1/2=0, float(1)/2=0.5)

Get Graphite run for a Django project

This post is intended first of all to memorize my experience of getting the Graphite system work.

Tab-completion for django-admin.py and manage.py

Download django_bash_completion file and save in some convenient place in your system. In .bash_profile (or .bashrc if it doesn't envoke .bash_profile) add

. ~/path/to/django_bash_completion

Settings will take effect the next time you log in.

Activate VE from Django's manage.py

To transparently activate virtual environment from project's manage.py insert the code below right after #!/usr/bin/env python

activate_this = '/ABSOLUTE/PATH/TO/ENVIRONMENT/FILES/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

Run Django via WSGI without web server

Recently I faced a problem - I needed to test updated Django WSGI-runner script. But I didn't have a web server on my computer. And I didn't want to install it. So, how to deal with that?

I found out about Werkzeug. It was supposed to solve my issue. To install it run system-wide

sudo pip install werkzeug

I made a runner script run.py that makes a dev server with Werkzeug and voila:

run.py

from werkzeug.serving import run_simple
from wsgi import application

run_simple('127.0.0.1', 4000, application)

where wsgi.py is former django.wsgi file.

Run Django with SQLite

In begin with check whether SQLite has been installed in your Ubuntu

sudo apt-get install sqlite

then in settings.py file:

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'path/to/db'

# the rest of the fields are not used

This trick very useful for development stuff because i.e. with Postgres tests run very slow, and etcetera

Run GAE SDK for Python and Django on Ubuntu Maverick

First, I make the following structure in my project's directory:
~/projects/gae/ - directory for GAE's projects
~/projects/gae/google_appengine/ - downloaded GAE sources
~/projects/gae/google_appengine/test/ - test project using GAE

So, to run this test project as well as other further projects that use GAE

1. Download Google App Engine SDK for Python and unpack it to ~/projects/gae/.
   
2. Install python-2.5
   

   sudo add-apt-repository ppa:fkrull/deadsnakes
   sudo apt-get update
   sudo apt-get install python2.5
   

3. Install easy_install for Python 2.5
   

   cd /tmp
   wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg
   sh setuptools-0.6c11-py2.5.egg
   

4. Install Django 1.1 for Python 2.5
   

   easy_install-2.5 django==1.1.1
   

5. Put into ~/projects/gae/google_appengine/test file
   

   echo "#!/bin/bash
   
   python2.5 ../google_appengine/dev_appserver.py ./" > run
   chmod a+x run
   

When you need to run development server for the Test project you should do

cd ~/projects/gae/google_appengine/test/
./run

Django model field inheritance

To inherit model field in Django is not enough simply inherit a field class. It is needed to add attribute __metaclass__ = models.SubfieldBase. After that, you can overload any method you want.

Django and Oracle support

To get work Django and Oracle client in Ubuntu:

Download cx_Oracle (choose for your system), Oracle InstantClient and Sql*Plus (in this page you can choose your OS and download the packages (in my case are oracle-instantclient11.2-basic-11.2.0.0.2-1.x86_64.rpm and oracle-instantclient11.2-sqlplus-11.2.0.0.2-1.x86_64.rpm).

Move the files into one directory for simplicity and run

alien -k *.rpm
dpkg -i *.deb

to convert all the downloaded packages to ubuntu deb-format and install them.

Then make symlink to lib, otherwise module won't be found by Django

cd /usr/lib/python2.6
ln -s site-packages/cx_Oracle.so

To avoid

Traceback (most recent call last):
  File "", line 1, in 
ImportError: libclntsh.so.10.1: cannot open shared object file: No such file or directory

add to ~/.bashrc (or /etc/bash.bashrc to system-wide apply)

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/oracle/11.2/client64/lib

If you see the message

Traceback (most recent call last):
  File "", line 1, in 
ImportError: libaio.so.1: cannot open shared object file: No such file or directory

run

sudo apt-get install -y libaio1

P.S. If you choose more modern (or older) version of Oracle IntstantClient you have to create symlinks for necessary version, e.g. you see the following error:

Traceback (most recent call last):
  File "", line 1, in 
ImportError: libclntsh.so.10.1: cannot open shared object file: No such file or directory

but in /usr/lib/oracle/11.2/client64/lib directory exists libclntsh.so.11.1. Just make it symbolic link with

ln -s libclntsh.so.11.1 libclntsh.so.10.1

Setup memcached cache backend in Django in Ubuntu 9

In order to run them together we have to:

1. Install necessary packages
   
   

   sudo apt-get install build-essential python-dev memcached libmemcache-dev python-dev
   

   
   
2. Install cmemcache (using easy_install)
   
   

   easy_install http://gijsbert.org/downloads/cmemcache/cmemcache-0.95.tar.bz2

   
   
3. And, optionally and for test only, run memcached
   
   

   memcached -d -m 1024 -i 0.0.0.0 11211

   
   

Django auto language selection

I want user can select language he prefers. But I didn't found how to do it in Django easy. And I stopped on the following decision.

At first we need middleware which will detects language parameter in GET (name is in settings.LANGUAGE_COOKIE_NAME) and sets session's paramater `django_language`.

import settings

class LocaleDetectMiddleware(object):
    """
    This middleware parse GET request and detects the language user chosen
    """
    def process_request(self, request):
        if settings.LANGUAGE_COOKIE_NAME in request.GET:
            request.session['django_language'] = request.GET[settings.LANGUAGE_COOKIE_NAME]

Note, this middleware must be included in settings.INSTALLED_APPS before django.middleware.locale.LocaleMiddleware, because we use Django's standard locale detection routine.

South for Django, or how to keep database up to date

South is a migration tool for Django.

There two ways to install South: system-wide or per-project. I choose the second way because I always know that it's works with the project.

1. Get from repository.
   

   hg clone http://bitbucket.org/andrewgodwin/south/
   hg update -C 0.6.2 #this is stable version now
           

2. copy south dir to project
3. Add it to INSTALLED_APPS
4. In order to create all system models
   

   ./manage syncdb 
           

5. Further you should create necessary apps and its models (don't forget to add them to INSTALLED_APPS)
6. To create initial migration for an application run this command
   

   ./manage.py startmigration <app name> --initial
           

7. After you changed the model
   

   ./manage.py startmigration <app name>
           

   to create migration
8. To migrate to current database state
   

   ./manage.py migrate [app]
           

These are the most often operations. More detailed info you get in official documentation.

Django authentication

How to login a user in Django? There are two ways.

First, "django way", use the function authenticate which returns User model with very necessary attribute backend. This way is good when you log a user in from a form when you know its username and password. But what do to do if needs to log the user in which loaded directly from User model, when you don't know its password (you just have a strongly encoded password's version)?

After a bit of googling I just was able to found a dirty hack - manually append backend attribute to User's object instance. Does anyone know a "django way" to this?

Here's the "Django way" code snippet

user = authenticate(username='user', password='password')
login(request, user)

And here's 'hack':

user = user_info.user
# a dirty hack ;(
user.backend = 'django.contrib.auth.backends.ModelBackend'
login(request, user)

Use objects as a django view functions

To use objects instead of simple functions I apply the following method:

class Page(object):
    # Variables
    template = "main.html"
    vars = {}
    redirect_url = ""

    def __call__(self,request,*args,**kwargs):
        # Before calling a "main" content-generating method
        self.before_content(request,*args,**kwargs)
        # "Main" content-generating method. Must be declared in descendants
        self.content(request,*args,**kwargs)
        # After generating content
        self.after_content(request,*args,**kwargs)

        # if need to redirect - do it
        if self.redirect_url:
            # if is present special redirect-field in GET
            url = request.GET.get(REDIRECT_FIELD_NAME)
            return HttpResponseRedirect(url if url else self.redirect_url)
        # Just return early declared template
        return render_to_response(self.template, context_instance=RequestContext(request, self.vars))

    def before_content(self,request,*args,**kwargs):
        pass
    def after_content(self,request,*args,**kwargs):
        pass


class PageAuth(Page):
    def __call__(self,request,*args,**kwargs):
        if request.user.is_authenticated():
            return super(PageAuth,self).__call__(request,*args,**kwargs)
        return HttpResponseRedirect("{0}?{1}={2}".format(reverse(settings.LOGIN_VIEW),REDIRECT_FIELD_NAME,request.path))

class MainView(Page):
    def content(self,request,*args,**kwargs):
        self.template = 'template_name.html'
        self.vars['list'] = []

And has been changed url declarations. An example:

url(r'^$', MainView(), name="main_page")

(I couldn't find decision without using name parameter).