Friday, February 26, 2016

Django query logging

Recently I had to optimise some piece of legacy code that worked slow. Among other refactoring procedures I wanted to check what queries gets invoked exactly. Having dug Django Docs and internets I came up with solution.

Using this, you'll get all executed queries in the console

For example:

2016-02-25 15:57:47 DEBUG (0.055) SET SQL_AUTO_IS_NULL = 0; args=None
2016-02-25 15:57:47 DEBUG (0.435) SELECT `table`.`id` FROM `table` ORDER BY `table`.`id` ASC LIMIT 10000; args=()

Add the code below to your (or whatever it's called in your project), make sure the include of this code is AFTER defining LOGGING variable.

DB_LOG = True

from settings import LOGGING
if DB_LOG:
    import logging, re

    class QueryFilter(logging.Filter):
        def filter(self, record):
            # exclude PyCharm debug window queries
            return 'LIMIT 21' not in getattr(record, 'sql', '')

    class SqlFormatter(logging.Formatter):
        def format(self, record):
            record.msg = re.sub(r"((?:u?')?\d+(?:u?')?)"
                                r'\1, ...', record.msg)
            return super(SqlFormatter, self).format(record)

    params = {
        'filters': {
            'db_query_filter': {
                '()': QueryFilter,
        'formatters': {
            'debugging': {
                '()': SqlFormatter,
                'format': '%(asctime)s %(levelname)-8s %(filename)s:%(lineno)d %(message)s',
                'datefmt' : '%Y-%m-%d %H:%M:%S',
        'handlers': {
            'db_console': {
                'level': 'DEBUG',
                'filters': ['require_debug_true', 'db_query_filter'],
                'class': 'logging.StreamHandler',
                'formatter': 'debugging',
        'loggers': {
            'django.db': {
                'level': 'DEBUG',
                'handlers': ['db_console'],
                'propagate': False,
    for key, param in params.items():
        except KeyError:
            LOGGING[key] = params

Thursday, February 25, 2016

No more commits with debugging code

Most of you have certainly found yourself in the situation when after committing and pushing you realise you've forgotten to delete some code you added for debugging purpose? I am prone to it and after yet another doing it I decided to let git itself monitor my commits and prevent me from wrongdoing. From me does it require to put a comment in the file I do not want to be committed before all the junk is gone.

To do so, you need to create .git/hooks/pre-commit file (or append to the existing one):


if git rev-parse --verify HEAD >/dev/null 2>&1
 # Initial commit: diff against an empty tree object

# Redirect output to stderr.
exec 1>&2

for F in $(git status --short -uno --porcelain | egrep -v '^D' | awk '{ printf $2"\n"; }')
 if grep -iq nocommit "$F"; then
  cat <<EOF
Error: '$F' containts "no commit" comment.
You must delete it before it can be committed.
  exit 1

And add NOCOMMIT (case insensetive) somewhere in your file that you don't want to commit, i.e.

import pdb
a = foo(1, 4)

Hope it'll save some nerves for you.

Tuesday, September 3, 2013

Install GeoIP from MaxMind on OS X

I could not find a package ready to use to get running GeoIP from MaxMind on OS X so I built it from sources. To do so, I
  • Downloaded sources
  • Unpacked it tar xvfz GeoIP-latest.tar.gz
  • cd GeoIP-1.5.1
  • ./configure
  • make
  • make check
  • make install
That is it. It's ready to go.

Friday, August 2, 2013

Parse any number string to float

function stringToFloat(str) {
    var fractional_part, integer, fractional = "", found = false, i;
    // remove leading and trailing non-digit symbols
    str = str.replace(/[^\d,\.]+$/, '').replace(/^[^\d,\.]+/, '');
    fractional_part = str.slice(-3);
    for (i = fractional_part.length - 1; i >= 0; i--) {
        if (/\d/.test(fractional_part[i])) {
            fractional = fractional_part[i] + fractional;
        } else {
            found = true;
    if (fractional && found) {
        integer = str.slice(0, i - 3);
    } else {
        integer = str;
        fractional = "";
    return parseFloat(integer.replace(/[^\d]/g, '') + "." + fractional);

Tuesday, July 2, 2013

Delete tags

To delete tags in git that match some regex, both locally and remotely, I use these two commands

# remove tags on origin
git ls-remote --tags  | awk '/tags\/release.*[^\}]$/ {print ":" $2}' | xargs git push origin

# remove locally
git tag -l | awk '/^release.*$/ {print $1}' | xargs git tag -d

Tuesday, May 14, 2013

Increase trackpad speed beyond system settings

If you are not satisfied with your trackpad speed and you've already set it to the maximum in System Settings pane, it could help you.

Go to Terminal and

open ~/Library/Preferences/.GlobalPreferences.plist

You need System Preferencies's maximum is 3. From my experience, 30 is insane. I use 15, it's enough to move cursor with one movement from the left side of left screen the right of the right one.

Thursday, April 25, 2013

Showing an image when dragging any tag in HTML

Chrome and Safari support displaying a ghost image for draggable objects out of the box, Firefox does not. It's not a big deal, I'd say a matter of one event handler.

This is a sample HTML:
Drag Me!

and JavaScript code:
$(function () {
    var dragImage = document.createElement("img");
    dragImage.src = "/image/to/show/when/dragging.jpg";
    function handleDragStart(e) {
        e.dataTransfer.effectAllowed = 'move';
        e.dataTransfer.setDragImage(dragImage, e.layerX, e.layerY);

    $("#draggableObject").get(0).addEventListener('dragstart', handleDragStart, false);

Native HTML5 Drag and Drop


Monday, April 15, 2013

Line numbers in Vim

I would like to cover all aspects known for me related to the line numbers in Vim.

move cursor to the first
move cursor to 42nd
move cursor to the last

Show line numbers in current file
:set number

Hide line numbers in current file
:set nonumber

Tuesday, April 9, 2013

Computed field on TabularInline (StackedInline)

Django admin doesn't provide list_display attribute on its inline admin classes. That's right, since the only supposed view for those classes is editable form. So, to display a result of custom function's call, it needs to add the name of the function (that can be declared either on the admin or model classed) to both fields and readonly_fields.

class StatsInline(admin.TabularInline):
    model = Stats
    fields = ('clicked', 'shown', 'avg')
    readonly_fields = ('avg',)
    verbose_name = 'Stats'
    verbose_name_plural = 'Stats'
    can_delete = False

    def avg(self, obj):
        return float(obj.clicked) / obj.shown if obj.shown else 0

P.S. float conversion is needed to avoid casting result to int (e.g. 1/2=0, float(1)/2=0.5)

Thursday, November 22, 2012

Reduce directory nesting

Once I encountered a problem with image files that were named according to their checksum of the content chunked by 2 symbols 6 times, so it gets like 2f/d4/e1/c6/7a/2d/28fced849ee1bb76e7391b93eb12.jpg.

To have this structure with approx. 13M files, I got short in inodes. After a quick research and calculations I realized that such nesting depth is redundant and easily can be reduced. So I need to turn all the existing files of form 2f/d4/e1/c6/7a/2d/28fced849ee1bb76e7391b93eb12.jpg to 2f/d4/e1c67a2d28fced849ee1bb76e7391b93eb12.jpg.

To do that, I wrote a script that renames the files and removes empty directories.

TARGET_DIR = '/full/path/to/dir/containing/dirs/to/reduce/'

last_dir = ''
for dirname, _, filenames in os.walk(TARGET_DIR):
    path = dirname[TARGET_DIR_LEN:].split('/')
    if filenames and len(path) >= CURRENT_DEPTH:
        cur_dir = '/'.join(path[:DESIRED_DEPTH - CURRENT_DEPTH])
        if last_dir != cur_dir:
            last_dir = cur_dir
        for filename in filenames:
            fullpath = os.path.join('/'.join(path), filename)
            parts = fullpath.split('/')
            prefix = ''
            if len(parts[0]) > 2: # cache
                prefix = parts[0]
                parts = parts[1:]
            old = os.path.join(dirname, filename)
            new = os.path.join(TARGET_DIR, prefix,
            os.rename(old, new)
        for i in range(CURRENT_DEPTH - DESIRED_DEPTH):
            d = os.path.join(TARGET_DIR,
                             dirname[0:(DESIRED_DEPTH - CURRENT_DEPTH + 1) * i]
                             if i else dirname)
            except OSError:

Sunday, October 7, 2012

Sync your development environment with a remote one

While the developing process you need sometimes to sync you local development database with a remote one (i.e. production).

Below there is a script allowing to recreate the development database and fill in with data from the remote server.


TMPFILE="$(mktemp db.XXXXXXX)"




TMPFILE="$(mktemp db.XXXXXXX)"



Sync media files
rsync -rltDvH $HOST:$PATH $LOCAL_DIR

Saturday, September 29, 2012

Favicon sizes

The browser can use the favicon in different situations, like to show it on the tab, for the bookmark and so forth. There are a few sizes the favicon can be. Here I gathered the most often and broadly used.

Favicon sizes

<link rel="icon" type="image/png" sizes="16x16" href="favicon@16.png">
<link rel="icon" type="image/png" sizes="32x32" href="favicon@32.png">
<link rel="icon" type="image/png" sizes="48x48" href="favicon@48.png">
<link rel="icon" type="image/png" sizes="64x64" href="favicon@64.png">
<link rel="icon" type="image/png" sizes="128x128" href="favicon@128.png">
<link rel="shortcut icon" href="favicon.ico">

Apple Icons (used sizes)

<link rel="icon" type="image/png" sizes="144x144" href="favicon@144.png">
<link rel="icon" type="image/png" sizes="512x512" href="favicon@512.png">
<link rel="apple-touch-icon" type="image/png" sizes="57x57" href="favicon@57.png">
<link rel="apple-touch-icon" type="image/png" sizes="144x144" href="favicon@144.png">

Friday, August 3, 2012

PyCharm can't run a project with MySQL database

I wanted to run my project in PyCharm IDE and got this error.

django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: dlopen(/Users/boris/env/lib/python2.7/site-packages/, 2): Library not loaded: libmysqlclient.18.dylib
  Referenced from: /Users/boris/env/lib/python2.7/site-packages/
  Reason: image not found

After googling a while I figured out there is a mess with DYLD_LIBRARY_PATH environment variable. So, first I needed to locate the path where libmysqlclient.18.dylib located.

$ locate libmysqlclient.18.dylib

Done. In Run/Debug Configurations in Environment variables I added (plus to PYTHONUNBUFFERED=1) DYLD_LIBRARY_PATH=/usr/local/mysql-5.5.22-osx10.6-x86_64/lib/:$DYLD_LIBRARY_PATH.

It started working.