New language injection in PyCharm

I love the PyCharm's ability to recognize one code type inside another. It is very convenient to use all the help to edit, say, SQL query inside a python function. However, I noticed if you have an HTML snippet inside the python code, it's not recognized as HTML, just as mere string. Time to fix it.

1. Go to Preference -> Language Injections


2. Press Add (the plus sign icon under the list of existing injections) and choose 2. Generic Python


3. Enter the following in the dialog.

4. Press OK to save all your changes

Now go to a python file and try it

Top 5 biggest tables in currently active scheme in PostgreSQL

SELECT C.relname                                     AS table_name,
       pg_size_pretty(pg_total_relation_size(C.oid)) AS total_size,
       S.n_live_tup                                  AS total_rows
FROM pg_class C
       LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
       LEFT JOIN pg_stat_user_tables S ON (C.relname = S.relname)
WHERE nspname NOT IN ('pg_catalog', 'information_schema')
  AND C.relkind <> 'i'
  AND nspname !~ '^pg_toast'
ORDER BY pg_total_relation_size(C.oid) DESC
LIMIT 5;

Drop all active connections to PostgreSQL server

SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'db' -- ! replace db with the target database name
  AND pid <> pg_backend_pid();

Questions for a Senior Python develoepr

A list of question to help you assess your Python seniority.

Questions

• When will the else part of try-except-else be executed?
• What are metaclasses in Python?
• What is monkey patching? How to use in Python? Example?
• What are the tools that help to find bugs or perform static analysis? What static code analyzers do you know/used?
• Whenever Python exits, why isn’t all the memory de-allocated?
• Explain how can you access a module written in Python from C? Vise versa?
• What do these mean to you: @classmethod, @staticmethod, @property?
• Is Python a functional language?
• What is the attribute __slots__?
• Is it possible to use the construction True = False?
• How to create a class without the class statement?
• Give an example of filter and reduce over an iterable object
• Is it possible to have a producer thread reading from the network and a consumer thread writing to a file, really work in parallel? What about GIL?
• How do you create a dictionary which can preserve the order of pairs?
• What does Python's code optimization do?
• What are descriptors? Code example?
• What is MRO in Python? How does it work?
• Mention what is the difference between Django, Pyramid, and Flask?
• Identify the pitfalls/limitations in the function code.
• Identify the pros and cons of the asynchronous code.
• How to package code in Python?
• What is wheels and eggs? What is the difference?
• How to package binary dependencies in Python?
• How can I reload a previously imported module?
• What is the process of compilation and linking in python?
• What id() function in Python is for?
• What method is used to pass variables into function?

Code involving questions:

• Explain how you reverse a generator?
• Let A and B be objects of class Foo. What methods and in what order are called when "print (A + B)" is executed?
• Place the following functions below in order of their efficiency. How would you test your answer?
  
  

  def f1(lIn):
      l1 = sorted(lIn)
      l2 = [i for i in l1 if i>0.5]
      return [i*i for i in l2]
  

  
  

  def f2(lIn):
      l1 = [i for i in lIn if i>0.5]
      l2 = sorted(l1)
      return [i*i for i in l2]
  

  
  

  def f3(lIn):
      l1 = [i*i for i in lIn]
      l2 = sorted(l1)
      return [i for i in l1 if i>(0.5*0.5)]
  

• Write a one-liner that will count the number of capital letters in a file. Your code should work even if the file is too big to fit in memory.
• Output? Why? Is this inheritance?
  
  

  class C:
      pass
  type (C ())
  type (C)

• Explain the output:
  

  big_num_1   = 1000
  big_num_2   = 1000
  small_num_1 = 1
  small_num_2 = 1
  big_num_1 is big_num_2
  >>> False
  small_num_1 is small_num_2
  >>> True
  

• How is this possible?
  

  _MangledGlobal__mangled = 23
  class MangledGlobal:
       def test(self):
           return __mangled
  >>> MangledGlobal().test()
  23
      

Inspired by this article.

Local docker deployment update with AWS CodeBuild

#!/usr/bin/env bash

# Prerequisites
# $WORKDIR/.env - environment variables
# awscli installed virtualenv env dir in $WORKDIR

set -e

WORKDIR=
PROJECT=project name
DOCKER_REPO=codebuild docker image
REPO_TAG=latest

cd $WORKDIR

while [[ $# -gt 0 ]]
do
key="$1"
case $key in
    -h|--help)
    echo '  -s|--skip-build - skip building image'
    exit 0
    ;;
    -s|--skip-build)
    SKIP_BUILD=yes
    ;;
    *)
    echo "Unknown option $2."
    exit 2
    ;;
esac
shift
done

print() {
    echo -e "\033[0;33m$1\033[0m"
}

if [ -z "$SKIP_BUILD" ]
then
    print 'Starting build'
    ARN=$(env/bin/aws codebuild start-build --project-name $PROJECT | python -c 'import json,sys;print(json.load(sys.stdin))["build"]["arn"]')
    print "Build started (ARN $ARN)"
    print "Waiting 3m"

    secs=180
    while [ $secs -gt 0 ]; do
        echo -ne "Time remaining: ${secs}s...\033[0K\r"
        sleep 1
        : $((secs--))
    done
    echo -ne '\033[0K\r'

    print "Checking if it's ready."
    while [ $(env/bin/aws codebuild batch-get-builds --ids "$ARN" | python -c 'import json,sys;print(json.load(sys.stdin))["builds"][0]["buildComplete"]') != 'True' ]
    do
        echo "...not ready yet. Waiting 5s."
        sleep 5s
    done
    print "Build's ready. Update docker container."
else
    print "Using existing build."
fi

sudo $(env/bin/aws ecr get-login --no-include-email --region us-east-1)
sudo docker pull $DOCKER_REPO:$REPO_TAG
sudo docker tag $DOCKER_REPO $PROJECT
sudo docker rm -f $PROJECT
sudo docker run -d --restart=always --env-file $WORKDIR/.env -p 8001:8000 --name=$PROJECT $PROJECT

print "Update finished."

Django query logging

Recently I had to optimise some piece of legacy code that worked slow. Among other refactoring procedures I wanted to check what queries gets invoked exactly. Having dug Django Docs and internets I came up with solution.

Using this, you'll get all executed queries in the console

For example:

2016-02-25 15:57:47 DEBUG    utils.py:89 (0.055) SET SQL_AUTO_IS_NULL = 0; args=None
2016-02-25 15:57:47 DEBUG    utils.py:89 (0.435) SELECT `table`.`id` FROM `table` ORDER BY `table`.`id` ASC LIMIT 10000; args=()

Add the code below to your local_settings.py (or whatever it's called in your project), make sure the include of this code is AFTER defining LOGGING variable.

DB_LOG = True

from settings import LOGGING
if DB_LOG:
    import logging, re

    class QueryFilter(logging.Filter):
        def filter(self, record):
            # exclude PyCharm debug window queries
            return 'LIMIT 21' not in getattr(record, 'sql', '')


    class SqlFormatter(logging.Formatter):
        def format(self, record):
            record.msg = re.sub(r"((?:u?')?\d+(?:u?')?)"
                                r"(?:,\s*(?:u?')?\d+(?:u?')?)+",
                                r'\1, ...', record.msg)
            return super(SqlFormatter, self).format(record)

    params = {
        'filters': {
            'db_query_filter': {
                '()': QueryFilter,
            },
        },
        'formatters': {
            'debugging': {
                '()': SqlFormatter,
                'format': '%(asctime)s %(levelname)-8s %(filename)s:%(lineno)d %(message)s',
                'datefmt' : '%Y-%m-%d %H:%M:%S',
            },
        },
        'handlers': {
            'db_console': {
                'level': 'DEBUG',
                'filters': ['require_debug_true', 'db_query_filter'],
                'class': 'logging.StreamHandler',
                'formatter': 'debugging',
            },
        },
        'loggers': {
            'django.db': {
                'level': 'DEBUG',
                'handlers': ['db_console'],
                'propagate': False,
            },
        }
    }
    for key, param in params.items():
        try:
            LOGGING[key].update(param)
        except KeyError:
            LOGGING[key] = params

No more commits with debugging code

Most of you have certainly found yourself in the situation when after committing and pushing you realise you've forgotten to delete some code you added for debugging purpose? I am prone to it and after yet another doing it I decided to let git itself monitor my commits and prevent me from wrongdoing. From me does it require to put a comment in the file I do not want to be committed before all the junk is gone.

To do so, you need to create .git/hooks/pre-commit file (or append to the existing one):

#!/bin/sh

if git rev-parse --verify HEAD >/dev/null 2>&1
then
 against=HEAD
else
 # Initial commit: diff against an empty tree object
 against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
fi

# Redirect output to stderr.
exec 1>&2

for F in $(git status --short -uno --porcelain | egrep -v '^D' | awk '{ printf $2"\n"; }')
do
 if grep -iq nocommit "$F"; then
  cat <<EOF
Error: '$F' containts "no commit" comment.
You must delete it before it can be committed.
EOF
  exit 1
 fi
done

And add NOCOMMIT (case insensetive) somewhere in your file that you don't want to commit, i.e.

# NOCOMMIT
import pdb
a = foo(1, 4)
pdb.set_trace()

Hope it'll save some nerves for you.

Install GeoIP from MaxMind on OS X

I could not find a package ready to use to get running GeoIP from MaxMind on OS X so I built it from sources. To do so, I

• Downloaded sources
  
• Unpacked it tar xvfz GeoIP-latest.tar.gz
  
• cd GeoIP-1.5.1
  
• ./configure
  
• make
  
• make check
  
• make install
  

That is it. It's ready to go.

Parse any number string to float

function stringToFloat(str) {
    var fractional_part, integer, fractional = "", found = false, i;
    // remove leading and trailing non-digit symbols
    str = str.replace(/[^\d,\.]+$/, '').replace(/^[^\d,\.]+/, '');
    fractional_part = str.slice(-3);
    for (i = fractional_part.length - 1; i >= 0; i--) {
        if (/\d/.test(fractional_part[i])) {
            fractional = fractional_part[i] + fractional;
        } else {
            found = true;
            break;
        }
    }
    if (fractional && found) {
        integer = str.slice(0, i - 3);
    } else {
        integer = str;
        fractional = "";
    }
    return parseFloat(integer.replace(/[^\d]/g, '') + "." + fractional);
}

Delete tags

To delete tags in git that match some regex, both locally and remotely, I use these two commands

# remove tags on origin
git ls-remote --tags  | awk '/tags\/release.*[^\}]$/ {print ":" $2}' | xargs git push origin

# remove locally
git tag -l | awk '/^release.*$/ {print $1}' | xargs git tag -d

Increase trackpad speed beyond system settings

If you are not satisfied with your trackpad speed and you've already set it to the maximum in System Settings pane, it could help you.

Go to Terminal and

open ~/Library/Preferences/.GlobalPreferences.plist

You need com.apple.trackpad.scaling. System Preferencies's maximum is 3. From my experience, 30 is insane. I use 15, it's enough to move cursor with one movement from the left side of left screen the right of the right one.

Showing an image when dragging any tag in HTML

Chrome and Safari support displaying a ghost image for draggable objects out of the box, Firefox does not. It's not a big deal, I'd say a matter of one event handler.

This is a sample HTML:

Drag Me!

and JavaScript code:

$(function () {
    var dragImage = document.createElement("img");
    dragImage.src = "/image/to/show/when/dragging.jpg";
    function handleDragStart(e) {
        e.dataTransfer.effectAllowed = 'move';
        e.dataTransfer.setDragImage(dragImage, e.layerX, e.layerY);
    }

    $("#draggableObject").get(0).addEventListener('dragstart', handleDragStart, false);
});

Native HTML5 Drag and Drop

MDN::dataTransfer

Line numbers in Vim

I would like to cover all aspects known for me related to the line numbers in Vim.

Command	Description
:0	move cursor to the first
:42 42gg 42G	move cursor to 42nd
:G	move cursor to the last

Show line numbers in current file

:set number

Hide line numbers in current file

:set nonumber

Computed field on TabularInline (StackedInline)

Django admin doesn't provide list_display attribute on its inline admin classes. That's right, since the only supposed view for those classes is editable form. So, to display a result of custom function's call, it needs to add the name of the function (that can be declared either on the admin or model classed) to both fields and readonly_fields.

Example:

class StatsInline(admin.TabularInline):
    model = Stats
    fields = ('clicked', 'shown', 'avg')
    readonly_fields = ('avg',)
    verbose_name = 'Stats'
    verbose_name_plural = 'Stats'
    can_delete = False

    def avg(self, obj):
        return float(obj.clicked) / obj.shown if obj.shown else 0

P.S. float conversion is needed to avoid casting result to int (e.g. 1/2=0, float(1)/2=0.5)

Reduce directory nesting

Once I encountered a problem with image files that were named according to their checksum of the content chunked by 2 symbols 6 times, so it gets like 2f/d4/e1/c6/7a/2d/28fced849ee1bb76e7391b93eb12.jpg.

To have this structure with approx. 13M files, I got short in inodes. After a quick research and calculations I realized that such nesting depth is redundant and easily can be reduced. So I need to turn all the existing files of form 2f/d4/e1/c6/7a/2d/28fced849ee1bb76e7391b93eb12.jpg to 2f/d4/e1c67a2d28fced849ee1bb76e7391b93eb12.jpg.

To do that, I wrote a script that renames the files and removes empty directories.

TARGET_DIR = '/full/path/to/dir/containing/dirs/to/reduce/'
TARGET_DIR_LEN = len(TARGET_DIR) + 1
CURRENT_DEPTH = 6
DESIRED_DEPTH = 2

last_dir = ''
for dirname, _, filenames in os.walk(TARGET_DIR):
    path = dirname[TARGET_DIR_LEN:].split('/')
    if filenames and len(path) >= CURRENT_DEPTH:
        cur_dir = '/'.join(path[:DESIRED_DEPTH - CURRENT_DEPTH])
        if last_dir != cur_dir:
            last_dir = cur_dir
        for filename in filenames:
            fullpath = os.path.join('/'.join(path), filename)
            parts = fullpath.split('/')
            prefix = ''
            if len(parts[0]) > 2: # cache
                prefix = parts[0]
                parts = parts[1:]
            old = os.path.join(dirname, filename)
            new = os.path.join(TARGET_DIR, prefix,
                               '/'.join(parts[:DESIRED_DEPTH]),
                               ''.join(parts[DESIRED_DEPTH:]))
            pass
            os.rename(old, new)
        for i in range(CURRENT_DEPTH - DESIRED_DEPTH):
            d = os.path.join(TARGET_DIR,
                             dirname[0:(DESIRED_DEPTH - CURRENT_DEPTH + 1) * i]
                             if i else dirname)
            try:
                os.rmdir(d)
            except OSError:
                break

Sync your development environment with a remote one

While the developing process you need sometimes to sync you local development database with a remote one (i.e. production).

Below there is a script allowing to recreate the development database and fill in with data from the remote server.

MySQL

#!/bin/sh

DBUSER_REMOTE="root"
DBPASSWORD_REMOTE="root"
DBUSER_LOCAL="root"
DBPASSWORD_LOCAL="root"
DBNAME_REMOTE=""
DBNAME_LOCAL=""
TMPFILE="$(mktemp db.XXXXXXX)"
HOST="www.example.com"

ssh $HOST "mysqldump -u$DBUSER_REMOTE -p$DBPASSWORD_REMOTE $DBNAME_REMOTE | gzip" | gunzip > $TMPFILE

mysqladmin -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL -f drop $DBNAME_LOCAL
mysqladmin -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL create $DBNAME_LOCAL
mysql -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL $DBNAME_LOCAL < $TMPFILE
rm $TMPFILE

PostgreSQL

#!/bin/sh

DBUSER_REMOTE="postgres"
DBUSER_LOCAL="postgres"
DBNAME_REMOTE=""
DBNAME_LOCAL=""
TMPFILE="$(mktemp db.XXXXXXX)"
HOST="www.example.com"

ssh $HOST "pg_dump -U $DBUSER_REMOTE -Fc $DBNAME_REMOTE" > $TMPFILE

dropdb -U $DBUSER_LOCAL $DBNAME_LOCAL
createdb -U $DBUSER_LOCAL $DBNAME_LOCAL
pg_restore -U $DBUSER_LOCAL -Fc -d $DBNAME_LOCAL $TMPFILE
rm $TMPFILE

Sync media files

$HOST="www.example.com"
$PATH="/absolute/path/to/media"
$LOCAL_DIR="."
rsync -rltDvH $HOST:$PATH $LOCAL_DIR

Favicon sizes

The browser can use the favicon in different situations, like to show it on the tab, for the bookmark and so forth. There are a few sizes the favicon can be. Here I gathered the most often and broadly used.

Favicon sizes

<link rel="icon" type="image/png" sizes="16x16" href="favicon@16.png">
<link rel="icon" type="image/png" sizes="32x32" href="favicon@32.png">
<link rel="icon" type="image/png" sizes="48x48" href="favicon@48.png">
<link rel="icon" type="image/png" sizes="64x64" href="favicon@64.png">
<link rel="icon" type="image/png" sizes="128x128" href="favicon@128.png">
<link rel="shortcut icon" href="favicon.ico">

Apple Icons (used sizes)

<link rel="icon" type="image/png" sizes="144x144" href="favicon@144.png">
<link rel="icon" type="image/png" sizes="512x512" href="favicon@512.png">
<link rel="apple-touch-icon" type="image/png" sizes="57x57" href="favicon@57.png">
<link rel="apple-touch-icon" type="image/png" sizes="144x144" href="favicon@144.png">

PyCharm can't run a project with MySQL database

I wanted to run my project in PyCharm IDE and got this error.

django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: dlopen(/Users/boris/env/lib/python2.7/site-packages/_mysql.so, 2): Library not loaded: libmysqlclient.18.dylib
  Referenced from: /Users/boris/env/lib/python2.7/site-packages/_mysql.so
  Reason: image not found

After googling a while I figured out there is a mess with DYLD_LIBRARY_PATH environment variable. So, first I needed to locate the path where libmysqlclient.18.dylib located.

$ locate libmysqlclient.18.dylib
/usr/local/mysql-5.5.22-osx10.6-x86_64/lib/libmysqlclient.18.dylib

Done. In Run/Debug Configurations in Environment variables I added (plus to PYTHONUNBUFFERED=1) DYLD_LIBRARY_PATH=/usr/local/mysql-5.5.22-osx10.6-x86_64/lib/:$DYLD_LIBRARY_PATH.

It started working.

Fatal Python error: Couldn't create autoTLSkey mapping

My last project I started with Django 1.4 and Django Compressor. Today I decided to deploy to the production its first version. I used nginx+Apache.

Everything seemed to be good but instead of normal page I got:

FilterError at /
Unable to apply CompilerFilter (lessc {infile} {outfile})

I tried to repeat the same with ./manage.py runserver. It works. So the issue not in configuration or lessc itself. Later I found here that it is Apache's issue and can be solved with

WSGIApplicationGroup %{GLOBAL}

directive in Apache's VirtualHost config.

It helped.

Install MySQL on Mac OS X to work with Python

I need to get mysql support for python on my Mac OS X Lion.

Download the latest MySQL .DMG distributive from somewhere (i.e. official website) and install it.

sudo pip install python-mysql

It requires superuser privileges to get installed.

What may happen:

while installing mysql-python

Downloading/unpacking mysql-python
  Downloading MySQL-python-1.2.3.tar.gz (70Kb): 70Kb downloaded
  Running setup.py egg_info for package mysql-python
    sh: mysql_config: command not found
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/Users/boris/projects/a1/th4x/build/mysql-python/setup.py", line 15, in <module>
        metadata, options = get_config()
      File "setup_posix.py", line 43, in get_config
        libs = mysql_config("libs_r")
      File "setup_posix.py", line 24, in mysql_config
        raise EnvironmentError("%s not found" % (mysql_config.path,))
    EnvironmentError: mysql_config not found
    Complete output from command python setup.py egg_info:
    sh: mysql_config: command not found

Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/Users/boris/projects/a1/th4x/build/mysql-python/setup.py", line 15, in <module>

    metadata, options = get_config()

  File "setup_posix.py", line 43, in get_config

    libs = mysql_config("libs_r")

  File "setup_posix.py", line 24, in mysql_config

    raise EnvironmentError("%s not found" % (mysql_config.path,))

EnvironmentError: mysql_config not found

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /Users/boris/.pip/pip.log

solution:

Add /usr/local/mysql/bin/ to PATH

export PATH=/usr/local/mysql/bin/:$PATH

(from my ~/.bashrc)

trying to run dev server

Unhandled exception in thread started by <bound method Command.inner_run of <django.contrib.staticfiles.management.commands.runserver.Command object at 0x10c59bc90>>
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/django/core/management/commands/runserver.py", line 88, in inner_run
    self.validate(display_num_errors=True)
  File "/Library/Python/2.7/site-packages/django/core/management/base.py", line 249, in validate
    num_errors = get_validation_errors(s, app)
  File "/Library/Python/2.7/site-packages/django/core/management/validation.py", line 35, in get_validation_errors
    for (app_name, error) in get_app_errors().items():
  File "/Library/Python/2.7/site-packages/django/db/models/loading.py", line 146, in get_app_errors
    self._populate()
  File "/Library/Python/2.7/site-packages/django/db/models/loading.py", line 64, in _populate
    self.load_app(app_name)
  File "/Library/Python/2.7/site-packages/django/db/models/loading.py", line 78, in load_app
    models = import_module('.models', app_name)
  File "/Library/Python/2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/boris/projects/a1/th4x/tradecontrol/models.py", line 4, in <module>
    from tradecontrol.db import auto_detect
  File "/Users/boris/projects/a1/th4x/tradecontrol/db/__init__.py", line 2, in <module>
    import MySQLdb as mysql
  File "/Library/Python/2.7/site-packages/MySQLdb/__init__.py", line 19, in <module>
    import _mysql
ImportError: dlopen(/Library/Python/2.7/site-packages/_mysql.so, 2): Library not loaded: libmysqlclient.18.dylib
  Referenced from: /Library/Python/2.7/site-packages/_mysql.so
  Reason: image not found

solution:

Add to your ~/.bashrc add

export DYLD_LIBRARY_PATH=/usr/local/mysql/lib

and then reread .bashrc

. ~/.bashrc