Thursday, August 10, 2017

Local docker deployment update with AWS CodeBuild

#!/usr/bin/env bash

# Prerequisites
# $WORKDIR/.env - environment variables
# awscli installed virtualenv env dir in $WORKDIR

set -e

WORKDIR=
PROJECT=project name
DOCKER_REPO=codebuild docker image
REPO_TAG=latest

cd $WORKDIR

while [[ $# -gt 0 ]]
do
key="$1"
case $key in
    -h|--help)
    echo '  -s|--skip-build - skip building image'
    exit 0
    ;;
    -s|--skip-build)
    SKIP_BUILD=yes
    ;;
    *)
    echo "Unknown option $2."
    exit 2
    ;;
esac
shift
done

print() {
    echo -e "\033[0;33m$1\033[0m"
}

if [ -z "$SKIP_BUILD" ]
then
    print 'Starting build'
    ARN=$(env/bin/aws codebuild start-build --project-name $PROJECT | python -c 'import json,sys;print(json.load(sys.stdin))["build"]["arn"]')
    print "Build started (ARN $ARN)"
    print "Waiting 3m"

    secs=180
    while [ $secs -gt 0 ]; do
        echo -ne "Time remaining: ${secs}s...\033[0K\r"
        sleep 1
        : $((secs--))
    done
    echo -ne '\033[0K\r'

    print "Checking if it's ready."
    while [ $(env/bin/aws codebuild batch-get-builds --ids "$ARN" | python -c 'import json,sys;print(json.load(sys.stdin))["builds"][0]["buildComplete"]') != 'True' ]
    do
        echo "...not ready yet. Waiting 5s."
        sleep 5s
    done
    print "Build's ready. Update docker container."
else
    print "Using existing build."
fi

sudo $(env/bin/aws ecr get-login --no-include-email --region us-east-1)
sudo docker pull $DOCKER_REPO:$REPO_TAG
sudo docker tag $DOCKER_REPO $PROJECT
sudo docker rm -f $PROJECT
sudo docker run -d --restart=always --env-file $WORKDIR/.env -p 8001:8000 --name=$PROJECT $PROJECT

print "Update finished."

Friday, February 26, 2016

Django query logging

Recently I had to optimise some piece of legacy code that worked slow. Among other refactoring procedures I wanted to check what queries gets invoked exactly. Having dug Django Docs and internets I came up with solution.

Using this, you'll get all executed queries in the console

For example:

2016-02-25 15:57:47 DEBUG    utils.py:89 (0.055) SET SQL_AUTO_IS_NULL = 0; args=None
2016-02-25 15:57:47 DEBUG    utils.py:89 (0.435) SELECT `table`.`id` FROM `table` ORDER BY `table`.`id` ASC LIMIT 10000; args=()

Add the code below to your local_settings.py (or whatever it's called in your project), make sure the include of this code is AFTER defining LOGGING variable.

DB_LOG = True

from settings import LOGGING
if DB_LOG:
    import logging, re

    class QueryFilter(logging.Filter):
        def filter(self, record):
            # exclude PyCharm debug window queries
            return 'LIMIT 21' not in getattr(record, 'sql', '')


    class SqlFormatter(logging.Formatter):
        def format(self, record):
            record.msg = re.sub(r"((?:u?')?\d+(?:u?')?)"
                                r"(?:,\s*(?:u?')?\d+(?:u?')?)+",
                                r'\1, ...', record.msg)
            return super(SqlFormatter, self).format(record)

    params = {
        'filters': {
            'db_query_filter': {
                '()': QueryFilter,
            },
        },
        'formatters': {
            'debugging': {
                '()': SqlFormatter,
                'format': '%(asctime)s %(levelname)-8s %(filename)s:%(lineno)d %(message)s',
                'datefmt' : '%Y-%m-%d %H:%M:%S',
            },
        },
        'handlers': {
            'db_console': {
                'level': 'DEBUG',
                'filters': ['require_debug_true', 'db_query_filter'],
                'class': 'logging.StreamHandler',
                'formatter': 'debugging',
            },
        },
        'loggers': {
            'django.db': {
                'level': 'DEBUG',
                'handlers': ['db_console'],
                'propagate': False,
            },
        }
    }
    for key, param in params.items():
        try:
            LOGGING[key].update(param)
        except KeyError:
            LOGGING[key] = params

Thursday, February 25, 2016

No more commits with debugging code

Most of you have certainly found yourself in the situation when after committing and pushing you realise you've forgotten to delete some code you added for debugging purpose? I am prone to it and after yet another doing it I decided to let git itself monitor my commits and prevent me from wrongdoing. From me does it require to put a comment in the file I do not want to be committed before all the junk is gone.

To do so, you need to create .git/hooks/pre-commit file (or append to the existing one):

#!/bin/sh

if git rev-parse --verify HEAD >/dev/null 2>&1
then
 against=HEAD
else
 # Initial commit: diff against an empty tree object
 against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
fi

# Redirect output to stderr.
exec 1>&2

for F in $(git status --short -uno --porcelain | egrep -v '^D' | awk '{ printf $2"\n"; }')
do
 if grep -iq nocommit "$F"; then
  cat <<EOF
Error: '$F' containts "no commit" comment.
You must delete it before it can be committed.
EOF
  exit 1
 fi
done

And add NOCOMMIT (case insensetive) somewhere in your file that you don't want to commit, i.e.

# NOCOMMIT
import pdb
a = foo(1, 4)
pdb.set_trace()


Hope it'll save some nerves for you.

Tuesday, September 3, 2013

Install GeoIP from MaxMind on OS X

I could not find a package ready to use to get running GeoIP from MaxMind on OS X so I built it from sources. To do so, I
  • Downloaded sources
  • Unpacked it tar xvfz GeoIP-latest.tar.gz
  • cd GeoIP-1.5.1
  • ./configure
  • make
  • make check
  • make install
That is it. It's ready to go.


Friday, August 2, 2013

Parse any number string to float

function stringToFloat(str) {
    var fractional_part, integer, fractional = "", found = false, i;
    // remove leading and trailing non-digit symbols
    str = str.replace(/[^\d,\.]+$/, '').replace(/^[^\d,\.]+/, '');
    fractional_part = str.slice(-3);
    for (i = fractional_part.length - 1; i >= 0; i--) {
        if (/\d/.test(fractional_part[i])) {
            fractional = fractional_part[i] + fractional;
        } else {
            found = true;
            break;
        }
    }
    if (fractional && found) {
        integer = str.slice(0, i - 3);
    } else {
        integer = str;
        fractional = "";
    }
    return parseFloat(integer.replace(/[^\d]/g, '') + "." + fractional);
}

Tuesday, July 2, 2013

Delete tags

To delete tags in git that match some regex, both locally and remotely, I use these two commands

# remove tags on origin
git ls-remote --tags  | awk '/tags\/release.*[^\}]$/ {print ":" $2}' | xargs git push origin

# remove locally
git tag -l | awk '/^release.*$/ {print $1}' | xargs git tag -d

Tuesday, May 14, 2013

Increase trackpad speed beyond system settings

If you are not satisfied with your trackpad speed and you've already set it to the maximum in System Settings pane, it could help you.

Go to Terminal and

open ~/Library/Preferences/.GlobalPreferences.plist

You need com.apple.trackpad.scaling. System Preferencies's maximum is 3. From my experience, 30 is insane. I use 15, it's enough to move cursor with one movement from the left side of left screen the right of the right one.

Thursday, April 25, 2013

Showing an image when dragging any tag in HTML

Chrome and Safari support displaying a ghost image for draggable objects out of the box, Firefox does not. It's not a big deal, I'd say a matter of one event handler.

This is a sample HTML:
Drag Me!

and JavaScript code:
$(function () {
    var dragImage = document.createElement("img");
    dragImage.src = "/image/to/show/when/dragging.jpg";
    function handleDragStart(e) {
        e.dataTransfer.effectAllowed = 'move';
        e.dataTransfer.setDragImage(dragImage, e.layerX, e.layerY);
    }

    $("#draggableObject").get(0).addEventListener('dragstart', handleDragStart, false);
});

Native HTML5 Drag and Drop

MDN::dataTransfer

Monday, April 15, 2013

Line numbers in Vim



I would like to cover all aspects known for me related to the line numbers in Vim.

CommandDescription
:0
move cursor to the first
:42
42gg
42G
move cursor to 42nd
:G
move cursor to the last


Show line numbers in current file
:set number

Hide line numbers in current file
:set nonumber

Tuesday, April 9, 2013

Computed field on TabularInline (StackedInline)

Django admin doesn't provide list_display attribute on its inline admin classes. That's right, since the only supposed view for those classes is editable form. So, to display a result of custom function's call, it needs to add the name of the function (that can be declared either on the admin or model classed) to both fields and readonly_fields.

Example:
class StatsInline(admin.TabularInline):
    model = Stats
    fields = ('clicked', 'shown', 'avg')
    readonly_fields = ('avg',)
    verbose_name = 'Stats'
    verbose_name_plural = 'Stats'
    can_delete = False

    def avg(self, obj):
        return float(obj.clicked) / obj.shown if obj.shown else 0

P.S. float conversion is needed to avoid casting result to int (e.g. 1/2=0, float(1)/2=0.5)

Thursday, November 22, 2012

Reduce directory nesting

Once I encountered a problem with image files that were named according to their checksum of the content chunked by 2 symbols 6 times, so it gets like 2f/d4/e1/c6/7a/2d/28fced849ee1bb76e7391b93eb12.jpg.

To have this structure with approx. 13M files, I got short in inodes. After a quick research and calculations I realized that such nesting depth is redundant and easily can be reduced. So I need to turn all the existing files of form 2f/d4/e1/c6/7a/2d/28fced849ee1bb76e7391b93eb12.jpg to 2f/d4/e1c67a2d28fced849ee1bb76e7391b93eb12.jpg.

To do that, I wrote a script that renames the files and removes empty directories.

TARGET_DIR = '/full/path/to/dir/containing/dirs/to/reduce/'
TARGET_DIR_LEN = len(TARGET_DIR) + 1
CURRENT_DEPTH = 6
DESIRED_DEPTH = 2

last_dir = ''
for dirname, _, filenames in os.walk(TARGET_DIR):
    path = dirname[TARGET_DIR_LEN:].split('/')
    if filenames and len(path) >= CURRENT_DEPTH:
        cur_dir = '/'.join(path[:DESIRED_DEPTH - CURRENT_DEPTH])
        if last_dir != cur_dir:
            last_dir = cur_dir
        for filename in filenames:
            fullpath = os.path.join('/'.join(path), filename)
            parts = fullpath.split('/')
            prefix = ''
            if len(parts[0]) > 2: # cache
                prefix = parts[0]
                parts = parts[1:]
            old = os.path.join(dirname, filename)
            new = os.path.join(TARGET_DIR, prefix,
                               '/'.join(parts[:DESIRED_DEPTH]),
                               ''.join(parts[DESIRED_DEPTH:]))
            pass
            os.rename(old, new)
        for i in range(CURRENT_DEPTH - DESIRED_DEPTH):
            d = os.path.join(TARGET_DIR,
                             dirname[0:(DESIRED_DEPTH - CURRENT_DEPTH + 1) * i]
                             if i else dirname)
            try:
                os.rmdir(d)
            except OSError:
                break

Sunday, October 7, 2012

Sync your development environment with a remote one

While the developing process you need sometimes to sync you local development database with a remote one (i.e. production).

Below there is a script allowing to recreate the development database and fill in with data from the remote server.

MySQL
#!/bin/sh

DBUSER_REMOTE="root"
DBPASSWORD_REMOTE="root"
DBUSER_LOCAL="root"
DBPASSWORD_LOCAL="root"
DBNAME_REMOTE=""
DBNAME_LOCAL=""
TMPFILE="$(mktemp db.XXXXXXX)"
HOST="www.example.com"

ssh $HOST "mysqldump -u$DBUSER_REMOTE -p$DBPASSWORD_REMOTE $DBNAME_REMOTE | gzip" | gunzip > $TMPFILE

mysqladmin -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL -f drop $DBNAME_LOCAL
mysqladmin -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL create $DBNAME_LOCAL
mysql -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL $DBNAME_LOCAL < $TMPFILE
rm $TMPFILE

PostgreSQL
#!/bin/sh

DBUSER_REMOTE="postgres"
DBUSER_LOCAL="postgres"
DBNAME_REMOTE=""
DBNAME_LOCAL=""
TMPFILE="$(mktemp db.XXXXXXX)"
HOST="www.example.com"

ssh $HOST "pg_dump -U $DBUSER_REMOTE -Fc $DBNAME_REMOTE" > $TMPFILE

dropdb -U $DBUSER_LOCAL $DBNAME_LOCAL
createdb -U $DBUSER_LOCAL $DBNAME_LOCAL
pg_restore -U $DBUSER_LOCAL -Fc -d $DBNAME_LOCAL $TMPFILE
rm $TMPFILE

Sync media files
$HOST="www.example.com"
$PATH="/absolute/path/to/media"
$LOCAL_DIR="."
rsync -rltDvH $HOST:$PATH $LOCAL_DIR

Saturday, September 29, 2012

Favicon sizes

The browser can use the favicon in different situations, like to show it on the tab, for the bookmark and so forth. There are a few sizes the favicon can be. Here I gathered the most often and broadly used.

Favicon sizes

<link rel="icon" type="image/png" sizes="16x16" href="favicon@16.png">
<link rel="icon" type="image/png" sizes="32x32" href="favicon@32.png">
<link rel="icon" type="image/png" sizes="48x48" href="favicon@48.png">
<link rel="icon" type="image/png" sizes="64x64" href="favicon@64.png">
<link rel="icon" type="image/png" sizes="128x128" href="favicon@128.png">
<link rel="shortcut icon" href="favicon.ico">

Apple Icons (used sizes)

<link rel="icon" type="image/png" sizes="144x144" href="favicon@144.png">
<link rel="icon" type="image/png" sizes="512x512" href="favicon@512.png">
<link rel="apple-touch-icon" type="image/png" sizes="57x57" href="favicon@57.png">
<link rel="apple-touch-icon" type="image/png" sizes="144x144" href="favicon@144.png">