Wednesday, March 6, 2019

New language injection in PyCharm

I love the PyCharm's ability to recognize one code type inside another. It is very convenient to use all the help to edit, say, SQL query inside a python function. However, I noticed if you have an HTML snippet inside the python code, it's not recognized as HTML, just as mere string. Time to fix it.

1. Go to Preference -> Language Injections


2. Press Add (the plus sign icon under the list of existing injections) and choose 2. Generic Python


3. Enter the following in the dialog.

4. Press OK to save all your changes

Now go to a python file and try it


Tuesday, February 12, 2019

Top 5 biggest tables in currently active scheme in PostgreSQL

SELECT C.relname                                     AS table_name,
       pg_size_pretty(pg_total_relation_size(C.oid)) AS total_size,
       S.n_live_tup                                  AS total_rows
FROM pg_class C
       LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
       LEFT JOIN pg_stat_user_tables S ON (C.relname = S.relname)
WHERE nspname NOT IN ('pg_catalog', 'information_schema')
  AND C.relkind <> 'i'
  AND nspname !~ '^pg_toast'
ORDER BY pg_total_relation_size(C.oid) DESC
LIMIT 5;

Drop all active connections to PostgreSQL server

SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'db' -- ! replace db with the target database name
  AND pid <> pg_backend_pid();

Monday, December 3, 2018

Questions for a Senior Python develoepr

A list of question to help you assess your Python seniority.

Questions
  • When will the else part of try-except-else be executed?
  • What are metaclasses in Python?
  • What is monkey patching? How to use in Python? Example?
  • What are the tools that help to find bugs or perform static analysis? What static code analyzers do you know/used?
  • Whenever Python exits, why isn’t all the memory de-allocated?
  • Explain how can you access a module written in Python from C? Vise versa?
  • What do these mean to you: @classmethod, @staticmethod, @property?
  • Is Python a functional language?
  • What is the attribute __slots__?
  • Is it possible to use the construction True = False?
  • How to create a class without the class statement?
  • Give an example of filter and reduce over an iterable object
  • Is it possible to have a producer thread reading from the network and a consumer thread writing to a file, really work in parallel? What about GIL?
  • How do you create a dictionary which can preserve the order of pairs?
  • What does Python's code optimization do?
  • What are descriptors? Code example?
  • What is MRO in Python? How does it work?
  • Mention what is the difference between Django, Pyramid, and Flask?
  • Identify the pitfalls/limitations in the function code.
  • Identify the pros and cons of the asynchronous code.
  • How to package code in Python?
  • What is wheels and eggs? What is the difference?
  • How to package binary dependencies in Python?
  • How can I reload a previously imported module?
  • What is the process of compilation and linking in python?
  • What id() function in Python is for?
  • What method is used to pass variables into function?

Code involving questions:
  • Explain how you reverse a generator?
  • Let A and B be objects of class Foo. What methods and in what order are called when "print (A + B)" is executed?
  • Place the following functions below in order of their efficiency. How would you test your answer?

    def f1(lIn):
        l1 = sorted(lIn)
        l2 = [i for i in l1 if i>0.5]
        return [i*i for i in l2]
    

    def f2(lIn):
        l1 = [i for i in lIn if i>0.5]
        l2 = sorted(l1)
        return [i*i for i in l2]
    

    def f3(lIn):
        l1 = [i*i for i in lIn]
        l2 = sorted(l1)
        return [i for i in l1 if i>(0.5*0.5)]
    
  • Write a one-liner that will count the number of capital letters in a file. Your code should work even if the file is too big to fit in memory.
  • Output? Why? Is this inheritance?

    class C:
        pass
    type (C ())
    type (C)
  • Explain the output:
    big_num_1   = 1000
    big_num_2   = 1000
    small_num_1 = 1
    small_num_2 = 1
    big_num_1 is big_num_2
    >>> False
    small_num_1 is small_num_2
    >>> True
    
  • How is this possible?
    _MangledGlobal__mangled = 23
    class MangledGlobal:
         def test(self):
             return __mangled
    >>> MangledGlobal().test()
    23
        

Inspired by this article.

Thursday, August 10, 2017

Local docker deployment update with AWS CodeBuild

#!/usr/bin/env bash

# Prerequisites
# $WORKDIR/.env - environment variables
# awscli installed virtualenv env dir in $WORKDIR

set -e

WORKDIR=
PROJECT=project name
DOCKER_REPO=codebuild docker image
REPO_TAG=latest

cd $WORKDIR

while [[ $# -gt 0 ]]
do
key="$1"
case $key in
    -h|--help)
    echo '  -s|--skip-build - skip building image'
    exit 0
    ;;
    -s|--skip-build)
    SKIP_BUILD=yes
    ;;
    *)
    echo "Unknown option $2."
    exit 2
    ;;
esac
shift
done

print() {
    echo -e "\033[0;33m$1\033[0m"
}

if [ -z "$SKIP_BUILD" ]
then
    print 'Starting build'
    ARN=$(env/bin/aws codebuild start-build --project-name $PROJECT | python -c 'import json,sys;print(json.load(sys.stdin))["build"]["arn"]')
    print "Build started (ARN $ARN)"
    print "Waiting 3m"

    secs=180
    while [ $secs -gt 0 ]; do
        echo -ne "Time remaining: ${secs}s...\033[0K\r"
        sleep 1
        : $((secs--))
    done
    echo -ne '\033[0K\r'

    print "Checking if it's ready."
    while [ $(env/bin/aws codebuild batch-get-builds --ids "$ARN" | python -c 'import json,sys;print(json.load(sys.stdin))["builds"][0]["buildComplete"]') != 'True' ]
    do
        echo "...not ready yet. Waiting 5s."
        sleep 5s
    done
    print "Build's ready. Update docker container."
else
    print "Using existing build."
fi

sudo $(env/bin/aws ecr get-login --no-include-email --region us-east-1)
sudo docker pull $DOCKER_REPO:$REPO_TAG
sudo docker tag $DOCKER_REPO $PROJECT
sudo docker rm -f $PROJECT
sudo docker run -d --restart=always --env-file $WORKDIR/.env -p 8001:8000 --name=$PROJECT $PROJECT

print "Update finished."

Thursday, February 25, 2016

Django query logging

Recently I had to optimise some piece of legacy code that worked slow. Among other refactoring procedures I wanted to check what queries gets invoked exactly. Having dug Django Docs and internets I came up with solution.

Using this, you'll get all executed queries in the console

For example:

2016-02-25 15:57:47 DEBUG    utils.py:89 (0.055) SET SQL_AUTO_IS_NULL = 0; args=None
2016-02-25 15:57:47 DEBUG    utils.py:89 (0.435) SELECT `table`.`id` FROM `table` ORDER BY `table`.`id` ASC LIMIT 10000; args=()

Add the code below to your local_settings.py (or whatever it's called in your project), make sure the include of this code is AFTER defining LOGGING variable.

DB_LOG = True

from settings import LOGGING
if DB_LOG:
    import logging, re

    class QueryFilter(logging.Filter):
        def filter(self, record):
            # exclude PyCharm debug window queries
            return 'LIMIT 21' not in getattr(record, 'sql', '')


    class SqlFormatter(logging.Formatter):
        def format(self, record):
            record.msg = re.sub(r"((?:u?')?\d+(?:u?')?)"
                                r"(?:,\s*(?:u?')?\d+(?:u?')?)+",
                                r'\1, ...', record.msg)
            return super(SqlFormatter, self).format(record)

    params = {
        'filters': {
            'db_query_filter': {
                '()': QueryFilter,
            },
        },
        'formatters': {
            'debugging': {
                '()': SqlFormatter,
                'format': '%(asctime)s %(levelname)-8s %(filename)s:%(lineno)d %(message)s',
                'datefmt' : '%Y-%m-%d %H:%M:%S',
            },
        },
        'handlers': {
            'db_console': {
                'level': 'DEBUG',
                'filters': ['require_debug_true', 'db_query_filter'],
                'class': 'logging.StreamHandler',
                'formatter': 'debugging',
            },
        },
        'loggers': {
            'django.db': {
                'level': 'DEBUG',
                'handlers': ['db_console'],
                'propagate': False,
            },
        }
    }
    for key, param in params.items():
        try:
            LOGGING[key].update(param)
        except KeyError:
            LOGGING[key] = params

Wednesday, February 24, 2016

No more commits with debugging code

Most of you have certainly found yourself in the situation when after committing and pushing you realise you've forgotten to delete some code you added for debugging purpose? I am prone to it and after yet another doing it I decided to let git itself monitor my commits and prevent me from wrongdoing. From me does it require to put a comment in the file I do not want to be committed before all the junk is gone.

To do so, you need to create .git/hooks/pre-commit file (or append to the existing one):

#!/bin/sh

if git rev-parse --verify HEAD >/dev/null 2>&1
then
 against=HEAD
else
 # Initial commit: diff against an empty tree object
 against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
fi

# Redirect output to stderr.
exec 1>&2

for F in $(git status --short -uno --porcelain | egrep -v '^D' | awk '{ printf $2"\n"; }')
do
 if grep -iq nocommit "$F"; then
  cat <<EOF
Error: '$F' containts "no commit" comment.
You must delete it before it can be committed.
EOF
  exit 1
 fi
done

And add NOCOMMIT (case insensetive) somewhere in your file that you don't want to commit, i.e.

# NOCOMMIT
import pdb
a = foo(1, 4)
pdb.set_trace()


Hope it'll save some nerves for you.

Tuesday, September 3, 2013

Install GeoIP from MaxMind on OS X

I could not find a package ready to use to get running GeoIP from MaxMind on OS X so I built it from sources. To do so, I
  • Downloaded sources
  • Unpacked it tar xvfz GeoIP-latest.tar.gz
  • cd GeoIP-1.5.1
  • ./configure
  • make
  • make check
  • make install
That is it. It's ready to go.


Friday, August 2, 2013

Parse any number string to float

function stringToFloat(str) {
    var fractional_part, integer, fractional = "", found = false, i;
    // remove leading and trailing non-digit symbols
    str = str.replace(/[^\d,\.]+$/, '').replace(/^[^\d,\.]+/, '');
    fractional_part = str.slice(-3);
    for (i = fractional_part.length - 1; i >= 0; i--) {
        if (/\d/.test(fractional_part[i])) {
            fractional = fractional_part[i] + fractional;
        } else {
            found = true;
            break;
        }
    }
    if (fractional && found) {
        integer = str.slice(0, i - 3);
    } else {
        integer = str;
        fractional = "";
    }
    return parseFloat(integer.replace(/[^\d]/g, '') + "." + fractional);
}

Tuesday, July 2, 2013

Delete tags

To delete tags in git that match some regex, both locally and remotely, I use these two commands

# remove tags on origin
git ls-remote --tags  | awk '/tags\/release.*[^\}]$/ {print ":" $2}' | xargs git push origin

# remove locally
git tag -l | awk '/^release.*$/ {print $1}' | xargs git tag -d

Tuesday, May 14, 2013

Increase trackpad speed beyond system settings

If you are not satisfied with your trackpad speed and you've already set it to the maximum in System Settings pane, it could help you.

Go to Terminal and

open ~/Library/Preferences/.GlobalPreferences.plist

You need com.apple.trackpad.scaling. System Preferencies's maximum is 3. From my experience, 30 is insane. I use 15, it's enough to move cursor with one movement from the left side of left screen the right of the right one.

Wednesday, April 24, 2013

Showing an image when dragging any tag in HTML

Chrome and Safari support displaying a ghost image for draggable objects out of the box, Firefox does not. It's not a big deal, I'd say a matter of one event handler.

This is a sample HTML:
Drag Me!

and JavaScript code:
$(function () {
    var dragImage = document.createElement("img");
    dragImage.src = "/image/to/show/when/dragging.jpg";
    function handleDragStart(e) {
        e.dataTransfer.effectAllowed = 'move';
        e.dataTransfer.setDragImage(dragImage, e.layerX, e.layerY);
    }

    $("#draggableObject").get(0).addEventListener('dragstart', handleDragStart, false);
});

Native HTML5 Drag and Drop

MDN::dataTransfer

Monday, April 15, 2013

Line numbers in Vim



I would like to cover all aspects known for me related to the line numbers in Vim.

CommandDescription
:0
move cursor to the first
:42
42gg
42G
move cursor to 42nd
:G
move cursor to the last


Show line numbers in current file
:set number

Hide line numbers in current file
:set nonumber