Thursday, November 22, 2012

Reduce directory nesting

Once I encountered a problem with image files that were named according to their checksum of the content chunked by 2 symbols 6 times, so it gets like 2f/d4/e1/c6/7a/2d/28fced849ee1bb76e7391b93eb12.jpg.

To have this structure with approx. 13M files, I got short in inodes. After a quick research and calculations I realized that such nesting depth is redundant and easily can be reduced. So I need to turn all the existing files of form 2f/d4/e1/c6/7a/2d/28fced849ee1bb76e7391b93eb12.jpg to 2f/d4/e1c67a2d28fced849ee1bb76e7391b93eb12.jpg.

To do that, I wrote a script that renames the files and removes empty directories.

TARGET_DIR = '/full/path/to/dir/containing/dirs/to/reduce/'
TARGET_DIR_LEN = len(TARGET_DIR) + 1
CURRENT_DEPTH = 6
DESIRED_DEPTH = 2

last_dir = ''
for dirname, _, filenames in os.walk(TARGET_DIR):
    path = dirname[TARGET_DIR_LEN:].split('/')
    if filenames and len(path) >= CURRENT_DEPTH:
        cur_dir = '/'.join(path[:DESIRED_DEPTH - CURRENT_DEPTH])
        if last_dir != cur_dir:
            last_dir = cur_dir
        for filename in filenames:
            fullpath = os.path.join('/'.join(path), filename)
            parts = fullpath.split('/')
            prefix = ''
            if len(parts[0]) > 2: # cache
                prefix = parts[0]
                parts = parts[1:]
            old = os.path.join(dirname, filename)
            new = os.path.join(TARGET_DIR, prefix,
                               '/'.join(parts[:DESIRED_DEPTH]),
                               ''.join(parts[DESIRED_DEPTH:]))
            pass
            os.rename(old, new)
        for i in range(CURRENT_DEPTH - DESIRED_DEPTH):
            d = os.path.join(TARGET_DIR,
                             dirname[0:(DESIRED_DEPTH - CURRENT_DEPTH + 1) * i]
                             if i else dirname)
            try:
                os.rmdir(d)
            except OSError:
                break

Sunday, October 7, 2012

Sync your development environment with a remote one

While the developing process you need sometimes to sync you local development database with a remote one (i.e. production).

Below there is a script allowing to recreate the development database and fill in with data from the remote server.

MySQL
#!/bin/sh

DBUSER_REMOTE="root"
DBPASSWORD_REMOTE="root"
DBUSER_LOCAL="root"
DBPASSWORD_LOCAL="root"
DBNAME_REMOTE=""
DBNAME_LOCAL=""
TMPFILE="$(mktemp db.XXXXXXX)"
HOST="www.example.com"

ssh $HOST "mysqldump -u$DBUSER_REMOTE -p$DBPASSWORD_REMOTE $DBNAME_REMOTE | gzip" | gunzip > $TMPFILE

mysqladmin -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL -f drop $DBNAME_LOCAL
mysqladmin -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL create $DBNAME_LOCAL
mysql -u$DBUSER_LOCAL -p$DBPASSWORD_LOCAL $DBNAME_LOCAL < $TMPFILE
rm $TMPFILE

PostgreSQL
#!/bin/sh

DBUSER_REMOTE="postgres"
DBUSER_LOCAL="postgres"
DBNAME_REMOTE=""
DBNAME_LOCAL=""
TMPFILE="$(mktemp db.XXXXXXX)"
HOST="www.example.com"

ssh $HOST "pg_dump -U $DBUSER_REMOTE -Fc $DBNAME_REMOTE" > $TMPFILE

dropdb -U $DBUSER_LOCAL $DBNAME_LOCAL
createdb -U $DBUSER_LOCAL $DBNAME_LOCAL
pg_restore -U $DBUSER_LOCAL -Fc -d $DBNAME_LOCAL $TMPFILE
rm $TMPFILE

Sync media files
$HOST="www.example.com"
$PATH="/absolute/path/to/media"
$LOCAL_DIR="."
rsync -rltDvH $HOST:$PATH $LOCAL_DIR

Friday, September 28, 2012

Favicon sizes

The browser can use the favicon in different situations, like to show it on the tab, for the bookmark and so forth. There are a few sizes the favicon can be. Here I gathered the most often and broadly used.

Favicon sizes

<link rel="icon" type="image/png" sizes="16x16" href="favicon@16.png">
<link rel="icon" type="image/png" sizes="32x32" href="favicon@32.png">
<link rel="icon" type="image/png" sizes="48x48" href="favicon@48.png">
<link rel="icon" type="image/png" sizes="64x64" href="favicon@64.png">
<link rel="icon" type="image/png" sizes="128x128" href="favicon@128.png">
<link rel="shortcut icon" href="favicon.ico">

Apple Icons (used sizes)

<link rel="icon" type="image/png" sizes="144x144" href="favicon@144.png">
<link rel="icon" type="image/png" sizes="512x512" href="favicon@512.png">
<link rel="apple-touch-icon" type="image/png" sizes="57x57" href="favicon@57.png">
<link rel="apple-touch-icon" type="image/png" sizes="144x144" href="favicon@144.png">

Friday, August 3, 2012

PyCharm can't run a project with MySQL database

I wanted to run my project in PyCharm IDE and got this error.

django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: dlopen(/Users/boris/env/lib/python2.7/site-packages/_mysql.so, 2): Library not loaded: libmysqlclient.18.dylib
  Referenced from: /Users/boris/env/lib/python2.7/site-packages/_mysql.so
  Reason: image not found

After googling a while I figured out there is a mess with DYLD_LIBRARY_PATH environment variable. So, first I needed to locate the path where libmysqlclient.18.dylib located.

$ locate libmysqlclient.18.dylib
/usr/local/mysql-5.5.22-osx10.6-x86_64/lib/libmysqlclient.18.dylib

Done. In Run/Debug Configurations in Environment variables I added (plus to PYTHONUNBUFFERED=1) DYLD_LIBRARY_PATH=/usr/local/mysql-5.5.22-osx10.6-x86_64/lib/:$DYLD_LIBRARY_PATH.

It started working.

Saturday, April 28, 2012

Fatal Python error: Couldn't create autoTLSkey mapping

My last project I started with Django 1.4 and Django Compressor. Today I decided to deploy to the production its first version. I used nginx+Apache.

Everything seemed to be good but instead of normal page I got:
FilterError at /
Unable to apply CompilerFilter (lessc {infile} {outfile})
I tried to repeat the same with ./manage.py runserver. It works. So the issue not in configuration or lessc itself. Later I found here that it is Apache's issue and can be solved with
WSGIApplicationGroup %{GLOBAL}
directive in Apache's VirtualHost config.

It helped.

Friday, March 30, 2012

Install MySQL on Mac OS X to work with Python

I need to get mysql support for python on my Mac OS X Lion.

Download the latest MySQL .DMG distributive from somewhere (i.e. official website) and install it.

sudo pip install python-mysql

It requires superuser privileges to get installed.

What may happen:


while installing mysql-python
Downloading/unpacking mysql-python
  Downloading MySQL-python-1.2.3.tar.gz (70Kb): 70Kb downloaded
  Running setup.py egg_info for package mysql-python
    sh: mysql_config: command not found
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/Users/boris/projects/a1/th4x/build/mysql-python/setup.py", line 15, in <module>
        metadata, options = get_config()
      File "setup_posix.py", line 43, in get_config
        libs = mysql_config("libs_r")
      File "setup_posix.py", line 24, in mysql_config
        raise EnvironmentError("%s not found" % (mysql_config.path,))
    EnvironmentError: mysql_config not found
    Complete output from command python setup.py egg_info:
    sh: mysql_config: command not found

Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/Users/boris/projects/a1/th4x/build/mysql-python/setup.py", line 15, in <module>

    metadata, options = get_config()

  File "setup_posix.py", line 43, in get_config

    libs = mysql_config("libs_r")

  File "setup_posix.py", line 24, in mysql_config

    raise EnvironmentError("%s not found" % (mysql_config.path,))

EnvironmentError: mysql_config not found

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /Users/boris/.pip/pip.log

solution:

Add /usr/local/mysql/bin/ to PATH
export PATH=/usr/local/mysql/bin/:$PATH
(from my ~/.bashrc)

trying to run dev server
Unhandled exception in thread started by <bound method Command.inner_run of <django.contrib.staticfiles.management.commands.runserver.Command object at 0x10c59bc90>>
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/django/core/management/commands/runserver.py", line 88, in inner_run
    self.validate(display_num_errors=True)
  File "/Library/Python/2.7/site-packages/django/core/management/base.py", line 249, in validate
    num_errors = get_validation_errors(s, app)
  File "/Library/Python/2.7/site-packages/django/core/management/validation.py", line 35, in get_validation_errors
    for (app_name, error) in get_app_errors().items():
  File "/Library/Python/2.7/site-packages/django/db/models/loading.py", line 146, in get_app_errors
    self._populate()
  File "/Library/Python/2.7/site-packages/django/db/models/loading.py", line 64, in _populate
    self.load_app(app_name)
  File "/Library/Python/2.7/site-packages/django/db/models/loading.py", line 78, in load_app
    models = import_module('.models', app_name)
  File "/Library/Python/2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/boris/projects/a1/th4x/tradecontrol/models.py", line 4, in <module>
    from tradecontrol.db import auto_detect
  File "/Users/boris/projects/a1/th4x/tradecontrol/db/__init__.py", line 2, in <module>
    import MySQLdb as mysql
  File "/Library/Python/2.7/site-packages/MySQLdb/__init__.py", line 19, in <module>
    import _mysql
ImportError: dlopen(/Library/Python/2.7/site-packages/_mysql.so, 2): Library not loaded: libmysqlclient.18.dylib
  Referenced from: /Library/Python/2.7/site-packages/_mysql.so
  Reason: image not found

solution:

Add to your ~/.bashrc add
export DYLD_LIBRARY_PATH=/usr/local/mysql/lib
and then reread .bashrc
. ~/.bashrc

Wednesday, March 7, 2012

Dealing with Tabs in vim

To insert space characters whenever the tab key is pressed, set the 'expandtab' option:
:set expandtab

With this option set, if you want to enter a real tab character use Ctrl-V<tab> key sequence.

To control the number of space characters that will be inserted when the tab key is pressed, set the 'tabstop' option. For example, to insert 4 spaces for a tab, use:

:set tabstop=4

After the 'expandtab' option is set, all the new tab characters entered will be changed to spaces. This will not affect the existing tab characters. To change all the existing tab characters to match the current tab settings, use:
:retab

To change the number of space characters inserted for indentation, use the 'shiftwidth' option:
:set shiftwidth=4

For example, to get the following coding style,
*No tabs in the source file.
*All tab characters are 4 space characters.

use the following set of options:
:set tabstop=4
:set shiftwidth=4
:set expandtab

Add the above settings to your .vimrc.

Friday, February 17, 2012

Allow remote connections to MySQL server

To enable remote access for MySQL server:

$ sudo vim /etc/mysql/my.cnf
### edit my.cnf
### set bind address and comment out or remove skip-networking (if exists)

[mysqld]
...
bind-address=SERVER-EXTERNAL-IP
# skip-networking

$ /etc/init.d/mysql restart

Now it works.

Friday, February 10, 2012

{less} mix function error

Documentation tells mix function takes only two parameters, @color1 and @color2. Actually there is a mistake. The correct function definition is

mix(@color1, @color2, value)

, where value denotes how much of @color1 should appear in the result color.

Tuesday, February 7, 2012

Trap Ctrl C in Bash

#!/bin/bash

# call trapped in case Ctrl+C pressed
trap trapped INT

function trapped() {
    echo "CTRL-C pressed"
}

while true; do
    sleep 1
    echo -n "."
done

Friday, February 3, 2012

Setup Apache2's mod_wsgi on Ubuntu

If you get this when restarting Apache2 on Ubuntu
Invalid command 'WSGIDaemonProcess', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
it means you don't have mod_wsgi module installed.

These commands install and enable it.
sudo apt-get install libapache2-mod-wsgi
sudo a2enmod mod-wsgi
sudo /etc/init.d/apache2 restart

Perl warning about locale in Ubuntu

If you get this warning
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
 LANGUAGE = (unset),
 LC_ALL = (unset),
 LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

it can be fixed with
locale-gen en_US.UTF-8
dpkg-reconfigure locales

Get Graphite run for a Django project

This post is intended first of all to memorize my experience of getting the Graphite system work.

Wednesday, February 1, 2012

Install RabbitMQ

install server itself (using port)

MacOs X:
sudo port install rabbitmq-server

Ubuntu:
sudo apt-get install rabbitmq-server

create user, vhost and set some permissions:
rabbitmqctl add_user username password
rabbitmqctl add_vhost /vhost
rabbitmqctl set_permissions -p /vhost username ".*" ".*" ".*"

done.

OSError: [Errno 38] Function not implemented

Server: Ubuntu 11.04

Run celeryd with info log level

$ ./manage.py celeryd -l info
[2012-02-01 01:16:45,779: ERROR/MainProcess] Unrecoverable error: OSError(38, 'Function not implemented')
Traceback (most recent call last):
  File "/var/www/th4x/.env/lib/python2.7/site-packages/celery/worker/__init__.py", line 268, in start
    component.start()
  File "/var/www/th4x/.env/lib/python2.7/site-packages/celery/concurrency/base.py", line 72, in start
    self.on_start()
  File "/var/www/th4x/.env/lib/python2.7/site-packages/celery/concurrency/processes/__init__.py", line 43, in on_start
    self._pool = self.Pool(processes=self.limit, **self.options)
  File "/var/www/th4x/.env/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 520, in __init__
    self._setup_queues()
  File "/var/www/th4x/.env/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 695, in _setup_queues
    self._inqueue = SimpleQueue()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 354, in __init__
    self._rlock = Lock()
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented
[2012-02-01 01:16:45,780: INFO/MainProcess] process shutting down

Python's _multithreading requires /dev/shm to work. In Ubuntu /dev/shm points to /run/shm by default. Rename (or delete, I doubt it is necessary anymore) existing /dev/shm.

Create /dev/shm directory
$ sudo -i  # get root
$ mkdir /dev/shm

Add to /etc/fstab mounting of /dev/shm command
tmpfs /dev/shm    tmpfs   defaults,noexec,nosuid     0     0

Mount all unmounted filesystems from /etc/fstab.
$ mount -a
There should be no errors.

Run again
$ ./manage.py celeryd -l info

It starts normally.

Monday, January 16, 2012

PostgreSQL autostart on Mac OS X

I installed PostgreSQL 9.0 from „One Click Installer” and didn't want it to launch as system starts.

As PostgreSQL has been installed launchctl takes responsibility to take care of always running PostgreSQL daemon. /Library/LaunchDaemons/com.edb.launchd.postgresql-9.0.plist file sets up the daemon's behavior.

There are two options that can prevent the daemon after start up.
<key>Disabled</key>
<false/>

<key>RunAtLoad</key>
<true/>

„Disabled”=True prevents the job from loading at all and there will be not possible start the job even manually.
„RunAtLoad”=Fasle tells the launchctl not to run the job after loading.

Here we are, set „RunAtLoad” to False and the daemon won't be run automatically.

By the way, there are useful commands below to run and stop the daemon though:

sudo launchctl list | grep postgresql
display current daemon's pid or exit code.

sudo launchctl start com.edb.launchd.postgresql-9.0
start the daemon

sudo launchctl stop com.edb.launchd.postgresql-9.0
and stop it

Wednesday, January 4, 2012

Tab-completion for django-admin.py and manage.py

Download django_bash_completion file and save in some convenient place in your system. In .bash_profile (or .bashrc if it doesn't envoke .bash_profile) add
. ~/path/to/django_bash_completion
Settings will take effect the next time you log in.