Linux Racker

Today a simple task such as regular crontab entry to schedule a home made script turned out to a learning experience

The script take the current date or any date for that matter as a parameter.
The date has got to be provided in the format YYYY-MM-DD

So I started with a crontab entry that looks like this

# Hourly job to update todays stats
1 * * * * root /home/tools/collect_stats.py --date `date -u +'%Y-%m-%d'`

I am waiting for my stats to get updated and nothing.
Funny thing is I copy paste the above life as it is into the shell and it execute perfectly and updates the stats.

So I know the command I have in crontab works.
I thought it might be something to do with setting up of the environment for the root user, when the script is run via cron. Eitherway, when there are error’s cron daemon sends out email. Dug out the emails, and this is what I found in there..

cron "Syntax error: EOF in backquote substitution"

So started looking Google for it, and came a across a couple of links that resolved the issue for me.

The gist of it is , one needs to escape the “%” symbol with an “\” in a crontab entry.
So now my updated version fo the cron job looks like

# Hourly job to update todays stats
1 * * * * root /home/tools/collect_stats.py --date `date -u +'\%Y-\%m-\%d'`

Fascinating to find out learn new things, even while doing the simplest of things such as scheduling a cron job. This is why I never ever claim to be an expert in almost anything.

Referencing the 2 links. which were handy in helping me to to the resolution
http://stackoverflow.com/questions/7068759/crontab-syntax-error
http://www.hcidata.info/crontab.htm

I keep using dd to write .iso files directly to usb drives, to create a bootable usb disk. Especially for ubuntu. I am for ever waiting around not sure how long to go or how far has it done etc. I have used pv for mysql dump and imports before, but never for dd. It seems to be working pretty well.

# ls -lh precise-desktop-amd64.iso 
-rw-r--r--@ 1 nobody  1989879963   702M Apr  7 14:33 precise-desktop-amd64.iso

# dd if=./precise-desktop-amd64.iso bs=16m | pv -s 702M | dd of=/dev/disk1 bs=16m
 201MB 0:01:51 [2.69MB/s] [=================>                                                 ] 28% ETA 0:04:37

If you are not aware of the size, for eg writing random data, you can omit the -s size option in the pv command. Mind you, the above was run from a mac osx terminal. So do validate the commands before you run them on linux.

Also came across the “bar” utility. This is pretty cool as well.

http://www.theiling.de/projects/bar.html

Lets start off by making sure we have a updated fresh debian squeeze install

# apt-get update
# apt-get -y upgrade
# reboot

Installation

We install the prerequisites necessary for graphite.

# apt-get install -y python2.6 python-pip python-cairo python-django python-django-tagging
# apt-get install -y libapache2-mod-wsgi python-twisted python-memcache python-pysqlite2 python-simplejson 

The remaining packages are not available in apt repos, but the latest version are available in pip. So we will be using pip to install them. This allows for upgrading them easily at a later point in time.

# pip install whisper
# pip install carbon
# pip install graphite-web

Configuration

Ok that is the installation done. Now to do the necessary configurations. I used the Graphite repo from github for a sample vhost configuration. There is a sample configuration example-graphite-vhost.conf in the examples directory. Saving the configuration as it is to the sites-available folder apache.

# wget https://raw.github.com/tmm1/graphite/master/examples/example-graphite-vhost.conf -O /etc/apache2/sites-available/graphite

Configuring graphite

Mostly making use of the the default configuration files that come with the installation of graphite. Finally set appropriate permissions for the storage directory so that apache doesn’t have any issues.

# cd /opt/graphite/conf/
# cp graphite.wsgi.example graphite.wsgi
# cp carbon.conf.example carbon.conf
# cp storage-schemas.conf.example storage-schemas.conf
# cd /opt/graphite/webapp/graphite
# python manage.py syncdb
# chown -R www-data:www-data /opt/graphite/storage/

We also need to make some fine adjustments to the sample vhost configuration we copied over.

Starting Graphite

Disable the default vhost and enable the graphite vhost. We need to start the carbon agent and restart apache.

# a2dissite default
# a2ensite graphite
# /opt/graphite/bin/carbon-cache.py start
# /etc/init.d/apache2 restart

Need to look for an init scrip or come up with one, to start carbon at system start up.

Installing Diamond Stats collector

I needed something to quickly and continuously populate graphite with some data, so that I could test some some of the capabilities of graphite. So the easiest option was to install the diamond collector which will collect system stats and populate graphite with the same information. You can clone the Diamond repo from github and build .deb or .rpm files.

# apt-get install python-configobj
# dpkg -i diamond_2.0.0_all.deb
# /etc/init.d/diamond restart

Additional Notes

Noticed serveral entries in the log files which didn’t seem right, see below.

#grep error /opt/graphite/storage/log/webapp/error.log
[Sat Mar 31 03:54:35 2012] [error] Could not import graphite.local_settings, using defaults!
[Sat Mar 31 03:54:44 2012] [error] Could not import graphite.local_settings, using defaults!
[Sat Mar 31 03:54:44 2012] [error] Could not import graphite.local_settings, using defaults!
[Sat Mar 31 07:48:52 2012] [error] No handlers could be found for logger "cache"
[Sat Mar 31 07:48:52 2012] [error] No handlers could be found for logger "cache"
[Sat Mar 31 07:49:11 2012] [error] No handlers could be found for logger "cache"
[Sat Mar 31 07:49:30 2012] [error] No handlers could be found for logger "cache"
[Sat Mar 31 17:23:51 2012] [error] No handlers could be found for logger "rendering"
[Sat Mar 31 17:24:06 2012] [error] No handlers could be found for logger "metric_access"
[Sat Mar 31 17:24:09 2012] [error] No handlers could be found for logger "rendering"

Did a quick search on the logger error. Initial indication is that this is because LOG_CACHE_PERFORMANCE = False in the settings.py configuration file for the webapp. As for the local_settings eror, I am guessing that the webapp looks for a local_settings.py file to override any of the default settings for graphite which i in settings.py. Will need to test the above 2 theories. Started getting a few more additional logger errors. So for now this is what I have to do to clean up these. Looks like the application is expecting it to be enabled.

# echo -e 'LOG_METRIC_ACCESS = True\nLOG_CACHE_PERFORMANCE = True\nLOG_RENDERING_PERFORMANCE = True' >> /opt/graphite/webapp/graphite/local_settings.py
#
# cat /opt/graphite/webapp/graphite/local_settings.py
LOG_METRIC_ACCESS = True
LOG_CACHE_PERFORMANCE = True
LOG_RENDERING_PERFORMANCE = True

The above will get rid of the error about the missing local_settings for the graphite webapp as well.
Almost forgot below are the references to the links that helped me come up with the above method to deploy a graphite server.

http://graphite.wikidot.com/


http://graphite.readthedocs.org/en/latest/index.html


https://github.com/BrightcoveOS/Diamond


http://drumcoder.co.uk/blog/2011/aug/26/installing-graphite-debian/


http://blog.4aiur.net/2012/01/installing-and-configuring-graphite-on-centos/


http://blog.defunct.ca/2012/01/23/fast-and-dirty-install-of-graphite-on-debian-squeeze/

I have been meaning to try and setup a encrypted parition for a while, now. To start off with I had a 10GB bootable partition within which the OS was entirely installed. Talking of OS, my virtual machine was running Debian Squeeze. (6.0) So i have an empty partition /dev/xvda2 which is available for me to play with.

# fdisk -l /dev/xvda

Disk /dev/xvda: 20.4 GB, 20401094656 bytes
224 heads, 56 sectors/track, 3176 cylinders
Units = cylinders of 12544 * 512 = 6422528 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x6f682d34

Device Boot Start End Blocks Id System
/dev/xvda1 * 1 1589 9960448 83 Linux
/dev/xvda2 1589 3176 9958400 83 Linux

Before we start off using this partition, we want to make sure that it is fill’d with random data. Typically this is not necessary, if you are using a virtual environment which is presenting you the disk as a sparse file.

dd if=/dev/urandom of=/dev/xvda2 bs=4M oflag=direct

dd: writing `/dev/xvda2': No space left on device
2432+0 records in
2431+0 records out
10197401600 bytes (10 GB) copied, 2047.1 s, 5.0 MB/s

apt-get install parted binutils dmsetup cryptsetup

We’ll aslo install a few packages which are necessary for setting this up. Cryptsetup requires the device mapper kernel module to mount or map an encrypted drive. Installed parted to make use of the partprobe utility to detect the newly created partition. As for binutils, I need the strings command, which I shall explain later. Verify that the device mapper module is loaded. if not load the device mapper module.

modprobe dm_mod

Let us now format or luksFomat the partition we would like to encrypt. There is no turning back.. So make sure and double check you have chosen the correct partition to format. YOu will be required to enter a LUKS passphrase which is the key that will be used to unlock/mount/decrypt/encrypt the partition in question. Do not loose this.

# cryptsetup luksFormat /dev/xvda2

WARNING!
========
This will overwrite data on /dev/xvda2 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:

So that is the LUKS device created, we can no use it as a normal disk, create a filesystem, mount it etc..  You can verify or get a bit of information about the luks formated drive by making use of the luksDump command.

# cryptsetup luksDump /dev/xvda2
LUKS header information for /dev/xvda2

Version: 1
Cipher name: aes
Cipher mode: cbc-essiv:sha256
Hash spec: sha1
Payload offset: 2056
MK bits: 256
MK digest: 36 92 9f de 46 eb ed 39 91 1f 7f 5a 2a 27 7a 23 11 c6 70 51
MK salt: ab cf 14 e6 c6 b2 7a 1f a7 43 b5 fb cb 3b fc 9e
 a8 7b e6 ad 49 9c 7d 72 59 ba 49 f0 e9 27 68 eb
MK iterations: 27500
UUID: d77760e7-6008-408d-9ae7-ade40229b36e

Key Slot 0: ENABLED
 Iterations: 110426
 Salt: f6 6b 79 37 30 1a e8 7b 39 13 ed 52 9f 48 fe 88
 ad f0 6b f0 68 26 a2 1e f1 b7 ec 1a ef 20 a0 12
 Key material offset: 8
 AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

The final step is now to create a mapper device, which can be used like a regular disk.  This achieved by making use of  luksOpen command. You will need to provide the LUKs passphrase to open the luks encrypted drive.

# cryptsetup luksOpen /dev/xvda2 securedev
Enter passphrase for /dev/xvda2: 

root@git:~# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 58 Feb 25 14:50 control
lrwxrwxrwx 1 root root 7 Feb 25 16:01 securedev -> ../dm-0

Once you have a mapper device created from this point on wards it is straightforward. Treat this mapper device as a regular disk drive.  You can format the device using mkfs /dev/mapper/securedev  and mount it store data and so on. Finally to verify or see if we can access any meaningful data.. from our encrypted block device.

I have formated /dev/mapper/securedev as ext4, and mounted it and copied over a bunch of text files.. So of them were apache log files, which contains text I can easily identify. Let unmount and close the encrypted block device. You can either do a hexdump of the block device, or make use of the strings command. Either way, you can attempt to parse the output of the strings or hexdump command and grep for wrong words, or string combinations, that you know exists within a file on disk.

umount /dev/mapper/securedev
cryptsetup luksClose securedev
hexdump -C /dev/xvda2 | less
strings /dev/xvda2 | grep <knows string>

This is just the tip of the ice berg. I have’t done proper comparison of how disk performance is and what happens in real life scenario migrating and this data, resizing disk, recovery from a snapshot based backup etc.. Lastly, I would like to acknowledge or reference the resources that proved to be quite handy in the process of figuring this out.

http://www.cromwell-intl.com/security/ec2-secure-storage.html

http://www.debian-administration.org/articles/469

http://code.google.com/p/cryptsetup/wiki/DMCrypt

http://code.google.com/p/cryptsetup/