Introducing NYMMS

I haven’t posted for a while (5 months, ugh), but that doesn’t mean I’ve been idle this whole time. I’ve been working on a new monitoring system, and I think it’s at the point where I want to share it with the world. It can do the basics - monitor services and send alerts when they fail. I’d love it if other people started to play with it so I can get some feedback/bugs.

Anyway, check it out:

• NYMMS docs
• NYMMS github

Also, be sure to checkout the Demo AMI page if you want to bring up an all-in-one NYMMS system with very basic configuration to play with.

(more) securely storing my AWS credentials

This is an update to my previous post, AWS/Boto Credentials and my prompt.

In that post I started storing my AWS credentials in my .bash_profile. It’s worked great for a while, but there’s been one thing that’s bothered me about it: my credentials were stored in plain text in my .bash_profile. Not awesome.

So I set out to fix it. The first step was to remove all the aws credential functions and instead move them to their own file. I named it .bash_aws_profiles. I then encrypted it with openssl using the aes-256 cipher with the following command:

openssl aes-256-cbc -a -salt -in .bash_aws_profiles -out .bash_aws_profiles.enc

It will ask for a password - pick a secure one, and don’t share it.

Next you need a way to get those functions back into your shell. I tried to do this by having source or eval read in the output from openssl unencrypting the file, but neither of them worked for me for various reasons. In the end I had to use a temporary file, which I’m not super excited about, but it only exists for a very short period of time, which is better than what we had before.

As my buddy Tim would say: “Don’t let best be the enemy of good!”

So I added these functions to my .bash_profile:

function load_aws () {
    AWSKEYFILE=$HOME/.bash_aws_profiles.enc
    TEMPFILE=`mktemp $HOME/.bash_aws_profiles.XXXXXXXXXXX`
    [ ! -f $AWSKEYFILE ] && echo "Error: Amazon key file ($AWSKEYFILE) not found." && return
    openssl aes-256-cbc -d -a -in $AWSKEYFILE -out $TEMPFILE
    source $TEMPFILE
    rm $TEMPFILE
    echo "! AWS keys loaded."
}

function unload_aws () {
    KEYFUNCS=`set | awk '/aws_[^ ]+ \(\)/ {print $1}'`
    for k in $KEYFUNCS
    do
        echo "! Unloading $k"
        unset $k
    done
}

Now whenever I want to use my AWS credentials I first have to type load_aws. It asks me for my password, then loads them into the environment. When I’m finished I can run unload_aws to delete them.

I’d really like it if there was a sort of boto-agent that would store the keys instead of the shell, and then expire them after a given amount of time. I’ll work on that someday.

Let me know if you have any questions, or any ideas for improvements. Thanks!

AWS/Boto Credentials and my prompt

I do a lot of stuff with Amazon AWS/EC2 everyday. I use many different amazon accounts with different credentials. For example I post this site to S3 in my personal AWS account. I also have multiple work accounts for various things. Most of my interactions with AWS are done through python & boto, sometimes through scripts and sometimes through the python shell. I used to keep all of my various credentials in my ~/.boto file, with only the one I was using uncommented.

This meant that I had to be extra careful and doublecheck my .boto file to make sure I was using the right account when doing anything with Amazon. After a couple of times where I ran commands (non-destructive) in the wrong account I complained to my coworker. He forwarded me a trick he’d been using, and I expanded on it.

One of boto’s features is that it can gather your credentials from a few places. One of those places is via the shell variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. So first I created functions in my ~/.bash_profile that allowed me to reset those variables easily. As well, I had those functions set another variable AWS_KEY_NAME. That isn’t used by boto at all, but it’s useful for me. For example, say I have two accounts: my personal account and my work account. I’d setup the following in my profile:

function aws_personal () {
    export AWS_KEY_NAME="personal"
    export AWS_ACCESS_KEY_ID=<my personal access key id>
    export AWS_SECRET_ACCESS_KEY=<my personal secret access key>
}

function aws_work () {
    export AWS_KEY_NAME="work"
    export AWS_ACCESS_KEY_ID=<my work access key id>
    export AWS_SECRET_ACCESS_KEY=<my work secret access key>
}

With that in place (and after resourcing my .bash_profile) I can now call either of those functions from my shell and I’m using those credentials in that shell until I choose to change them again.

That’s useful, but now it’d be really good to know what credentials I’m currently using. There are two places I really need to know that:

• My shell
• My python interactive interpreter

So lets start with the first. In bash it’s easy enough to change your prompt. Since I set the AWS_KEY_NAME variable I can use that in my prompt. That said, my prompt is already pretty long. I finally gave in and decided I needed to have a two line prompt.

Since I was going to do a little surgery on my prompt, and I don’t do that very often, I decided to add a couple of features to it. Those were:

• multiline
• unobtrusive color unless a previous command failed, then make sure I know
• adding my current AWS credentials if they are set

The first is easy. The second I used a variable I set in my PROMPT_COMMAND that I discussed in a previous article Keeping bash history forever. The third I used the new AWS_KEY_NAME variable. In the end, this is what I ended up with for my new prompt:

COLOR_RESET="\e[00m"
COLOR_GRAY="\e[01;30m"
COLOR_RED="\e[22;31m"

PROMPT_COMMAND="ECODE=\$?; TODAY=\`date +%Y%m%d\`;[ -d $HOME/.history ] || mkdir -p $HOME/.history; echo : [\$(date)] $USER \$ECODE \$PWD\; \$(history 1 | sed -E 's/^[[:space:]]+[0-9]*[[:space:]]+//g') >> $HOME/.history/bash_history-\$TODAY"

PS1="${COLOR_GRAY}[\$([ \$ECODE -ne 0 ] && echo \"${COLOR_RED}\$ECODE${COLOR_GRAY} \")\u@\h \$([ ! -z \$AWS_KEY_NAME ] && echo \"aws:\$AWS_KEY_NAME \")\w]${COLOR_RESET}\n> "

With that set I have a two line, dark grey prompt that will display which of my Amazon credentials I am currently using and will also display the exit code of my last command in red only if the last command exited with a non-zero status (thx to my buddy John for that idea). Here’s an example:

Now my shell is displaying things, but I also play around in the python interactive interpreter a lot as well, so it’d be good to see what credentials I was using in there as well. Fortunately you can modify that prompt by setting sys.ps1 in the ~/.pythonrc.py file. The trick is that we want the prompt to be dyamic - fortunately sys.ps1 allows you to do that by calling str() on any non-string object you give it. This means you can pass an object with a dynamic __str__ method. Here’s the excerpt from my .pythonrc.py that does this:

import os
import sys

class AWSCredentialsPrompt(object):
    def __init__(self, base_prompt):
        self.base = base_prompt

    def __str__(self):
        try:
            return "(aws:%s) %s " % (os.environ['AWS_KEY_NAME'], self.base)
        except KeyError:
            return "%s " % (self.base)

sys.ps1 = AWSCredentialsPrompt('>>>')
sys.ps2 = AWSCredentialsPrompt('...')

That’s it! If I were using the work credentials and I enter my python interpreter, here’s what you’d see:

So that’s all there is to it. I’m working on a few functions in python to act the same as the shell functions to switch credentials. It shouldn’t be too difficult.

Let me know if you have any questions or something doesn’t work for you. Oh, and if you have any ideas for enhancements leave them in the comments!

shortcut your email on iOS

Ok, this is probably something people have already thought to do, but I only today figured it out. What’s screwed up is that I already knew about the shortcut feature in iOS (I hated the fact it defaulted ‘omw’ to ‘On my way!’).

Basically I end up having to type my email address fairly often, and typing email addresses on the iPhone isn’t the easiest. Soooo, today I setup a couple of shortcuts. You can set these up by going into Settings > General > Keyboard and then clicking on Add New Shortcut.

I setup the following shortcuts:

@@

My personal email address

@w

My work email address

@p

My phone number

Now I can just type those whenever I need to give my email address.

I’m trying to think of other shortcuts - if you have any ideas, share them in the comments!

disabling aliases in pyYaml

Lately I’ve been working with a python script to act as an external node classifier for Puppet. Puppet expects the output of the ENC to be in YAML, so naturally I reached for PyYAML. Everything was going great, but then I tried to put the same bit of information in both the classes AND parameters section. See this script for example:

import yaml

services = {
        'enabled': ['nginx', 'webapp'],
        'disabled': ['postgres'],
}

data = {'classes': {'services': services}, 'parameters': services}

print yaml.safe_dump(data, default_flow_style=False)

When you run this script you end up with the following output:

classes:
  services: &id001
    disabled:
    - postgres
    enabled:
    - nginx
    - webapp
parameters: \*id001

The issue with this is the use of aliases. Basically since the data I had in the ‘services’ dictionary was the same as what I had in parameters pyYAML recognized that and created the alias &id001 pointing at that data, then referenced it rather than copying that data into parameters.

Unfortunately the version of puppet that I am running (2.7.11, the default in Ubuntu 12.04) can’t handle aliases, even though they’re a definitely a part of the YAML syntax. Honestly in most cases I’d be super happy that PyYAML was smart enough to do this for me automatically. But in this case it was causing issues.

After digging around, I found out a way to work around it. Unfortunately it’s not as simple as an argument to safe_dump. Instead you need to provide your own instance of a Dumper object, and then override ignore_aliases so that it always returns True.

Here’s the updated script:

import yaml

services = {
        'enabled': ['nginx', 'webapp'],
        'disabled': ['postgres'],
}

data = {'classes': {'services': services}, 'parameters': services}

noalias_dumper = yaml.dumper.SafeDumper
noalias_dumper.ignore_aliases = lambda self, data: True
print yaml.dump(data, default_flow_style=False, Dumper=noalias_dumper)

All the changes are in the last three lines. First I create an object which is a copy of the SafeDumper class. Then I patch the ignore_aliases method on that object to always return True. Finally I use dump instead of safe_dump (safe_dump just calls dump, but specifies the SafeDumper) and pass along the newly created dumper which is based on SafeDumper. In the end, this is the new output:

classes:
  services:
    disabled:
    - postgres
    enabled:
    - nginx
    - webapp
parameters:
  disabled:
  - postgres
  enabled:
  - nginx
  - webapp

And now puppet can read it just fine.

retina macbook pro & screenshots

A while back I got a new Macbook Pro with the flashy retina display. It’s probably the most powerful computer that I’ve ever owned, and the display made the pain of being forced from my usual 17” laptop to the smaller 15” laptop a whole lot easier to stomach (also man is this thing easier to carry around in my laptop bag!)

One of the things that’s proven to be a pain with it though is that all those extra pixels make for massive screenshots. This is made even worse when I try to take a screenshot then post it to this site. As of now I’ve made due with using the really great website Pixlr, but that isn’t really ideal because it’s an extra step, and I’m super lazy.

Unfortunately despite quite a bit of searching I haven’t found a solution that provides for the following requirements:

• Acts like the regular screenshot command (shift-command-4 in other words)
• Automatically scales the image quality down to something reasonable (I realize that ‘reasonable’ isn’t a great definition).

Honestly I don’t know if such a tool exists, and maybe the haziness of the second requirement makes it impractical. Still, if anyone out there has found something that would work, I’d love to hear about it.

In the meantime I think I’m going to give the following workaround for posting the screenshots to this page a try. It’s not perfect, because it means I’m using larger images than necessary, but at least it lets laziness prevail.

One of the cool features of Sphinx (the thing that renders the RST documents I write into the HTML that you see, by way of Tinkerer) is that you can specify image dimensions. This isn’t all that amazing honestly, since HTML has allowed for this for a while, but its still useful. So for now, I plan on posting my images as normal, and then using something similar to the following to make sure the image doesn’t look GIGANTIC on the page:

.. image:: /_static/images/example.jpg
    :width: 540px

Anyway, like I said its not optimal, and if I take any gigantic screenshots I’ll still use Pixlr to shrink them down before uploading them, but for most of my screenshots I think using the width should be fine.

If you have any other suggestions I’m all ears!

current python class & method in vim status line

I’ve been looking through some (awesome) third party modules lately, and while doing so I kept running into the same issue fairly often. Whenever I was looking at a module that had large classes (spanning multiple screens) I would find myself getting lost. For the longest time I’ve just sucked it up, but today I decided there has to be a better way! Enter the Python Helper vim plugin.

After downloading that module and dropping it into my .vim/plugin/ directory now whenever I’m in a python file it does this for me:

Sooo much easier, and now I can save some of the precious few brain cells I have for other things.

copy and paste without formatting

So one thing that’s been annoying me more and more lately is that when you copy and paste text from a style-aware (say Chrome) app to another style-aware (say Mail.app) app in Mac OS X that it copies the style. There are shortcuts you can use to paste without style, but in Mail.app the shortcut is Shift-Option-Command-V (which doesn’t seem like all that much of a shortcut to my fingers).

Anyway, I went hunting for how to fix this and I found a way. Here it is.

The CLI version didn’t work for me, but the System Preferences route worked like a charm.

Moving to S3

Sooooo I finally decided it was time to shut down the server I’ve ran for the past 8 years. In doing that I’ve had to start moving the various sites and services I ran on it to other places. That includes this site.

In the end, since this is a static website, I decided to go ahead and host it on Amazon S3, especially since they recently announced that they have given sites on S3 the ability to host their root zone as a website there.

So I went and looked for directions on Amazon’s website and found this walkthrough. It’s pretty easy to follow. The only slightly annoying thing is that you need to use Route53 for your nameservers, which meant I had to move it away from my registrar. That didn’t bother me too much in the end - I don’t have any particular love for GoDaddy these days, and the domain for this website is tiny.

So first things first. At first I considered writing my own script for pushing the files on my site up to Amazon. After playing with it in boto, I decided to have a look around and see what else was out there. That’s when I found s3cmd.

It filled all my needs, specifically the sync functionality. So I installed it on my Mac laptop using Homebrew. Once that was done, I went ahead and wrote a simple script that would handle a few things for me:

1. Run Tinkerer to compile the static site.
2. Updated my sitemap using the Google Sitemap Generator
3. Call s3cmd to push my site up to S3.

I’ve gone ahead and pushed that script up to github (along with the rest of the site). You can find it here: s3push.

One other file that’s worth looking at is my .gitignore - I updated it so that I wouldn’t push my refreshed sitemap everytime I update the site.

Anyway, as you can see from the script, this is all super easy to do, and now you’re reading this site through the magic of S3.

Let me know if you have any questions or need a hand if you try to do the same and I’ll do my best!

Mountain Lion + Chrome + 2 Finger Swipe == ugh...

I’m not sure when it started happening, but recently I’ve noticed that when using the 2-finger-swipe to scroll left & right in chrome it sometimes instead choses to interpret that gesture as me wanting to either go back or forward in my browser history. Talk about annoying. Fortunately you can disable this, but rather than doing it in Chrome you have to do it System Preferences > Trackpad > More Gestures. The first option is ‘Swipe between pages’. Uncheck that and things go back to normal.

VIM Crosshairs

VIM is my editor of choice. I’ve tried a few others, but I’m too used to using vi keybindings (I even use them for command line editing in bash) and so I have a really hard time walking away from it.

Since I write python code, indent levels become very important. I try to keep my functions nice and short, but sometimes when looking at other people’s code I’ll find long functions and it becomes increasingly hard to tell which indent level I am at or should be at.

A useful way I’ve found for dealing with this is by having crosshairs that cross over where my cursor is. The best way to expalin this is by an example, so here it is:

Setting this up is actually pretty easy. The full explanation of it can be found on the vim wikia site. The way I have it setup is almost exactly like the example they give - when you hit ‘c’ the crosshairs are toggled on or off. To get that going, put the following in your .vimrc or your custom colors file:

hi CursorLine   cterm=NONE ctermbg=235
hi CursorColumn cterm=NONE ctermbg=235
nnoremap <Leader>c :set cursorline! cursorcolumn!<CR>

Checking git commit message format before commit’ing

UPDATE: You can’t actually use the python script directly as your commit hook due to the need for a TTY. Instead I’ve updated these instructions with a way to call it from a simple shell script which provides a TTY. Sorry for the misinformation!

Seems like I’ve been on a ‘style guide’ tear lately. Once I got git yelling at me when I tried to check in non-pep8 compliant code a coworker of mine pointed out that git commit messages have a ‘style guide’ of sorts. Of course now I needed to make sure that my git commit messages were good as well.

Fortunately the style guide for git commit messages is pretty simple, so I wrote my own really basic checker. If you’re interested, keep reading.

Keeping bash history forever

I spend a lot of time in a terminal. More specifically, I spend a lot of time in bash - being a sysadmin/developer type guy kind of necessitates it. One of my favorite features of most modern shells is the fact that they record the commands you run, allowing you to easily repeat commands or to go back and look at how you did something.

What’s even better is that bash saves your history to a file whenever you logout of your shell. While that’s awesome, it’s little less awesome that the size of your history file is limited (dictated by the HISTFILESIZE environment variable, by default to 500 lines, at least on Mountain Lion). There are tons of times when I go to look back through my history and the command I needed was missing because of this.

The problem becomes even worse when you have multiple terminals open.

Wouldn’t it be awesome if after every command that command was dropped in a file? What if there was no limit to the # of history lines you could record? What if you could record other useful data, like say what directory you were in when you executed the command, the time you executed it, and what its return code was? Finally, wouldn’t it be sweet if you could easily add notes to that history file?

If you answered yes to any of the above, then look no further!

Comments now work!

Thanks to Vlad Riscutia for putting out a hotpatch (and new release) of Tinkerer that fixes the Disqus bug issue signal0 now has comments. This may actually end up being a bad thing, but I still want to thank Vlad for being so responsive on this awesome piece of code!

Calibrating i(phone|pad|pod) home button

I’ve had an iPhone for a long time, and this year got the iPad. I love my Apple toys, but one thing that’s been driving me nuts about my iPhone for the past few months is that the ‘home’ button seems to have become less responsive. It’s really frustrating, and I thought at first that my poor iPhone 4 was on its last legs. I figured I’d be stuck dealing with this till Apple FINALLY released the iPhone 5.

When it started happening on my new iPad I was livid, so I went ahead and looked around online. Sure enough there’s a way to fix it, and it worked perfectly for both my iPad and iPhone.

Check it out: http://www.redmondpie.com/how-to-recalibrate-iphone-home-button-to-make-it-more-responsive/

examples of pygment styles

So in this blog post I talked about how you’d have to test each of the pygments styles for yourself. Well, turns out that’s just not true, thanks to Favio Manriquez Leon over at http://favrik.com/. He posted a cool page with an example of all the built in pygments styles at http://blog.favrik.com/2011/02/22/preview-all-pygments-styles-for-your-code-highlighting-needs/

Enjoy!

signal0 now on github

Some friends asked that I share my site source and so I figured it was probably worth sharing with the world. I’ve added a link over on the sidebar to my github repository for the signal0.com source code (and learned how to add my own sidebar entries - I’ll put a blog post up about that soon).

I figure I might as well give a quick outline of how I got the code into github just in case folks were curious.

Syntax+pep8 checking before committing in git

UPDATE 2012/08/13: I’ve updated the git hook code per suggestions from folks in the comments.

I write quite a bit of code in python. As a considerate python programmer I tend to try to follow PEP8 as best I can. That said, even though I’ve been working with python & pep8 for years it’s still easy to forget the little things.

While PEP8 is great while you’re working alone, it’s even more important when you’re working on code with other people. Wouldn’t it be nice if you could make sure that every piece of python code you push into your repository that other people might see is pep8 compliant? Fortunately this is very possible, and you can even add some syntax checking as well thanks to Tarek Ziade’s awesome flake8 script.

Modifying Tinkerer Themes

Update: After I wrote this I ended up going back to the ‘native’ pygments style. The reason was that monokai didn’t seem to produce css for the shell console code-block type, and I wanted those blocks to be pretty like the rest of them.

So one of the reasons it took me so long to create this site was that I had a really hard time finding a system I liked to create it with. My thought process on that is probably the content of another blog post, but since I’m in the middle of working on making this site the way I want it I figured I’d share some of the things I’ve found about the static blog generator I settled on, Tinkerer.

While for the most part I really enjoyed Tinkerer’s default theme (modern5), one thing I wanted to mess around with was the theme it used for syntax highlighting.

signal0?

Or ‘Where did the name signal0 come from?’

It comes from a ‘trick’ I like to use in unix shell scripts.

Basically when you want to check to see if a process is running with a given process ID in unix there are a lot of ways to check - but the way I think is the best is to execute a ‘kill -0 <pid>’. On my laptop I want to check if there is a process with a PID of 777:

$ kill -0 777
-bash: kill: (777) - No such process
$ echo $?
1

Or, used in a simple shell script that tells your nginx process is running:

1
2
3
4
5
6
7
8
9

#!/bin/sh

_pid=`cat /var/run/nginx.pid`
if kill -0 $_pid 2> /dev/null
then
    echo "nginx pid $_pid is running."
else
    echo "nginx pid $_pid is not running."
fi

A while back (like 2010?) I was looking to start a blog to write about various techy things I was working on and so I started looking for domains. Took a look, and sure enough signal0.com was available.

Of course I bought the domain, but then I never got around to actually writing that blog. Well here we are two years later. Lets see if I’ll keep up on this.