The Standard Output

For many years I have been an ADSL subscriber receiving 3 Mbps service from AT&T. The line came into the house and into a simple ADSL modem that connected to a Linksys WRT54GL running DD-WRT firmware, which was setup to serve as the Internet gateway for all computers in the house. I won’t go into the virtues of this particular model of Linksys router running this particular firmware, but suffice it to say that those who know what I’m talking about will need no further explanation.

Recently, we moved to a new house and AT&T would not let me transfer the ADSL line to the new address, but they said they would be happy to set me up with this shinny new digital service they call UVerse. After doing some checking, I noticed that the reviews from customers were mixed, but that the speeds provided were pretty decent. I was leaning towards cable service, but the wife likes AT&T, so we signed up for the 12 Mbps UVerse Internet plan.

The AT&T technician came to the house and set it all up in the phone box outside — inside the house he hooked the line up to a 2Wire 3600HGV device. When he saw my Linksys router laying nearby, he said “With this beauty right here…” and he showed me the 2Wire device, “… you won’t need that Linksys router of yours.” That should have been my fist sign that something was amiss.

The one redeeming quality of the 2Wire device is that it does have a really strong radio signal, but other than that the firmware running on it is really crippled and perhaps can do a little more than 10% of what the DD-WRT firmware is capable of. The conspiracy theorist in me says that the 2Wire hardware and software are probably more capable than AT&T is willing to document and let the users have access to,

Anyway, the ideal situation for me was to turn the 2Wire into a dumb modem and use my Linksys as the gateway, just like the old ADSL days… but that was easier in concept than in practice. There are all kinds of gotchas and tricks to get it setup, like the Router Behind Router detection. After much trial and error, I persevered, and below is the result of my research into how to get this setup to work.

At this point I recommend that you make a backup of your current DD-WRT setup by going to Administration and then the Backup menu and then clicking the Backup button.

The first step is to get the Linksys router ready for the setup. Make sure you can reach it and connect to it with a browser by going to http://192.168.1.1/. I used an Ethernet cable directly from my laptop and into one of the 4 LAN ports on the Linksys. Once you logged in to the Linksys, go to Setup and then Basic Setup. Under WAN Setup, make sure that the Connection Type is DHCP and give the device a unique Router and Host name. Under the Network Setup area, change the Local IP Address to something like 10.0.0.1 — just make sure that the IP you choose is in a different subnet than 192.168.1.X. With this setup we are trying to avoid conflicts between the Linksys DHCP and the 2Wire DHCP servers and in general we want the 2 devices in separate networks anyway.

The next step is to connect to the 2Wire device. I unplugged the Ethernet cable from the Linksys and connected it to one of the LAN ports on the 2Wire. Now you should be able to reach the 2Wire setup interface by pointing your browser to http://192.168.1.254/. If at first you are unable to connect, then check your current IP address. Since the Linksys was changed to a different network, you may have gotten an IP in the range 10.0.0.X. You can fix this by releasing and renewing your IP lease. Also, in order to apply any changes to the 2Wire device you will need to know the System Password which should printed on a label on the device.

Before moving on, I want you to consider the above image for a moment. Here’s what’s going on:

1. The blue ethernet cable connects the the 2Wire device to the WAN Port on the Linksys.
2. The yellow ethernet cable was used to connect the laptop directly to the 2Wire device and also to the Linksys previously.
3. The green RJ11 cable is the digital data cable connecting to the Internet at large.
4. The white RJ11 cable is for the phone/voice service and goes into the telephone set.

As you can see in the above image, the blue cable connected to the 2Wire device terminates into the WAN port of the Linksys device.

Now, back from this little tangent, I was about to connect to the 2Wire device with the browser. Once connected, go to Settings and the LAN tab and click on the Wireless link and ensure that the value under Wireless Interface is set to Disabled. I want the Linksys in charge of providing Wireless and DHCP services. Once that’s done, click the Save button at the bottom of the page.

Next I need some way for the Linksys to receive all inbound Internet traffic into the house. This can be done by putting the Linksys in DMZPlus mode in the Firewall setting of the 2Wire device. Go to the Settings tab in the 2Wire setup interface and then click on the Firewall tab and click on the Applications, Pinholes and DMZ. Choose the DD-WRT (or whatever name you gave your device) from the list under Select a computer. Then further down on that same settings page under Edit firewall settings for this computer select the choice Allow all applications (DMZplus mode). Make sure to click the Save button at the bottom of this page when all done.

You can verify that the 2Wire setup is correct by going to the Status page under the Settings and Firewall sections and seeing that the DD-WRT device is listed under the Current Applications, Pinholes and DMZ Settings: Custom and that it was assigned a public Internet IP.

One more side bar is called for at this point in the setup process. If at any point in time during the setup you are faced with a warning from the 2Wire device that it has detected a Router-Behind-Router setup, please just ignore it as the final outcome of the setup I am doing will render that type of detection irrelevant. As a matter of fact, while on the 2Wire setup interface, just go to the Settings tab and then the System Info tab and the Event Notifications link and make sure that the option Enable detection of router-behind-router conditions is not checked. Save your change before moving on.

Let’s do a quick recap: We first put the Linksys device in DHCP mode at the WAN level and gave it a recognizable name, then we gave it a different subnet IP than the default 2Wire IP. Next we disable the Wireless service on the 2Wire device and then we put the Linksys device in DMZPlus mode in the 2Wire configuration screens. We also verified that the Linksys is getting a public IP from the 2Wire perspective.

Next we need to ensure that the Linksys is getting the public IP address from it’s own perspective (see above image). Go to the Status tab and then the WAN tab on the DD-WRT setup interface. Under the WAN section on that page ensure that the IP Address value is the same as the one assigned to the DD-WRT in the 2Wire interface. I obfuscated the public values here for obvious reasons.

This is where I was a little stumped when I went through these steps for the first time. If at first your Linksys/DD-WRT doesn’t recognize the public IP, try using the DHCP Release button or rebooting the Linksys. I had to reboot the Linksys twice initially, but it has been working for me without any issues for a week now.

I hope this write up is useful to you…

I recently moved to a new house and during the move something really peculiar occured with my computer: It seems that the UEFI chip reset to the factory defaults. The 2 visible outcomes of this was that the computer clock was reset back to 2009-01-01 00:00:00 and that the Hardware Virtualization on the CPU was turned off. The latter manifested itself as an error when I tried to run virtual machines with Virtual Box. This was the error generated by Virtual Box:

To fix these errors I had to first set the correct system date and timezone offset and then I had to boot into the UEFI setup utility and enable hardware virtualization for the Intel i7 CPU. This setting is going to be slightly different on every motherboard, but after I enabled it VBox started to run the VMs as usual.

If you have ever played around with the Debian distribution proper, you will quickly notice that Firefox is not available as a browser and in its place is something called IceWeasel. Here I will show you how to install the real Firefox browser on Debian 7.

The first step is to remove the IceWeasel package if it’s installed:

    $ sudo apt-get remove iceweasel

Next, we will make use of a Linux Mint package repository that target Debian proper. To do this add the following line to you /etc/apt/sources.list file:

    deb http://packages.linuxmint.com debian import

Now when you update your package list you will see an error like so:

    $ sudo apt-get update
    ... a ton of output ...
    W: GPG error: http://packages.linuxmint.com debian Release: The following
    signatures couldn't be verified because the public key is not available:
    NO_PUBKEY 3EE67F3D0FF405B2

This happens because you don’t have the proper key for the Linux Mint repository. To a valid PGP key do the following:

    $ sudo gpg --keyserver pgp.mit.edu --recv-keys 3EE67F3D0FF405B2
    $ sudo gpg --export 3EE67F3D0FF405B2 > 3EE67F3D0FF405B2.gpg
    $ sudo apt-key add ./3EE67F3D0FF405B2.gpg
    $ sudo rm ./3EE67F3D0FF405B2.gpg

Now you should be able to update the package list successfully. Then what’s left to do is to install firefox and you can do that like so:

    $ sudo apt-get install firefox firefox-l10n-en-us

Enjoy firefox!

Just recently I rebuilt a system that I use for work. After I installed Cygwin on it, I started getting some weird errors when using git with ssh to do some source control tasks.

When doing a fetch, pull or push to/from origin, or really any git command that uses ssh to communicate, the ssh client would complain with the error:

    Could not create directory '/home/jgg/.ssh'.
    The authenticity of host 'github.com (204.232.175.90)' can't be established.
    RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
    Are you sure you want to continue connecting (yes/no)?

Once I enter yes, I get the following:

    Failed to add the host to the list of known hosts (/home/ME/.ssh/known_hosts).

This was happening everytime I was using git under cygwin and very quickly the situation became completely unsustainable. But after doing a little research, I found a solution that is pretty simple:

1. Locate the passwd file, which is usually at C:\cygwin\etc\
2. Open it with your prefered editor
3. On the line that starts with your username, change the section that reads /home/ME to read /cygdrive/c/path/to/home/folder/ME or wherever your home folder is.
4. Save the changes and restart the Cygwin Terminal

Expect to see the authenticity check one final time and see the ssh client actually be able to write to the known_hosts file.

Over the years I have written several of these articles and I find that every 2 to 3 years I need to re-write these notes because the installation process diverges enough from the previous set of steps that it warrants a re-write.

The main Linux distribution that I have settled on lately is Xubuntu, which is a XFCE-based version of Ubuntu. I dabble in other distros also and I am experimenting a lot lately with Arch and other Debian-based distros. One thing is clear, the versions of GNU Emacs available with each distribution are different. So, as a standard I will always try to download the latest stable GNU Emacs sources and build it on the system I am using at the moment. This ensures that I have a homogeneous set of features across the different machines and also that I am pretty close to the bleeding edge.

In order to get to this state with your Emacs install you can follow the below steps. These steps are focused on a Debian-based distro using APT. I will assume that you have a base Debian or Ubuntu system already installed with a common GUI environment. Again, I wrote these notes for a Xubuntu system running XFCE.

Let’s start by downloading and installing the tools needed to build Emacs:

    $ sudo apt-get install build-essential git git-core curl
    $ sudo apt-get install automake autogen autoconf ssh

Next, you need to get the sources for Emacs. There is two ways to go about doing this: 1) Clone the sources from an Emacs repository directly or 2) Download the latest compressed source tarball from a FSF/GNU mirror site.

To clone the Emacs source from a repo, do the following:

    $ cd ~/Downloads
    $ git clone git://git.savannah.gnu.org/emacs.git

Cloning will take several minutes, because git will get all the history and changes and we all know that Emacs has a long history. The clone method is the way to go for those who will hack on emacs itself. The easier way to do this, however, and the way that I prefer is to pull the source tarball from a mirror. A list of available mirrors can be found at the FSF site. You can pick any of these sites and download from it. The command to do that will differ based on the mirror you pick, but should look something like this:

    $ curl -O http://mirror.sdunix.com/gnu/emacs/emacs-24.3.tar.gz

While you are downloading the tarball or cloning the repo you should open another terminal window and start installing some needed Emacs dependencies in parallel:

    $ sudo apt-get install texinfo libncurses5-dev libgtk2.0-dev libgif-dev
    $ sudo apt-get install libjpeg-dev libpng-dev libxpm-dev libtiff4-dev
    $ sudo apt-get install libxml2-dev librsvg2-dev libotf-dev libm17n-dev
    $ sudo apt-get install libgpm-dev libgnutls-dev libgconf2-dev libdbus-1-dev

Once the above deps are installed and you finished retrieving the sources for Emacs, it’s time to do the actual build. Decompress the source if you downloaded it and change directory into the Emacs source:

    $ tar -zxvf emacs-24.3.tar.gz
    $ cd ~/Downloads/emacs-24.3/

Configure the build according to what you want your installation to look like:

    $ sudo ./autogen.sh
    $ ./configure

I use the vanilla configuration, but you can change it to what suits you best and you can view all alternatives by using ./configure --help … If you don’t like what configure has done for you (read the configure output) you can also clean things up and start over with make distclean.

The key thing that I am trying to achieve is a message from configure that looks very similar to the below output. Note all the yes values next to all the stuff that configure checks for; this is a direct outcome of installing all the dependency packages above. Emacs absolutely needs texinfo, ncurses, an X toolkit and the image libraries to even build. The rest of the stuff is nice to have, but if you don’t install it you will get a no from configure for that feature check.

    ... a ton of output

    Configured for `x86_64-unknown-linux-gnu'.

      Where should the build process find the source code?    ~/Downloads/emacs-24.3
      What compiler should emacs be built with?               gcc -std=gnu99 -g3 -O2
      Should Emacs use the GNU version of malloc?             yes
          (Using Doug Lea's new malloc from the GNU C Library.)
      Should Emacs use a relocating allocator for buffers?    no
      Should Emacs use mmap(2) for buffer allocation?         no
      What window system should Emacs use?                    x11
      What toolkit should Emacs use?                          GTK2
      Where do we find X Windows header files?                Standard dirs
      Where do we find X Windows libraries?                   Standard dirs
      Does Emacs use -lXaw3d?                                 no
      Does Emacs use -lXpm?                                   yes
      Does Emacs use -ljpeg?                                  yes
      Does Emacs use -ltiff?                                  yes
      Does Emacs use a gif library?                           yes -lgif
      Does Emacs use -lpng?                                   yes
      Does Emacs use -lrsvg-2?                                yes
      Does Emacs use imagemagick?                             no
      Does Emacs use -lgpm?                                   yes
      Does Emacs use -ldbus?                                  yes
      Does Emacs use -lgconf?                                 yes
      Does Emacs use GSettings?                               yes
      Does Emacs use -lselinux?                               no
      Does Emacs use -lgnutls?                                yes
      Does Emacs use -lxml2?                                  yes
      Does Emacs use -lfreetype?                              yes
      Does Emacs use -lm17n-flt?                              yes
      Does Emacs use -lotf?                                   yes
      Does Emacs use -lxft?                                   yes
      Does Emacs use toolkit scroll bars?                     yes

    ... more output

Now you can go into the actual build of Emacs by issuing the following command:

    $ make

And then last step is to install the Emacs you just finished building:

    $ sudo make install

… and run it:

    $ emacs &

Enjoy hacking with Emacs!

P. S. One last step that is personal and relevant to me only is to install my Emacs configuration. Execute the following to accomplish that:

    $ rm -r .emacs.d
    $ git clone git@github.com:gorauskas/.emacs.d.git

You will of course need to have the correct encryption key in order to clone the repo like the line above.

Recently I was faced with the task of looking for the occurences of a variable name in all files in a directory recursively. Here is a quick bash incantation that accomplishes that:

$ find . -type f -print0 | xargs -0 grep -l variable-name

The above can be understood better if we break it down at the pipe character. To the left of the pipe we use the find command to generate a list of file names recursively starting at the current directory. We use find with the -print0 action because we are passing the output to another program via a pipe and this way the list of file names come out delimited by a null character. Next, we feed the output of find to the input of xargs which will split the long list of file names into sublists and calls grep once for every sublist. The grep command then filters the list of file names and returns only the ones that contain the variable-name string in them.

Pretty cool, eh?

Here is a quick way to convert any integer n into a string that represents the n integer in binary notation:

def dec_to_bin(n, bits=32):
    return ''.join([str((n >> y) & 1) for y in range(bits - 1, -1, -1)])

What’s going on in the above code? This can be better explained if we read the return line in the dec_to_bin function backwards. First we create a list of integers from 31 to 0 in order to iterate through it. This list is created with the built-in range function and it uses the number of bits that we passed into the function, with a default being 32. Next, we iterate through the list with each number being represented by y. Then we take the number parameter n, which is the number we want to transform to binary, and we shift its bits to the right by y places. Then we apply a bitwise and operation on the result of that right shift and the literal value of 1. This and operation tells us if the bit at location y is turned on or off. We then make a string out of the value and append it to the string that we are building to return to the user. This is a perfect example of Python’s conciseness and the power of list comprehensions.

You can use the code above like so:

>>> print dec_to_bin(255)
00000000000000000000000011111111

Enjoy!

If you spent any time in Emacs you will sooner or later run into keyboard macros. In this article I will cover the basics of keyboard macros by going through a few practical, real-world examples. If you are new to Emacs or don’t know about keyboard macros, then this is a great place to start.

In my work I am querying databases and writing reports for people on a daily basis. I like to use Emacs SQL mode as my front end for connecting to a Microsoft SQL Server backend because it allows me to write queries and manipulate the results in very powerful ways.

A scenario where I use keyboard macros goes something like this: A business user gives me a list of invoices and she needs to check if they where processed in our invoicing application. She would send the list in email or IM and it looks like this:

001000403456
001000404567
001000405678
001000405679
001000405680
001000406789
001000406790
001000407890
001000407891
001000407892

In the database we have a table called InvoiceHeader where we keep the invoice data. The InvoiceNumber column is of a nvarchar(30) type. This means that if I want to use the list of invoices above in a IN clause of a transact SQL query, I need to wrap each item with single quotes and delimit then with a comma. That’s not too much work if this is a one-off and your list has only 10 items like the list above. But what if the list has 100 or 1000 items and you were doing these types of queries several times per week? Well, this is where keyboard macros come to the rescue.

Suppose that your SQL code looks like this:

SELECT h.InvoiceNumber
     , h.CustomerName
     , h.DueDate
     , h.Status
     , SUM(d.Cost)
FROM InvoiceHeader h (NOLOCK)
JOIN InvoiceDetail d (NOLOCK) ON h.RowID = d.InvoiceID
WHERE h.InvoiceNumber IN (
-- PASTE INVOICE NUMBER LIST HERE
) GROUP BY h.InvoiceNumber, h.CustomerName, h.DueDate, h.Status

I paste the list of invoice numbers from the email to where I indicated in the comment in the query above. The invoice numbers are structured one per line and each looking like this 001000407892 but I need them to look like this '001000407892',. To achieve that using Emacs keyboard macros, place the point on the line with the first invoice number and follow these keystrokes:

Go ahead and try it now. Once completed, the key sequences above allowed you to record the macro, and now you will want to apply the macro to all the lines in the rest of list of invoices. You do that by keying in the following chord: C-x e. This key sequence invokes the Emacs command kmacro-end-and-call-macro which calls the last defined keyboard macro.

This is certainly a lot more efficient than applying the single quotes and commas to each line manually. And this is just one way to make macros work for you. As you probably already know, there is more than one way to skin a cat in Emacs. Lets see how we can make our macro example even more efficient for our SQL coding situation.

Macros Reloaded

What I really want to do in the situation above is to yank my list of invoice numbers into the WHERE clause of the query and then mark the region. Once I have the region marked, I want to apply the macro to the whole region in one key chord. Starting with the following SQL code:

SELECT h.InvoiceNumber
     , h.CustomerName
     , h.DueDate
     , h.Status
     , SUM(d.Cost)
FROM InvoiceHeader h (NOLOCK)
JOIN InvoiceDetail d (NOLOCK) ON h.RowID = d.InvoiceID
WHERE h.InvoiceNumber IN (
001000403456
001000404567
001000405678
001000405679
001000405680
001000406789
001000406790
001000407890
001000407891
001000407892
) GROUP BY h.InvoiceNumber, h.CustomerName, h.DueDate, h.Status

This is what I do:

1. Place the point a the start of the line of the first invoice number
2. Record the macro by follonwing the key chords in the table above (if you haven’t already done so)
3. Undo the editing you done by recording the macro
4. Mark the region encompassing all the invoice numbers
5. Use the key chord C-x C-k r to apply the macro to each line in the region
6. Delete that last comma behind the last invoice number so you won’t get a syntax error in you SQL code

This is the result of performing the steps above:

SELECT h.InvoiceNumber
     , h.CustomerName
     , h.DueDate
     , h.Status
     , SUM(d.Cost)
FROM InvoiceHeader h (NOLOCK)
JOIN InvoiceDetail d (NOLOCK) ON h.RowID = d.InvoiceID
WHERE h.InvoiceNumber IN (
'001000403456',
'001000404567',
'001000405678',
'001000405679',
'001000405680',
'001000406789',
'001000406790',
'001000407890',
'001000407891',
'001000407892'
) GROUP BY h.InvoiceNumber, h.CustomerName, h.DueDate, h.Status

This is a simple example of the power of keyboard macros in Emacs. There are many other more complex tasks you can accomplish using keyboard macros that I have not covered here. Things such as parsing log files and outputing lines based on a regular expression or doing string substitution on a buffer while asking the user for the input to replace the tokens with.

Things to Remember

In order to write effective and flexible macros that work everywhere always remember to use your anchor key chords. These key chords allow you to place the point in the buffer with a great degree of flexibility. To name a few, they are: C-a, C-e, C-n, C-p, etc.

If you will perform the task you captured in the macro on a regular basis, then you will want to save it to your .emacs file and maybe even bind it to a key chord of your choosing. First, key in the chord C-x C-k n and give your keyboard macro a unique name. Open up your .emacs file and place the point in the buffer where you can yank some content. Use the elisp function to yank the macro definition from the macro ring to the point location by typing M-x insert-kbd-macro and then providing the name you just created. If you are saving the macro definition we have been working on in the article, then you should see code similar to this:

(fset 'prep-list-items-for-sql
   (lambda (&optional arg) "Keyboard macro."
     (interactive "p")
     (kmacro-exec-ring-item
      (quote ("^A'^E',^N" 0 "%d")) arg)))

Finally, you can bind your macro to a key chord to make it easily accessible in any situation. I have the following code in my .emacs file to bind Shift plus the F10 key to the macro:

(global-set-key [S-f10] 'prep-list-items-for-sql)

Until next time and happy emacs hacking!

This is off-topic for this venue and the fact that I decided to include it says a lot about how strongly I feel: I just watched the trailer for the upcoming movie Hyde Park on Hudson, starring Bill Murray as F.D.R., and based on that alone, I am hereby predicting that Mr. Murray will win an Academy Award for the performance.

If you want to get the latest version from a Git repository and you don’t care about any local changes, do the following:

git reset --hard HEAD
git clean -f
git pull

The above will copy the latest code from the repo and overwrite the local copy.