The Standard Output (bash)

Tracking down where disk space has gone on Linux

Jonas Gorauskas — Tue, 23 Sep 2014 20:03:26 GMT

When administering Linux systems I often find myself struggling to track down the culprit after a partition goes full. Here's a solution which relies on standard Linux commands to find where all that disk space is being consumed.

du -h / | grep '[0-9\.]\+G'

One caveat: Don't go straight to du /. Use df to find the partition that's hurting you, and then try du commands.

Basic WGET usage examples

Jonas Gorauskas — Mon, 24 Feb 2014 08:13:49 GMT

If you have ever used a relatively modern flavor of UNIX, you likely used a tool called wget. It comes as a standard piece of almost every single UNIX variant, Linux included. Here are some ways that I use it on a daily basis:

Download file

wget http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz

Downloading files with different names

wget http://ftp.gnu.org/gnu/wget/{wget-1.15.tar.gz,wget-1.15.tar.gz.sig}

Multiple downloads using different protocols

wget http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz.sig

Using an input file

You can use an input file that contains a list of URLs to download, like so:

wget -i ~/Downloads/urls-for-wget.txt

Resume incomplete download

Sometimes a download is interrupted in the middle. To resume a partial download use the -c option, like so:

wget -c http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz

Download files in the background

Using the option -b sends wget to the background and also saves the output to a log file:

wget -b ~/Downloads/wget-log.txt http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz

Throttling download speeds

To limit the download speed, use the --limit-rate option:

wget --limit-rate=100k http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz

Performing downloads with authentication

Some sites restrict access to content by requiring a username and password. wget can handle that:

wget --user=joeschmoe --password=p4sswOrd! http://ftp.gnu.org/gnu/wget/

More information

Of course, you can mix and match all of these options into a single command, like so:

wget -c -b -a ./wget-log --limit-rate=100k http://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz

You can also go much further than what I described above. The script below will check the web page for the latest listed version of wget and then download it and its signature file if the online version is newer than the local one.

wget -qN $(wget -qO- http://ftp.gnu.org/gnu/wget/ \
          | grep tar | cut -d\" -f6 \
          | tail -n4 | grep gz \
          | sed "s|^|http://ftp.gnu.org/gnu/wget/|")

For more information, check online or use man or info.

Resume interrupted downloads with cURL in Linux

Jonas Gorauskas — Mon, 14 Oct 2013 04:59:23 GMT

I have recently been trying to download an ISO of a certain Linux distro, but I find myself at the wrong end of a crappy connection. I tried downloading it with both Firefox and Chrome but both choked very early in the process and current speed would go down to 0 and after a while the connection would just cutoff with no way to resume the download from where I left off.

Fear not because [cURL][curl] comes to the rescue. After typing Up-Arrow and Enter more time than I care to admit, I decided to let the computer do the hard work and automate this thing. So I created a small Bash script called acurl.sh (for Automated cURL) that looks like this:

export ec=1;

while [ $ec -gt 0 ]; do
    /usr/bin/curl -O -C - $1;
    export ec=$?;
done

if [ $ec -eq 0 ]; then
    echo "Downloaded $1";
fi

What's going on above is that I have a while loop that is checking the exit code of curl. This exit code is captured by assigning the value of $? to the variable $ec. If curl exit code is larger than 0 then continue trying to download the file, which is represented by parameter $1, from where we left off, which is accomplished by passing the parameter -C to curl.

Assuming that the script above is located somewhere in the $PATH, then using it is as simple as this:

acurl.sh http://distrodomain.org/downloads/linux-distro-version-platform.iso

This is definitely a brute force approach, but it does the work.

[curl]: http://curl.haxx.se/ "cURL"

Grepping Multiple Files Recursively

Jonas Gorauskas — Thu, 18 Apr 2013 04:26:25 GMT

Recently I was faced with the task of looking for the occurences of a variable name in all files in a directory recursively. Here is a quick bash incantation that accomplishes that:

$ find . -type f -print0 | xargs -0 grep -l variable-name

The above can be understood better if we break it down at the pipe character. To the left of the pipe we use the find command to generate a list of file names recursively starting at the current directory. We use find with the -print0 action because we are passing the output to another program via a pipe and this way the list of file names come out delimited by a null character. Next, we feed the output of find to the input of xargs which will split the long list of file names into sublists and calls grep once for every sublist. The grep command then filters the list of file names and returns only the ones that contain the variable-name string in them.

Pretty cool, eh?

Quick Note on Bash History Substitution

Jonas Gorauskas — Sat, 02 Jul 2011 21:01:49 GMT

Expanding the last argument (!$):

$> cat .emacs
$> cat !$

Expanding the last command (!!):

$> ls -alF /etc
$> !! | less

Changing the hostname of a Debian server

Jonas Gorauskas — Mon, 30 May 2011 17:54:45 GMT

I am doing web development using Python on a Linux environment and the plan for the final product is to be hosted at Linode.

As such, I want to setup a integration test environment that looks a little like the stuff at Linode and namely, the cheap VM specs that they offer. So I created a default VM in VirtualBox on my development workstation that looks like the Linode 512 VM. This default VM is a template whose purpose is to be cloned into other VMs and moved into the test server for execution. I will go into the software specs (what's installed in it) some other time.

Suffice it to say that moving the VM to the test server has 2 caveats that I haven't quite addressed in a graceful manner yet. One of the issues happens when I boot the new VM in the test server using the cloned VDI file and it loses the eth0 network interface. The other issue is related to the hostname of the VM and I will discuss this here today.

When I created the default VM on the workstation I chose the hostname change-hostname for it in an effort to thwart any inclinations of not changing such setting. I use debian-based operating systems exclusively and in the case of the VM, it is a Ubuntu server 10.04. In order to change the hostname in such machines, you need to follow these steps:

Open the /etc/hostname file with your favorite editor. On Debian-based systems this file constains the hostname of the current machine.
Delete the current name and change it to whatever you like
Save the file and exit your editor
Reset the hostname by running the following command:
```
sudo /etc/init.d/hostname start
```

This is it... Pretty easy, but I want to automate it a little more; Too many steps to do manually.

How to resize a VirtualBox VDI file

Jonas Gorauskas — Sat, 28 May 2011 09:32:29 GMT

I am running Ubuntu Lucid on my workstation at home, but sometimes I still need to go back to Windows to get something done that I haven't quite figured out how to do in Linux yet. For that I am running VirtualBox on the workstation from which I spin up a Windows 7 VM.

I initially created the VM with 32 GB of hard drive space. That turned out to be too little and after the base Win7 OS and a few applications I was using over 15 GB. So I figured that I needed more hard disk space and I thought that I could just enlarge the pesky VDI file for the Win7 VM (it stands for Virtual Desktop Image and that's the default VM image file format used by VBox).

Before

Well, it turns out that in nature you can't really pick up a hard disk, open it up, and throw in a new platter. In a virtual world, however, you get to play God and manipulate it any way you choose. All you have to do is clone and modify your original VDI file. Here's what you need:

VirtualBox. Read this to install it on a Debian-based Linux distro.
Windows 7... But any other OS ISO image or disk will do.
GParted, the Gnome partition editor.

Assuming that you already downloaded the GParted ISO, installed VirtualBox and created some VM of which you want to resize the VDI file, here's how you do it:

Open a terminal (we'll be using the command line for this).
Find the VDI file you want to resize. This is usually under ~/VirtualBox VMs
Now clone the VDI by issuing the following command:
```
VBoxManage clonevdi Win7_64bit.vdi Win7_64bit_Larger.vdi
```
Where the first file name is the input file and the second file name the output.
Then resize the VDI (I grew mine from 32 to 64 GB):
```
VBoxManage modifyvdi Win7_64bit_Larger.vdi --resize 65536
```
Where the first parameter is the input file name and the resize parameter takes the new size in megabytes.
In the graphical VirtualBox manager, create a new VM using the cloned and expanded VDI file. I would pay special attention to use the same setting as the original VM that you cloned from.
Now boot the new VM using the GParted ISO image that you downloaded earlier. We will use it to resize the partition inside the VDI.
On the boot screen you will want to choose to boot using the Live GParted instance. We will use it to resize the partition in the VDI.
Once the live CD has booted up and the GParted application is running, select the partition you want to resize by clicking on it and then click the resize button.
Then select the handle at the end of the partition and slide it to fill up the entire unused space area. Click the resize button.
Next click the apply button on the GParted toolbar and watch the magic happen. Once done, click close and reboot the VM.
The first time you boot after performing the steps above, Windows will likely want to check the disk for consistency. Let it happen and Windows will reboot itself.

After

After this process, my Windows 7 VM was able to see a much larger drive. Let me know in the comments if you have questions or suggestions about how to improve this process.

VirtualBox on Debian-based distros

Jonas Gorauskas — Sat, 28 May 2011 06:35:42 GMT

There is a really great write up on how to install and maintain the latest Oracle VirtualBox package on a Debian-based Linux distribution.

Here are the steps in a nutshell:

Add a line to the /etc/apt/sources.list file:

deb http://download.virtualbox.org/virtualbox/debian lucid contrib non-free

Download and add the Oracle public key:

wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -

Install VirtualBox:

sudo apt-get update
sudo apt-get install virtualbox-4.0

That's basically it... Do refer to the link above for extra details and some caveats.

Restoring Installed Packages in Debian

Jonas Gorauskas — Thu, 19 May 2011 23:15:40 GMT

I have an older laptop that runs Ubuntu Linux and one of the issues that comes up regularly is that Ubuntu releases a new version every 6 months and I usually flatten the disks on that machine and do a fresh install. The base system is great and very usable, but there are several packages that I install manually after the fact, most of them being programming tools. There is a quick way for you to list out all installed packages and then restore them after upgrading and I will cover that technique here.

Note: The instructions listed here are for Debian based distributions that use .deb based packages.

Backing up a list of installed packages

In order to list out all installed packages and save them to a text file, you can do this from the shell:

$ dpkg --get-selections > installed-packages.txt

Let's deconstruct the above command:

dpkg is the Debian package manager
--get-selections is the parameter or action that lists all the currently installed packages. You can also pass a pattern to this action to generate different types of package lists. Look at the man pages for details.
> textfilename.txt will redirect the standard output to the text file specified.

Restoring the list of installed packages

When you need to restore the installed packages from a previous state, you can run the following command from the shell:

$ dpkg --set-selections < installed-packages.txt

The command above is a little different and those differences mean:

--set-selections is the action that sets the package selections using a file read in from standard in.
< textfilename.txt will read the contents of the file and send them to standard in.

Conclusion

Backing up and restoring your package installation list is a really simple procedure in Debian based distributions. You can even script it out and run it on a schedule and then save it to a cloud storage service such as Dropbox. Let me know what you think and if you have a better process.

Mounting a USB Stick in Linux

Jonas Gorauskas — Thu, 22 Jul 2010 21:50:00 GMT

Here is a quick way to mount a USB thumb drive in Linux. Most distros will recognize the USB stick when you plug it in, but it case they don't then this command comes in handy:

sudo mount -t vfat /dev/sdf1 /mnt/cruzer/

Now you can access and browse the contents of the usb stick by change the current directory to /mnt/cruzer/. Your mileage may vary and your Linux distro may have a different file system hierarchy.

Automating Emacs Build and Install from Source

Jonas Gorauskas — Fri, 14 May 2010 15:15:00 GMT

A little while back I wrote up my notes on how to build and install Emacs from souce on Ubuntu.

I have now automated the task in the form of the crude script provided below. The script takes one parameter: gui. This parameter tells the build whether you intend to do a GUI or command line only build of Emacs. It also writes out some messages to a file called emacs-build.log ... Enjoy!

#!/bin/bash
echo "Building @`date`" | tee -a ~/emacs-build.log
mkdir ~/emacs-src
cd ~/emacs-src

echo "Installing pre-requisites" | tee -a ~/emacs-build.log
apt-get -q -y install build-essential gcc git-core texinfo libncurses5-dev

if [ "$1" = gui ]; then
    echo "Installing GUI pre-requisites" | tee -a ~/emacs-build.log
    apt-get -q -y install libgtk2.0-dev libtiff4-dev libgif-dev libjpeg62-dev libpng12-dev libxpm-dev
fi

echo "Cloning Emacs Git Repo" | tee -a ~/emacs-build.log
git clone git://git.savannah.gnu.org/emacs.git

cd ./emacs

echo "Configuring build" | tee -a ~/emacs-build.log
if [ "$1" = gui ]; then
    ./configure
else
    ./configure --without-x
fi

echo "Building source code" | tee -a ~/emacs-build.log
make
echo "Installing emacs" | tee -a ~/emacs-build.log
make install
echo "Clean up" | tee -a ~/emacs-build.log
cd ~/
rm -rf ~/emacs-src
echo "All Done!" | tee -a ~/emacs-build.log