The Standard Output (python)

A note on migrating python virtual environments

Jonas Gorauskas — Wed, 18 Feb 2015 09:13:52 GMT

The engine that builds this blog is called Nikola it's written in Python. I write the blog and run the Nikola compiler to build the site inside a Python virtual environment.

Up until recently I was doing this work on a machine running the latest Arch Linux and Python 2.7.8, but I have recently rebuilt that machine to run Debian 8, whose default Python interpreter is currently at version 2.7.6 and I copied my virtual environments to the new OS install.

This caused me a problem with my virtual environments that manifested itself as the following error when I tried to run any Python code from within the active virtual environment:

$ pip freeze
Traceback (most recent call last):
  File "/home/jonasg/.virtualenvs/tso/bin/pip", line 7, in <module>
    from pip import main
  File "/home/jonasg/.virtualenvs/tso/local/lib/python2.7/site-packages/pip/__init__.py", line 11, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/home/jonasg/.virtualenvs/tso/local/lib/python2.7/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/home/jonasg/.virtualenvs/tso/local/lib/python2.7/site-packages/pip/download.py", line 2, in <module>
    import email.utils
  File "/usr/lib/python2.7/email/utils.py", line 32, in <module>
    from email._parseaddr import quote
  File "/usr/lib/python2.7/email/_parseaddr.py", line 16, in <module>
    import time, calendar
  File "/usr/lib/python2.7/calendar.py", line 9, in <module>
    import datetime
ImportError: No module named datetime

$ nikola build
...
ImportError: No module named datetime

This happens because my virtual environments had a different version of /usr/bin/python2.7 (the one from Arch Linux) that — unlike the new binary — does not include the datetime built-in, and therefore generates an error when it cannot find it on disk anywhere. The new interpreter seems to import it without any file I/O (try running it under strace to check).

The fix for this issue is to activate the virtual environment you are having problems with and run the following:

$ cp /usr/bin/python2.7 $(which python2.7)

There are other ways to fix this issue, but I like the one above rather than re-initializing the virtualenv because I use virtualenvwrapper which does NOT gererate the virtual environment within the present directory.

Hope this helps!

Free GeoIp location web service

Jonas Gorauskas — Thu, 04 Sep 2014 05:54:04 GMT

This weekend I found a really cool web service at FreeGeoIp.net. Their API is really simple... just send a HTTP GET request to:

http://freegeoip.net/{format}/{ip_or_hostname}

Here are some of the implementation details:

SSL is supported
Formats are csv, json, and xml
IP is optional. It defaults to your current IP.

I quickly implemented a comnmand line client written in Python 3. It uses the awesome requests library and ElementTree to parse the resulting XML response. Here's the code...

def main():
    args = parse_args()

    MYIP_URL = 'http://freegeoip.net/xml/'
    MYIP_HEADERS = {'Accept': 'application/xml'}

    if args.ip is not None:
        MYIP_URL = MYIP_URL + args.ip

    rspn = requests.get(MYIP_URL, headers=MYIP_HEADERS)
    tree = etree.ElementTree(etree.fromstring(rspn.text))
    root = tree.getroot()

    ip = root.find('Ip')
    cntr = root.find('CountryName')
    reg = root.find('RegionName')
    city = root.find('City')

    print('Your Current IP: %s' % ip.text)
    print('You appear to be in %s, %s, %s' % (city.text, reg.text, cntr.text))

Or you can get the whole code from my github profile.

Quick Ref: Python Virtualenv use

Jonas Gorauskas — Fri, 06 Jun 2014 07:55:33 GMT

Supposing you have a clean install of a Debian-based distro such as Debian 7 proper or Xubuntu 14.04, this notes will walk you through setting up a Python development environment where you can leverage the power of Virtualenv through Virtualenvwrapper.

Why virtualenv?

When you code in python you will quickly find out that different projects have different requirements for the python version you need to use and also the versions of libraries you need to use. Virtualenv will allow you to compartmentalize the requirements on a per project basis without making you install libraries and python versions globally or system-wide.

Ok, ok... Here are the goods...

You have your new shinny install of Debian which comes with a fairly recent version of python pre-installed (somewhere above 2.7.3). Now you need to install a package manager for python and the one I use is pip.

sudo apt-get install python-pip

Once you have pip installed, you will need to also install virtualenv, like so:

sudo pip install virtualenvwrapper

Virtualenvwrapper is a very nice layer on top of virtualenv that makes working with it much more intuitive and convenient.

Next, you may want to automate a couple of things like I did. First, create a ~/.virtualenvs directory. This is where virtualenvwrapper will keep track of your environments. Then open you .bashrc file and add a few lines to it:

export WORKON_HOME=$HOME/.virtualenvs
if [ -f /usr/local/bin/virtualenvwrapper_lazy.sh ]; then
    . /usr/local/bin/virtualenvwrapper_lazy.sh
fi

Now you are ready to start working, so open a new terminal, setup a virtual environment, and start cranking out some python code. Here's a quick reference for virtualenvwrapper:

Start new virtual environment?

mkvirtualenv [env-name]

Leave the current virtual environment?

deactivate

List all available virtual environments?

workon

lsvirtualenv -b

Enter an existing virtual environment?

workon [env-name]

List packages installed in current virtual environment?

pip freeze

Remove a virtual environment?

rmvirtualenv [env-name]

Start a virtual environment with specific python version?

mkvirtualenv [env-name] -p /usr/bin/python3.4

Replace [env-name] above with your actual environment name and enjoy!

Convert an Integer to a Binary Representation

Jonas Gorauskas — Mon, 15 Apr 2013 21:17:49 GMT

Here is a quick way to convert any integer n into a string that represents the n integer in binary notation:

def dec_to_bin(n, bits=32):
    return ''.join([str((n &gt;&gt; y) &amp; 1) for y in range(bits - 1, -1, -1)])

What's going on in the above code? This can be better explained if we read the return line in the dec_to_bin function backwards. First we create a list of integers from 31 to 0 in order to iterate through it. This list is created with the built-in range function and it uses the number of bits that we passed into the function, with a default being 32. Next, we iterate through the list with each number being represented by y. Then we take the number parameter n, which is the number we want to transform to binary, and we shift its bits to the right by y places. Then we apply a bitwise and operation on the result of that right shift and the literal value of 1. This and operation tells us if the bit at location y is turned on or off. We then make a string out of the value and append it to the string that we are building to return to the user. This is a perfect example of Python's conciseness and the power of list comprehensions.

You can use the code above like so:

>>> print dec_to_bin(255)
00000000000000000000000011111111

Enjoy!

Delicious to Blogger Python Script

Jonas Gorauskas — Sat, 09 Oct 2010 21:19:00 GMT

Readers of this blog (that means my wife and my mom) will notice that there is a new post with a weird title right before this one (in chronological order). What's going on with that?

On a weekly basis I save several links to my Delicious account and I consider that to be a pretty good representation of what is going on in mind at the time. The only problem is that sometimes I link to something and then it gets lost in the black hole that Delicious can sometimes be. There are situations when I saved a link and didn't go back to look for it until a year later.

So I thought that I wanted to group these links together and bring them up front and center in my radar. I just want to have a little better visibility into what I am thinking at the time. This is a task that can be easily automated and I wrote a script in Python to do that for me. You can look at the code here.

Basically, the script looks at the links posted to Delicious since a certain date, downloads the information about those links and creates a blog post for my Blogger account based on the Delicious links.

Errors with the prepopulate_from SlugField parameter in Django 1.2.1

Jonas Gorauskas — Tue, 06 Jul 2010 02:26:00 GMT

I recently upgraded to Django version 1.2.1 and I immediatelly noticed that some of my models were broken with the following error:

C:\test&gt;python manage.py syncdb

Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_manager(settings)
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\core\management\__init__.py", line 438, in execute_manager
    utility.execute()
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\core\management\__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\core\management\base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\core\management\base.py", line 217, in execute
    self.validate()
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\core\management\base.py", line 245, in validate
    num_errors = get_validation_errors(s, app)
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\core\management\validation.py", line 28, in get_validation_errors
    for (app_name, error) in get_app_errors().items():
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\db\models\loading.py", line 146, in get_app_errors
    self._populate()
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\db\models\loading.py", line 61, in _populate
    self.load_app(app_name, True)
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\db\models\loading.py", line 78, in load_app
    models = import_module('.models', app_name)
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\utils\importlib.py", line 35, in import_module
    __import__(name)
  File "..\tso\blog\models.py", line 50, in <module>
    class Category(models.Model):
  File "..\tso\blog\models.py", line 53, in Category
    slug = models.SlugField(unique=True, prepopulate_from=('title',), help_text='Used in the category URL. Must be unique')
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\db\models\fields\__init__.py", line 988, in __init__
    super(SlugField, self).__init__(*args, **kwargs)
  File "C:\Python26\lib\site-packages\django-1.2.1-py2.6.egg\django\db\models\fields\__init__.py", line 542, in __init__
    super(CharField, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'prepopulate_from'

So I did a little bit of research on this problem and I found the following explanation for what is going on:

Changed prepopulate_from to be defined in the Admin class, not database field classes

As of [4446], the prepopulate_from option to database fields no longer exists. It's been discontinued in favor of the new prepopulated_fields option on class Admin. The new prepopulated_fields option, if given, should be a dictionary mapping field names to lists/tuples of field names. This change was made in an effort to remove admin-specific options from the model itself. Here's an example comparing old syntax and new syntax:
# OLD:

class MyModel(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)
    slug = models.CharField(max_length=60, prepopulate_from=('first_name', 'last_name'))

    class Admin:
        pass

# NEW:

class MyModel(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)
    slug = models.CharField(max_length=60)

from django.contrib import admin

class MyModelAdmin(admin.ModelAdmin):
    prepopulated_fields = {'slug': ('first_name', 'last_name')}

admin.site.register(MyModel, MyModelAdmin)

The above is taken from the Django wiki page that talks about the newforms-admin branch

This is why the command line will never die

Jonas Gorauskas — Sat, 19 Jun 2010 09:07:00 GMT

A while back I wrote about the address line of the browser being the new command line. It turns out I was wrong. The command line itself is the newcommand line.

I ran into this Google Code project today via my tweeter feed. GoogleCL allows the nerd in me to interact directly with several Google services from a shell prompt. It includes interfaces to Blogger, Picasa, Calendar, Contacts, Docs, and YouTube. You can find several scripting examples under the project's Wiki.

The fact that this is a python script and it's open source is really interesting to me. I am thinking that I need to integrate all of this into an Emacs mode.

Generate .NET Assemblies from Iron Python

Jonas Gorauskas — Fri, 26 Mar 2010 14:32:00 GMT

The other day I was talking to a fellow programmer and the question of compiling Iron Python code into .NET assemblies came up. So what if you have an Iron Python script that you want to run and then have the ipy interpreter output an assembly that you can run on other machines. How do you do that?

There are several ways to approach the problem. One way is to use an IDE, but IDE support for Iron Python is still a bit lacking. One IDE that provides Iron Python great support is SharpDevelop create a Iron Python project, write your code as you normally would and then build your project. The compile assemblies can then be found in the build output folders for that project.

The other way to do this is through a tool that ships with IronPython called Pyc or the Python Command-Line Compiler. If you installed the latest version 2.6 of IronPython, then pyc.py will be available at C:\Program Files (x86)\IronPython 2.6\Tools\Scripts or wherever you installed IronPython on your system. If you have earlier versions of IronPython then pyc.py will be available as a separate download in the samples package.

With pyc.py you can create console or Windows assemblies from your python scripts. Basic usage looks like this:

ipy pyc.py /out:myprogram.exe /main:mainfile.py /target:exe program.py support.py

The above will generate an assembly called myprogram.exe (/out) which is a console application (/target) and will execute the code in mainfile.py first (/main) and will also include code from program.py and support.py in the assembly.

Some Notes on Serializing Objects in Python

Jonas Gorauskas — Sat, 16 Jan 2010 18:40:00 GMT

I was playing with .NET serialization at work the other day and got curious about how Python does it. Serialization is a little confusing in the .NET world, but it's not an insurmountable task to grasp it. For one, there is more than a single implementation of serialization within the .NET base class library, or namely, System.Xml.Serialization and System.Runtime.Serialization, which respectively implement XML and binary serialization. The techniques used in each implementation are also disparate, having the binary serialization make heavy use of class attributes, while the XML implementation uses a method call to XmlSerializer.Serialize.

The Python implementation of serialization is much simpler, concise and easier to understand. It is implemented as a Standard Library module called Pickle. The actions to serialize and deserialize classes are implemented as simple function class and there is no need to put attributes on classes. Let's see how it works.

First import the pickle module and then declare a class called Person as in the code below:

import pickle

class Person(object):
    def __init__(self, first_name=None, last_name=None, age=None):
        self.first_name = first_name
        self.last_name = last_name
        self.age = age

Now create two instances of the Person class above and place them in alist.

p1 = Person('Jane', 'Doe', 26)
p2 = Person('John', 'Hancock', 33)
people = []
people.append(p1)
people.append(p2)

Next serialize the list to a file and then read it back into a new list. First serialize the list:

fname = 'peoplelist.dat'
f1 = open(fname, 'wb')
pickle.dump(people, f1)
f1.close()

Finally, read the contents of the serialized file back into a new list and print out the name and age of each person:

f2 = open(fname)
new_people = pickle.load(f2)
for person in new_people:
    print '%s %s is %d years old.' % (person.first_name, person.last_name, person.age)

That's it... Serialization in Python is just too easy!

Some thoughts on Python and Unladen Swallow

Jonas Gorauskas — Sat, 16 Jan 2010 09:16:00 GMT

Between Twitter and the blogosphere, I have been hearing a lot about Unladen Swallow lately. For those who don't know, Unladen Swallow is an experimental branch of Python that aims at improving performance of the language. In their own words, Unladen Swallow is "An optimization branch of CPython, intended to be fully compatible and significantly faster."

I wanted to find out more and started reading their Project Plan page on Google Code. I think their goals are commendable, as you may see for yourself, and number 5 below explains why I added bold face the word branch above:

Produce a version of Python at least 5x faster than CPython.
Python application performance should be stable.
Maintain source-level compatibility with CPython applications.
Maintain source-level compatibility with CPython extension modules.
We do not want to maintain a Python implementation forever; we view our work as a branch, not a fork.

This is all fine and dandy, and the list above has made the rounds on the blogs. But what does it all mean? What follows is my impressions of the most important points that the Unladen Swallow branch is addressing.

A New Virtual Machine

The goal is to eventually replace the Python 2.6.1 virtual machine with a just-in-time compiler built for the LLVM. The rest of the Python runtime would be left untouched. The key benefits of this approach are that is a register-based machine and those perform better than stack machines, which is what the current Python VM is implemented as.

The internals of the implementation will assume at the outset that the machine has multiple cores. For instance, very aggressive optimization of code is assigned to a secondary cores while compilation occurs on other cores. The garbage collector for Unladen Swallow will also be implemented to utilize multiple cores.

The Global Interpreter Lock

While Python has had threading for a while, it is not a true multi-threading implementation. This is because of the existence of the GIL. Dave Beazley has written about the GIL and how it works several times and you should read his "The Python GIL Visualized" article to find out more about why the GIL keeps Python from having a real multi-threaded runtime.

I bring up the GIL here because the folks working on Unladen Swallow plan on removing the GIL from Python, although they are not very optimistic about it. And even if they are not able to remove the GIL completely there may be other optimizations in the garbage collector reference counting mechanism that may yield some improvements in the threading area.

Anyway, these are the two major points I take away from the Unladen Swallow plan of record. These changes seem pretty big to me and a major risk of doing this kind of work is that your changes are rejected by the community. However, the Unladen Swallow team is sponsored by Google who also employs Guido, so I'm sure that those guys are talking amongst themselves.

Thanks for reading this, go read the project plan and let me know what you think.

Iron Python: How to talk to the .NET Framework

Jonas Gorauskas — Tue, 26 May 2009 06:00:00 GMT

Iron Python is an implementation of the Python programming language that runs on the Common Language Runtime and can use the classes of the .NET Framework.

Here are some quick notes on how to interface with the framework from Iron Python:

Open the Iron Python REPL.
Import the clr module.
Add a reference to the .NET Namespaces you want to interact with.
Start using the .NET classes as if they were Python classes.

Below I transcribed a little code snippet for you to use as an example. I also provide a screen capture of the sample and the outcome of running it.

 import clr
 clr.AddReference("System.Windows.Forms")
 from System.Windows.Forms import Application, Form, Button, MessageBox

 f = Form()
 b = Button()

 f.Text = "Hello World!"
 b.Text = "Click Me!"

 def clicked(sender, args):
     MessageBox.Show("Hello, Again...")

 b.Click += clicked
 f.Controls.Add(b)
 f.ShowDialog()

Enjoy!

Ternary Operator in Python

Jonas Gorauskas — Thu, 16 Apr 2009 00:13:00 GMT

C-like languages such as C# have a ternary operator which uses the symbols: ? and :, while the actual syntax looks something like this:

string s = myBoolValue ? 'True' : 'False';

Python doesn’t feature a ternary operator as in the C-like languages, but the underlying behavior is supported. You can achieve this in one of two ways:

s = 'True' if myBoolValue else 'False'
s = myBoolValue and 'True' or 'False'

The if form in number 1 above was introduced in Python 2.5 to address feature requests for a ternary conditional form (see PEP 308).

The form in number 2 above is a clever way to achieve the same effect using short circuiting. The conditional form in number 1 is the preferred form as the technique in number 2 is error-prone.

Update: 2009-09-25

With the ternary example number 2 above there is a small caveat that didn't occur to me until today. Consider the following code:

(a,b) = ('abc','xyz')
print True and a or b
print False and a or b

The above code is fine and behaves as expected. But what if the value of a turned out to be equal to a Python False? That would give us a wrong result... Consider the following code:

>>> (a,b) = ('', 'xyz')
>>> print True and a or b
xyz
>>> print False and a or b
xyz

See the conundrum? The real trick is to make sure that the value of a is never false. One way to fix this is to turn a and b into lists and the whole expression into a tuple and then read out the first element returned, like so:

>>> (a,b) = ('', 'xyz')
>>> print (True and [a] or [b])[0]

>>> print (False and [a] or [b])[0]
xyz

The first print statement now returns an empty string as expected.

XO: First Impressions

Jonas Gorauskas — Sun, 09 Mar 2008 07:12:00 GMT

I had the XO for a few days now and I had the opportunity to play with it a little bit. The device is small; smaller than I expected. And, yet, it packs a lot of punch for the size and the components inside. It's definitely slower than your top-of-the-line two core machine, but you are not left twiddling your thumbs while waiting for things to load.

The screen presents great sharpness of image and is big enough for 99% of the tasks that I will ever perform with it. I'm not going to use the XO for extensive programming tasks, for instance. But the XO is great to take on trips with the family just to check on email and surf the web.

I have large hands, so the keyboard is unusable for me. Thankfully, I can easily plug my HHKB into one of the three USB ports and use it instead.

It comes with a built-in wireless network card that connects right up to my wireless network at home. I also read a little about the mesh networking capabilities of the XO, but haven't had the chance to actually play with it first-hand yet. Mesh networking is a fascinating technology, though.

A little perk that comes attached to the XO version that I have is the free year of T-Mobile HotSpot access. I activated mine today and it works like a charm.

The OS is a modified version of Fedora Linux, so I'm sure it is highly customizable. This weekend and the next few days I will be doing a lot of reading at the Laptop wiki site. They have an amazing amount of really great information about the machine itself, as well as projects and lessons to do with the kids. I also found some very useful posts by Bill Clementson about his experiences with his XO.

Anyway, I will keep you posted.

It's the Python

Jonas Gorauskas — Fri, 04 Jan 2008 06:24:00 GMT

This is how I really feel about python: http://xkcd.com/353/