Python Simple

September 20th, 2009 by Python Simple 1 comment »

What is the Python Simple website about?

Python Simple is all about combining simplicity with the power and elegance of Python to provide everyday value. Study Python tutorials and Python examples from RealLife™,  get recommendations for Python books, and discover other useful Python stuff. Simplify your everyday computer life with Python Simple!

From a operating system perspective, and not surprisingly, Python Simple‘s focus is on Windows: To provide the most everyday value it is logical to choose an everyday system – and Windows is by far the most widely used OS in personal computing.1

In case you don’t know: Python is a dynamic, object-oriented, general-purpose programming language that combines clarity and expressiveness with power and versatility to an extraordinary degree. From the Python website:

Python is a programming language that lets you work more quickly and integrate your systems more effectively. You can learn to use Python and see almost immediate gains in productivity and lower maintenance costs.

Excellent language features that allow for an extremely succinct coding style (read “with Python you code less to accomplish more”) are complemented by it’s “batteries included” property (meaning Python comes with a ton of extremely powerful libraries for a vast spectrum of programming areas2). With all that power, Python programming is really more about solving the real problem at hand (in contrast to struggling with language/library insufficiencies) than most other programming languages. As such, Python is very well suited for large-scale software development (think BIG applications), and it is being used by an increasing number of major organizations like NASA, Google, Industrial Light & Magic, the The New York Stock Exchange (NYSE), and many more.

Google and YouTube use Python because it’s highly adaptable, easy to maintain, and allows for rapid development.

Here at SIAC and NYSE Python is recognized by management to give results that other languages just can’t achieve.

But Python is also very easy to learn, especially if you are a non-programmer:

A user-friendly, object-oriented language, Python is quickly becoming the favorite introductory programming language among students and instructors. Many find Python to be a more lucid language than Java but with much of the functionality and therefore the ideal first language for those entering the world of Computer Science.

LBNL, Python is also an excellent choice for everyday scripting tasks. That is, Python is ideal for automating tedious day-to-day routine, be it updating your downloaded inventory from the internet, testing your software, or hacking your way into anything on a computer.

Python is fast becoming the programming language of choice for hackers, reverse engineers, and software testers because it’s easy to write quickly, and it has the low-level support and libraries that make hackers happy.

This, above all, is where Python Simple weighs in: Helping you lifehack your everyday computer life as much as possible.

  1. We do not necessarily like the fact about Windows’ market share, but we can’t ignore it. []
  2. Where the default libraries are not enough, myriads of freely available Python packages and Python extensions are available on the Web. []

ActivePython 2.6 Installation Path Caveat

March 10th, 2010 by Python Simple 1 comment »

pathIn a prior reading we recommended to install Python on Windows into C:\Program Files\Python\X.Y (with X.Y being the Python version number). Alas, due to a bug in one of the included ActiveState’s tools, there is a minor issue with this installation location. Fortunately, there is also a very simple workaround.

The Problem

The ActivePython 2.6 distribution includes ActiveState’s new Python Package Manager, PyPM. PyPM is relatively young, it’s labeled “beta”, and – not surprisingly – is the source of mentioned issues: It contains a bug that effectively prevents running PyPM from a path that contains a space.1

The problem can be tracked down to a simple application invocation bug: On Windows, to execute a program path that contains a space, the program path must be enclosed in quotes. Hence

"C:\Program Files\Foo\Bar.exe"

will work, but

C:\Program Files\Foo\Bar.exe

will try to execute C:\Program instead (with “Files\Foo\Bar.exe” as argument) and fail.2

Note that this is only an issue with Windows in languages where “Program Files” actually contains a space. For example, the German version of Windows XP uses C:\Programme as default program directory, so the issue doesn’t manifest.

Problem Analysis

We investigated the problem further and discovered that it is not directly related to the path where Python is installed, but to the path how Python is invoked. (The two are usually, but not necessarily the same; see the solution.)

When ActivePython is installed, it adds the installation path to the program search path (the PATH environment variable), so you can execute “python” from the command line from every directory. Even when you execute a script (like pypm) on the command line, python.exe is searched on the PATH, and it is invoked from there. The python invocation path is stored in sys.executable, from where it is obviously used by PyPM – in a wrong way.

The Solution

The simple solution that worked for us was to change the python invocation path without reinstalling it. Our solution is rather generic, it fixes the whole family of issues connected to “Program Files” containing a space, not just the PyPM issue that gave rise to it.

This is how to do it:

Step 1: Create a symbolic link to C:\Program Files that does not contain a space, call it Programs. On Vista, this can be done using the command line:

mklink /j C:\Programs "C:\Program Files"

On Windows XP, you can use the junction utility to create symbolic links to directories:

junction C:\Programs "C:\Program Files"

Step 2: Replace the PATH entry C:\Program Files\Python\X.Y with C:\Programs\Python\X.Y. (Here’s how.) For first-time python installs, you can install it into C:\Programs\Python\X.Y right away.

This way, the standard program location is preserved and the blanks-in-paths issue is resolved.

Update: The author of PyPM, Sridhar Ratakumar, has kindly provided an alternative and easy to apply fix for the concrete PyPM issue. See his comment below. Keep in mind though that, of course, it does not help when other programs do not handle spaces in paths correctly (happens more often than one might expect).

Conclusion

We still advise to stick with the recommended installation locations. For one, PyPM will mature, and “child diseases” like the one described here will go away. Meanwhile, the shown directory link hack will do the trick.

To prevent future issues with Program Files you may want to install all your programs using the newly created link.

  1. This is a type of bug that should be a matter of the past given how long paths with spaces exist, but… []
  2. We have never understood why Microsoft has chosen a two-word name, “Program Files”, for their default program location, when “Programs” would have prevented so many of the early – and obviously still occsionally recurring – problems with spaces in paths. []

Python Data Processing

November 4th, 2009 by Python Simple No comments »

Python Data ProcessingOne particular strength of Python is it’s ability to process data in a variety of ways. While many programming languages would impose a great overhead here, in Python most data processing tasks are hacked away within (far) less than 100 lines of code.

What do we mean by data processing?

From a certain perspective, most programming tasks can be related to either user interaction or data processing. User interaction is basically GUI programming. While it delivers the data to process (for example by the user selecting a file), the data processing part makes something useful with it (transforming it or performing computations upon it). Finally, the results of this data processing are passed to the user interaction software parts for presentation to the user.

Usually, GUI programming is pretty involved, but it is worth the effort for any kind of serious software product intended for public use. In everyday personal computing however, where you are the only target user for your everyday automation, great results can be achieved with data processing alone. (In future posts, we will present examples and tools that demonstrate this.)

Why is data processing in Python so easy?

For one, Python is a programing language with so-called dynamic typing. Particularly data processing tasks (from database access to complex data transformations) benefit from dynamic typing: Functions and algorithms can be formulated very succinctly, with a few lines doing wonders.

For another, the Python language includes very high level language constructs for dealing with data sets (as opposed to dealing with singular data elements). Traditionally, features for dealing with data sets were found in special-purpose database languages such as SQL. Python was probably the first wide-spread programming language to include such features as part of the language. Today, many follow this paradigm shift, as exemplified by LINQ.

Finally, Python is packed with libraries that are particularly aimed at easy and efficient data processing. From sophisticated string handling and regular expressions to streamlined SQL database interfaces to comprehensive XML and HTML processing, a whole universe of ready-to-use code libraries is there to help you complete your everyday tasks quickly and elegantly.

Python data processing example

One such everyday task is inserting a piece of text into an email or such as a quote. (You email program may support it already.) This implies prefixing every line of text with “> “, and can be easily done using the Windows clipboard as the only user interface – in only two lines of code.

>>> from WinClipboard1 import getText, putText
>>> putText(''.join( '> ' + line for line in getText().splitlines(1) ))

If you don’t understand the code yet, don’t worry. Here is a short explanation.»

You simply copy the (unquoted) text to the clipboard, run this little program, then paste the clipboard contents into your application/document/whatever.

Conclusion

Python is an excellent choice for everyday personal automation tasks that naturally involve a lot of data processing. Both its language features and its comprehensive libraries make lifehacking your everyday computer life a breeze.

# get clipboard text
  1. Note: WinClipboard is a small wrapper that we made for the win32clipboard built-in Python module. It will be presented at another occasion. []

The second line of the above code could be rewritten like this:

# get clipboard text:
text = getText()
# split text into lines, preserving line breaks at their end:
lines = text.splitlines(1)
# prefix each line with '> ':
lines = ( '> ' + line for line in ...''.join(...) )
# rejoin lines:
text = ''.join(lines)
# copy modified text back to clipboard:
putText(text)

For readability sake, this would probably be the preferred way to write the code anyway, but we wanted to demonstrate how succinct and powerful a single line of Python code can be.

Powered by Hackadelic Sliding Notes 1.6.5

Python Site-Packages – What They Are And Where To Put Them

October 30th, 2009 by Python Simple 1 comment »

Papiroflexia modularIn this post we will show how to set up an advanced directory structure for painless maintenance of version-specific and version-independent Python packages.

In Python, reusable code is distributed in so-called modules and packages. A Python package is a folder containing several Python modules and possibly sub-packages.

When a module X is imported, Python searches its module search path and loads the first module named X that it finds. This is a very powerful and far-reaching concept, because it facilitates and proliferates reuse much more than approaches where module dependencies are resolved explicitly (using some kind of central configuration file or such).

Python expands on this concept further with the notion of site-packages directories. A site-packages directory is where you install additional python packages, including your own reusable code.1 There can be more than one site-packages directory.

When python installs, a default site-packages folder is set up. On Windows, the folder is created inside the Python installation directory tree. (For example as C:\Program Files\Python\2.6\Lib\site-packages.) Hence, every installed Python version has a different site-packages directory. We call this the version-specific site-packages directory.

This is not where you want to install pure Python modules and packages!

Here’s why:

Python differentiates between “pure-python” modules and “native” or “built-in” modules. The latter are modules compiled for a specific python version. They won’t work with any other Python version.

These are the modules that belong into the version-specific Python site-packages directory.

But pure-python modules are usually inter-operable across Python versions. There is no sense maintaining redundant copies of them.

Hence, we set up a version-independent site-packages directory as well.

On Windows Vista, a good place is either C:\Users\Public\Libs\Python\site-packages or C:\Users\<your name>\Libs\Python\site-packages. (We prefer the former.) If you installed Python into C:\Users\Public\Python\X.Y you should use C:\Users\Public\Python\Libs\site-packages instead. On Windows XP, you may also consider C:\Program Files\Python\Libs\site-packages or C:\Program Files\Python\Any\Libs\site-packages (to contrast with C:\Program Files\Python\X.Y\Libs\site-packages).

Where ever you put that folder, there are two important steps to take:

  1. In each version-specific site-packages directory, create a PTH file (this is just an ordinary text file; you may call it xsites.pth or whatever you want as long it has the suffix “.pth”), and enter in it the location of your version-independent site-packages directory.
  2. In the version-independent site-packages directory, create a file named sitecustomize.py with the following content:
import os, site ; site.addsitedir( os.path.dirname(__file__) )

Now when you start Python, both the version-specific and the common, version-independent modules and packages will be accessible.

This way, you have to maintain mostly just a single collection of pure-python modules and packages. And when you switch to a newer Python version, you only need to re-install version-specific stuff.

A real time-saver, that much we can tell.

  1. Besides being on the Python module search path itself, a site-packages directory may contain so-called PTH files to control the inclusion of additional directories to the search path. All PTH files in the site-packages directory are automatically processed at Python startup. That way, packages can reside at any nesting subdirectory level as long as a corresponding PTH entry exists. []

Python Install

October 5th, 2009 by Python Simple 1 comment »

Hello World!Before you can use any of the code or apply any of the recipes around here, you’ll need to make sure Python is in place on your machine. Here’s are instructions how to obtain and install Python.

Do You Need To Install Python?

If you are a Linux user, Python is almost certainly already installed. Mac users have a pre-installed python, too. However, both Linux distributions as well as Max OS X may come packaged with an outdated version of Python, so a fresh Python install is worthwhile anyway. To check the version of Python, open a terminal window and type in

python --version

If the output is something like “Python X.Y.Z”, then Python version X.Y.Z is installed on your machine. The Python versions current at time of this writing are 2.6.3 and 3.1.1. For any serious work, your Python version should not be below 2.5.1.

$ python --version
Python 2.6.3

Windows users must download Python and install it – nothing pre-installed here.

Python Install For Non-Programmers

On Windows, non-programmers should head for a standard Python installation. Python downloads can be obtained from the Python website. Grab an installer, execute it and follow the instructions. Vista users should ensure they run the Python install with admin privileges.

Mac users should check their version and possibly update it from the above Python website. Leopard has Python 2.5.1 built in, so there is no urgent need to upgrade (yet).

Linux users are likely to have an up-to-date Python install included. Otherwise, use the package manager of your system to upgrade (most Linux package managers will tell you there is an upgrade available).

Python Install For (Casual) Programmers

If you intend to hack Python scripts yourself, or learn python, a standard Python installation is not sufficient. You’ll need a Python IDE (IDE = Integrated Development Environment). Python IDEs will be subject to another article. However, Windows users should not install a standard Python distribution, but go for the ActiveState’s ActivePython distro. It includes a full python distribution with a couple of nice extras. In particular on Windows, ActivePython comes with a simple but cool IDE called PythonWin – our favorite for day-to-day scripting.

Where To Install Python On Windows

On Windows, the installation location proposed by the installer is C:\PythonXY, where XY stands for version X.Y. Example, for Python 2.6, the location C:\Python26 is proposed.

We suggest a different approach. Use C:\Program Files\Python\X.Y instead. There will be occasions when you’ll want more than one Python version installed simultaneously, but besides that, there will be some version-independent installations as well that you don’t want repeated for each Python version.

On Windows Vista and later, you may consider something like C:\Users\Public\Programs\Python\X.Y if you want unrestricted access to the Python installation directory tree.

A Word Of Caution About Python Versions

The Python community is pretty agile, and new Python versions are ahead of many useful Python libraries outside the python distribution. This is not usually a problem with “pure Python” libraries, as backward compatibility and smooth transitions between Python versions are highly valued in the Python world. We don’t recall a Python upgrade to have broken any pure Python code ever. There are mechanisms and procedures in place that give developers and library maintainers planty of time and oporunity to adjust their code. But there are “mixed code” libraries that can only work with the specific Python version they’ve been built for, and they are naturally somewhat behind the newest Python version.

It may be wise to resist the urge to install the very newest version of Python while it is fresh.

Here’s how to decide when you should upgrade:

In case you don’t know a version X.Y.Z consist of the major version number X, a minor version number Y, and the release number Z. (Some call Z a build number.) Now here’s the rule:

When an increase of the minor version number (Y) happens, so that the version looks like X.Y.0, wait until it is X.Y.1 or X.Y.2.

When an increase of the major version number (X) happens, so that the version looks like X.0.0, wait until they increse the minor version number and the release number. That is, wait until version X.1.1 or X.1.2.

This way you can save yourself from some “Oops! Why did this stop working?” experiences.

Now what are you waiting for? Go get Python, and start your pythonic journey!

This website uses a Hackadelic PlugIn, Hackadelic Sliding Notes 1.6.5.