Python Programming/Tips and Tricks

There are many tips and tricks you can learn in Python:

Strings

• Triple quotes are an easy way to define a string with both single and double quotes.
• String concatenation is expensive. Use percent formatting and str.join() for concatenation:

(but don't worry about this unless your resulting string is more than 500-1000 characters long)

print "Spam" + " eggs" + " and" + " spam"               # DON'T DO THIS
print " ".join(["Spam","eggs","and","spam"])            # Much faster/more
                                                        # common Python idiom
print "%s %s %s %s" % ("Spam", "eggs", "and", "spam")   # Also a pythonic way of
                                                        # doing it - very fast

Optimized C modules

Several modules have optimized versions written in C, which provide an almost-identical interface and are frequently much faster or more memory-efficient than the pure Python implementations. Module behavior generally does differ in some respects, often minor, and thus C versions are frequently used.

This is primarily a Python 2.x feature, which has been largely removed in Python 3, with modules automatically using optimized implementations if available. However, the cProfile/profile pair still exists (as of Python 3.4).

importing

The C version of a module named module or Module is called cModule, and frequently imported using import...as to strip off the prefix, as:

import cPickle as pickle

For compatibility, one can try to import the C version and fall back to the Python version if the C version is not available; in this case using import...as is required, so the code does not depend on which module was imported:

try:
  import cPickle as pickle
except ImportError:
  import pickle

Examples

Notable examples include:

• (Python 2.x) cPickle for pickle, up to 1000× faster.
• (Python 2.x) cStringIO for StringIO, replaced by io.StringIO in Python 3
• cProfile for profile – the Python profile adds significant overhead, and thus cProfile is recommended for most use.
• (not needed in Python 3.3+) cElementTree for ElementTree, 15–20 times faster and uses 2–5 times less memory; not needed in Python 3.3+, which automatically uses a fast implementation if possible.

List comprehension and generators

• List comprehension and generator expressions are very useful for working with small, compact loops. Additionally, it is faster than a normal for-loop.

directory = os.listdir(os.getcwd())       # Gets a list of files in the
                                          # directory the program runs from
filesInDir = [item for item in directory] # Normal For Loop rules apply, you
                                          # can add "if condition" to make a
                                          # more narrow search.

• List comprehension and generator expression can be used to work with two (or more) lists with zip or itertools.izip

[a - b for (a,b) in zip((1,2,3), (1,2,3))]  # will return [0, 0, 0]

Data type choice

Choosing the correct data type can be critical to the performance of an application. For example, say you have 2 lists:

list1 = [{'a': 1, 'b': 2}, {'c': 3, 'd': 4}, {'e': 5, 'f': 6}]
list2 = [{'e': 5, 'f': 6}, {'g': 7, 'h': 8}, {'i': 9, 'j': 10}]

and you want to find the entries common to both lists. You could iterate over one list, checking for common items in the other:

common = []
for entry in list1:
    if entry in list2:
        common.append(entry)

For such small lists, this will work fine, but for larger lists, for example if each contains thousands of entries, the following will be more efficient, and produces the same result:

set1 = set([tuple(entry.items()) for entry in list1])
set2 = set([tuple(entry.items()) for entry in list2])
common = set1.intersection(set2)
common = [dict(entry) for entry in common]

Sets are optimized for speed in such functions. Dictionaries themselves cannot be used as members of a set as they are mutable, but tuples can. If one needs to do set operations on a list of dictionaries, one can convert the items to tuples and the list to a set, perform the operation, then convert back. This is often much faster than trying to replicate set operations using string functions.

Other

• Decorators can be used for handling common concerns like logging, db access, etc.
• While Python has no built-in function to flatten a list you can use a recursive function to do the job quickly.

def flatten(seq, list = None):
    """flatten(seq, list = None) -> list

    Return a flat version of the iterator `seq` appended to `list`
    """
    if list == None:
        list = []
    try:                          # Can `seq` be iterated over?
        for item in seq:          # If so then iterate over `seq`
            flatten(item, list)      # and make the same check on each item.
    except TypeError:             # If seq isn't iterable
        list.append(seq)             # append it to the new list.
    return list

• To stop a Python script from closing right after you launch one independently, add this code:

print 'Hit Enter to exit'
raw_input()

• Python already has a GUI built in: Tkinter, based on Tcl's Tk. More are available, such as PyQt4, pygtk3, and wxPython.

• Ternary Operators:

[on_true] if [expression] else [on_false]

x, y = 50, 25

small = x if x < y else y

• Booleans as indexes:

b = 1==1
name = "I am %s" % ["John","Doe"][b]
#returns I am Doe