Python

Python
Python (programming language) - Wikipedia - http://en.wikipedia.org/wiki/Python_(programming_language)
 * Python is an interpreted, general-purpose high-level programming language whose design philosophy emphasizes code readability. Python aims to combine "remarkable power with very clear syntax", and its standard library is large and comprehensive. Its use of indentation for block delimiters is unique among popular programming languages.
 * Python supports multiple programming paradigms, primarily but not limited to object-oriented, imperative and, to a lesser extent, functional programming styles. It features a fully dynamic type system and automatic memory management, similar to that of Scheme, Ruby, Perl, and Tcl. Like other dynamic languages, Python is often used as a scripting language, but is also used in a wide range of non-scripting contexts.
 * The reference implementation of Python (CPython) is free and open source software and has a community-based development model, as do all or nearly all of its alternative implementations. CPython is managed by the non-profit Python Software Foundation.
 * Python interpreters are available for many operating systems, and Python programs can be packaged into stand-alone executable code for many systems using various tools.

Author:
 * "Python was conceived in the late 1980s and its implementation was started in December 1989 by Guido van Rossum at CWI in the Netherlands as a successor to the ABC programming language (itself inspired by SETL) capable of exception handling and interfacing with the Amoeba operating system. Van Rossum is Python's principal author, and his continuing central role in deciding the direction of Python is reflected in the title given to him by the Python community, Benevolent Dictator for Life (BDFL)."

Name:
 * "The language is named after the BBC show “Monty Python’s Flying Circus” and has nothing to do with reptiles. Making references to Monty Python skits in documentation is not only allowed, it is encouraged!"

We as people are "Pythonites"

The Zen of Python
"The core philosophy of the language is summarized by the document 'PEP 20 (The Zen of Python)'"
 * PEP 20 -- The Zen of Python - http://www.python.org/dev/peps/pep-0020/

Abstract

Long time Pythoneer Tim Peters succinctly channels the BDFL's   guiding principles for Python's design into 20 aphorisms, only 19 of which have been written down.

The Zen of Python

Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those!

Easter Egg

>>> import this

Other Sources:
 * http://www.python.org/doc/humor/
 * The Zen of Python – as a Poster « CODE POETRY - http://codepoetry.wordpress.com/2010/03/23/the-zen-of-python-as-a-poster/
 * Zen of Python (PDF) http://codepoetry.files.wordpress.com/2010/03/zen-of-python.pdf

Documentation

 * Overview — Python v2.7.1 documentation - http://docs.python.org/index.html
 * Download — Python v2.7.1 documentation - http://docs.python.org/download.html


 * Overview — Python v3.2 documentation - http://docs.python.org/py3k/
 * Download — Python v3.2 documentation - http://docs.python.org/py3k/download.html


 * PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/

Tutorials

 * The Python Tutorial — Python v2.7.1 documentation - http://docs.python.org/tutorial/


 * Python 101 -- Introduction to Python - http://www.rexx.com/~dkuhlman/python_101/python_101.html


 * 4. More Control Flow Tools — Python v2.7.1 documentation - http://docs.python.org/tutorial/controlflow.html

Linux Installation
RHEL Distribution installation: yum -y install python python-setuptools

Ubuntu Distribution installation: sudo apt-get install mercurial
 * 1) enable 'universe' in the sources.list
 * 2) "Mercurial is included in the Universe repository - activate this repo first"

CentOS 5 Option Recommended Dependencies: yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel


 * 1) this leaves: 2.6.6: _bsddb, bsddb185, dl, imageop, sunaudiodev
 * 2) this leaves: 2.7.1: _bsddb, bsddb185, dl, imageop, sunaudiodev
 * 3) this leaves: 3.2: none

Ubuntu 12.04 LTS 64bit dependencies: apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev


 * 1) maybe: lib64readline-dev libsqlite3-dev tk-dev


 * 1) this leaves: 2.7: _bsddb, bsddb185, dl, imageop, sunaudiodev

2.x
CentOS dependencies: yum -y install gcc make sudo readline-devel openssl-devel ncurses-devel zlib-devel gdbm-devel sqlite-devel bzip2-devel tk-devel tcl-devel

Ubuntu 12.04 LTS 64bit dependencies: apt-get install gcc make sudo lib64bz2-dev tcl-dev lib64ncurses5-dev libssl-dev libreadline-dev tk-dev libsqlite3-dev libgdbm-dev

2.7.15: PYVER=2.7.15 mkdir -p ~/.src ; cd ~/.src wget --no-check-certificate http://www.python.org/ftp/python/$PYVER/Python-$PYVER.tgz tar -zvxf Python-$PYVER.tgz cd Python-$PYVER ./configure --prefix=/opt/python-$PYVER && make clean && make sudo make install
 * 1) PYVER=2.6.6
 * 2) PYVER=2.7.5
 * 3) PYVER=2.7.6
 * 4) PYVER=2.7.9
 * 5) PYVER=2.7.10

sudo /opt/python-$PYVER/bin/python -m ensurepip --upgrade sudo /opt/python-$PYVER/bin/pip install --upgrade pip sudo /opt/python-$PYVER/bin/pip install requests

export PATH=/opt/python-$PYVER/bin:$PATH echo -e "\n\nexport PATH=/opt/python-$PYVER/bin:\$PATH\n" >> ~/.bash_profile
 * 1) echo -e "\n\nexport PATH=/opt/python27/bin:\$PATH\n" >> ~/.bashrc


 * 1) NOTE: remove any old versions from your .bash_profile

Bootstrapping the pip installer - https://docs.python.org/2/library/ensurepip.html

3.x
3.7.0: VER=3.7.0 mkdir -p ~/.src ; cd ~/.src wget http://www.python.org/ftp/python/${VER}/Python-${VER}.tgz tar -zvxf Python-${VER}.tgz cd Python-${VER} ./configure --prefix=/opt/python${VER} make sudo make install

export PATH=/opt/python${VER}/bin:$PATH echo -e "\n\nexport PATH=/opt/python${VER}/bin:\$PATH\n" >> ~/.bash_profile

sudo apt install ... yum install libffi-devel openssl-devel
 * 1) DEPENDENCIES

---

3.2.0: mkdir -p ~/src ; cd ~/src wget http://www.python.org/ftp/python/3.2/Python-3.2.tgz tar -zvxf Python-3.2.tgz cd Python-3.2 ./configure --prefix=/opt/python32 make sudo make install

export PATH=/opt/python32/bin:$PATH echo -e "\n\nexport PATH=/opt/python32/bin:\$PATH\n" >> ~/.bash_profile

3.3.0: mkdir -p ~/src ; cd ~/src wget http://www.python.org/ftp/python/3.3.0/Python-3.3.0.tgz tar -zvxf Python-3.3.0.tgz cd Python-3.3.0 ./configure --prefix=/opt/python3.3.0 make sudo make install

export PATH=/opt/python3.3.0/bin:$PATH echo -e "\n\nexport PATH=/opt/python3.3.0/bin:\$PATH\n" >> ~/.bash_profile

3.3.4: mkdir -p ~/.src ; cd ~/.src wget http://www.python.org/ftp/python/3.3.4/Python-3.3.4.tgz tar -zvxf Python-3.3.4.tgz cd Python-3.3.4 ./configure --prefix=/opt/python3.3.4 make clean make sudo make install

export PATH=/opt/python3.3.4/bin:$PATH echo -e "\n\nexport PATH=/opt/python3.3.4/bin:\$PATH\n" >> ~/.bash_profile

3.4.0: mkdir -p ~/.src ; cd ~/.src wget http://www.python.org/ftp/python/3.4.0/Python-3.4.0.tgz tar -zvxf Python-3.4.0.tgz cd Python-3.4.0 ./configure --prefix=/opt/python3.4.0 make sudo make install

export PATH=/opt/python3.4.0/bin:$PATH echo -e "\n\nexport PATH=/opt/python3.4.0/bin:\$PATH\n" >> ~/.bash_profile

setuptools
Purpose: to install pip

Note: required for several 'python setup.py install' installations.

Python Package Index : setuptools 0.6c11 - http://pypi.python.org/pypi/setuptools

By Package Manager: yum install python-setuptools

By Source: VER=3.4.4 mkdir -p ~/.src ; cd ~/.src wget --no-check-certificate http://pypi.python.org/packages/source/s/setuptools/setuptools-$VER.tar.gz tar -zvxf setuptools-$VER.tar.gz cd setuptools-$VER sudo /opt/python-2.7.9/bin/python setup.py install
 * 1) VER=2.0.1
 * 2) VER=2.2
 * 1) sudo python setup.py install

Fix header: sed -i 's#!/usr/bin/python#!/usr/bin/env python#g' /opt/python279/bin/easy_install
 * 1) Only needed on older versions

Install pip: sudo /opt/python-2.7.9/bin/easy_install pip

Sample pip usage: sudo /opt/python-2.7.9/bin/pip install paramiko

IDLE and Interactive Mode
Interactive Mode - "When commands are read from a tty, the interpreter is said to be in interactive mode. In this mode it prompts for the next command with the primary prompt, usually three greater-than signs (>>>); for continuation lines it prompts with the secondary prompt, by default three dots (...)." 

Can use '_' to reference last output: >>> 5 + 6 11 >>> print _ 11

References:
 * 2. Using the Python Interpreter — Python v2.7.3 documentation - http://docs.python.org/2/tutorial/interpreter.html

Python 2.7 EOL
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

Hello World
shebang (also called a hashbang) of Python: print "Hello World"
 * 1) !/usr/bin/env python

NOTE: can also just use "#!/usr/bin/python"

White Space
Python uses white space to indicate code blocks. Instead of { use a : and indent consistently (4 spaces).

if a == b:    print 'equal'

Convert tabs to spaces: sed -i 's/\t/   /g' *.py

Remove trailing whitespace: sed -i 's/\s*$//' *.py sed --in-place 's/space:\+$//' fileName

Clean project: find. -name "*py" -exec dos2unix {} \; -exec sed -i 's/\t/   /g' {} \; -exec sed -i 's/\s*$//' {} \;
 * 1) find . -name "*py" -exec dos2unix {} \;
 * 2) find . -name "*py" -exec sed -i 's/\t/    /g' {} \;
 * 3) find . -name "*py" -exec sed -i 's/\s*$//' {} \;

Print
print - http://docs.python.org/tutorial/inputoutput.html

Print: # include new line print("Hello World") print "Hello World"  # python 2 only

Suppress new line: print('#', end='')   # python 3 print "Hello World", # python 2 only (does produce a single separating white space though)

import sys sys.stdout.write('.') # no separating white space

import sys sys.stdout.flush
 * 1) flush output
 * 2) optional, if new line won't come for some time

Multiple with separator: print("hello", "world") # separated by spaces print("hello", "world", sep = '.') # separated by period

Formatting: mystr = "abc" myint = 5 myflt = 3.141592 print( "%s %d %.3f" % (mystr, myint, myflt) ) print( "{0} {1} {2}".format(mystr, myint, myflt) ) # string's format function (python > 2.7) print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding

Padded floating point decimal number: (rounds) print "%06.2f" % 2.319        # "002.32" print "%6.2f" % 2.311         # "  2.31" print "{:06.2f}".format(2.319) # "002.32" print "{:6.2f}".format(2.319) # "  2.32"

Comments
x = y # this is a comment
 * 1) this is a comment
 * 2) this is a comment

Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended) """ multi line comment """

 multi line comment can comment out """ 

This can be used like a C "#if 0" concept to comment out whole sections of code.

Using false block to comment out a block code: (does not ignore bad syntax though) if False: # code in this block is ignored

doc strings
Using a non assigned string constant as a comment: (also used for doc strings, not generally recommended) """ multi line comment """

Help
dir - list functions and properties of objects: import math dir(math) # module dir(dir)  # function dir("")   # object  - "", [], {}, etc... dir(__builtins__) # built in functions dir     # list imported objects, and dunder methods

Note: dunder (double under) methods determine special functionality (eg. __add__ for +)

help - manual printout: (class doc string, function doc string, data, etc) import math help(math)   # full module info help(len)    # full function info help(["foo"]) # full object info

Pydoc: (command line, same as 'import math, help(math)' $ pydoc math

Documentation String: (aka docstring, doc string) print(math.pow.__doc__) # module doc string print(len.__doc__)      # function doc string

Object or function type: type(x)

mylist = [] isinstance(mylist, list) # True type(s) is list          # True

Note: isinstance and type not equivalent when objects and subclass get involved

IDLE interactive help: help

Variables
Basic Data Types str = "string" # string num = 5        # integer flt = 5.0      # float nothing = None # None  (not the same as undefined)

Undefined. Variables start off in an 'undefined' state. It is always better to initially set the variable to None to avoid this state. To check if undefined: try: x except NameError: x = None

Rules: PEP8 - Method Names and Instance Variables
 * any length
 * letters, numbers, underscore
 * first character can't be number
 * case sensitive
 * cannot use keywords

Multiple assignment: x = y = z = 0             # Zero x, y and z x, y, z = 1, "two", 3.0    # x = 1, y = "two", z = 3.0

Swap variables: (done in parallel) a, b = b, a

Unpack variables: (requires equal items on each side) a, b, c = [1, 2, 3]

Other Data Types:
 * tuple
 * list
 * dictionary
 * set
 * objects

Everything in Python is an object that has:
 * identity: id(obj)
 * type: type(obj)
 * value: obj
 * mutable - id stays the same on modification (dictionary, lists, objects)
 * immutable - id changes on modification as new object is created (string, integer, tuple)

Check if Variable Exists
try: a # does a exist in the current namespace except NameError: a = 10 # nope

'a' in vars or 'a' in globals 'a' in vars(__builtins__)

if hasattr(a, 'property'): doStuff(a.property) else: otherStuff

Ref:

Numbers
num = 5 print "num: %d" % num

>>> print "Today's stock price: %f" % 50.4625  1 50.462500 >>> print "Today's stock price: %.2f" % 50.4625 2 50.46 >>> print "Change since yesterday: %+.2f" % 1.5 3 +1.50

Arithmetic: + - * / // # integer division %  # modulus ** # exponent

Compact Arithmetic: i+=1 # Increment - Note, there is no 'i++' i*=2 # Multiply i%=2 # modulus ...

Math Functions: import math math.ceil(x) math.floor(x) math.trunc(x) math.pow(x,y)

abs(x) # built in

String interpolation: %% % character %d integer %x hex %X hex (upper) %f float %s string

x = 1/81 print('value: %.2f' % x) # 0.01 print('value: %.5f' % x) # 0.012345 print('%d %d %d' % (1, 2, 3))

Type check: if not isinstance(myvar, (int, float)): ...

Strings
STRINGS ARE IMMUTABLE! (to modify convert to a list)

Strings: (String Constants) str = "Hello" str = 'Hello' str = """  Hello   World """      # can also use ...

String literals can span multiple lines: (no wrapping when \ used) hello = "hello\ world"

Length: len("str")

Raw string: (back slashes are not escaped) myrawstr = r"hello\nworld\ how are you" myrawstr = r"C:\" # ERROR - limitation of what won't be escaped

Concatenation: str = 'hot' + 'dog' print "Hello" "World"

Can't concat numbers str = 5 + 'test' # error num = 5 str = num + 'test' # error str = str(5) + str(num) + 'test' # ok

Repeated Concatenation: str = 10 * 'ha' str = 'ha' * 10 # same print '-'*60 #

String Indexing: s = 'hello' print( s[0] )   # is 'h' print( s[-1] )   # is 'o' print( s[len(s) - 1 ] )  # is also 'o'

Modify Strings: s[1] = 'x' # ERROR! ERROR: can't modify string by index s = list("Hello World") s[0] = 'C' print( "".join(s) )
 * 1) solution:

String Slicing: s = 'hello' print( s[0:3] ) # 'hel' i = 1 print( s[i:i+1] ) # 'e' print( s[:3] )  # 0:3 'hel' print( s[4:] ) # 4:end 'o' print( s[:] )  # start:end 'hello' print( s[-2:] ) # -2:end 'lo'

For loop: s = 'hello' for c in s:  print c  # c = 'h' first time

Escape characters: \\ \'  \"  \n  \r  \t

Alternate loop: s = 'hello' for i in range( len(s) ): print s[i]

Ord and Char (ASCII): ord('a') # 97 chr(97)  # 'a'

0-9 : 48-57 A-Z : 65-90 a-z : 97-122

List Functions: (dir on any string) dir('') dir(str)

Help: help('') help(str)

Padding: "a".rjust(2, '0') # '0a' "1".ljust(4, '0') # '1000'

String Test Functions: "HELLO".isupper  # True "hello".islower  # True "hello".isalpha  # True "12.22".isdigit  # False (because of '.') "12.22".isnumeric # False (because of '.') "hel2".isalnum   # True "e" in "hello"     # True "hello".startswith('he') # True "hello".endswith('lo')   # True

Note: To check for floats: def is_number(s): try: float(s) return True except ValueError: return False

String Modify Functions: (sort of... Strings are immutable) "HELLO".lower          # 'hello' "hello".upper          # 'HELLO' " hello ".strip        # 'hello' remove leading/tailing whitespace " hello ".lstrip       # 'hello ' remove left whitespace " hello ".rstrip       # ' hello' remove right whitespace "www.com".split('.')     # ['www', 'com']   <- list "www.com".partition('.') # ('www', '.', 'com')  <- tuple "".join( ('a','b','c') ) # "abc"  join strings, list or tuple "-".join( ('a','b','c') ) # "a-b-c" join with separator "23".zfill(4)            # "0023"

String Search Functions: "hello".find('lo')  # 3  (-1 on fail) "hello".index('lo') # 3 (exception on fail) "hello".rfind('lo') # 3 right to left (-1 on fail) "hello".rindex('lo') # 3 right to left (exception on fail)

Formatting: (form of concatenation) mystr = "abc" myint = 5 myflt = 3.141592 print( "%s %d %.3f" % (mystr, myint, myflt) ) print( "{0} {1} {2}".format(mystr, myint, myflt) ) # pep 3101 style - string's format function (python > 2.7) print( "{a} {b} {c}".format( b = myint, a = mystr, c = myflt) ) # by name print( "={a:4s}={b:3d}={c:10.3f}=".format( b = myint, a = mystr, c = myflt) ) # type and padding

Split multi line string: for line in str.split('\n'): print(line) lines = str.splitlines     # returns list lines = str.splitlines(True) # returns list (each includes new line characters)

Input a string: name = input('What is your name: ').strip # python 3 name = raw_input('What is your name: ').strip # python 2

Leet Speak: leet_table = ''.maketrans('EIOBT', '31087') 'BE COOL. SPEAK LEET!'.translate(leet_table)

String Functions: newstr = mystr.strip            # remove white space on both sides newstr = mystr.rstrip           # remove white space on right side newstr = mystr.lstrip           # remove white space on left side newstr = mystr.replace('\n', ' ') # replace string with another newstr = mystr.replace('jello', 'hello') # replace string with another

String Formatting
PyFormat: Using % and .format for great good! - https://pyformat.info/

Zero Padding
3 zero padding: "{:03}".format(1) # 003

3 zero padding, specify position: "{1:03}".format(1, 2) # 002

Old method: "%03d" % 2 # 002

chr and ord
Ord and Char (ASCII): ord('a') # 97 chr(97)  # 'a'

Range
Creates a list of numbers. Remember: range always ends one less than you want!

Count: ten = 10 one = 1 range(ten)              # [0,1,...,9]   starts with zero by default range(one, ten + 1)     # [1,...,10]    actually get 1 to 10 range(ten, one - 1, -1) # [10,9,...,1]  count down range(2, 10 + 1, 2)     # [2,4,...,10]  even numbers

drange
Solution for decimal range step value: def drange(start, stop, step): r = start while r < stop: yield r   r += step

sub_ten = drange(0.0, 1.0, 0.1)

References:
 * floating point - Python decimal range step value - Stack Overflow - http://stackoverflow.com/questions/477486/python-decimal-range-step-value

Casting
Convert: float(3) str(85) int(8.6) round(5.5)

Check Type: type(5)    #  type(5.0)  #  type('5')  #  type(None) #  type(print) #  type()   #  type([])   # 

mytype = int if type(1) is mytype: ...

Boolean Logic
True and False: bol = True bol = False

Boolean Operators: (in order precedence) a == b a != b not a a and b a or b

Note: Short-circuit evaluated - 'and' and 'or'

Return Boolean: return myint >= 16

Get Boolean from string:
 * 1) Note: no built in way to do this

myString = "false" val = (myString == "true")

True if myString=="True" else False

s in ['true', '1', 't', 'y', 'yes', 'yeah', 'yup', 'certainly', 'uh-huh']

def str2bool(v): return v.lower in ("yes", "true", "t", "1")

None
None is Python's version of NULL or NIL.

None is a singleton (Python only has one copy of None in the interpreter).

if myvar is None: ...

If Conditionals
Operators: ==, !=, not, and, or, <, >, <>, <=, >=, in, not in, is, is not

If statement: if a == b: ... elif c < age < d: ... else: ...

Shorthand conditional: (ternary operator) val = True if a == b else False

is - checks for identical objects - same id if myvar is None: ...
 * 1) None is a singleton, so this works great for None

Other shorthand: def mycmp(x, y): if x > y: return True else: return False

Switch/Case statement:
 * there is no built in switch/case statement, use "if ... elif ... elif ..." sequence instead

For Loop
for is used for iterating over an iterable sequence (eg. lists, dictionaries, tuples). (for each)


 * 1) range(start, end + 1, skip)

for i in range(10): print(i) # 0 .. 9

for i in range(5, 10): print(i) # 5 .. 9

for i in range(10, 0, -1): print(i) # 10 .. 1

for i in [1, 2, 3]: # iterate over list print(i) # 1 2 3

Discard variable: for _ in range(10): # repeat and discard variable (good for pep8 check) print "hello world"

for key in my_dict.keys: ... for value in my_dict.values: ... for key, value in my_dict.items: ...

Compact version: for i in range(10): print(i)   # 0 .. 9

Access index: for idx, val in enumerate(mylist): print idx, val
 * 1) using enumeration

for ix in range(len(mylist)): print mylist[ix]
 * 1) using range

[ (ix, mylist[ix]) for ix in range(len(mylist))]
 * 1) list comprehension

While Loop
i = 0 while i < 10: i = i + 1

while True: ... # infinite loop

Breaking Loops
break statement: while True: if s == 'done': break # jump out of the loop ...

continue statement: (works with while too) for i in range(10): if i == 2: continue # skip 2 ...

Functions
No return value: (None) x = print('')
 * 1) x is 'None'

Note: specifying no return value is the same as this: return None

No Operation - 'pass': def noop: pass

Function: def area(radius): """ This is a doc string """  # can be " ... " also... import math return math.pi * radius ** 2

Function Doc string: print(area.__doc__) foo.func_doc

Function name: foo.func_name

Local variable: name = "old" def cname: name = "new" print(name) # is still 'old'

Global variable: name = "old" def cname: global name name = "new" cname print(name) # is now 'new'

By convention starting point is: (not required) def main:

Parameters: (with default) def foo(x, y, z = "default"):

Note: don't use mutable for defaults, as the object is remembered across runs! (unless that is what you want) def t(foo=[]): foo.append('1') print(foo)
 * 1) each call to t will increase list

def named_param(a, foo=None): foo = foo or [] if not foo: foo.append(a)
 * 1) fix mutable issue

Parameters by keyword: def foo(x = '1', y = '2', z = '3'): ... foo(y = 'a', x = 'b')  # order is not important with keywords

Trick for optional parameter: def do(x = None): if x == None:

Function tests - functions are callable callable(myfunc)

Pass
Pass is a null operation (noop):

Do nothing, noop, take no action: pass

Great for creating minimal classes: class MyEmptyClass: pass

Greate for making stub functions/methods: def initlog(*args): pass # remember to implement this!

Slice
Take take a slice of strings, lists, tuples, etc... a = [0,1,2,3,4,5] a[0]   # 0 - not a slice a[-1]  # 5 - not a slice a[0:1] # [0] a[1:3] # [1, 2] a[:2]  # [0, 1] a[:-1] # [0, 1, 2, 3, 4] - all but last a[2:]  # [2, 3, 4, 5] a[::2] # [0, 2, 4] - stride a[::-1] # [5, 4, 3, 2, 1, 0] - reverse range(0,10)[::2] # [0, 2, 4, 6, 8] - stride

a = '012345' a[1:3] # '12' a[::2] # '024'

Compound Statements
Compound statements: (generally discouraged) fun1; fun2; fun3

Compound with statements: with A as a, B as b: ...

with A as a:    with B as b:         ...
 * 1) is equivalent to

PEP 0343 - The “with” statement

f = open('file') with f: ...

Try: (quick ignore all exceptions, very dangerous) try: my_broken_function except: pass

Passing Command Line Arguments
Parameters are stored in the 'sys.argv' variable: import sys print( sys.argv[0] ) # name of script print( sys.argv[1] ) # first argument for arg in sys.argv: print( arg ) # list all arguments

Print script name: os.path.basename(sys.argv[0])

Print path to script: print sys.path[0]

Paths: import os,sys print "CWD: ",os.getcwd print "Script: ",sys.argv[0] print ".EXE: ",os.path.dirname(sys.executable) print "Script dir: ", os.path.realpath(os.path.dirname(sys.argv[0])) pathname, scriptname = os.path.split(sys.argv[0]) print "Relative script dir: ",pathname print "Script dir: ", os.path.abspath(pathname)

os.path.realpath(__file__)

print 'sys.argv[0] =', sys.argv[0] pathname = os.path.dirname(sys.argv[0]) print 'path =', pathname print 'full path =', os.path.abspath(pathname)

Environment Variables
import os a = os.environ.get('MYVAR')              # get environment var a = os.environ.get('MYVAR','default')    # get environment var a = os.environ['MYVAR']                  # get environment var os.environ['MYVAR'] = 'something'        # set environment var os.environ.update({'MYVAR':'something'}) # set environment var

for env in os.environ: print "%s = %s" % (env, os.environ[env])

with open("environment.txt", "w") as f:  for env in os.environ: f.write("%s = %s\n" % (env, os.environ[env]))

PYTHONPATH
Environment Search path: PYTHONPATH=...

From Code: sys.path.append('...')

Get site-package directory from shell: python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib)"

Import and Modules
Import and Modules - http://docs.python.org/tutorial/modules.html

Import module: import package           # standard import import package.code      # nested name space from package import code # import specifics from module import module            # standard import from math import sin     # import specifics from module from module import myfunc as fun1 # alias function/object import math as other_math # alias module import longname as ln    # alias module from module import *     # BAD BAD BAD!

import math math.pow(x,y)

Group and organize by:
 * stdlib libraries
 * local libraries
 * 3rd party libraries

Remove import: del math

from math import * pow(x,y)

from math import pow pow(x,y)

import math as math_lib  # rename math_lib.pow(x,y)

List functions in module: import math dir(math)

List all built in functions: dir(__builtins__)

Note: Modules have the extension .py

Simply module Example:

def hello: """ this is a doc string """ print "Hello World"
 * 1) mymod.py:

import mymod mymod.hello:
 * 1) Use the module:

from mymod import * hello
 * 1) Use everything in the the module, except for names beginning with '_':  (bad practice!)

from mymod import hello, hi hello
 * 1) Use specific functions in the module:

h = mymod.hello h
 * 1) Function assignment:

mymod.__name__  # mymod __name__        # used inside of module will print current module or "__main__" if directly called
 * 1) module name

Note: careful when building own modules, as the import literally imports and executes the code, so any code not in functions is executed!

Check if being called directly or indirectly: if __name__ == "__main__": # execute tests...

Standard Script Setup: def main(args): # do logic if __name__ == "__main__"; sys.exit(main(sys.argv))

Easter egg: import this
 * 1) The Zen of Python

Search path:
 * environment variable 'PYTHONPATH'  (on Unix, this is usually .:/usr/local/lib/python)
 * sys.path (list of paths, initialized to PYTHONPATH)
 * sys.path.append('path')

Module folder: /kenlib/kenmod.py
 * 1) so you can do "import kenlib.kenmod"
 * 2) create file '__init__.py' in kenlib/

Check if package exists: try: import argparse except ImportError: print >> sys.stderr, "Please install argparse from http://code.google.com/p/ argparse." exit(1)

ifmain
if __name__ == '__main__': sys.exit(main(sys.argv))

if __name__ == '__main__': sys.exit(main(sys.argv[1:]) or 0)

Lists

 * Lists can be modified (mutable)
 * Tuples are constant (immutable)

List: (mutable or modifiable) x = [] # empty list x = [5] # singleton list

s = list("Hello World") print(s) # s = ['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd'] s[0] = 'X'

s.append(x)    # append x s.insert(i, x)  # insert x at i del(s[i])       # delete item at i s.sort        # sort (modify in place!) s.reverse    # reverse s.index(x)     # index of x in s new = old[:]    # create copy of list new = list(old) # create copy of list

Check if item in list: if x in s: ...

Filter list of strings based on contents: res = [k for k in lst if 'ab' in k]

List comprehension [n * n for n in range(1, 11)]  # squares of numbers 1 to 10 [c for c in 'pizza']           # list of string 'pizza' [c.upper for c in 'pizza']   # list of string 'pizza' uppercase [n for n in nums if n > 0]     # filters out only positive numbers in nums list [fname for fname in os.listdir(path) # on multiple lines if os.path.isfile(fname) if fname.endswith('.py')]

Enumeration for index, value in enumerate(mylist): print index, value

Length: len(x)

Concatenation: x + s x * 2

Convert list to string: "".join(mylist) # no separator " ".join(mylist) # space separator ",".join(mylist) # comma separator

Test Membership x in s

Multi dimensional list: mylist = [ [0,"hi"], [1,"bye"] ] print( mylist[0][1] )

Sorting: (see lambda) data.sort # alpha numerica sort data.reverse # alpha numerica sort

data = [1, 5, 3, 9] def mylistcmp(x, y): if x == y: return 0 if x < y:   return -1 else: # swap 1 and -1 here to reverse return 1
 * 1) custom sort

data.sort(cmp=mylistcmp)

Slices: lst = ['1a', '2b', '3c', '4d', '5e', '6f'] lst[0]    # '1a' lst[-1]   # '6f' last item lst[0:2]  # ['1a', '2b'] lst[0:2] = ['11', '22']  # lst = ['11', '22', '3c', '4d', '5e', '6f'] - assignment to slice lst[0:2] = []  # remove items lst[0:0] = ['1a', '2b']  # insert items (at beginning) lst[-1:-1] = ['7g', '8h']  # insert items (at item BEFORE last), not really what you want, use append lst[-1:] = ['7g', '8h']  # replace last item lst[:] = []  # clear entire list del(lst[0:2]) # delete slice items

Note: it is not safe to modify a sequence while being iterated over. Create a copy first: for x in mylst[:]: # make a slice copy of the entire list if len(x) > 6: a.insert(0,x)

Make list from tuple: list( range(10) )

Merge multiple lists: def merge(seq): merged = [] for s in seq: for x in s:           merged.append(x) return merged

foo = 'a', 'b'],['c'],['d', 'e', 'f' print merge(foo) >>>['a', 'b', 'c', 'd', 'e', 'f']
 * 1) sample usage:

Tuple
Tuple: (immutable or constant) x =    # empty tuple x = (5,) # singleton tuple x = (5)  # NOTE: integer with parenthesis, not tuple s = tuple("Hello World") print(s) # s = ('H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd')
 * 1) ERROR: s[0] = 'X'

Make list from tuple: list( range(10) )

Use in conditionals: if answer in ('y', 'ye', 'yes'): ...

Convert tuple string to tuple: x = "(1,2,3)" t = tuple(int(v) for v in re.findall("[0-9]+", x))

Another Convert tuple string to tuple: import ast ast.literal_eval("(1,2,3,4)") # (1,2,3,4)

Named Tuple
What are "named tuples" in Python? - Stack Overflow - http://stackoverflow.com/questions/2970608/what-are-named-tuples-in-python

from collections import namedtuple Point = namedtuple('Point', 'x y') pt1 = Point(1.0, 5.0) pt2 = Point(2.5, 1.5) print p1    # point(x=1, y=5) print p1.x  # 1.0 print p1[0] # 1.0

Dictionary
NOTE: Dictionaries are also known as associative arrays, maps or hash tables.

Dictionary: (mutable or modifiable)

color = {}                     # empty dictionary color = {'red' : 1, 'blue', 2} print( color['red'] )          # 1 print( color.get('red') )      # 1 print( color.get('red', 'default') )      # red if found, or return 'default' color['green'] = 3             # able to add new item on the fly! 'red' in color                 # True - check if index exists del(color["red"])              # delete item len(color)                     # 2 - count of items

List in dictionary d.setdefault(key,[]).append(member) # tricky!

from from collections import defaultdict d = defaultdict(list) d[key].append(member)
 * 1) more tricky: set default type as list

Enumeration: for index, key in enumerate(mylist): print index, key

functions: d.clear d.copy d.get(key) d.get(key, default_val) d.update(e)  # update with (key, value) pairs in e d.setdefault(key, value)  # only update if not exist, and return set value

Views: (adjust on the fly as dictionary changes) d.items (key, values) for k, v in d.items: print(k, v) d.keys for k in d.keys: print(k) d.values for v in d.values: print(v)

Sorting: (see also lambda) data = [dict(number=x) for x in '036149'] # create list of dictionaries def mydictcmp(x, y): if x['number'] == y['number']: return 0 if x['number'] < y['number']: return -1 else: # swap 1 and -1 here to reverse return 1
 * 1) custom sort of dictionaries in list


 * 1) data.sort(cmp=mydictcmp)  # modifies data

new_data = sorted(data, cmp=mydictcmp) # better solution

Sorting: my_dict = {'a':1, 'b':4, 'c':3, 'd':9} my_dict_keys = my_dict.keys

def my_dict_sort(x, y): if my_dict[x] == my_dict[y]: return 0 if my_dict[x] < my_dict[y]: return -1 if my_dict[x] > my_dict[y]: return 1

sorted_tuple = sorted(my_dict_keys, cmp=my_dict_sort) print "Sorted Dictionary:" for key in sorted_tuple: print "%s: %d" % (key, my_dict[key])
 * 1) sorted_my_dict = sorted(my_dict_keys, cmp=my_dict_sort, reverse=True)


 * a: 1, c: 3, b: 4, d: 9,
 * 1) reverse: d: 9, b: 4, c: 3, a: 1,

Sort and keys: keys = sorted(mydict.keys)

Sort by value: import operator x = {1: 2, 3: 4, 4:3, 2:1, 0:0} sorted_x = sorted(x.iteritems, key=operator.itemgetter(1))

Sort options:
 * 1) LIST.sort(cmp=None, key=None, reverse=False) -- stable sort *IN PLACE*;
 * 2) sorted(iterable, cmp=None, key=None, reverse=False)

Another: (works well!) sorted(my_dict.items, key=lambda x: x[1])

See: http://wiki.python.org/moin/HowTo/Sorting/

Sets
Sets are special lists with unique values (good for removing duplicates) lst = [1, 1, 6, 8, 1, 5] s = set(lst) print(s) # set([8, 1, 5, 6]) s.add(item) item in s # true s.remove(item)

Union (return combination of all): s.union(t)

Intersection (return matching): s.intersection(t)

Symmetric Difference (return only items not matching either, inverse of intersection): s.symmetric_difference(t) list( set(mylist).symmetric_difference(set(mylist2)) )

Operation		  Equivalent	Result len(s) 	 				cardinality of set s x in s 	  				test x for membership in s x not in s 	  			test x for non-membership in s s.issubset(t) 		   s <= t       test whether every element in s is in t s.issuperset(t) 	   s >= t 	test whether every element in t is in s s.union(t) 		   s | t 	new set with elements from both s and t s.intersection(t)	   s & t 	new set with elements common to s and t s.difference(t) 	   s - t	new set with elements in s but not in t s.symmetric_difference(t)  s ^ t 	new set with elements in either s or t but not both s.copy 	 			new set with a shallow copy of s

References:
 * sets — Unordered collections of unique elements - http://docs.python.org/library/sets.html

Files
Open file, get file descriptor: fd = open('/dev/null')

File Modes: 'r' read (default) 'w' write (truncate and create if needed) 'a' append (create if needed) 'b' binary 't' text (default) '+' read and write

Common Functions: (fd)

f.read      # read whole file to string f.read(n)     # read n bytes from file to string f.readline  # read single line as string (while loop) f.readlines # read all lines to list

f.write(str)  # write string to file (does not auto include new line) f.writelines(list) # write list to file (does not auto include new lines)

f.seek(0)     # seek to start of file f.seek(pos)   # seek to position f.truncate  # truncate file

f.close     # close file (fd) f.closed      # True/False flag that reports if file is closed

Help: help(open('/dev/null')) pydoc file

Read Text File Line by Line f = open(fname, 'r') # ('r' optional) for line in f: # or f.readlines print(line) f.close # optional

Write list to file (with new lines): f.writelines('\n'.join(mylist))

One liner read whole file: print( open( fname, 'r' ).read )

Write text file: f = open(fname, 'w') f.write("Hello\n") f.write( str(10) ) # convert numbers to strings first f.writelines(["line one\n", "line two\n"]) f.close
 * 1) f = open(fname, 'a')  # append text file

Read binary file: def is_gif(fname): f = open(fname, 'br') # not sure 'b' is correct first4 = tuple(f.read(4)) return first4 == (0x47, 0x49, 0x46, 0x38)

Read all characters of file: c = f.read(1) while c:  # if c == ... logic c = f.read(1)

Reading file in while loop: f=open("file") while True: line=f.readline if not line: break print line f.close

with - implicit close - PEP 343 -- The "with" Statement: (python >= 2.5) with open("/etc/passwd") as f:  # ..., f.read, etc

With statement: (promptly close open handle at end of block) with open(fname, 'r') as f:  for ...

f = open("/etc/passwd") f.__enter__ try: ... finally: f.__exit__
 * 1) eqivalent to:

cat: (for grabbing a configuration line) def cat(filename): with open(filename) as f:       return f.readline.strip

With
PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/

'with' structure: with EXPR as VAR: BLOCK

translates to: VAR = EXPR VAR.__enter__ try: BLOCK finally: VAR.__exit__

Note: The "as VAR" part is optional. Good for locking: with locking(myLock): BLOCK

with - implicit close - PEP 343 -- The "with" Statement: (python >= 2.5) with open("/etc/passwd") as f:  ...

With statement: (promptly close open handle at end of block) with open(fname, 'r') as f:  for ...

See also

References:
 * PEP 343 -- The "with" Statement - http://www.python.org/dev/peps/pep-0343/
 * PEP 340 -- Anonymous Block Statements - http://www.python.org/dev/peps/pep-0340/

Try Exception Handling
Exception Handling - http://docs.python.org/release/2.6.6/tutorial/errors.html

throw execption: raise [EXCEPTION] raise IOError("This is a test!")

Catch exceptions: try: except ( [EXCEPTION], [EXCEPTION] ): except [EXCEPTION]: except [EXCEPTION] as err: # use "[EXCEPTION], err" in python < 2.6 print("Error: {}".format(err)) except: raise  # reraise caught error that is not handled else:    # if no exception caught finally: # always executed

Note: Usually a good idea to re-raise exception if you don't handle it (just 'raise')

General Exception: (all exceptions inherit from Exception) try: raise Exception('spam', 'eggs') except Exception as e:  pass

Custom Exception Class: class MyError(Exception): """My Error Exception"""

class MyException(Exception): pass

rasie MyError("My Error")

class ValidationError(Exception): def __init__(self, message, Errors):

# Call the base class constructor with the parameters it needs Exception.__init__(self, message)

# Now for your custom code... self.Errors = Errors def __str__(self): return "MyErrors - %s" % self.Errors

Rename standard exception: class Failure(StandardError): """Exception for unexpected failures."""

Note: a good place to hide the exception classes in the __init__.py file.

exception line number
Python When I catch an exception, how do I get the type, file, and line number? - Stack Overflow import sys, os

try: raise NotImplementedError("No error") except Exception as e:   exc_type, exc_obj, exc_tb = sys.exc_info fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1] print(exc_type, fname, exc_tb.tb_lineno)

traceback
import traceback try: ...   except Exception as E:        ... print 'Traceback: %s', traceback.format_exc

Catching multiple exceptions and getting properties from the exception: import traceback import sys

class MyError(Exception): """My Error Exception"""

try: #raise ValueError("a value error") #  raise MyError("a my error")        #  raise Exception("won't be caught") #  except (ValueError, MyError) as e: print type(e)       #  print e            # __str__ = a my error msg = str(e)       # msg = 'a my error' print e.__doc__    # My Error Exception print e.__str__  # a my error print e.__repr__ # MyError('a my error',) print e.args       # ('a my error',) x = e              # __getitem__ allows args to be unpacked directly # x, y = e         # if two args were passed, use len(e.args) to see count

# manually print stack trace (import traceback, sys) print "Exception in user code:" print '-'*60 traceback.print_exc(file=sys.stdout) # stack_trace = traceback.format_exc print '-'*60

Object Oriented Programming
Terms:
 * encapsulates
 * inheritance
 * polymorphism
 * method overriding

class
class animal.py: class Animal(object): # default "(object)" not required " Animal class " def __init__(self, name): self.name = name      # name is private variable self.__iam = "animal" # __iam is private variable def __str__(self):      # string representation return "name = '%s'" % (self.name) def __repr__(self):     # object representation (just typing instance in IDLE, or repr) return 'Animal(%s)' % str(self) # returns __str__ def speak(self, msg='Nothing to say'): print(msg)

Note: if no __str__ defined, but __repr__ is, then: __str__ = __repr__

Usage: from animal import Animal a = Animal('good') print(a.__doc__) # Animal class print(a)         # name = 'good' a                # Animal(name = 'good') print(a.name)    # good a.speak('hi')    # hi print(a._Animal__iam)  # animal - access to private variable (shouldn't do)

subclass dog.py: from animal import Animal class Dog(Animal): " Dog class " def __init__(self, name): self.name = name     # name is private variable self.__iam = "dog"   # __iam is private variable # NOTICE: no __str__, inherited from super class def __repr__(self):    # override to reflect "dog" return 'Dog(%s)' % str(self) # returns __str__ def speak(self, msg='Nothing to say'):  # override method super(Dog, self).speak('Dog says: ' + msg)    # call super class method

Subclass usage: from dog import Dog d = Dog('bad') print(d.__doc__)   # Dog class print(d)           # name = 'bad' d                  # Dog(name = 'bad') print(d.name)      # bad d.speak('hi')      # Dog says: hi print(d._Dog__iam)  # dog - access to private variable (shouldn't do)

Super: super.__init__
 * 1) python 3.0

super(self.__class__, self).__init__ super([CLASSNAME], self).__init__
 * 1) python 2.7

class Rectangle(Polygon): def __init__(self, id, width, height): # super(Rectangle, self).__init__(id) super(self.__class__, self).__init__(id) self.shape = (width, height)

Get class name: self.__class__.__name__ obj.__class__.__name__

Full name: obj.__module__ + "." + obj.__class__.__name__

Private variables: (uses name mangling) self.__age # direct access: _classname__varname  (eg. p._Person__age = 44)

Tests: isinstance(myobj, myclass) # check if myobj is a class or subclass of myclass issubclass(A, B) # check if A is subclass of B issubclass(Exception, BaseException)  # True

Multiple Inheritance: (methods searched in order) class DerviedClass(Base1, Base2, Base3): pass

Standard startup script: class MyClass: ...    @classmethod def main(cls, argv=None): ...        # return cls # creator if __name__ == "__main__"; sys.exit(MyClass.main(sys.argv))

Raise an error if a method is not overridden (not implemented) def _get_dist(self): """Find linux disto. Implemented in subclasses.""" raise NotImplementedError

Multiple Constructors
class Animal(object):

def __init__(self, name): self.name = name

@classmethod def create_fido(cls): return cls("fido")

Descriptor Classes
"In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol: __get__, __set__, and __delete__. If any of those methods are defined for an object, it is said to be a descriptor." 

object.__get__(self, instance, owner)

Called to get the attribute of the owner class (class attribute access) or of an instance of that class (instance attribute access). owner is always the owner class, while instance is the instance that the attribute was accessed through, or None when the attribute is accessed through the owner. This method should return the (computed) attribute value or raise an AttributeError exception.

object.__set__(self, instance, value)

Called to set the attribute on an instance instance of the owner class to a new value, value.

Example: class Celsius(object): def __init__(self, value=0.0): self.value = float(value) def __get__(self, instance, owner): return self.value def __set__(self, instance, value): self.value = float(value)

class Temperature(object): celsius = Celsius

t = Temperature t.celsius = 5   # calls Celsius.__set__ print t.celsius # calls Celsius.__get__

Note: Also appears in 'pydoc' under "Data descriptors defined here" with object's docstring. Trick: the __doc__ can be overridden in the __init__ method, which makes for dynamic docstrings!

References:
 * Descriptor HowTo Guide — Python v2.7.5 documentation - http://docs.python.org/2/howto/descriptor.html
 * 3.4.2.2. Implementing Descriptors¶ - 3. Data model — Python v2.7.5 documentation - http://docs.python.org/2/reference/datamodel.html#implementing-descriptors

exec code
DANGEROUS!

Execute code in string: exec( 'print("hi")' )

Example of executing a startup script for IDLE: if os.path.isfile('.pythonrc.py'): exec(open('.pythonrc.py').read)

import os filename = os.environ.get('PYTHONSTARTUP') if filename and os.path.isfile(filename): exec(open(filename).read)

eval
VERY DANGEROUS

Will execute a string as though it were pure python code!

eval("print 'hello'")

Common Header Format
__author__ = "software team" __date__ ="$Feb 15, 2011 1:42:17 PM$" __version__ = "1.1.0.441 bandelier@66d9fe1edf3e"

Next should be authorship information. This information should follow this format:

__author__ = "Rob Knight, Gavin Huttley, and Peter Maxwell" __copyright__ = "Copyright 2007, The Cogent Project" __credits__ = ["Rob Knight", "Peter Maxwell", "Gavin Huttley", "Matthew Wakefield"] __license__ = "GPL" __version__ = "1.0.1" __maintainer__ = "Rob Knight" __email__ = "rob@spot.colorado.edu" __status__ = "Production" Source: Python Coding Guidelines - http://bayes.colorado.edu/PythonGuidelines.html

These will show up in the help like such: Help on module test:

NAME test

FILE /home/kenneth/test.py

FUNCTIONS test

DATA __author__ = 'test author' __version__ = '1.0'

VERSION 1.0

AUTHOR test author

References:
 * Python: What is the common header format? - Stack Overflow - http://stackoverflow.com/questions/1523427/python-what-is-the-common-header-format

Common Environment Variables
1. Command line and environment — Python v2.7.2 documentation - http://docs.python.org/using/cmdline.htm

PYTHONUNBUFFERED
 * If this is set to a non-empty string it is equivalent to specifying the -u option.
 * -u - Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.


 * See also stdout - Python output buffering - Stack Overflow - http://stackoverflow.com/questions/107705/python-output-buffering

PYTHONVERBOSE
 * If this is set to a non-empty string it is equivalent to specifying the -v option. If set to an integer, it is equivalent to specifying -v multiple times.
 * -v - Print a message each time a module is initialized, showing the place (filename or built-in module) from which it is loaded.

PYTHONPATH
 * Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored.

Queue
Good for threading.

"The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics." 

import Queue

bucket = Queue.Queue bucket.put("something")

while True: try: item = bucket.get(block=False) except Queue.Empty: pass else: # do stuff with queue

Important methods: Queue.qsize Queue.empty Queue.put(item[, block[, timeout]]) Queue.get([block[, timeout]])

-

Example: http://docs.python.org/2/library/queue.html def worker: while True: item = q.get do_work(item) q.task_done

q = Queue for i in range(num_worker_threads): t = Thread(target=worker) t.daemon = True t.start

for item in source: q.put(item)

q.join      # block until all tasks are done

-

"The module implements three types of queue, which differ only in the order in which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the heapq module) and the lowest valued entry is retrieved first."

Queue Types: class Queue.Queue(maxsize=0)         # FIFO Queue class Queue.LifoQueue(maxsize=0)     # LIFO Queue class Queue.PriorityQueue(maxsize=0) # Priority Queue

References:
 * 8.10. Queue — A synchronized queue class — Python v2.7.5 documentation - http://docs.python.org/2/library/queue.html

Remove Numbers from String
Using list comprehension: result = ''.join([i for i in s if not i.isdigit])

Using pattern matching: >>> re.compile(r'\D+').findall("test 1234 test 1234") ['test ', ' test ']

>>> re.compile(r"\d").sub("", "test 1234 test 1234") 'test test '

user input
line = raw_input # Python 2 line = input('What is your name: ') print(": " + line.strip + " :")
 * 1) line = input    # Python 2

sys - stdout stderr stdin
STDOUT: import sys sys.stdout.write("Hello ") # does not write new line sys.stdout.write("World\n")

STDERR: import sys sys.stderr.write("ERROR!\n")

STDIN: import sys line = sys.stdin.readline

line = input  # use raw_input in Python 2 line = input('What is your name: ') print(": " + line.strip + " :")

Read piped stdin line by line:
 * 1) cat /etc/passwd | ./readlines.py     # linux
 * 2) type file.txt | python readlines.py  # windows

import fileinput for line in fileinput.input: print(": " + line + " :")
 * 1) fileinput.input - returns fileinput object which is iterative

import sys for line in sys.stdin.readlines: print(": " + line + " :")
 * 1) sys.stdin.readlines - returns list of strings

import sys for line in sys.stdin: print(": " + line + " :")
 * 1) sys.stdin - returns object which is iterative

while True: try: s = raw_input("Say something ") except EOFError: break print ": %s" % s
 * 1) ugly while loop:

Check if stdin has any waiting data: import sys import select


 * 1) if select.select([sys.stdin,],[],[],0.0)[0]:  # alternative old method

if not sys.stdin.isatty: print "Have data!" print sys.stdin.readlines else: print "No data"

if len(sys.argv) > 1: message = ' '.join(sys.argv[1:]) elif not sys.stdin.isatty: # read from pipe in   message = sys.stdin.readline else: # read from user sys.stdout.write("Message: ") message = raw_input.strip

Regular Expressions
Import: import re # user regular expressions

Match vs Search: re.match("cde", "abcdef") # no match (match searches beginning of string, first match) re.match("abc", "abcdef") # match (match searches beginning of string, first match) re.search("cd", "abcdef") # match (matches any location, first match)

Both re.match and re.search return objects that are not intuitive to use. You can check the object's group method for what matched, but I prefer re.findall.

Match: s = 'done' if re.match('done|quit', s) != None # True

Search: s = 'done' if re.search('do', s) != None # True

Replace: msg = 'jello world' msg = re.sub('jello', 'hello', msg)

For case intensive searches append 3rd parameter "re.I".

Find all: re.findall("[a-z]+", "aaa111bbb222")      # ['aaa', 'bbb'] re.findall("111([a-z]+)", "aaa111bbb222") # ['bbb'] re.findall("hello", "HELLO", re.I)        # ['HELLO'] - Case Insensitive

Split: >>> re.split('[a-z]+','111aaa222bbb') ['111', '222', '']

>>> re.split('([a-z]+)','111aaa222bbb') # parenthesis cause return of all ['111', 'aaa', '222', 'bbb', '']

Replacement Substitution: re.sub(r'\\', r'/', r'\\data\files')
 * 1) re.sub(pattern, repl, string[, count, flags])

Compiled regular express for repeated use: (performance) valid = re.compile(r"[0-9]+") valid.match("aaa111bbb")              # not valid match = valid.search("aaa111bbb222")  # valid print(match.group)                  # what matched: '111'

Date and Time
Sleep in seconds: import time time.sleep(seconds)

Use the following functions to convert between time representations:

Date formats: >>> time.strftime("%Y-%m-%d %H:%M:%S", time.localtime) '2014-04-04 12:22:43'

>>> time.time 1396635763.0581281

>>> datetime.datetime.now datetime.datetime(2014, 4, 4, 12, 22, 43, 58456)
 * 1) datetime(year, month, day, hour, minute, second, microsecond)

>>> time.localtime time.struct_time(tm_year=2014, tm_mon=4, tm_mday=4, tm_hour=12, tm_min=22, tm_sec=43, tm_wday=4, tm_yday=94, tm_isdst=1)

Current epoch time: int(time.time)

Time zone offset: standard_tz_offset = - time.timezone / 3600 # 7 daylight_tz_offset = - time.altzone / 3600 # 6

Is daylight saving time: import time time.localtime _.tm_isdst # 0 or 1

mst_epoch_time = time.time # server in MST/MDT if time.localtime.tm_isdst: # are we in daylight saving time? utc_epoch_time = mst_epoch_time + time.altzone else: utc_epoch_time = mst_epoch_time + time.timezone

Format current time: time.strftime('%d%b%y') # '12May11' time.strftime('%Y.%m.%d %H:%M:%S') # '2013.08.09 14:01:28'

Time conversions: epoch_time = time.time   # 1375992088.7867229 dt = datetime.datetime.fromtimestamp( epoch_time )   # datetime.datetime(2013, 8, 8, 14, 1, 28, 786723) date_str = dt.strftime("%Y-%m-%d %H:%M:%S")   # '2013-08-08 14:01:28' tz_offset = - time.altzone / 3600 if tz_offset < 0: tz_offset_str = "-%02d00" % abs(tz_offset) else: tz_offset_str = "+%02d00" % abs(tz_offset) print date_str + " " + tz_offset_str # '2013-08-08 14:01:28 -0600'
 * 1) epoch to str

gm_epoch_time = calendar.timegm(time.gmtime)   # 1375992114 dt = datetime.utcfromtimestamp( gm_epoch_time )   # datetime.datetime(2013, 8, 8, 20, 1, 54) dt.strftime("%Y-%m-%d %H:%M:%S +0000")   # '2013-08-08 20:01:54 +0000' dt.strftime("%Y-%m-%dT%H:%M:%SZ")   # '2013-08-08 20:01:54 +0000'
 * 1) UTC epoch to str

iso_string='2013-08-08T14:01:28' epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S' ) )   # 1375992088.0
 * 1) Local time str to epoch

iso_string='2013-08-08T14:01:28 MDT' epoch_time = time.mktime( time.strptime( iso_string, '%Y-%m-%dT%H:%M:%S %Z' ) )   # 1375992088.0


 * 1) not sure how to specify numerical time zone and pull it in!

iso_string = '2013-08-08T20:01:54Z' epoch_time = calendar.timegm( time.strptime( iso_string.replace('Z', 'GMT'), '%Y-%m-%dT%H:%M:%S%Z' ) ) print epoch_time   # 1375992114 print "%sZ" % datetime.utcfromtimestamp(timestamp).isoformat # 2013-08-08T20:01:54Z
 * 1) UTC (Zulu) str to epoch, and back again

epoch_time = 1375992114 tz_offset = - time.altzone / 3600 gm_epoch_time = epoch_time - tz_offset * 3600   # 1376013714
 * 1) Convert epoch from Local to UTC

gm_epoch_time = 1376013714 tz_offset = - time.altzone / 3600 epoch_time = gm_epoch_time + tz_offset * 3600   # 1375992114
 * 1) Convert epoch from UTC to Local

struct_time: time.localtime # local zime zone time.gmtime    # UTC time

today = time.localtime today_year = today.tm_year today_month = today.tm_mon today_day = today.tm_mday

time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime)

Zulu time to epoch time: s = '2008-09-03T20:56:35.450686Z' event_time_struct = time.strptime(s.split('.')[0]+ "UTC", "%Y-%m-%dT%H:%M:%S%Z") epoch_time = calendar.timegm(event_time_struct)

Parse string to time.struct_time (like gmtime): time.strptime(time.ctime) # current time - default format is ctime format: "%a %b %d %H:%M:%S %Y" time.strptime(time.ctime(time.time)) # default format is ctime format: "%a %b %d %H:%M:%S %Y" time.strptime("30 Nov 00", "%d %b %y")

Build from ticks: localtime = time.localtime( time.time ) # build time.struct_time from ticks print time.asctime( localtime ) # Tue Jan 13 10:17:09 2009

dt = datetime.datetime.fromtimestamp( time.time ) # build datetime.datetime struct from ticks dt.strftime("%Y-%m-%d %H:%M:%S")  # '2013-07-31  23:07:48' time.strptime( time.time, ")

from datetime import date now = date.today   # now = datetime.date(2003, 12,2) now.strftime("%m-%d-%y")  # '12-02-03' birthday = date(1964, 7, 31) age = now - birthday age.days # 14368
 * 1) dates support calendar arthimetic

ts = (datetime.datetime.now.strftime("%Y%m%d%H%M"))        # 201105191118 tstxt = (datetime.datetime.now.strftime("%m-%d-%Y %H:%M")) # 05-19-2011 11:18

Seconds since epoch: int(time.time)   # 1305842304

Zulu and back again: (Datetime hell) import time from calendar import timegm from datetime import datetime if __name__ == '__main__': iso_string = '2013-06-05T15:19:10Z' timestamp = timegm(       time.strptime( iso_string.replace('Z', 'GMT'), '%Y-%m-%dT%H:%M:%S%Z' )   )    print iso_string print timestamp print '-> %sZ' % datetime.utcfromtimestamp(timestamp).isoformat

http://wiki.python.org/moin/TimeTransitionsImage?action=AttachFile&do=get&target=v1.png

References:
 * WorkingWithTime - PythonInfo Wiki - http://wiki.python.org/moin/WorkingWithTime
 * Datetime hell. Time zone aware to UNIX timestamp. | About me and the things I do - http://aboutsimon.com/2013/06/05/datetime-hell-time-zone-aware-to-unix-timestamp/
 * WorkingWithTime - Python Wiki - http://wiki.python.org/moin/WorkingWithTime

---

Something I cooked up...

def utc_time(self, years_offset=60): mst_epoch_time = time.time # server is in MST if time.localtime.tm_isdst: # are we daylight saving time? utc_epoch_time = mst_epoch_time + time.altzone else: utc_epoch_time = mst_epoch_time + time.timezone return utc_epoch_time

def tz_time(self, tz_offset, daylight_adjust=True): """ return the future time based on timezone offset           @tz_offset is timezone offset, eg -7 for MST            @daylight_adjust is flag to auto adjust for daylight time""" if daylight_adjust: if time.localtime.tm_isdst: # are we daylight saving time? _tz_offset = tz_offset - 1 else: _tz_offset = tz_offset _utc_epoch_time = self.utc_time _adj_time = _utc_epoch_time + _tz_offset * 60 * 60 return _adj_time

def future_time(self, tz_offset=-7, daylight_adjust=True, years_offset=60): """ Calculate some future year """ _tz_time = self.tz_time(tz_offset) future_utc_epoch_time = _tz_time + years_offset * 365 * 24 * 60 * 60 # account for leap years cur_year = datetime.datetime.now.year cur_year_leap = cur_year / 4 future_year = cur_year + years_offset future_year_leap = future_year / 4 leap_days = future_year_leap - cur_year_leap future_utc_epoch_time += leap_days * 24 * 60 * 60 return future_utc_epoch_time

Math
Math Functions: import math math.ceil(x) math.floor(x) math.trunc(x) math.pow(x,y)

abs(x) # built in

Random
import random random.choice(['a', 'b', 'c']) random.sample(range(100), 10) random.random # random float 0 < x < 1 random.randrange(3)    # integer in [0, 1, 2] random.randrange(1, 3) # integer in [1, 2] random.randint(1, 3)   # integer in [1, 2, 3]

Operating System Operations
Import OS Library: import os

Common Functions: os.getcwd      # get current directory os.listdir(p)    # get list of directory contents os.chdir(p)      # change directory os.mkdir(p)      # make directory os.makedirs(p)   # make directory and all intermediate os.rmdir(p)      # remove empty directory, see shutil.rmtree os.removedirs(p) # remove empty directories, see shutil.rmtree os.path.exists(p) # does file/directory exist? os.path.isdir(p) # does directory exist os.path.join(folder1, folder2, file, etc) # portable path join
 * 1) directory functions

os.remove(p)     # remove file os.unlink(p)     # same as os.remove os.path.exists(p) # does file/directory exist? os.path.isfile(p) # does file exist os.stat(fname)   # get file stats os.stat(fname).st_size # file size os.link(src, dst) # create hard link os.readlink(p)   # read symbolic link os.rename(old, new) # rename/move file or directory os.tempnam(p)    # temporary unique name for target path os.tmpfile     # create temporary file os.tmpnam      # temporary file name os.path.getsize(filename) # get file size
 * 1) file functions

os.fork        # fork process os.getpid      # get process id os.getppid      # get parent's process id os.kill(pid, sig) # kill a process os.getlogin    # get user's login name
 * 1) system functions

os.system(cmd)   # output to stdout, return exit_status os.popen(cmd)    # returns a pipe to command
 * 1) execution

Note: to get path to script: sys.path[0]

Get directory and file names: os.path.dirname(path) # '/root/test.txt' -> '/root' os.path.basename(path) # '/root/test.txt' -> 'test.txt' os.path.splitext(path) # '/root/test.test.txt' -> ('/root/test.test', '.txt') - extract extention filename.split(".")[-1] # '/root/test.test.txt' -> 'txt' - extract extention

Normalize path, eliminating double slashes, etc. os.path.normpath(path) # '/root/..' -> '/' or 'test/..' -> '.' os.path.abspath(path)  # '/root/..' -> '/' or 'test/..' -> '/root/test'

List dir: (returns list) os.listdir( os.getcwd )

List of files: [p for p in os.listdir(os.getcwd) if os.path.isfile(p)]

List of folders: [p for p in os.listdir(os.getcwd) if os.path.isdir(p)]

Build path for local OS: (using appropriate slash) pathstr = os.path.join("folder", "folder", "file_or_folder")

File and directory management with shutil import shutil shutil.copyfile('file1', 'file2') shutil.move('file1', 'folder2/file1')

File Wildcards: import glob glob.glob('*.py') # return list of files

Walk directory tree: import os for dirname, dirnames, filenames in os.walk('.'): for subdirname in dirnames: print os.path.join(dirname, subdirname) for filename in filenames: print os.path.join(dirname, filename)

ugly 'deltree' (by kenneth) import os def deltree(path): files = os.listdir(path) for file in files: file_path = os.path.join(path, file) if os.path.isdir(file_path): deltree(file_path) os.rmdir(file_path) else: os.remove(file_path)

Cleaner 'deltree' (by Kep) import shutil if os.path.isdir("xxx"): shutil.rmtree("xxx")
 * 1) Blow away (if needed), and reclone to proper version

System Platform
import platform

if platform.system == "Linux": if platform.system == "Windows": else:

Temp Files
import tempfile filename = tempfile.mktemp f = open(filename, 'w') ... f.close import os os.remove(filename) # delete temp file

Pickling
Pickling (serializing, making objects persistent): import pickle grades = {'bob' : [4, 5, 6], 'sue' : [3, None, 7, 7]} outfile = open(fname, 'wb') pickle.dump(grades, outfile)

infile = open(fname, 'rb') grades = pickle.load(infile)

Logging

 * Logging HOWTO — Python v3.3a0 documentation - http://docs.python.org/dev/howto/logging.html
 * Logging Cookbook — Python v3.3a0 documentation - http://docs.python.org/dev/howto/logging-cookbook.html
 * Logging Cookbook — Python v2.7.1 documentation - http://docs.python.org/howto/logging-cookbook.html

Log Levels: import logging msg = "i broke" logging.debug(msg) logging.info(msg) logging.warning(msg)  # WARNING:root:i broke logging.error(msg)    # ERROR:root:i broke logging.critical(msg) # CRITICAL:root:i broke

log.setLevel(level) # debug, info, warn/warning, error, critical log.setLevel(logging.DEBUG)

A good convention to use when naming loggers is to use a module-level logger, in each module which uses logging, named as follows: logger = logging.getLogger(__name__)

Simple example: log_format='%(asctime)s | %(levelname)s | %(message)s' log_date_format='%Y-%m-%d %H:%M:%S' logging.basicConfig(format=log_format, datefmt=log_date_format) log = logging.getLogger(__name__) log.setLevel(logging.INFO)

log.info('hi')

Formatter function: def makeFormatter(format_str="", date_format="%Y-%m-%d %H:%M:%S"): if not format_str: format_str = "%(asctime)s | %(levelname)s | %(filename)s:%(lineno)d:%(funcName)s | %(message)s" return logging.Formatter(fmt=format_str, datefmt=date_format)

Medium example: log_format='%(asctime)s | %(levelname)s | %(message)s' log_date_format='%Y-%m-%d %H:%M:%S' log_formatter = logging.Formatter(fmt=log_format, datefmt=log_date_format) log = logging.getLogger(__name__) log.setLevel(logging.DEBUG)
 * 1) Create logger

ch = logging.StreamHandler ch.setLevel(logging.INFO) ch.setFormatter(log_formatter) log.addHandler(ch)
 * 1) Log to console

fh = logging.FileHandler('my_logger.log') fh.setLevel(logging.DEBUG) fh.setFormatter(log_formatter) log.addHandler(fh)
 * 1) Log to file

Application Setup (not basic log configuration carries across imported modules): import logging def main: logging.basicConfig(filename='myapp.log', level=logging.INFO) logging.info('Started') ...   logging.info('Finished')

if __name__ == '__main__': main

Log multiple lines: for line in output.splitlines: log.debug(line)

Set log level: (warning is default) logging.basicConfig(level=logging.DEBUG)

Set log level from string value: numeric_level = getattr(logging, loglevel.upper, None) if not isinstance(numeric_level, int): raise ValueError('Invalid log level: %s' % loglevel) logging.basicConfig(level=numeric_level, ...)
 * 1) assuming loglevel is bound to the string value obtained from the
 * 2) command line argument. Convert to upper case to allow the user to
 * 3) specify --log=DEBUG or --log=debug

Log to file: logging.basicConfig(filename='example.log',level=logging.DEBUG) logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG) # do not append

Format output: (once set, can't be changed?) logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
 * 1) WARNING:MESSAGE

logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')
 * 1) 05/23/2011 12:50:02 PM MESSAGE

logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')
 * 1) 05/23/2011 12:55:28 PM : WARNING : MESSAGE

logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
 * 1) 2011-05-23 13:05:30 : WARNING : MESSAGE

logging.basicConfig(format='%(asctime)s | %(levelname)s | %(filename)s:%(funcName)s | %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
 * 1) 2011-05-23 13:05:30 | WARNING | FILE:FUNC | MESSAGE

Log record attributes - http://docs.python.org/library/logging.html#logrecord-attributes

Format Example: FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' # doesn't work! logging.basicConfig(format=FORMAT) d = { 'clientip' : '192.168.0.1', 'user' : 'fbloggs' } logger = logging.getLogger('tcpserver') logger.warning('Protocol problem: %s', 'connection reset', extra=d)

import logging

logger = logging.getLogger('spam_application') logger.setLevel(logging.DEBUG)
 * 1) create logger with 'spam_application'

fh = logging.FileHandler('spam.log') fh.setLevel(logging.DEBUG)
 * 1) create file handler which logs even debug messages

ch = logging.StreamHandler ch.setLevel(logging.ERROR)
 * 1) create console handler with a higher log level

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') fh.setFormatter(formatter) ch.setFormatter(formatter)
 * 1) create formatter and add it to the handlers

logger.addHandler(fh) logger.addHandler(ch)
 * 1) add the handlers to the logger

logger.info('Message to loggers and handlers')

Templating
from string import Template t = Template('${name}'s fav dish is $food') t.substitute(name = 'ken', food = 'beans') # throws exception if missing one, but can have more t.safe_substitute(name = 'ken') # will fill in what it can

Change delimiter: class MYT(Template): delimiter = '%' minput = input("File name format (%d - date, %s - size): ") myt = MYT(minput) myt.substitute(%d=date, %s=size)

Threading
Basic thread: import thread def someFunc: print "someFunc was called" thread.start_new_thread(someFunc, )

More complete basic thread: import thread import threading import time

def run_thread (threadname, sleeptime): print "Thread start: " + threadname time.sleep(sleeptime) print "Thread end: " + threadname thread.interrupt_main  # throw keyboard interrupt to kill main program

if __name__ == "__main__": thread.start_new_thread(run_thread, ("Thread1", 3)) try: while 1: pass except: print "Thread1 exited...."

Can use a global counter instead of exception to track number of threads: import thread import threading import time

threadcount = 3

def run_thread (threadname, sleeptime): global threadcount print "Thread start: " + threadname time.sleep(sleeptime) print "Thread end: " + threadname threadcount -= 1

if __name__ == "__main__": thread.start_new_thread(run_thread, ("Thread1", 3)) thread.start_new_thread(run_thread, ("Thread2", 4)) thread.start_new_thread(run_thread, ("Thread3", 2)) while 1: if threadcount == 0: break else: pass print "completed"

---

Thread object: import threading t1 = threading.Thread(target=someFunc) t1.start t1.join
 * 1)  with parameters: Thread(target=myfunction, args=('MyStringHere',1)).start

Basic threading with parameters: import threading t = threading.Thread(None, someFunc, None, (param1, param2)) t.start t.join

t = threading.Thread(target=client_recv, args=(s, name)) while True: t.join(timeout=2) if t.isAlive: print "Thread is dead..." break

Threading as a class: import threading class FuncThread(threading.Thread): def __init__(self, target, *args): self._target = target self._args = args threading.Thread.__init__(self) def run(self): self._target(*self._args) def someOtherFunc(data, key): print "someOtherFunc was called : data=%s; key=%s" % (str(data), str(key)) t1 = FuncThread(someOtherFunc, [1,2], 6) t1.start t1.join
 * 1) Example usage

Threading as a class #2: import threading import time import sys

class MyT(threading.Thread): def __init__(self, name, time): threading.Thread.__init__(self) self.name = name self.time = time def run(self): print("Thread Start: " + self.name) for i in range(self.time): print("[" + self.name + "]", end='') sys.stdout.flush time.sleep(1) print("Thread Stop: " + self.name)

myt1 = MyT('1', 10) myt1.start

myt2 = MyT('2', 15) myt2.start

myt3 = MyT('3', 5) myt3.start

myt4 = MyT('4', 10) myt4.start

myt1.join myt2.join myt3.join myt4.join

Locking: lock=thread.allocate_lock

lock.acquire lock.release
 * 1) pass lock to thread, and use in thread as such:
 * 1) some operation

Returning data from a thread (using a queue): import sys import threading import Queue
 * 1) Pass exception back to parent

class ExcThread(threading.Thread): def __init__(self, bucket): threading.Thread.__init__(self) self.bucket = bucket def run(self): try: raise Exception('An error occured here.') except Exception: self.bucket.put(sys.exc_info)

def main: bucket = Queue.Queue thread_obj = ExcThread(bucket) thread_obj.start while True: try: exc = bucket.get(block=False) except Queue.Empty: pass else: exc_type, exc_obj, exc_trace = exc # deal with the exception print "Exception Caught:", exc_type, exc_obj print "Exception Trace:", exc_trace thread_obj.join(0.1) if thread_obj.isAlive: continue else: break

if __name__ == '__main__': main

Exception Caught:  An error occured here. Exception Trace: 

Another Exception catching queue example: (kenneth's example) def run_thread(queue, item): ... queue.put( Exception("Failure on item:" + item) )
 * 1) thread function

threads = [] queue = Queue.Queue

for item in items: t = Thread(None, run_thread, None, (queue, item)) t.start threads.append(t)
 * 1) start threads

for t in threads: t.join
 * 1) wait for threads

while not queue.empty: raise(queue.get)
 * 1) catch and reraise exceptions

Returning data from a thread (using a callback function):
 * "Another approach is to pass a callback function to the thread. This gives a simple, safe and flexible way to return a value to the parent, anytime from the new thread."

import threading import time import sys

class MyThread(threading.Thread): def __init__(self, cb): threading.Thread.__init__(self) self.callback = cb   def run(self): for i in range(10): self.callback(i) time.sleep(1)

def count(x): print x   sys.stdout.flush

t = MyThread(count) t.start

References:
 * 16.2. threading — Higher-level threading interface — Python v2.7.1 documentation - http://docs.python.org/library/threading.html
 * An Introduction to Thread Programming - http://www.prasannatech.net/2008/08/introduction-to-thread-programming.html
 * Python - Multithreaded Programming - http://www.tutorialspoint.com/python/python_multithreading.htm
 * Running Functions as Threads in Python « Software Ramblings - http://softwareramblings.com/2008/06/running-functions-as-threads-in-python.html
 * Understanding Threading in Python LG #107 - http://linuxgazette.net/107/pai.html
 * multithreading - Simple threading in Python 2.6 using thread.start_new_thread - Stack Overflow - http://stackoverflow.com/questions/849674/simple-threading-in-python-2-6-using-thread-start-new-thread
 * python - Return value from thread - Stack Overflow - http://stackoverflow.com/questions/1886090/return-value-from-thread

---

Interrupt main thread:
 * 1) source: http://stackoverflow.com/questions/4591917/python-how-to-quit-cli-when-stuck-in-blocking-raw-input

import thread import time

def main: try: m = thread.start_new_thread(killable_input, tuple) while 1: time.sleep(0.1) except KeyboardInterrupt: print "exception"

def killable_input: w = thread.start_new_thread(normal_input, tuple) i = thread.start_new_thread(wait_sometime, tuple)

def normal_input: s = raw_input("input:")

def wait_sometime: time.sleep(4) # or any other condition to kill the thread print "too slow, killing imput" thread.interrupt_main

if __name__ == '__main__': main

urllib
See Python/urllib

Command line arguments
import sys for x in sys.argv: print x

Execute other python script
import sys sys.argv = ['', 'my', 'args', 'here'] execfile('myscript.py') # does not support arguments directly

Exit Code
sys.exit(n) http://docs.python.org/library/sys.html: import sys sys.exit(1) # exit code of 1 (default is 0) sys.exit("some error") # exit code of 1, and message displayed

syslog
import syslog syslog.openlog('[WATER]') syslog.syslog('Processing started') if error: syslog.syslog(syslog.LOG_ERR, 'Processing started')

import syslog syslog.openlog('[WATER]') def log(msg): syslog.syslog(str(msg)) log("Starting...")

Subprocess
Subprocess - http://docs.python.org/library/subprocess.html

Methods of subprocess (or external command execution):
 * subprocess (only method to rule/replace all others)
 * os.system
 * os.spawn*
 * os.popen*
 * popen2.*
 * commands.*

My Favorite: def exe(cmd): p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) #output = " | ".join(p.stdout.readlines) output = "\n".join(p.stdout.readlines) rc = p.wait return (rc, output)

def exe_checked(cmd, checked=True): p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) #output = " | ".join(p.stdout.readlines) output = "\n".join(p.stdout.readlines) rc = p.wait if rc and checked: raise Exception("Command returned non zero [{0}]: {1}".format(rc, cmd)   return (rc, output)

def exe_with_stderr(cmd): p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subproc ess.PIPE) output = " | ".join(p.stdout.readlines) outputerr = " | ".join(p.stderr.readlines) rc = p.wait return (rc, output, outputerr)

def exec_program(cmd): s = subprocess.Popen(cmd.split, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr = s.communicate result = s.wait return stdout, stderr, result

Pass along output and return true error code: import subprocess retcode = subprocess.call(["ls", "-l"]) # true return code, output sent to STDOUT

retcode = subprocess.call(["foo.sh","args"],shell=True) # Execute in shell, with shell processing retcode = subprocess.call(['C:\\Temp\\a b c\\Notepad.exe']) retcode = subprocess.call(["D:/Python24/Python.exe", "Skripta.py", "arg1", "arg2"])

Note: shell=True is equivalent to: subprocess.Popen(['/bin/sh', '-c', args[0], args[1], ...])

Process STDOUT and STDERR and get return code: import subprocess import string
 * 1) !/usr/bin/env python

cmd = "cat /etc/passwd | grep ken" p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) # stderr=subprocess.STDOUT to combine

out = string.join(p.stdout.readlines) outerr = string.join(p.stderr.readlines) retcode = str(p.wait)

print("STDOUT: " + out) print("STDERR: " + outerr) print("RETCODE: " + retcode)

Above with stderr redirected to stdout (combined): import subprocess p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) for line in p.stdout.readlines: print line, retval = p.wait

Replacing shell backtick: from subprocess import Popen output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate[0]
 * 1) shell code:
 * 2) OUTPUT=`mycmd myarg`

Replacing shell pipeline from subprocess import Popen p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close # Allow p1 to receive a SIGPIPE if p2 exits. output = p2.communicate[0]
 * 1) shell code:
 * 2) OUTPUT=`dmesg | grep hda`

Replacing os.system: sts = os.system("mycmd" + " myarg")
 * 1) old:

p = Popen("mycmd" + " myarg", shell=True) sts = os.waitpid(p.pid, 0)[1]
 * 1) new:

Control an executed process: cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,       stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

while p.poll == None:
 * 1) cycle while process continues alive

# check if input if select.select([p.stdout.fileno], [], [], 1)[0]: while select.select([p.stdout.fileno], [], [], .1)[0]: line = p.stdout.readline # ... process

# write something to process p.stdin.write("tp %s %s\n" % (user1, user2))

...

# time to force a quit: (choose which you need) p.kill # SIGKILL p.terminate # SIGTERM os.kill(p.pid, signal.SIGTERM) # same as first, but not as clean

Other Command Execution
Old method with os.system: import os encoded_wait_code = os.system("some command") # output sent to STDOUT
 * 1) Note: returns code of 0 if command returns 0, but wait encoded value for all other values

Quick and dirty old method (prints stderr, captures stdout) import os out = string.join( os.popen(cmd).readlines )

Quick and dirty old method (prints stderr, captures stdout, captures ugly wait encoded return code) import os p = os.popen(cmd) stdout = "".join( p.readlines ) encoded_wait_code = p.close # None if 0 rcode = os.WEXITSTATUS(ugly_rcode) # fix ugly wait encoded rcode print(stdout) print(rcode)

if os.popen(cmd).close: print "error with cmd: %s" % cmd sys.exit(1)

???: os.spawnvp(os.P_WAIT,"/usr/bin/mplayer",["","-ao","pcm",file]) os.spawnvp(os.P_NOWAIT,"/usr/bin/mplayer",["","-ao","pcm",file])
 * 1) or

Replacing os.system: sts = os.system("mycmd" + " myarg")
 * 1) old:

p = Popen("mycmd" + " myarg", shell=True) sts = os.waitpid(p.pid, 0)[1]
 * 1) new:

More realistic example replacing os.system: try: retcode = call("mycmd" + " myarg", shell=True) if retcode < 0: print >>sys.stderr, "Child was terminated by signal", -retcode else: print >>sys.stderr, "Child returned", retcode except OSError, e:   print >>sys.stderr, "Execution failed:", e

select
Select:

select.select(rlist, wlist, xlist[, timeout])
 * This is a straightforward interface to the Unix select system call. The first three arguments are sequences of ‘waitable objects’: either integers representing file descriptors or objects with a parameterless method named fileno returning such an integer
 * The optional timeout argument specifies a time-out as a floating point number in seconds. When the timeout argument is omitted the function blocks until at least one file descriptor is ready. A time-out value of zero specifies a poll and never blocks.

if select.select([sys.stdin.fileno], [], [], .1)[0]: line = p.stdout.readline

full wrapped execution
Process timeout (by Kenneth): import subprocess import time import os import signal

def cmd(cmd, timeout=300, retry=0): # Execute command, time progress, and kill on timeout

# (status, output) = commands.getstatusoutput(cmd) start_time = None status = None output = None for retry_count in range(retry+1): if retry_count > 0: print("RETRY: " + str(retry_count) + " " + cmd) start_time = time.time p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,               stderr=subprocess.STDOUT) timeout_fail = False while p.poll == None: if time.time - start_time > timeout: # kill process print("TIMEOUT: " + str(timeout) + "sec - killing (" + str(p.pid) + ") - " + cmd) try: os.kill(p.pid, signal.SIGTERM) except OSError as e:                   print("Kill exception: " + str(e)) timeout_fail = True break #print(".") time.sleep(1) status = p.poll output = p.stdout.read if timeout_fail == False: break seconds = int(time.time - start_time) return (status, output, seconds)

---

Process input and output wrapper (from minecraft): import subprocess import select import re import sys

cmd = "/usr/bin/java -Xms256M -Xmx1024M -jar minecraft_server.jar nogui"

p = subprocess.Popen(cmd, shell=True,       stdin=subprocess.PIPE,        stdout=subprocess.PIPE,        stderr=subprocess.STDOUT)

while p.poll == None:
 * 1) cycle while process continues alive

if select.select([p.stdout.fileno], [], [], 1)[0]: while select.select([p.stdout.fileno], [], [], .1)[0]: line = p.stdout.readline # -tp joe match = re.findall("<(.*)> -tp (.*)", line) if match: user1 = match[0][0] user2 = match[0][1] print ">> ACTION: Teleporting %s to %s" % (user1, user2) p.stdin.write("tp %s %s\n" % (user1, user2)) if line: print ">> SERVER: %s" % line.strip else: print "The server died!" sys.exit(1)

if select.select([sys.stdin.fileno], [], [], .1)[0]: line = sys.stdin.readline print ">> CONSOLE: %s" % line.strip p.stdin.write("%s" % line)

Command With Timeout
python - Using module 'subprocess' with timeout - Stack Overflow - http://stackoverflow.com/questions/1191374/using-module-subprocess-with-timeout

import subprocess, threading

class Command(object): def __init__(self, cmd): self.cmd = cmd self.process = None

def run(self, timeout): def target: print 'Thread started' self.process = subprocess.Popen(self.cmd, shell=True) self.process.communicate print 'Thread finished'

thread = threading.Thread(target=target) thread.start

thread.join(timeout) if thread.is_alive: print 'Terminating process' self.process.terminate thread.join print self.process.returncode

command = Command("echo 'Process started'; sleep 2; echo 'Process finished'") command.run(timeout=3) command.run(timeout=1)

My version: import subprocess import threading

class Command(object): def __init__(self, cmd, capture=False, shell=True): self.cmd = cmd self.process = None self.thread = None self.shell = shell if type(cmd) == list: self.shell = False self.rc = None self.pid = None self.stdout = None self.stderr = None self.capture = capture # capture stdout?

def run(self, timeout=None, wait=True): def target: print '>> Thread started (%s)' % self.cmd if self.capture: self.process = subprocess.Popen(self.cmd, shell=self.shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE) else: self.process = subprocess.Popen(self.cmd, shell=self.shell) print '>> Thread PID: %d' % self.process.pid self.pid = self.process.pid self.stdout, self.stderr = self.process.communicate self.rc = self.process.returncode print '>> Thread finished: %d' % self.rc

self.thread = threading.Thread(target=target) self.thread.start

if wait: self.wait(timeout) return self # so you can do: c = Command.run

def wait(self, timeout=None): print '>> Waiting for thread...' if timeout: self.thread.join(timeout) else: self.thread.join if self.thread.is_alive: print '>> Terminating process' self.process.terminate self.rc = self.process.returncode self.thread.join if self.thread.is_alive: print '>> Failed to terminate' else: print '>> Thread completed' #print self.process.returncode

def is_alive(self ): return self.thread.is_alive

command = Command('do something').run

command2 = Command(['do', 'something', 'long']).run(wait=False) command2.wait(timeout=10)

command3 = Command('want output', capture=True).run print command3.stdout

System call
os.system("some_command with args") os.system("some_command < input_file | another_command > output_file") stream = os.popen("some_command with args")

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:

1.
 * os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example,
 * os.system("some_command < input_file | another_command > output_file")
 * However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.
 * http://docs.python.org/lib/os-process.html

2.
 * stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.
 * http://docs.python.org/lib/os-newstreams.html

3.
 * The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say
 * print Popen("echo Hello World", stdout=PIPE, shell=True).stdout.read
 * instead of
 * print os.popen("echo Hello World").read
 * but it is nice to have all of the options there in one unified class instead of 4 different popen functions.
 * http://docs.python.org/lib/node528.html

4.
 * The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives you the return code. For example:
 * return_code = call("echo Hello World", shell=True)
 * http://docs.python.org/lib/node529.html

5.
 * The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.


 * The subprocess module should probably be what you use.

Source:

Executing Code from Command Line
python -c [cmd]
 * 1) -c cmd : program passed in as string (terminates option list)

python -c "import sys;exec(sys.stdin.read)" << "EOF" def test1: print "test1" test1 print "test" EOF

Integer Return Code Class
class RC(int): def __new__(cls, rc, stdout=, stderr=): """We can't use use __init__ because int is immutable.""" return super(RC, cls).__new__(cls, rc) def __init__(self, rc, stdout=, stderr=): super(self.__class__, self).__init__(rc) self.stdout = stdout self.stderr = stderr def __repr__(self): """Optional string output""" return "(%s, '%s', '%s')" % (self, self.stdout, self.stderr)

sample: >>> rc1 = RC(5) >>> rc2 = RC(0, 'out', 'err') >>> rc1 (5, , ) >>> rc2 (5, 'out', 'err') >>> rc2.stdout 'out' >>> int(rc1) 5 >>> 0 + rc1 5 >>> if rc1 <= 5: print "yes" ... yes >>> if rc2: print "non-zero" ...

select
usage: select.select(rlist, wlist, xlist[, timeout])
 * rlist: wait until ready for reading
 * wlist: wait until ready for writing
 * xlist: wait for an “exceptional condition” (see the manual page for what your system considers such a condition)
 * The optional timeout argument specifies a time-out as a floating point number in seconds.

if select.select([p.stdout.fileno], [], [], 1)[0]:

ANSI Color

 * ANSI escape code - http://en.wikipedia.org/wiki/ANSI_escape_code

---

unicode - Print in terminal with colors using Python? - Stack Overflow - https://stackoverflow.com/questions/287871/print-in-terminal-with-colors-using-python

class bcolors: HEADER = '\033[95m' OKBLUE = '\033[94m' OKGREEN = '\033[92m' WARNING = '\033[93m' FAIL = '\033[91m' ENDC = '\033[0m' BOLD = '\033[1m' UNDERLINE = '\033[4m'

print bcolors.WARNING + "Warning: No active frommets remain. Continue?" + bcolors.ENDC

class colors: black = 0 red = 1 green = 2 yellow = 3 blue = 4 magenta = 5 cyan = 6 white = 7 fg = 30 # foreground add bg = 40 # background add bfg = 90 # bold foreground add bbg = 100 # bold background add

print '\033[%dm red foreground \033[0m' % (colors.red + colors.fg) print '\033[%dm bold red foreground \033[0m' % (colors.red + colors.bfg) print '\033[%dm blue background \033[0m' % (colors.blue + colors.bg) print '\033[%dm bold blue background \033[0m' % (colors.blue + colors.bbg)

---

Before you start spouting escape sequences, you should check that stdout is a tty. You can do this with sys.stdout.isatty

sys.stdout.isatty

def hilite(string, status, bold): attr = [] if status: # green attr.append('32') else: # red attr.append('31') if bold: attr.append('1') return '\x1b[%sm%s\x1b[0m' % (';'.join(attr), string)

Python | change text color in shell - Stack Overflow - https://stackoverflow.com/questions/2330245/python-change-text-color-in-shell

---

For reference, a small bash script which prints a table of escape sequences: colortable16.sh: (The 256 color mode of xterm, The 90s called; they want their 256 colors back)
 * 1) !/bin/bash

echo echo Table for 16-color terminal escape sequences. echo Replace ESC with \\033 in bash. echo echo "Background | Foreground colors" echo "-" for((bg=40;bg<=47;bg++)); do	for((bold=0;bold<=1;bold++)) do		echo -en "\033[0m"" ESC[${bg}m  | " for((fg=30;fg<=37;fg++)); do			if [ $bold == "0" ]; then echo -en "\033[${bg}m\033[${fg}m [${fg}m " else echo -en "\033[${bg}m\033[1;${fg}m [1;${fg}m" fi done echo -e "\033[0m" done echo "- " done
 * 1) prints a color table of 8bg * 8fg * 2 states (regular/bold)

echo echo

Python version: (modified by me)
 * 1) !/usr/bin/env python

import sys

print print "Table for 16-color terminal escape sequences." print "Replace ESC with \\033 in bash." print print "Background | Foreground colors" print "-" for bg in range(40, 47+1): for bold in range(0, 1+1): sys.stdout.write("\033[0m"" ESC[{bg}m  | ".format(bg=bg)) for fg in range(30, 37+1): if bold == 0: sys.stdout.write("\033[{bg}m\033[{fg}m [{fg}m ".format(bg=bg, fg=fg)) else: sys.stdout.write("\033[{bg}m\033[1;{fg}m [1;{fg}m".format(bg=bg, fg=fg)) sys.stdout.write("\033[0m\n") print "- "
 * 1) prints a color table of 8bg * 8fg * 2 states (regular/bold)

print print

Execute External Application
Execute program: (and send output to STDOUT) encoded_wait_code = os.system('mkdir mydir') # output sent to STDOUT

Get output and ugly return code: (STDERR goes to STDOUT) p = os.popen(cmd) print p.readlines encoded_wait_code = p.close

Execute other python script import sys sys.argv = [, 'my', 'args', 'here'] execfile('myscript.py') # does not support arguments directly

Subprocess:
 * See

Program Layout
PackageName/ README setup.py   bin/ script docs/ test/ # some include in package_name pacakgename/ __init__.py       code1.py        subpackage/ __init__.py

__init__.py
Purpose: Makes the folder into a package library

NOTE: This file is most commonly blank

Good place to put your package functions. def load: ...
 * 1) ken/__init__.py

import ken ken.load

Can import other modules to then use as package modules: from ken.client import Server
 * 1) ken/__init__.py

import ken ken.Server

Good place to put your Exceptions. class MyException(Exception): """ My Error Exception """

Use the __all__ to specify what gets imported on 'from package import *' __all__ = ['func1', 'func2']

Metadata
VERSION = (0, 2, 1) or __version__ = '0.2.1'

Also See

---

__version__ = '1.10' __author__ = 'Joshua Roesslein' __license__ = 'MIT'

setup.py
See

Distributing Python Modules with Distutils
Distributing Python Modules — http://docs.python.org/distutils/index.html

Uses setup.py for package installation. python setup.py install

Simple example
To install foo.py: from distutils.core import setup setup(name='foo',     version='1.0',      py_modules=['foo'],      )

Source: http://docs.python.org/distutils/introduction.html

klib example
klib setup.py: from distutils.core import setup

setup(name = "klib",   version = "101",    description = "klib library",    url = "http://www.kennethburgener.com/klib",    author = "Kenneth",    author_email = "kenneth@kennethburgener.com",    license = "Kenneth License",    long_description = """Really long text here.""",    platforms = "Any Platform",

packages = ['klib','klib.messenger'], )

Installed to: /opt/python26/lib/python2.6/site-packages/klib/ /opt/python26/lib/python2.6/site-packages/klib-100-py2.6.egg-info

Metadata: (/opt/python26/lib/python2.6/site-packages/klib-101-py2.6.egg-info) Metadata-Version: 1.0 Name: klib Version: 101 Summary: klib library Home-page: http://www.kennethburgener.com/klib Author: Kenneth Author-email: kenneth@kennethburgener.com License: Kenneth License Description: Really long text here. Platform: Any platform

Any fields not included will be marked: Platform: UNKNOWN

References:
 * 2. Writing the Setup Script — Python v2.7.2 documentation - http://docs.python.org/distutils/setupscript.html
 * http://wiki.python.org/moin/Distutils/Tutorial

Uninstall Package
Collect files during install: python setup.py install --record install-record.txt sudo rm $(cat install-record.txt)

References:
 * installation - python setup.py uninstall - Stack Overflow - http://stackoverflow.com/questions/1550226/python-setup-py-uninstall

Distributing Python Modules with Setuptools
import setuptools
 * 1) !/usr/bin/env python

setuptools.setup(name="tweepy",     version="1.10",      description="Twitter library for python",      license="MIT",      author="Joshua Roesslein",      author_email="tweepy@googlegroups.com",      url="http://github.com/tweepy/tweepy",      packages = setuptools.find_packages,      keywords= "twitter library",      zip_safe = True)

Decorators
Decorators wrap functions

Function Counter: def fcount(func): def wrapper(*args, **kw): wrapper.count = wrapper.count + 1 return func(*args, **kw) wrapper.count = 0 return wrapper

@fcount def f(n): return n*n

print "f.count = ", f.count print "f(10) = ", f(10) print "f.count = ", f.count print "f(3) = ", f(3) print "f.count = ", f.count print "f(12) = ", f(12) print "f.count = ", f.count print "f(15) = ", f(15) print "f.count = ", f.count


 * 1) Output:
 * 2) f.count =  0
 * 3) f(10) =  100
 * 4) f.count =  1
 * 5) f(3) =  9
 * 6) f.count =  2
 * 7) f(12) =  144
 * 8) f.count =  3
 * 9) f(15) =  225
 * 10) f.count =  4

Great reference: Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

Example 1: @dec1 def func(arg1, arg2, ...): pass

def func(arg1, arg2, ...): pass func = dec1(func)
 * 1) equivalent to:

Example 2: @dec2 @dec1 def func(arg1, arg2, ...): pass

def func(arg1, arg2, ...): pass func = dec2(dec1(func))
 * 1) equivalent to:

Example 3: @decomaker(argA, argB, ...) def func(arg1, arg2, ...): pass

func = decomaker(argA, argB, ...)(func)
 * 1) equivalent to:

@accepts(int,int) @returns(float) def bar(low,high): pass
 * 1) example:

Example 4: (overrides and ignores passed in function - bad, unless desired) def spamrun(fn): def sayspam(*args): print "spam, spam, spam" return sayspam

@spamrun def useful(a, b): print a**2 + b**2

Example 5: (Decorator that modifies behavior of undecorated func) def addspam(fn): def new(*args): print "spam, spam, spam" return fn(*args) return new

@addspam def useful(a, b): print a**2 + b**2

My Example 1: (print args and continue to function) def showargs(fn): def doshowargs(*args, **kw): print "Args for:", fn.__name__, args, kw       return fn(*args, **kw) return doshowargs

@showargs def useful(a, b): print a**2 + b**2

Another Example: def makebold(fn): def wrapped(*args, **kw): return "" + fn(*args, **kw) + "" return wrapped

def makeitalic(fn): def wrapped(*args, **kw): return "" + fn(*args, **kw) + "" return wrapped

@makebold @makeitalic def hello(msg): return msg

print hello("hello world")
 * 1) output: hello world

References:
 * PEP 318 -- Decorators for Functions and Methods - http://www.python.org/dev/peps/pep-0318/
 * Charming Python: Decorators make magic easy - http://www.ibm.com/developerworks/linux/library/l-cpdecor/index.html
 * What are some common uses for Python decorators? - Stack Overflow - http://stackoverflow.com/questions/489720/what-are-some-common-uses-for-python-decorators
 * Understanding Python decorators - Stack Overflow - http://stackoverflow.com/questions/739654/understanding-python-decorators

---

Utah Python July 2013 Meeting - Decorator Examples:

Example of uses:
 * Count function calls
 * Add additional output (debugging)
 * Timing code
 * Thread synchronization (locks)
 * Type checking

def shout(f): def wrapper(*args): print('This is before {name}'.format(name=f.__name__)) ret = f(*args) print('This is after {name}'.format(name=f.__name__)) return ret wrapper.__name__ = f.__name__ return wrapper
 * 1) Simple wrapping call

@shout def echo(text): print(text)

echo('this is a test')

def count(f): def wrapper(*args): wrapper.counter += 1 print("Function called {count} times".format(count=wrapper.counter)) return f(*args) wrapper.counter = 0 # change wrapper name, or it will just be 'wrapper' (for inspection) wrapper.__name__ = f.__name__ return wrapper
 * 1) Counting function calls

@count def donothing: pass

donothing donothing print("Function called a total of {count} times".format(count=donothing.counter))

import time
 * 1) Timing your code

def timer(f): def wrapper(*arg): t = time.time ret = f(*arg) print("Took {elapsedtime} seconds".format(elapsedtime=time.time-t)) return ret return wrapper

@timer def fact(n): prod = 1 while n > 0: prod *= n       n -= 1 return prod

total = fact(10000)

import threading
 * 1) Synchronization

def synchronized(lock): def wrap(f): def wrapper(*args, **kwargs): lock.acquire try: return f(*args, **kwargs) finally: lock.release return wrapper return wrap

mylock = threading.Lock

@synchronized(mylock) def myfunc: pass

def myMethod(ID, name):  Old way  if not (isinstance(ID, 'uint') and isinstance(name, 'utf8string')): raise Exception
 * 1) Type checking

def accepts(*typeargs):  Decorator way  def wrap(f): def wrapper(*args): for a, t in zip(args, typeargs): if not isinstance(a, t): raise Exception('{a} not of type {t}'.format(a=a, t=t)) return f(*args) return wrapper return wrap

@accepts(int) def add1(n): return n + 1

def memoize(f): def wrapper(*args): if not wrapper.ret: wrapper.ret = f(*args) return wrapper.ret wrapper.ret = False wrapper.__name__ = f.__name__ return wrapper
 * 1) Memoize

import time

@memoize def sumto10: count = 0 sum = 0 while count < 10: sum += count time.sleep(1) count += 1 return sum

print sumto10 print sumto10
 * 1) first time will count slow
 * 1) second time will return cached value instantly

Memoize: "Caches a function's return value each time it is called. If called later with the same arguments, the cached value is returned (not reevaluated)"

import time @shout @memoize def test1: print "calculating" time.sleep(3) return 100
 * 1) Nested decorators

Order is important! For tested decorators the execution order is from the top to the bottom. If you reverse @shout and @memoize, the @shout will not run because of the way @memoize was written.

registry = [] @shout def registry(f): registry.append(f) return f
 * 1) registry
 * 1) can also decorate decorators:

@register def test2: pass

Decorator is just a function that returns a function.

Can use functools to automatically update decorator wrappers with __name__, etc import functools def my_decorator(f): @functools.wraps(f) def wrapper(*args, **kwds): print 'Calling decorated function' return f(*args, **kwds) return wrapper
 * 1) Auto syntax sygar the wrapper for you

Class Method Decorators
Static methods:
 * Similar to C++ and Java static methods

class C: @staticmethod def say(msg): print(msg)

C.say('hi')

Class method:
 * A class method receives the class as implicit first argument, just like an instance method receives the instance.
 * Class methods are different than C++ or Java static methods. If you want those, see staticmethod in this section.

@classmethod def setup_parser(cls, parser):

See also

Class Properties
Properties: (not using decorators) class C(object): def __init__(self): self._x = None def getx(self): print("get x") return self._x def setx(self, value): print("set x") self._x = value def delx(self): print("del x") del self._x x = property(getx, setx, delx, "I'm the 'x' property.")

c = C c.x = 5 print(c.x) del(c.x)
 * 1) usage

Getter and Setter Property Decorators: class C(object):   # make sure to inherit from object for these properties to work! def __init__(self): self._x = None @property def x(self): """I'm the 'x' property.""" print("get x") return self._x @x.setter  # python >= 2.6 def x(self, value): print("set x") self._x = value @x.deleter # python >= 2.6 def x(self): print("del x") del self._x

c = C c.x = 5 print(c.x) del(c.x)
 * 1) usage

Note: If you don't add the @x.setter, the property will be read only, and will throw a "AttributeError: can't set attribute" if attempted to be set.

Using x.getter example: class Cls(object): def __init__(self): self._x = 100

x = property

@x.getter def x(self): """I'm the 'x' property.""" return self._x

@x.setter def x(self, value): self._x = value

@x.deleter def x(self): del self._x

Custom Decorators
decorator @dec1 def func(arg1, arg2, ...): pass

equivalent to: def func(arg1, arg2, ...): pass func = dec1(func)

Basic layout (names are arbitrary): def my_decorator(function_name): # update wrapper.__doc__ and .func_name # or @functools.wraps(my_wrapper) def mywrapper(*args, **kw): # do something before result = function_name(*args, **kw) # do something after return result return my_wrapper

Example: def limit(length): def decorator(function): def wrapper(*args, **kw): result = function(*args, **kw) result = result[:length] return result return wrapper return decorator

@limit(3) def echo(foo): return foo

echo('123456') # '123'

assert
assert True assert False # exception assert 1 == 1 assert 1 == 2, "some message" # AssertionError: some message

Snake Case
Convert Camel Case to Snake Case:

def snake_case(name): s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', name) return re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1).lower

>>> convert('CamelCase') 'camel_case'

Camel Case
Camel Case:

>>> ''.join(x for x in 'make IT camel CaSe'.title if not x.isspace) 'MakeItCamelCase'

To titleize a sentence: >>> "this is a test".title "This Is A Test"

Lambda
Simple functions mul = lambda a, b: a * b

def mul(a, b): return a * b
 * 1) equivalent to:

Sort usage: data = [dict(number=x) for x in '036149'] data.sort(key=lambda x: float(x['number']))

Compare usage: data = [dict(number=x) for x in '036149'] data.sort(cmp=lambda x,y: cmp(x['number'], y['number']))

Boolean: lambda x: x % 3 == 0

Multiple parameter: lambda x, y: x + y

Call another function: lambda word: len(word)

Strings: lambda line: line.split[2]

Embedded: def transform(n): return lambda x: x + n

No parameters: lambda: 1

Default parameters: lambda x=1: x+1

References:
 * Python: Lambda Functions - http://www.secnetix.de/olli/Python/lambda_functions.hawk

Map
Apply a function to items of a sequence: map(str, range(3))  # ['0', '1', '2']

Note: list comprehensions can replace map and filter

Reduce
Apply a function to pairs of the sequence: import operator reduce(operator.mul, [1,2,3,4])  # 24   - ((1*2)*3)*4)

Note: sum or for loop can replace reduce

Filter
Return a sequence of items for which function(item) is True filter(lambda x: x >= 0, [0, -1, 3]) # [0, 3]

Note: list comprehensions can replace map and filter

Find matching items in two lists: a = ['a','b','c'] b = ['c','d'] filter(lambda x: x in a, b) # returns only 'c' as it is in both lists

List comprehension
Nothing more than Syntactic Sugar

5. Data Structures — Python v2.7.5 documentation - http://docs.python.org/2/tutorial/datastructures.html#list-comprehensions
 * "List comprehensions provide a concise way to create lists."
 * "A list comprehension consists of brackets containing an expression followed by a for clause, then zero or more for or if clauses. "

Remove newlines from each item in a list: mylist = [x.strip for x in mylist]

Simple: doubled = [x*2 for x in mylist] squares = [x**2 for x in range(10)]

If: [ 2*x for x in range(1,10) if x >= 5 ]

Complex: [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

Expanding PI: [str(round(pi, i)) for i in range(1, 6)]

Return list of words my_list = [current_word.lower for current_word in words]

Utah Python July 2013 Meeting: (http://basepi.net) squares = [] for x in range(10): squares.append(x**2)
 * 1) Iterative
 * [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

squares = map(lambda x: x**2, range(10))
 * 1) Functional

squares = [x**2 for x in range(10)] even_squares = [x**2 for x in range(10) if x%2 == 0]
 * 1) List comprehension

import sha strings = ['Python is cool!', 'I love python'] hashes = [sha.sha(x).digest for x in strings]
 * 1) get hash of strings in list:

matrix = 1, 2, 3], [4, 5, 6 [x for row in matrix for x in row]
 * 1) flatten matrix:
 * [1, 2, 3, 4, 5, 6]

[(x,y) for x in range(10) for y in range(10)] [(x,y) for x in range(10) for y in range(10) if x%2 != y%2]
 * 1) build matrix:
 * 1) matrix of not both odd and not both even

[line.strip.split("\t") for line in open("my_file.tab") if not line.startswith('#')]
 * 1) suck lines of files into list, ignoring commented lines

References:
 * 5.1.4. List Comprehensions - Data Structures — Python v2.7.2 documentation - http://docs.python.org/tutorial/datastructures.html#list-comprehensions
 * Utah Python July 2013 Meeting - YouTube - https://www.youtube.com/watch?v=uh9uhzitdvw

Mass Parameter Passing
Using: *args and **kw
 * *args a tuple of remaining unnamed parameters
 * **kw a dictionary of remaining named parameters

def param_func(a, b='b', *args, **kw): print( [a, b, args, kw] )

WARNING: If not careful, this can result in this error: "TypeError: param_func got multiple values for keyword argument 'myvar'". Might be better to just use:

def param_func(*args, **kw): print( [args, kw] )

param_func(1, 2, 3, 4, 5) # [1, 2, (3, 4, 5), {}] param_func(1, 2, 3, 4, 5, joe='test') # [1, 2, (3, 4, 5), {'joe': 'test'}] mylist = ("a", "c", "g", "h", 5) param_func(mylist) # [('a', 'c', 'g', 'h', 5), 'b',, {}] param_func(*mylist) # ['a', 'c', ('g', 'h', 5), {}]  *tricky* Note: *mydict will strip out just keys and treat like list Note: **mydict will treat like named parameters

def mywrapper(*args, **kw): # do something before result = function_name(*args, **kw)

Passing arbitrary parameters: def fun2(**kw): print kw # {'a': '1', 'b': '2', 'c': '3'}

def fun1(c='3', **kw): print kw # {'a': '1', 'b': '2'} fun2(c='3', **kw)

fun1(a='1', b='2')

Iteration Protocol
PEP 234 -- Iterators

Making a class iterable: class mycount(object): def __init__(self, max): self.x = 0 self.max = max def __iter__(self): # return object that implements __next__ or next return self def next(self): if self.x < self.max: self.x += 1 return self.x   else: self.x = 0 raise StopIteration

for i in mycount(10): print i    # 1 ... 10 print [x for x in mycount(10) ]  # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10) for i in c: print i   # 1 ... 10 [ x for x in c ]      # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Generators
Generators are functions that return to their previous state (using 'yield') when iterated over, which can be used in for loops.

Yield
Improve on Iterators:
 * Infinite data (infinite iterators)
 * Memory-intensive iteration
 * Cooperative Multi-threading

Generators are functions that return to their previous state when iterated over, which can be used in for loops.

Note: Generators throw StopIteration when completed

Note: the 'yield' keyword will stash away the state of the function and return the value. (Continuations)

def gen_range(end): cur = 0 while cur < end: yield cur # return here next time cur += 1

for i in gen_range(10): print i
 * 1) for loop iteration

my_gen = gen_range(10) try: while True: print my_gen.next except StopIteration: pass
 * 1) manual iteration

numbers = [x for x in gen_range(10)]
 * 1) list comprehension

Note: one can place an empty 'return' in a generator if needed. A 'return' with argument is invalid.

Making a class generate: class mycount(object): def __init__(self, max): self.max = max def __iter__(self): return self.next     # change here def next(self):          # change here x = 0 while x < self.max: x += 1 yield x

for i in mycount(10): print i    # 1 ... 10 print [x for x in mycount(10) ]  # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

c = mycount(10) for i in c: print i   # 1 ... 10 [ x for x in c ]      # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

itr = mycount(10).next itr.next # 1 itr.next # 2  - eventually throws StopIteration exception

Utah Python July 2013 Meeting:
 * Iterators are used to traverse a list
 * Generators are more powerful Iterators

def firstn(n): num = 0 nums = [] while num < n:       nums.append(num) num += 1 return nums
 * 1) simple 'n' number in functional form

def firstn(n): num = 0 while num < n:       yield num num += 1
 * 1) simple 'n' number in generator form

-

def counter: num = 0 while True: yield num num += 1
 * 1) Infinite list generator

def fib: num1 = 0 num2 = 1 yield num1 while True: num1, num2 = num2, num1 + num2 yield num1
 * 1) Fibbonacci generator

def fib(n): num1 = 0 num2 = 1 count = 0 yield num1 while count < n:       count += 1 num1, num2 = num2, num1 + num2 yield num1
 * 1) Fibbonacci generator 2

Utah Python July 2013 Meeting:
 * 1) Cooperative Multi-Threading:
 * 2) Cooperative multithreading relies on the threads themselves to relinquish control once they are at a stopping point.

def mythread(name): print('Starting Thread: {name}...'.format(name=name)) counter = 10 while counter > 0: print('Thread: {name} doing some work!'.format(name=name)) counter -= 1 yield

thread1 = mythread('Alice') thread2 = mythread('Bob') thread3 = mythread('Trudy')

try: while True: thread1.next thread2.next thread3.next except StopIteration: pass

References:
 * Utah Python July 2013 Meeting:
 * http://files.basepi.net/talks/pyarmory.zip

dunder methods
Some dunder (double under) methods determine what will happen when + (__add__) or other similar numerical symbols used.

class str_add(object): def __init__(self, name): self.name = name def __add__(self, other): self.name += other.name return self # note: only need to return self if you want to go beyond 'a + b'

a = str_add('a_') b = str_add('b_') c = str_add('c_') d = a + b + c print(a.name) # a_b_c_ print(b.name) # b_ print(c.name)  # c_ print(d.name)  # a_b_c_

Other common methods to override:
 * __add__ for +
 * __sub__ for -
 * __mul__ for *
 * __div__ for /
 * __mod__ for %

References:
 * Emulating numeric types - http://docs.python.org/2/reference/datamodel.html#emulating-numeric-types

doctest
test.py: """ This is the "example" module.
 * 1) !/usr/bin/env python

>>> double(5) 10 """

def double(n): """ This is double function

>>> double(3) 6 """  return n * 2

if __name__ == "__main__": import doctest doctest.testmod

unittest
unittest.TestCase

Here is a short script to test three functions from the random module: import random import unittest

class TestSequenceFunctions(unittest.TestCase):

def setUp(self): self.seq = range(10)

def test_shuffle(self): # make sure the shuffled sequence does not lose any elements random.shuffle(self.seq) self.seq.sort self.assertEqual(self.seq, range(10))

# should raise an exception for an immutable sequence self.assertRaises(TypeError, random.shuffle, (1,2,3))

def test_choice(self): element = random.choice(self.seq) self.assertTrue(element in self.seq)

def test_sample(self): with self.assertRaises(ValueError): random.sample(self.seq, 20) for element in random.sample(self.seq, 5): self.assertTrue(element in self.seq)

if __name__ == '__main__': unittest.main

The unittest module can be used from the command line to run tests from modules, classes or even individual test methods: python -m unittest test_module1 test_module2 python -m unittest test_module.TestClass python -m unittest test_module.TestClass.test_method python -m unittest -v test_module # verbose python -m unittest -h  # help

References:
 * 25.3. unittest — Unit testing framework — Python v2.7.4 documentation - http://docs.python.org/2/library/unittest.html
 * Python Debugging Tips - Google Drive - https://docs.google.com/presentation/d/130Wl--IzbDh9ymF6KHMu0WrsIwz6k32pav1gFbsw7NI/edit?usp=sharing

Internet Access
from urllib.request import urlopen for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'): line = line.decode('utf-8') if 'EST' in line or 'EDT' in line: # look for eastern time print(line)
 * 1) Python 3

Simple SMTP
Requires local STMP server.

import smtplib server = smtplib.SMTP("localhost") server.sendmail('mrken@demo.oeey.com', 'kenneth@demo.oeey.com', """To: kenneth@demo.oeey.com From: mrken@demo.oeey.com Subject: This is fun

Beware of the dog. """) server.quit

s = smtplib.SMTP(smtp_server)

with TLS: s = smtplib.SMTP("smtp.gmail.com",587) s.ehlo s.starttls s.ehlo

SSL: s = smtplib.SMTP_SSL('smtp.mail.yahoo.com', 465)

Data Compress
zlib, gzip, bz2, zipfile, tarfile

import zlib s = b'hello world' # only in python >3 len(s) t = zlib.compress(s) zlib.decompress(t) zlib.crc32(s)

gzip
import gzip content = "Lots of content here" f = gzip.open('file.txt.gz', 'wb') f.write(content) f.close

import gzip f_in = open('file.txt', 'rb') f_out = gzip.open('file.txt.gz', 'wb') f_out.writelines(f_in) f_out.close f_in.close

Performance Timer
Time a task repeated 10 times: from timeit import Timer Timer('t=a; a=b; b=t', 'a=1; b=2').timeit Timer('print("hi")').timeit(10) # repeat 10 times and time

Simple method: start_time = int(time.time) ... task ... compute_time = int(time.time) - start_time

Python Version Check
Solution:

__init__.py: import sys assert sys.version_info >= (2, 6, 0), 'requires Python 2.6 or higher' del sys
 * 1) Check Python Version:

resulting error if not > 2.6: >>> import nucleon Traceback (most recent call last): File " ", line 1, in ? File "nucleon/__init__.py", line 9, in ? assert sys.version_info >= (2, 6, 0), 'nucleon requires Python 2.6 or higher' AssertionError: nucleon requires Python 2.6 or higher

Other solutions:

Note: this will throw syntax error on 2.4 due to ternary import sys if sys.version_info < (2, 4): raise "must use python 2.5 or greater" else: # syntax error in 2.4, ok in 2.5 x = 1 if True else 2 print x

maybe solution: try: eval("1 if True else 2") except SyntaxError: # doesn't have ternary

>>> try: ...    from my_module import twoPointSixCode ... except Exception: ...    print "can't import, probably because your python is too old!" >>>

import sys if sys.version_info<(2,6,0): sys.stderr.write("You need python 2.6 or later to run this script\n") exit(1)
 * 1) !/usr/bin/env/python

>>> import sys >>> sys.hexversion # hex printed as integer 33817584 >>> '%x' % sys.hexversion # output as hex, not zero fill  2.4.3 '20403f0' >>> sys.hexversion < 0x02060000 # test if version less than 2.6 True >>> print "0x%08x" % sys.hexversion # print as hex and zero fill and prepend '0x' 0x020403f0

import sys if sys.hexversion < 0x02060500: print >> sys.stderr, "This script requires python 2.6.5 or higher." exit(1)

>>> import sys >>> sys.version_info (2, 6, 4, 'final', 0) >>> if not sys.version_info[:2] == (2, 6): ...   print "Error, I need python 2.6" ... else: ...   from my_module import twoPointSixCode >>>

References:
 * http://stackoverflow.com/questions/446052/python-best-way-to-check-for-python-version-in-program-that-uses-new-language-fe
 * http://stackoverflow.com/questions/1093322/how-do-i-check-what-version-of-python-is-running-my-script

Struct - Binary Data Record Layout
Format Characters: Format	C Type		Python type 	Standard size x	pad byte	no value c	char		str of len 1	1 b	signed char	integer		1 B	unsigned char	integer 	1 ?	_Bool		bool 		1 h	short		integer 	2 H	unsigned short 	integer 	2 i	int		integer 	4 I	unsigned int 	integer 	4 l	long		integer 	4 L	unsigned long	integer		4 q	long long	integer		8 Q	unsigned long long   integer	8 f	float		float		4 d	double		float		8 s	char[]		string p	char[]		string P	void *		integer

H # two byte unassigned number I # four byte unassigned number < # standard size and in little-endian byte order

import struct data = struct.pack('>> struct.pack("<5sh", "helloo", 11) 'hello\x0b\x00' >>> struct.unpack("<5sh", 'hello\x0b\x00') ('hello', 11)

pdb interactive debugger

 * 1) uncomment the following to use the interactive debugger:
 * 2) import pdb; pdb.set_trace

Telnet
create: class telnetlib.Telnet([host[, port[, timeout]]])

Note: The instance is initially not connected by default; the open method must be used to establish a connection. Alternatively, the host name and optional port number can be passed to the constructor, to, in which case the connection to the server will be established before the constructor returns.

open if not initially done so: Telnet.open(host[, port[, timeout]])

Functions: Telnet.read_until(expected[, timeout])  # Read until a given string, expected, is encountered or until timeout seconds have passed. # on timeout return whatever is in queue, possibly empty string Telnet.close        # close connection Telnet.read_all     # Read all data until EOF; block until connection closed. Telnet.write(buffer)  # Write a string to the socket, doubling any IAC characters. This can block if the connection is blocked. Telnet.expect(list[, timeout]) # Read until one from a list of a regular expressions matches. # Return a tuple of three items: the index in the list of matches; the match object; and the text read to match.

Read until end of line: line = tn.read_until("\n")

import getpass import sys import telnetlib

HOST = "localhost" user = raw_input("Enter your remote account: ") password = getpass.getpass

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ") tn.write(user + "\n") if password: tn.read_until("Password: ") tn.write(password + "\n")

tn.write("ls\n") tn.write("exit\n")

print tn.read_all


 * 1) !/usr/bin/env python

import sys import telnetlib

HOST = "216.119.193.252" USER = "admin" PASS = "@lt12345"

tn = telnetlib.Telnet(HOST)

tn.read_until("login: ") tn.write(USER + "\n") tn.read_until("Password: ") tn.write(PASS + "\n") tn.read_until("# ")

tn.write("terminal length 0\n") tn.read_until("# ")

tn.write("show running-config\n") configlines = tn.read_until("# ")

tn.write("exit\n") tn.read_all

print(configlines)

SSH
Cheap method: os.system('ssh user@server command')

SecureShell - PythonInfo Wiki - http://wiki.python.org/moin/SecureShell
 * There are several ways to use Secure Shell (SSH) in Python


 * pyssh - http://pyssh.sourceforge.net/
 * paramiko - Paramiko is a native Python implementation of SSH. - http://www.lag.net/paramiko/
 * conch - conch is another native implementation of SSH and part of the Twisted Matrix project - http://twistedmatrix.com/projects/conch
 * SSH wrapper - Some tools just wrap around existing ssh/sftp implementations
 * keyphrene - http://www.keyphrene.com/products/org.keyphrene/
 * keyphrene SSH wrapper - http://membres.lycos.fr/fredp/python/pyscp.html
 * Fabric - http://docs.fabfile.org/en/1.4.0/index.html
 * pexpect - Noah.org - http://www.noah.org/wiki/Pexpect

SFTP:
 * SFTP in Python: Really Simple SSH | The best of Zeth - http://zeth.net/post/332/

pexpect: wget http://pexpect.sourceforge.net/pexpect-2.3.tar.gz tar xzf pexpect-2.3.tar.gz cd pexpect-2.3 sudo python ./setup.py install

SFTP
SFTP in Python? (platform independent) - Stack Overflow - http://stackoverflow.com/questions/432385/sftp-in-python-platform-independent

paramiko: ssh2 protocol for python - http://www.lag.net/paramiko/

pip install paramiko

mkdir -p ~/.src ; cd ~/.src wget http://www.lag.net/paramiko/download/paramiko-1.7.7.1.zip unzip paramiko-1.7.7.1.zip cd paramiko-1.7.7.1 python setup.py install

SFTP Methods:
 * chdir
 * chmod
 * chown
 * close
 * file
 * from_transport
 * get
 * get_channel
 * getcwd
 * listdir
 * listdir_attr
 * logger
 * lstat
 * mkdir
 * normalize
 * open
 * put
 * readlink
 * remove
 * rename
 * request_number
 * rmdir
 * sock
 * stat
 * symlink
 * truncate
 * ultra_debug
 * unlink
 * utime

import paramiko

host = "THEHOST.com"                   #hard-coded port = 22 transport = paramiko.Transport((host, port))

password = "THEPASSWORD"               #hard-coded username = "THEUSERNAME"               #hard-coded transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

import sys path = './THETARGETDIRECTORY/' + sys.argv[1]   #hard-coded localpath = sys.argv[1] sftp.put(localpath, path)

sftp.close transport.close print 'Upload done.'

VMware Example:
 * 1) !/usr/bin/env python

import paramiko import sys

host = "sftp2.engx.vmware.com" port = 443 transport = paramiko.Transport((host, port))

username = "xxxx" password = "xxx" transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

print dir(sftp)

print sftp.chdir("build-uploader") print sftp.listdir

sftp.close transport.close

paramiko
paramiko: ssh2 protocol for python - http://www.lag.net/paramiko/

pip install paramiko

Github: git clone https://github.com/paramiko/paramiko cd paramiko git checkout v1.12.2 python setup.py install

Old archive: mkdir -p ~/.src ; cd ~/.src wget http://www.lag.net/paramiko/download/paramiko-1.7.7.1.zip unzip paramiko-1.7.7.1.zip cd paramiko-1.7.7.1 python setup.py install

---

[Python] paramiko examples - https://gist.github.com/mlafeldt/841944

scp_demo.py import sys, paramiko if len(sys.argv) < 5: print "args missing" sys.exit(1) hostname = sys.argv[1] password = sys.argv[2] source = sys.argv[3] dest = sys.argv[4] username = "root" port = 22 try: t = paramiko.Transport((hostname, port)) t.connect(username=username, password=password) sftp = paramiko.SFTPClient.from_transport(t) sftp.get(source, dest) finally: t.close
 * 1) !/usr/bin/env python

---

ssh_demo.py: import sys, paramiko if len(sys.argv) < 4: print "args missing" sys.exit(1) hostname = sys.argv[1] password = sys.argv[2] command = sys.argv[3] username = "admin" port = 22 try: client = paramiko.SSHClient client.load_system_host_keys client.set_missing_host_key_policy(paramiko.WarningPolicy) client.connect(hostname, port=port, username=username, password=password) stdin, stdout, stderr = client.exec_command(command) print stdout.read, finally: client.close
 * 1) !/usr/bin/env python

---

SSH Key auto add: client.set_missing_host_key_policy(paramiko.AutoAddPolicy)

Get Password
Prompt for password: getpass.getpass(prompt='Password: ')

import getpass user = raw_input("Enter your remote account: ") password = getpass.getpass

ini files
ini_filename = 'my.ini'

import ConfigParser parser = ConfigParser.SafeConfigParser parser.read(ini_filename)

sections = parser.sections # get sections for section in sections: print "[" + section + "]" for name, value in parser.items(section): print name + " = " + value

if parser.has_option(section, option): print( parser.get(section, option) )

To have the option names not convert to lower case: parser.optionxform = str

Ignore section: ini_str = '[root]\n' + open(ini_path, 'r').read ini_fp = StringIO.StringIO(ini_str) config = ConfigParser.RawConfigParser config.readfp(ini_fp)

Get option: config.has_option('root', 'some_key') config.get('root', 'some_key')

config.items('root') # list of Key Value tuples

optparse - Option Parser
[http://docs.python.org/library/optparse.html - Option Parser

Note: Deprecated since version 2.7: The optparse module is deprecated and will not be developed further; development will continue with the argparse module.

import optparse parser = optparse.OptionParser

parser.add_option('--name', default=None)

parser.add_option(   '-l', '--loglevel',    help='Log Level (default: info)',    default="info", action='store', type='string')

(options, args) = parser.parse_args

if len(sys.argv) == 1: parser.print_help sys.exit(1)
 * 1) if no parameters passed show help

if args: options.target = args[0]
 * 1) if a stand alone parameter passed, override option

print(options.loglevel)
 * 1) show option:

Note: Order is important when listing add_options!

Pretty Description: parser = optparse.OptionParser(description="Purpose: to do something") # or parser.description = "Purpose: to do something"

Parse arguments: (options, args) = parser.parse_args Note: If you like, you can pass a custom argument list to parse_args, but that’s rarely necessary: by default it uses sys.argv[1:].

Change Usage Message: usage = "%prog [options]" # default message
 * 1) Usage: [options]"  # where is replaced with the script name

parser = OptionParser(usage=usage) parser.usage = usage # default message parser.set_usage(usage)
 * or
 * or

Remove option: (does not remove default value from 'options' variable!!! - stupid!) parser.remove_option('--loglevel')

Modify an option: (changes are saved to parser automatically, unless in an option group) option = parser.get_option('--loglevel') option.help = 'something else' option.default = 'debug'

# or parser.remove_option("--loglevel") parser.add_option("--loglevel", help="something else", default="info")

Modify an option in an option group: opt_group = parser.get_option_group("--loglevel") opt_group.remove_option("--loglevel") opt_group.add_option("--loglevel", help="something else", default="info")

Types: "string", "int", "long", "choice", "float" and "complex" Defaults:
 * Default actions is 'store'.
 * Default type is 'string'.
 * Default dest is the long name of the parameter (or short name if no long name provided)
 * Default value is None

from optparse import OptionParser [...] parser = OptionParser

parser.add_option("-f", "--file", dest="filename",                 help="write report to FILE", metavar="FILE")
 * 1) string

parser.add_option("-q", "--quiet",                 action="store_true", dest="verbose", default=False,                  help="don't print status messages to stdout")
 * 1) boolean - action is what to do when flag found

parser.add_option(   "-m", "--minutes",    help="Minutes (default: 60)",    default=60, type='int')
 * 1) integer

(options, args) = parser.parse_args

print "Filename: " + options.filename

fake handling of required option: if not option.filename: parser.print_help parser.error("filename is required")

change default usage: parser = OptionParser(usage="Usage: %prog [args] arg1 arg2")

Really ugly way to print help usage :-( def Usage:    pname = sys.argv[0]    cmd='%s -h' % (pname)    os.system(cmd)

if (len(sys.argv) < 2): Usage sys.exit(1)

Groups: group = OptionGroup(parser, "Dangerous Options",                   "Caution: use these options at your own risk.  "                    "It is believed that some of them bite.") group.add_option("-g", action="store_true", help="Group option.") parser.add_option_group(group)
 * 1)  OptionGroup(parser, title, description=None)

Hide an option:
 * Option.help
 * "Help text to print for this option when listing all available options after the user supplies a help option (such as --help). If no help text is supplied, the option will be listed without help text. To hide this option, use the special value optparse.SUPPRESS_HELP."

parser.add_option("--secret", help=SUPPRESS_HELP)

Issues with Option Parser:
 * No handling for positional arguments
 * No adding help for positional arguments
 * No handling for required field

argparse
argparse - http://docs.python.org/library/argparse.html

argparse — Parser for command-line options, arguments and sub-commands

NOTE: MUCH MORE POWERFUL THAN OPTION PARSER!

Note: New in version 2.7.

Installation on <= 2.6 mkdir -p ~/.src ; cd ~/.src wget http://argparse.googlecode.com/files/argparse-1.2.1.tar.gz tar -zvxf argparse-1.2.1.tar.gz cd argparse-1.2.1 sudo python setup.py install
 * 1) Install argparse: - http://code.google.com/p/argparse/
 * 2) REQUIRES SETUPTOOLS

The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv. The argparse module also automatically generates help and usage messages and issues errors when users give the program invalid arguments.

import argparse

parser = argparse.ArgumentParser(description='Process some integers.') parser.add_argument('integers', metavar='N', type=int, nargs='+',                  help='an integer for the accumulator') parser.add_argument('--sum', dest='accumulate', action='store_const',                  const=sum, default=max,                   help='sum the integers (default: find the max)')

args = parser.parse_args print args.accumulate(args.integers)

import argparse

argparser = demoproject.util.argparse.ArgumentParser(   usage='Kick off demoproject benchmarks.') argparser.add_argument(   '--card',    help='What cards to run on.  Depends on Jenkins NODE_LABEL environment'        'variable for correct mappings.') argparser.add_argument(   '--suite', metavar='SUITE',    help='Specify the appropriate benchmark suite as defined in '        'the benchmark ini file.  Defaults to standard_perf',    default='standard_perf') args = argparser.parse_args

print args.card

Remove an argument: def remove_options(parser, options): """ remove_options(parser, ['--some-option', '--some-option2']) """ for option in options: for action in parser._actions: if vars(action)['option_strings'][0] == option: parser._handle_conflict_resolve(None,[(option,action)]) break

commands
35.16. commands — Utilities for running commands — Python v2.7.1 documentation - http://docs.python.org/library/commands.html

Note: Deprecated since version 2.6: The commands module has been removed in Python 3.0. Use the subprocess module instead.

The commands module contains wrapper functions for os.popen which take a system command as a string and return any output generated by the command and, optionally, the exit status.

Get status and output: status, output = commands.getstatusoutput(cmd)

Get output ignoring status: output = commands.getoutput(cmd)

NOTE: DO NOT USE getstatus AS IT DOES NOT DO WHAT YOU THINK IT DOES! status = commands.getstatus(file)

>>> import commands

>>> commands.getstatusoutput('ls /bin/ls') (0, '/bin/ls')

>>> commands.getstatusoutput('cat /bin/junk') (256, 'cat: /bin/junk: No such file or directory')

>>> commands.getstatusoutput('/bin/junk') (256, 'sh: /bin/junk: not found')

>>> commands.getoutput('ls /bin/ls') '/bin/ls'

>>> commands.getstatus('/bin/ls') '-rwxr-xr-x 1 root        13352 Oct 14  1994 /bin/ls'

Windows Registry
Note: The _winreg module has been renamed to winreg in Python 3.0.


 * 32.3. winreg – Windows registry access — Python v3.2 documentation - http://docs.python.org/py3k/library/winreg.html
 * 34.3. _winreg – Windows registry access — Python v2.7.1 documentation - http://docs.python.org/library/_winreg.html

Read from registry: from _winreg import * key = OpenKey(HKEY_LOCAL_MACHINE, r'Software\Microsoft\Outlook Express', 0, KEY_ALL_ACCESS) QueryValueEx(key, "InstallRoot")

Write to registry: keyVal = r'Software\Microsoft\Internet Explorer\Main' try: key = OpenKey(HKEY_CURRENT_USER, keyVal, 0, KEY_ALL_ACCESS) except: key = CreateKey(HKEY_CURRENT_USER, keyVal) SetValueEx(key, "Start Page", 0, REG_SZ, "http://www.blog.pythonlibrary.org/") CloseKey(key)

Enumerate registry keys: from _winreg import EnumKey, HKEY_USERS try: i = 0 while True: subkey = EnumKey(HKEY_USERS, i)       print subkey i += 1 except WindowsError: # WindowsError: [Errno 259] No more data is available pass

Another enumeration example:
 * 1) File: winreg-example-1.py

import _winreg

explorer = _winreg.OpenKey(   _winreg.HKEY_CURRENT_USER,    "Software\\Microsoft\\Windows\\CurrentVersion\\Explorer"    )

try: i = 0 while 1: name, value, type = _winreg.EnumValue(explorer, i)       print repr(name), i += 1 except WindowsError: print
 * 1) list values owned by this registry key

value, type = _winreg.QueryValueEx(explorer, "Logon User Name")

print print "user is", repr(value)

Enable Remote Desktop Example: (by Kenneth) from _winreg import * key = OpenKey(HKEY_LOCAL_MACHINE, r'SYSTEM\CurrentControlSet\Control\Terminal Server', 0, KEY_ALL_ACCESS) SetValueEx(key, "fDenyTSConnections", 0, REG_DWORD, 0) ValueTypeTuple = QueryValueEx(key, "fDenyTSConnections")  # (0, 4) print("Please reboot the system to complete the changes")

References:
 * Python’s _winreg: Editing the Windows Registry « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/03/20/pythons-_winreg-editing-the-windows-registry/

JSON
JSON - http://docs.python.org/library/json.html

ENCODING TO JSON STRING (type str)

import json json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
 * 1) '["foo", {"bar": ["baz", null, 1.0, 2]}]'

compact encoding: import json json.dumps([1,2,3,{'4': 5, '6': 7}], separators=(',',':'))
 * 1) '[1,2,3,{"4":5,"6":7}]'

Pretty printing: import json print json.dumps({'4': 5, '6': 7}, sort_keys=True, indent=4) # { # }
 * 1)     "4": 5,
 * 2)     "6": 7

DECODING JSON STRING TO PYTHON LIST OR DICTIONARY OBJECT

import json json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')
 * 1) [u'foo', {u'bar': [u'baz', None, 1.0, 2]}]

Load from file: f = open(filename) j = json.load(f) print j["kind"] # "dual-controller-adapter" #{ #}
 * 1)   "kind": "dual-controller-adapter",

From Command Line: echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

echo '{"test":1,"test2":2}' | python -c 'import sys,json;data=json.loads(sys.stdin.read); print data["test"]'
 * 1) get value

python -c "import json;f=open('file.json');data=json.load(f);print json.dumps(data,indent=2)" python -c "import json;print json.dumps(json.load(open('file.json')),indent=2)"
 * 1) read large file

Load from web: import httplib import json

def get_http_json(server, path): """ This performs an HTTP GET against a server and path and returns the       resulting page data.

server The server we're hitting, e.g. demo.com path  The path we're hitting, e.g. api/part/MA12345/ """   connection = httplib.HTTPConnection(server)    try:        url = 'http://%s/%s' % (server, path)        print 'Querying url %s' % url        connection.request('GET', url)        response = connection.getresponse        if response.status == httplib.OK:            return json.loads(response.read)        else:            print 'Unable to get %s. %s %s' % (url, response.reason,                    response.status)            return None    finally:        connection.close

j = get_http_json(server, path)

Pretty printing from the command line: echo '{"json":"obj"}' | python -mjson.tool #{ #}
 * 1)   "json": "obj"

References:
 * 18.2. json — JSON encoder and decoder — Python v2.7.1 documentation - http://docs.python.org/library/json.html

Sockets
See Python/Sockets

base64
import base64

Encode: encoded = base64.b64encode('data to be encoded')

Decode: base64.b64decode(encoded)

daemonize
Creating a daemon the Python way « Python recipes « ActiveState Code - http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

import os import sys

try: pid = os.fork except OSError, e:   raise Exception, "%s [%d]" % (e.strerror, e.errno)

if (pid != 0): # parent os._exit(0)

os.setsid
 * 1) create new process session

os.open('/dev/null', os.O_RDWR) os.dup2(0, 1) os.dup2(0, 2)
 * 1) redirect stdin, stdout, stderr


 * 1) ... tasks ...

My Solution: import os import sys import signal import syslog

HUP = False

def log(msg): syslog.syslog(str(msg)) syslog.openlog('[WATER]')

def kill_signal_handler(signal, frame): log("Being killed off: {0} {1}".format(signal, frame)) log("Shutting Down!") sys.exit(0) signal.signal(signal.SIGTERM, kill_signal_handler) signal.signal(signal.SIGALRM, kill_signal_handler)

def hup_signal_handler(signal, frame): global HUP log("Signal reload received: {0} {1}".format(signal, frame)) HUP=True signal.signal(signal.SIGHUP, hup_signal_handler)

def main_loop: global HUP

while True: # ... handle HUP=True ...

# ... main code ...

# wait 60 seconds before we start the next loop time.sleep(60) # signals will interrupt this

if __name__ == "__main__": log("Starting Daemon...") child_id = os.fork if child_id: log("Daemonized!") sys.exit(0) os.setsid # redirect stdin, stdout, stderr os.open('/dev/null', os.O_RDWR) os.dup2(0, 1) os.dup2(0, 2)

# write pid with open("water.pid", "w") as f:       f.write(str(os.getpid))

while True: try: main_loop except Exception as e:           log("Critical Failure Caught: %s" % str(e)) log("Critical Failure - Restarting!") time.sleep(30) #raise e

ln -s /opt/pyton27/bin/python pyton-water ./python-water water.py killall python-water

fork
import os import sys import time

child_id = os.fork if child_id: print "parent exiting" sys.exit(0)

while True: time.sleep(3) print "child"

For windows, to hide the console, see

csv
13.1. csv — CSV File Reading and Writing — Python v2.7.2 documentation - http://docs.python.org/library/csv.html

Read: import csv spamReader = csv.reader(open('eggs.csv', 'rb'), delimiter=' ', quotechar='|') for row in spamReader: print ', '.join(row)

Write: import csv spamWriter = csv.writer(open('eggs.csv', 'wb'), delimiter=' ',   quotechar='|', quoting=csv.QUOTE_MINIMAL) spamWriter.writerow(['Spam'] * 5 + ['Baked Beans']) spamWriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])

SQLite
11.13. sqlite3 — DB-API 2.0 interface for SQLite databases — Python v2.7.2 documentation - http://docs.python.org/library/sqlite3.html

sqlite3 - New in version 2.5.

import sqlite3

conn = sqlite3.connect('/tmp/example')

c = conn.cursor

c.execute('''create table stocks (date text, trans text, symbol text, qty real, price real)''')
 * 1) Create table

c.execute("""insert into stocks         values ('2006-01-05','BUY','RHAT',100,35.14)""")
 * 1) Insert a row of data

print c.rowcount
 * 1) Show affected row count

conn.commit
 * 1) Save (commit) the changes

c.close
 * 1) We can also close the cursor if we are done with it

parameter substitution: symbol = 'IBM' c.execute("... where symbol = '%s'" % symbol)
 * 1) Never do this -- insecure!

t = (symbol,) c.execute('select * from stocks where symbol=?', t) c.execute('select * from stocks where symbol=?', (symbol,))
 * 1) Do this instead

for t in [('2006-03-28', 'BUY', 'IBM', 1000, 45.00), ('2006-04-05', 'BUY', 'MSOFT', 1000, 72.00), ('2006-04-06', 'SELL', 'IBM', 500, 53.00), ]:   c.execute('insert into stocks values (?,?,?,?,?)', t)
 * 1) Larger example

select results: c = conn.cursor c.execute('select * from stocks order by price')

for row in c: print row  # row is a tuple


 * 1) OR (but not both)

rows = c.fetchall   # rows is list of tuples print len(rows) for row in rows: print row

Show which version of sqlite3:
 * 1) !/usr/bin/python
 * 2) -*- coding: utf-8 -*-

import sqlite3 as lite import sys

con = None

try: con = lite.connect('test.db')

cur = con.cursor cur.execute('SELECT SQLITE_VERSION')

data = cur.fetchone

print "SQLite version: %s" % data

except lite.Error, e:

print "Error %s:" % e.args[0] sys.exit(1)

finally:

if con: con.close

Or simply: (3, 3, 6)
 * 1) echo -e "import sqlite3\nprint sqlite3.sqlite_version_info" | python

MySQL Database
Ubuntu packages:
 * python-mysqldb - Python interface to MySQL
 * python-mysql.connector - pure Python implementation of MySQL Client/Server protocol

apt-get install python-mysqldb >>> import _mysql

apt-get install python-mysql.connector PIP:
 * https://pypi.python.org/pypi/MySQL-python
 * https://pypi.python.org/pypi/mysqlclient

pip install MySQL-python

pip install mysqlclient

---

_mysql - http://mysql-python.sourceforge.net/MySQLdb.html

apt-get install python-mysqldb

import _mysql

db=_mysql.connect(host="localhost",user="user",                 passwd="pass",db="db")

db.query("""SELECT spam, eggs, sausage FROM breakfast        WHERE price < 5""")

r=db.store_result r=db.use_result
 * 1) ...or...

row = r.fetch_row

couchdb
Getting started with couchdb-python — couchdb-python v0.8 documentation - http://packages.python.org/CouchDB/getting-started.html

couch = couchdb.Server('http://127.0.0.1:5984/')

db = couch.create('test') # create new db = couch['mydb'] # or get existing
 * 1) database

for db in couch: print db
 * 1) print databases:

doc = {'foo': 'bar'} doc_id, doc_rev = db.save(doc) doc = db['9278f7691c3421227a9f8436540008f9']
 * 1) (u'9278f7691c3421227a9f8436540008f9', u'1-4c6114c65e295552ab1019e2b046b10e')
 * 2) note: this also updates doc with _rev and _id

db[doc.id] = doc # this will write immediately to the database
 * 1) better to create a document using an id you want, instead of the randomly generated one

for id in db: print id
 * 1) print all docs:

db.delete(doc)
 * 1) delete doc

couch.delete('test')
 * 1) delete database

Search database: map_fun = function(doc) { emit(doc.name, null); }
 * 1) show all records


 * 1) row = 
 * 2) emit(key, value) - returns the values for key and value

map_fun = ''' function(doc) { if (doc.username == 'virt-01') emit(doc.username, null); } '''

for row in db.query(map_fun): print row.id
 * 1) row = 

for row in db.query(map_fun, descending=True): print row.id

couchdb-python - CouchDB Python Library - Google Project Hosting - http://code.google.com/p/couchdb-python/

Installation: mkdir -p ~/.src ; cd ~/.src wget http://pypi.python.org/packages/source/C/CouchDB/CouchDB-0.8.tar.gz#md5=e32b39e459e1fa2586ea252712a11a59 tar -zvxf CouchDB-0.8.tar.gz cd CouchDB-0.8 python setup.py install

Test: echo "import couchdb" | python

nose
nose: nose extends unittest to make testing easier

nose : Python Package Index - http://pypi.python.org/pypi/nose

Install: mkdir -p ~/.src ; cd ~/.src wget http://pypi.python.org/packages/source/n/nose/nose-1.1.2.tar.gz#md5=144f237b615e23f21f6a50b2183aa817 tar -zvxf nose-1.1.2.tar.gz cd nose-1.1.2 sudo python setup.py install

skeleton: mkdir skeleton cd skeleton mkdir bin NAME tests docs touch NAME/__init__.py touch tests/__init__.py touch tests/NAME_tests.py

nose test skeleton: from nose.tools import * import klib
 * 1) tests/NAME_tests.py

def setup: print "SETUP!"

def teardown: print "TEAR DOWN!"

def test_basic: print "I RAN!"


 * 1) def test_something_else:

Usage: nosetests

See Also:
 * 
 * 

pydoc
25.1. pydoc — Documentation generator and online help system — Python v2.7.2 documentation - http://docs.python.org/library/pydoc.html

pydoc - The pydoc module automatically generates documentation from Python modules.

pydoc - same as 'import [library], help([library])' $ pydoc [library]

Note: lists some special 'module metadata variables' separately (e.g. version, author).

special pydoc recognized metadata sections: __version__ __author__ __credits__ __date__

pydoc function module
Help on package klib:

NAME klib - This is the name

FILE /root/klib/klib/__init__.py

DESCRIPTION These are module doc strings that give helpful details

PACKAGE CONTENTS markone

FUNCTIONS func1 This is a doc string for func1

DATA PI = 3.141 __all__ = ['func1', 'PI'] __author__ = 'Kenneth Burgener <kenneth@kennethburgener.com>' __copyright__ = 'Copyright 2007, The Project' __credits__ = 'Credits go to me' __date__ = '26 February 2001' __email__ = 'ken@k.ttak.org' __license__ = 'GPL' __maintainer__ = 'Ken Burgener' __status__ = 'Production' __version__ = '$Revision: 83492 $'

VERSION 83492

DATE 26 February 2001

AUTHOR Kenneth Burgener <kenneth@kennethburgener.com>

CREDITS Credits go to me

Source: """ This is the name

These are module doc strings that give helpful details """

__version__ = "$Revision: 83492 $" __author__ = "Kenneth Burgener <kenneth@kennethburgener.com>"
 * 1) __version__ = "1.0"

__date__ = "26 February 2001" __credits__ = "Credits go to me"

__all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license' __license__ = "GPL" __copyright__ = "Copyright 2007, The Project" __maintainer__ = "Ken Burgener" __email__ = "ken@k.ttak.org" __status__ = "Production"

def func1: """   This is a doc string    for func1    """

pydoc class module
Sample: Help on module class_pydoc:

NAME class_pydoc - This is the name

FILE /root/class_pydoc.py

DESCRIPTION These are module doc strings that give helpful details

CLASSES Toad

class Toad | doc string for object |    |  Methods defined here: |    |  __init___(self, name) |     Constructor |    |  instance_function(self) |     Instance Function |    |  --     |  Class methods defined here: |    |  class_function(cls) from __builtin__.classobj |     Class Function |    |  --     |  Data and other attributes defined here: |    |  my_class_attribute = 'sample class attribute'

DATA PI = 3.141 __author__ = 'Kenneth Burgener <kenneth@kennethburgener.com>' __copyright__ = 'Copyright 2007, The Project' __credits__ = 'Credits go to me' __date__ = '26 February 2001' __email__ = 'ken@k.ttak.org' __license__ = 'GPL' __maintainer__ = 'Ken Burgener' __status__ = 'Production' __version__ = '1.0'

VERSION 1.0

DATE 26 February 2001

AUTHOR Kenneth Burgener <kenneth@kennethburgener.com>

CREDITS Credits go to me

Code: """ This is the name

These are module doc strings that give helpful details """

__version__ = "1.0" __author__ = "Kenneth Burgener <kenneth@kennethburgener.com>"
 * 1) __version__ = "$Revision: 83492 $"

__date__ = "26 February 2001" __credits__ = "Credits go to me"


 * 1) __all__ = ['func1', 'PI']

PI = 3.141

__copyright__ = 'hello license' __license__ = "GPL" __copyright__ = "Copyright 2007, The Project" __maintainer__ = "Ken Burgener" __email__ = "ken@k.ttak.org" __status__ = "Production"

class Toad: """ doc string for object """

my_class_attribute = 'sample class attribute'

def __init___(self, name): """ Constructor """

def instance_function(self): """ Instance Function """

@classmethod def class_function(cls): """ Class Function """

username
User ID: os.getuid

User Name: os.getlogin
 * 1) fails it stdin piped to program

User Name: import sys import pwd def get_username: return pwd.getpwuid( os.getuid )[ 0 ]

Environment: os.environ['USER']

hostname
import socket print(socket.gethostname)

os.uname[1]

hostname = os.system('hostname')

Get Hostname
import socket print(socket.gethostname)

Convert hostname to ip address
>>> socket.gethostbyaddr('demo') ('demo.oeey.com', [], ['10.10.10.10'])

/usr/local/bin/whatip: (or nametoip)
 * 1) !/usr/bin/env python


 * 1) whatip - convert hostname to ip address
 * 2) author: Kenneth Burgener <kenneth@kennethburgener.org> (FEB 2012)
 * 1) author: Kenneth Burgener <kenneth@kennethburgener.org> (FEB 2012)

import socket import sys

if len(sys.argv) != 2: print "Convert Hostname to IP Address" print "Usage: %s HOSTNAME" % sys.argv[0].split('/')[-1] sys.exit(1)

hostname = sys.argv[1] try: result = socket.gethostbyaddr(hostname) print result[2][0] except (socket.gaierror, socket.herror) as e: print "Error: Unable to resolve '%s'" % hostname sys.exit(1)

Get Public IP Address
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s.connect(("gmail.com",80)) print(s.getsockname[0]) s.close

networking - Finding local IP addresses using Python's stdlib - Stack Overflow - http://stackoverflow.com/questions/166506/finding-local-ip-addresses-using-pythons-stdlib

Python Network Server

 * 1) !/usr/bin/python          # This is server.py file

import socket              # Import socket module

s = socket.socket        # Create a socket object host = socket.gethostname # Get local machine name port = 12345               # Reserve a port for your service. s.bind((host, port))       # Bind to the port

s.listen(5)                # Now wait for client connection. while True: c, addr = s.accept    # Establish connection with client. print 'Got connection from', addr c.send('Thank you for connecting') c.close               # Close the connection

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
 * 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

Python Network Client

 * 1) !/usr/bin/python          # This is client.py file

import socket              # Import socket module

s = socket.socket        # Create a socket object host = socket.gethostname # Get local machine name port = 12345               # Reserve a port for your service.

s.connect((host, port)) print s.recv(1024) s.close                    # Close the socket when done

Source: http://www.tutorialspoint.com/python/python_networking.htm

References:
 * 17.2. socket — Low-level networking interface — Python v2.7.5 documentation - http://docs.python.org/2/library/socket.html

Run Library Module as Script
Run python with -m option: -m mod : run library module as a script (terminates option list)

Example: cat [json_file] | python -m json.tool cat [json_file] | python /opt/python26/lib/python2.6/json/tool.py
 * 1) same as:

Good for when library is buried deep in the library directories

Hex
>>> hex(10) # print integer as hex string '0xa' >>> sys.hexversion # python version as integer 33817584 >>> hex(sys.hexversion) # python version as hex '0x20403f0' >>> print "0x%08x" % sys.hexversion # python version as hex '0x20403f0' >>> int('0xa', 16) # hex string to integer 10 >>> int('0a', 16) # hex string to integer 10 >>> int('0xa', 0) # hex string to integer auto detected du to '0x' 10

Binary
Integer to Binary String with Negative Number Support: (2's complement) def bin(x): if x < -32768 or x > 65535: raise Exception("Out of Range Error: %d" % x)   if x < 0: x = 65535 + x + 1 bin_str = '' bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(15, 11, -1)) bin_str += ' ' bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(11, 7, -1)) bin_str += ' ' bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(7, 3, -1)) bin_str += ' ' bin_str += ''.join(x & (1 << i) and '1' or '0' for i in range(3, -1, -1)) return bin_str
 * 1) source: http://www.gossamer-threads.com/lists/python/python/645216

Integer to Binary String: def bin(x): if x 65535: raise Exception("Out of Range Error: %d" % x)   bin_str = '' bin_str = ''.join(x & (1 << i) and '1' or '0' for i in range(15, 7, -1)) bin_str = bin_str + ' ' bin_str = bin_str + ''.join(x & (1 << i) and '1' or '0' for i in range(7, -1, -1)) return bin_str
 * 1) source: http://www.gossamer-threads.com/lists/python/python/645216

print bin(0)     # '00000000 00000000' print bin(1)     # '00000000 00000001' print bin(255)   # '00000000 11111111' print bin(256)   # '00000001 00000000' print bin(43690) # '10101010 10101010' print bin(65535) # '11111111 11111111' print bin(65536) # '00000000 00000000' - rolled over to next set!

Integer to Binary: bin(170) # '0b10101010'

Binary to Integer: int('10101010', 2) # 170

bitwise
See

BitwiseOperators - Python Wiki - http://wiki.python.org/moin/BitwiseOperators

The Operators:

x << y   Returns x with the bits shifted to the left by y places (and new bits on the right-hand-side are zeros). This is the same as multiplying x by 2**y. x >> y   Returns x with the bits shifted to the right by y places. This is the same as //'ing x by 2**y. x & y   AND - Does a "bitwise and". Each bit of the output is 1 if the corresponding bit of x AND of y is 1, otherwise it's 0. x | y   OR - Does a "bitwise or". Each bit of the output is 0 if the corresponding bit of x AND of y is 0, otherwise it's 1. ~ x   Returns the complement of x - the number you get by switching each 1 for a 0 and each 0 for a 1. This is the same as -x - 1. x ^ y   XOR - Does a "bitwise exclusive or". Each bit of the output is the same as the corresponding bit in x if that bit in y is 0, and it's the complement of the bit in x if that bit in y is 1.

Script Path
if __name__ == "__main__": # Change directory to where this script lives so we can find relative # paths from here regardless of where the working directory was when we   # ran the script. # APP_DIR = os.path.dirname(sys.argv[0]) APP_DIR = os.path.dirname(os.path.abspath(sys.argv[0)) if APP_DIR: os.chdir(APP_DIR)

normpath(path) # Normalize path, eliminating double slashes, etc. abspath(path)   # Return an absolute path

Path to a module: (couldn't fine a_module) import a_module import os path = os.path.dirname(a_module.__file__)

Path to module: import inspect, os print inspect.getfile(inspect.currentframe) # script filename (usually with path) print os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe))) # script directory

Even better: print os.path.realpath(__file__) print os.path.dirname(__file__) print os.path.basename(__file__)

references:
 * http://stackoverflow.com/questions/595305/python-path-of-script

keywords: scriptpath script path program path pathname

Singleton
GLOBAL OBJECT!

None is a singleton (Python only has one copy of None in the interpreter).

Make your own singleton:

override the new method like this: class Singleton(object): _instance = None def __new__(cls, *args, **kwargs): if not cls._instance: cls._instance = super(Singleton, cls).__new__(                               cls, *args, **kwargs) return cls._instance

if __name__ == '__main__': s1=Singleton s2=Singleton if(id(s1)==id(s2)): print "Same" else: print "Different"

Using decorator -implementation from PEP318? Implementing the singleton pattern with a decorator: def singleton(cls): instances = {} def getinstance: if cls not in instances: instances[cls] = cls return instances[cls] return getinstance

@singleton class MyClass: ...


 * Python and the Singleton Pattern - Stack Overflow - http://stackoverflow.com/questions/42558/python-and-the-singleton-pattern

XML
from xml.dom.minidom import parse, parseString

dom1 = parse('c:\\temp\\mydata.xml') # parse an XML file by name

datasource = open('c:\\temp\\mydata.xml') dom2 = parse(datasource)  # parse an open file

dom3 = parseString(' Some data some more data ')

Pretty Print: import xml.dom.minidom

xml = xml.dom.minidom.parse(xml_fname) # or xml.dom.minidom.parseString(xml_string) pretty_xml_as_string = xml.toprettyxml

Pretty Print with etree import lxml.etree as etree

x = etree.parse("filename") print etree.tostring(x, pretty_print = True)

References:
 * 19.7. xml.dom.minidom — Lightweight DOM implementation — Python v2.7.3 documentation - http://docs.python.org/library/xml.dom.minidom.html
 * Pretty printing XML in python - Stack Overflow - http://stackoverflow.com/questions/749796/pretty-printing-xml-in-python

---

Simply don't include self.output.startElement and self.output.endElement for elements you don't want:

simple_filter_parser.py: (my version of ) import os import sys import optparse import fnmatch from sys import argv import xml.sax, xml.sax.saxutils
 * 1) this parses out the time element of a gpx for Google Earth

class my_filter_xml(xml.sax.ContentHandler): def __init__(self, outfile, verbose=False): self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8') self.indent = 0 self.ignore = False self.output.startDocument

def startElement(self, name, attrs): print " " * self.indent + "start: " + name if attrs.keys: keys = " " * (self.indent + 2) + "attrs:" for key in attrs.keys: keys = keys + " " + key + "=" + attrs[key] print keys self.indent = self.indent + 2 #self.output.startElement(name, {}) if name == "time": self.ignore = True #if not self.ignore: self.output.startElement(name, attrs)

def characters(self, ch): if ch.strip: print " " * self.indent + "ch: " + ch.strip if not self.ignore: self.output.characters(ch) else: if not self.ignore: self.output.characters("\n")

def endElement(self, name): self.indent = self.indent - 2 print " " * self.indent + "/end: " + name #if not self.ignore: self.output.endElement(name) if name == "time": self.ignore = False

if __name__ == '__main__': source = open("tracks.gpx") dest = open("tracks_out.gpx", 'w+') xml.sax.parse(source, my_filter_xml(dest, True))

---

fixdescriptor.py: import os import sys import optparse import fnmatch from sys import argv import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler): def __init__(self, outfile, verbose=False): self._outfile = outfile self._verbose = verbose self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8') self._filter = [ 'payload', 'payloads', 'file-list', 'file', 'checksum' ] self._keep = [ 'vib', 'maintenance-mode' ] self._inside = None

def startElement(self, name, attrs): # Certain elements are kept unchanged if name in self._keep: if self._verbose: print '%s element start found' % (name) self.output.startElement(name, {}) if name == 'maintenance-mode': if self._verbose: print 'inside maintenance-mode' self._inside = name return # Other elements are filtered (removed) elif name in self._filter: if self._verbose: print '%s element start found' % (name) self._inside = name return self.output.startElement(name, attrs)

def characters(self, ch): if self._inside == 'maintenance-mode': if self._verbose: print 'new content for maintenance-mode' self.output.characters('false') elif self._inside not in self._filter: self.output.characters(ch)

def endElement(self, name): # If it's not a checksum element, only print it       if name in self._filter + self._keep: self._inside = None if self._verbose: print '%s element end found' % (name) if name in self._filter: return self.output.endElement(name)

if __name__ == '__main__': #   parser = optparse.OptionParser(usage='%prog ...') parser.add_option('-v', '--verbose',                      action='store_true', default=False,                       help='Print more information.') parser.add_option('--input-xml',                      type='str', default='descriptor.xml',                       help='Name of input XML [default: %default]') parser.add_option('--output-xml',                      type='str', default='fixed_descriptor.xml',                       help='Name of output XML [default: %default]') opts, args = parser.parse_args(argv[1:]) source = open(opts.input_xml) dest = open(opts.output_xml, 'w+') xml.sax.parse(source, fix_descriptor(dest, opts.verbose))

--

Simple read and output parsing: import os import sys import optparse import fnmatch from sys import argv import xml.sax, xml.sax.saxutils

class fix_descriptor(xml.sax.ContentHandler): def __init__(self, outfile, verbose=False): self._outfile = outfile self._verbose = verbose self.output = xml.sax.saxutils.XMLGenerator(outfile, 'utf-8') self._inside = None

def startElement(self, name, attrs): # attrs.items is a list of tuples (name, value) print "start:", name, "/", attrs.items self.output.startElement(name, attrs)

def characters(self, ch): print "char:", self._inside, "/", ch       self.output.characters(ch)

def endElement(self, name): print "end:", name self.output.endElement(name)

if __name__ == '__main__': parser = optparse.OptionParser(usage='%prog ...') parser.add_option('-v', '--verbose',                      action='store_true', default=False,                       help='Print more information.') parser.add_option('--input-xml',                      type='str', default='descriptor.xml',                       help='Name of input XML [default: %default]') parser.add_option('--output-xml',                      type='str', default='fixed_descriptor.xml',                       help='Name of output XML [default: %default]') opts, args = parser.parse_args(argv[1:]) source = open(opts.input_xml) dest = open(opts.output_xml, 'w+') xml.sax.parse(source, fix_descriptor(dest, opts.verbose))

Unicode
Unicode string: u'Hello World !'

Raw Unicode string: >>> ur'Hello\u0020World !' u'Hello World !'

Convert: u = u'hello\xe4' str(u) # UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 5: ordinal not in range(128) s = u.encode('utf8') # 's' is now type str

References:
 * Introduction to Unicode Strings - http://docs.python.org/tutorial/introduction.html#unicode-strings

Zip
See Python/Zip

signals
signal.SIGTERM : kill  [pid] signal.SIGALRM : kill -ALRM  [pid] signal.SIGHUP  : kill -HUP  [pid]

import signal, os

def handler(signum, frame): print 'Signal handler called with signal', signum raise IOError("Couldn't open device!")

signal.signal(signal.SIGALRM, handler) signal.alarm(5)
 * 1) Set the signal handler and a 5-second alarm

fd = os.open('/dev/ttyS0', os.O_RDWR)
 * 1) This open may hang indefinitely

signal.alarm(0)         # Disable the alarm

Note: use signal.SIGTERM to handle "kill" command

References:
 * 17.4. signal — Set handlers for asynchronous events — Python v2.7.5 documentation - http://docs.python.org/2/library/signal.html

Ctrl+C
import signal import sys def signal_handler(signal, frame): print 'You pressed Ctrl+C!' sys.exit(0) signal.signal(signal.SIGINT, signal_handler) print 'Press Ctrl+C'
 * 1) !/usr/bin/env python

signal.pause
 * 1) linux only:


 * 1) windows alternative:
 * 2) import time
 * 3) time.sleep(60)

References:
 * control - How do I capture SIGINT in Python? - Stack Overflow - http://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python

Non-blocking raw_input for Python
Non-blocking raw_input for Python:
 * 1) source: http://www.garyrobinson.net/2009/10/non-blocking-raw_input-for-python.html

import signal

class AlarmException(Exception): pass

def alarmHandler(signum, frame): raise AlarmException

def nonBlockingRawInput(prompt='', timeout=20): signal.signal(signal.SIGALRM, alarmHandler) signal.alarm(timeout) try: text = raw_input(prompt) signal.alarm(0) return text except AlarmException: print '\nPrompt timeout. Continuing...' signal.signal(signal.SIGALRM, signal.SIG_IGN) return ''

email
import smtplib import email.mime.text

msg_text = "Hello World" msg = email.mime.text.MIMEText(msg_text)

msg['Subject'] = 'The contents of %s' % textfile msg['From'] = me msg['To'] = you
 * 1) me == the sender's email address
 * 2) you == the recipient's email address

s = smtplib.SMTP('localhost') s.sendmail(me, [you], msg.as_string) s.quit
 * 1) Send the message via our own SMTP server, but don't include the
 * 2) envelope header.

my nagios fake /bin/mail:
 * 1) !/usr/bin/env python

import smtplib import email.mime.text import optparse import os import sys import pwd

os.system('logger mail called')

parser = optparse.OptionParser parser.add_option(   '-s', '--subject',    help='Subject',    default="") parser.add_option(   '-t', '--to',    help='To',    dest="send_to",    default=None) parser.add_option(   '-f', '--from',    help='From',    dest="send_from",    default=None) parser.add_option(   '-S', '--server',    help='server',    default="prime") (options, args) = parser.parse_args

def get_username: return pwd.getpwuid( os.getuid )[ 0 ]

if options.send_to: send_to = options.send_to else: if len(args) == 1: send_to = args[0] else: parser.print_help sys.exit(1)

if options.send_from: send_from = options.send_from else: username = get_username hostname = os.uname[1] send_from = username + "@" + hostname

msg_text = sys.stdin.read msg = email.mime.text.MIMEText(msg_text)

msg['Subject'] = options.subject msg['From'] = send_from msg['To'] = send_to

s = smtplib.SMTP(options.server) s.sendmail(send_from, [send_to], msg.as_string) s.quit

Authenticated (with gmail): import smtplib import email.mime.text

from_email = "Kenneth Burgener <kenneth@demo.oeey.com>" to_emails = ["Kenneth Burgener <kenneth@demo.oeey.com>", "Kenneth Cell <xxx@xxx.com>"]
 * 1) from_email = "kenneth@demo.oeey.com"
 * 2) to_email = "kenneth@demo.oeey.com"

msg_text = "hello world" msg = email.mime.text.MIMEText(msg_text) msg['Subject'] = 'Test text' msg['From'] = from_email msg['To'] = ", ".join(to_emails) print "sending..." s = smtplib.SMTP_SSL('smtp.gmail.com', '465') s.login('xxx', 'xxx') s.sendmail(from_email, to_emails, msg.as_string) s.quit print "sent!"
 * 1) msg['To'] = to_email
 * 1) s = smtplib.SMTP('localhost')
 * 1) s.sendmail(from_email, [to_email], msg.as_string)

Source:
 * 20.12. smtplib — SMTP protocol client — Python v2.7.5 documentation - http://docs.python.org/2/library/smtplib.html
 * 18.1.11. email: Examples — Python v2.7.5 documentation - http://docs.python.org/2/library/email-examples.html

pprint
8.18. pprint — Data pretty printer — Python v2.7.5 documentation - http://docs.python.org/2/library/pprint.html

import pprint pprint.pprint(myVar)

import pprint pp = pprint.PrettyPrinter(indent=4) pp.pprint(myVar)

Note: Useless on objects!

Dump object: def dumpobject(obj): print "<<<---" print type(obj), obj.__class__.__name__ for attr in dir(obj): if attr[0] == "_": continue # ignore under/dunder properties if str(type(getattr(obj, attr))) == "<type 'instancemethod'>": print attr, ": METHOD" else: print attr, ":", type(getattr(obj, attr)), ":", getattr(obj, attr) print "END --->>>"

Curses
Curses Programming with Python — Python v3.3.2 documentation - http://docs.python.org/3/howto/curses.html

"The curses library supplies a terminal-independent screen-painting and keyboard-handling facility for text-based terminals; such terminals include VT100s, the Linux console, and the simulated terminal provided by various programs. Display terminals support various control codes to perform common operations such as moving the cursor, scrolling the screen, and erasing areas. Different terminals use widely differing codes, and often have their own minor quirks.

In a world of graphical displays, one might ask “why bother”? It’s true that character-cell display terminals are an obsolete technology, but there are niches in which being able to do fancy things with them are still valuable. One niche is on small-footprint or embedded Unixes that don’t run an X server. Another is tools such as OS installers and kernel configurators that may have to run before any graphical support is available."

---

Thread from Utah Python User's Group mailing list:

-

Hello, I am trying to learn more about curses in python for a desktop application. Does anybody have any recommendations on tutorials or documentation?

-Corban

-

We've use urwid at work. Seems to be a good approach if you aren't already tied to something. Learning curve can be a bit steep (well, compared to what you'd think it should be). Good luck!

http://excess.org/urwid/

Gabriel Gunderson

-

This video series looks pretty good:

http://www.youtube.com/user/pythoncursestutorial?feature=watch

Official HOWTOs from the python.org site:

http://docs.python.org/2/howto/curses.html http://docs.python.org/3/howto/curses.html

As Gabe mentioned, there are other libraries that provide similar or greater functionality, so if you aren't doing this specifically to learn curses, it's worth at least looking at a few of them to compare.

There's a brief comparison of three possibilities here:

http://stackoverflow.com/questions/8349085/python-ncurses-cdk-urwid-difference

Daniel Fackrell

-

Here's another option that looks promising:

https://pypi.python.org/pypi/blessings/

Daniel Fackrell

dialog
pythondialog - http://pythondialog.sourceforge.net/
 * "pythondialog is a Python wrapper for the dialog utility originally written by Savio Lam, and later rewritten by Thomas E. Dickey. Its purpose is to provide an easy to use, pythonic and comprehensive Python interface to dialog. This allows one to make simple text-mode user interfaces on Unix-like systems (including Linux). "

Note: The latest versions (2.12 and later) only support Python 3; users who really want to stick to Python 2 should use version 2.11 for now.

Install:
 * 1) latest:
 * 2) wget http://sourceforge.net/projects/pythondialog/files/latest/download?source=dlp

wget http://sourceforge.net/projects/pythondialog/files/pythondialog/2.11/python2-pythondialog-2.11.tar.gz/download tar -zvxf python2-pythondialog-2.11.tar.gz cd pythondialog-2.11

import dialog d = dialog.Dialog if d.yesno("Continue?") == d.DIALOG_OK: print "Continuing..."

---

My solution

kiloforce / dialog / source / — Bitbucket - https://bitbucket.org/kiloforce/dialog


 * 1) !/usr/bin/env python

""" Wrapper for Linux 'dialog' command """

__author__ = "Kenneth Burgener <kenneth@k.ttak.org>" __copyright__ = "(c) 2013" __date__ = "$Sep 27, 2013$" __version__ = "1.0" __credits__ = r""" XXXXXXXXXXXXXXXXXX X Dialog Wrapper # """

import subprocess import time import sys

class Dialog: """ Python wrapper for Linux 'dialog' command

dialog - display dialog boxes from shell scripts

>>> import dialog >>> dlg = dialog.Dialog >>> rc = dlg.msgbox('hello world')

Return codes: YES = 0 OK = 0 EXIT = 0 NO = 1 CANCEL = 1 ESC = 255 """

YES = 0 OK = 0 EXIT = 0 NO = 1 CANCEL = 1 ESC = 255

def __init__(self): rc = self._exe('which dialog > /dev/null')[0] if rc != 0: print "Error: failed to find Linux dialog command" sys.exit(1)

def _exe(self, cmd, strip=True): p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE) output = "\n".join(p.stderr.readlines) if strip: output = output.strip rc = p.wait return (rc, output)

def rc_to_str(self, rc): if rc == 0: return "YES" elif rc == 1: return "NO" elif rc == 255: return "ESX" else: raise Exception("Unknown return code: %s" % rc)

def clear(self): """       Clear screen

Example: >>> dialog.clear"""       rc, output = self._exe('clear')        assert rc in (0,), "clear rc"        return rc

def msgbox(self, msg): """       Simple message box

@param msg (str) = display message @return: rc (int)

Example: >>> rc = dialog.msgbox('hello world')"""       rc, output = self._exe('dialog --msgbox "{msg}" 0 0'.format(msg=msg))        assert rc in (self.YES, self.ESC), "msgbox rc"        return rc

def yesno(self, msg): """       Yes/No dialog

@param msg (str) = display message @return: rc (int)

Example: >>> rc = dialog.yesno('continue')"""       rc, output = self._exe('dialog --yesno "{msg}" 0 0'.format(msg=msg))        assert rc in (self.YES, self.NO, self.ESC), "yesno rc"        return rc

def infobox(self, msg): """       Dialog that does not wait for user's response        (do your own pause)

@param msg (str) = display message @return: rc (int) always zero

Example: >>> for i in range(10): >>>  dialog.infobox('counting: ' + str(i)) >>>  time.sleep(3)"""        rc, output = self._exe('dialog --infobox "{msg}" 0 0'.format(msg=msg))        assert rc in (self.YES,), "infobox rc"        return rc

def inputbox(self, msg): """       Get user input

@param msg (str) = display message @return: rc (int), output (str/None)

Example: >>> rc, output = dialog.inputbox('name:')"""       rc, output = self._exe('dialog --inputbox "{msg}" 0 0'.format(msg=msg))        assert rc in (self.YES, self.NO, self.ESC), "inputbox rc"        if not output:            output = None        return rc, output

def textbox(self, filename): """       Display a file

@param filename (str) = file to display @return: rc (int)

Example: >>> dialog.textbox('/etc/passwd')"""       rc, output = self._exe('dialog --textbox {filename} 0 0'.format(filename=filename))        assert rc in (self.YES, self.ESC), "textbox rc"        return rc

def menu(self, msg, item_list): """       Single selection checkbox menu

@param msg (str) = display message @param item_list (str list) = list of items to display @return: rc (int), selection (int/None)

Example: >>> rc, selection = dialog.menu('Select multiple options:',       ...     ('option 1', 'option 2'))"""        item_str = ""        for i in range(len(item_list)):            item_str += " " + str(i+1) + " \"" + item_list[i] + "\""        rc, output = self._exe('dialog --menu "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))        if output:            selection = int(output)        else:            selection = None        assert rc in (self.YES, self.NO, self.ESC), "menu rc"        return rc, selection

def checklist(self, msg, item_list, checked_list=None): """       Multiple selection checkbox menu

@param msg (str) = display message @param item_list (str list) = list of items @param checked_list (int list) = default items to check @return: rc (int), selections (int list/None)

Example: >>> rc, selections = dialog.checklist('Select multiple options:',       ...     ('option 1', 'option 2')        """        item_str = ""        assert type(item_list) in (type(None), type(), type([])), "checklist: item_list must be a list"        assert type(checked_list) in (type(None), type(), type([])), "checklist: checked_list must be a list"        for i in range(len(item_list)):            item_str += " " + str(i+1) + " \"" + item_list[i] + "\""            if checked_list and i+1 in checked_list:                item_str += " on"            else:                item_str += " off"        rc, output = self._exe('dialog --checklist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))        print item_str        selections = []        for choice in output.split('"'):            if choice.strip:                selections.append(int(choice))        if selections:            selections = tuple(selections) else: selections = None assert rc in (self.YES, self.NO, self.ESC), "checklist rc" return rc, selections

def radiolist(self, msg, item_list, checked_item=None): """       Single selection radio menu, with default

@param msg (str) = display message @param item_list (str list) = list of items @param checked_item (int) = default item to select @return: rc (int), selection (int)

Example: >>> rc, selection = dialog.radiolist('Select one option:',       >>>     ('option 1', 'option 2'), 2)"""        item_str = ""        for i in range(len(item_list)):            item_str += " " + str(i+1) + " \"" + item_list[i] + "\""            if checked_item and i+1 == int(checked_item):                item_str += " on"            else:                item_str += " off"        rc, selection = self._exe('dialog --radiolist "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str))        if not selection:            selection = None        assert rc in (self.YES, self.NO, self.ESC), "radiolist rc"        return rc, selection

def gauge(self, msg, percent): """       Progress bar (does not wait for user's input)

@param msg (str) = display message @param percent (int) = progress @return: rc (int) always zero

Example: >>> for i in range(0, 100+1, 2): >>>  dialog.gauge('Progress:', i)        >>>   time.sleep(.1)"""        rc, output = self._exe('echo {percent} | dialog --gauge "{msg}" 0 0'.format(percent=percent, msg=msg))        assert rc in (self.YES,), "guage rc"        return rc

def progress(self, msg, percent): """       Alias for self.guage        """ return self.gauge(msg=msg, percent=percent)

def form(self, msg, item_list): """       Multiple item entry form

@param msg (str) = display message @param item_list (str list) = list of items @return: rc (int), answers (str list)

Example: >>> rc, answers = dialog.form('Enter details',       ...     ('first name', 'last name'))"""        item_str = ""        name_len = 1        for i in range(len(item_list)):            if len(item_list[i])+3 > name_len:                name_len = len(item_list[i])+3        for i in range(len(item_list)):            item_str += " \"" + item_list[i] + ":\" " + str(i+1) + " 1 \"\" " + str(i+1) +\                " " + str(name_len) + " 40 0"        rc, answers = self._exe('dialog --form "{msg}" 0 0 0 {item_str}'.format(msg=msg, item_str=item_str),                                strip=False)        answers = answers.split('\n\n')        if not answers:            answers = None        assert rc in (self.YES, self.NO, self.ESC), "form rc"        return rc, answers

def test(self): """       Test features

>>> import dialog >>> dlg = dialog.Dialog >>> dlg.test """

def wait_user(msg): self.msgbox("%s" % msg)

# rc_to_str wait_user("Testing return code...") print self.rc_to_str(self.YES) # YES print self.rc_to_str(self.NO) # NO        print self.rc_to_str(self.ESC)  # ESX try: print self.rc_to_str(111) # exception raise Exception("Did not throw exception") except Exception as rc: pass

# msgbox wait_user("Testing msgbox...") rc = self.msgbox("msgbox") if rc not in (self.YES, self.ESC): raise Exception("Unknown return code for msgbox: %s" % rc)

# yesno wait_user("Testing yesno...") rc = self.yesno("yesno") if rc not in (self.YES, self.NO, self.ESC): raise Exception("Unknown return code for yesno: %s" % rc)

# infobox wait_user("Testing infobox...") rc = self.infobox("infobox sleep 1 sec") if rc not in (self.YES,): raise Exception("Unknown return code for infobox %s" % rc) time.sleep(1)

# inputbox wait_user("Testing inputbox...") rc, output = self.inputbox("inputbox") if rc not in (self.YES, self.NO, self.ESC): raise Exception("Unknown return code for inputbox: %s" % rc) wait_user("Result: %s" % output)

# textbox wait_user("Testing textbox...") rc = self.textbox("/etc/passwd") if rc not in (self.YES, self.ESC): raise Exception("Unknown return code for textbox: %s" % rc)

# menu wait_user("Testing menu...") rc, selection = self.menu("menu", ("option 1", "option 2")) if rc not in (self.YES, self.NO, self.ESC): raise Exception("Unknown return code for menu: %s" % rc) wait_user("Result: %s" % selection)

# checklist wait_user("Testing checklist...") rc, selection = self.checklist("menu", ["option 1", "option 2"]) if rc not in (self.YES, self.NO, self.ESC): raise Exception("Unknown return code for checklist: %s" % rc) wait_user("Result: %s" % str(selection)) wait_user("Testing checklist with default selection...") rc, selection = self.checklist("menu", ["option 1", "option 2"], [1, 2]) if rc not in (self.YES, self.NO, self.ESC): raise Exception("Unknown return code for checklist: %s" % rc) wait_user("Result: %s" % str(selection))

# radiolist wait_user("Testing radiolist...") rc, selection = self.radiolist("menu", ["option 1", "option 2"]) if rc not in (self.YES, self.NO, self.ESC): raise Exception("Unknown return code for radiolist: %s" % rc) wait_user("Result: %s" % str(selection)) wait_user("Testing radiolist with default selection...") rc, selection = self.radiolist("menu", ["option 1", "option 2"], 2) if rc not in (self.YES, self.NO, self.ESC): raise Exception("Unknown return code for radiolist: %s" % rc) wait_user("Result: %s" % str(selection))

# guage wait_user("Testing guage...") for i in range(4+1): rc = self.gauge("test guage", i*25) # rc = self.progress("test progress", i*25) if rc not in (self.YES,): raise Exception("Unknown return code for guage: %s" % rc) time.sleep(1)

# form wait_user("Testing form...") rc, answers = self.form("form", ["option 1", "option 2"]) if rc not in (self.YES, self.NO, self.ESC): raise Exception("Unknown return code for form: %s" % rc) wait_user("Result: %s" % str(answers))

# done with test, cleanup wait_user("Done with tests...") self.clear

if __name__ == "__main__": # dialog = Dialog # dialog.test pass

Busy Loop and Thread.yield
"I want to tell my Python threads to yield, and so avoid hogging the CPU unnecessarily. In Java, you could do that using the Thread.yield function"

Solution: time.sleep(0) time.sleep(0.00001) time.sleep(epsilon)

References:
 * multithreading - In there something similar to Java's Thread.yield in Python? Does that even make sense? - Stack Overflow - http://stackoverflow.com/questions/1908206/in-there-something-similar-to-javas-thread-yield-in-python-does-that-even-ma
 * multithreading - How does a threading.Thread yield the rest of its quantum in Python? - Stack Overflow - http://stackoverflow.com/questions/787803/how-does-a-threading-thread-yield-the-rest-of-its-quantum-in-python

USB
Programming with PyUSB 1.0 - http://pyusb.sourceforge.net/docs/1.0/tutorial.html

PyUSB - http://pyusb.sourceforge.net/

git clone https://github.com/walac/pyusb

import usb.core dev = usb.core.find

dev = usb.core.find(idVendor=0xfffe, idProduct=0x0001)
 * 1) find our device

"Dealing with multiple identical devices - Sometimes you may have two identical devices connected to the computer. How can you differentiate them? Device objects come with two additional attributes which are not part of the USB Spec, but are very useful: bus and address attributes. First of all, it is worth to say that these attributes come from the backend and a backend is free to not support them, in which case they are set to None. That said, these attributes represent the bus number and bus address of the device and, as you might already have imagined, can be used to differentiate two devices with the same idVendor and idProduct attributes."

Tkinter
Python GUI Bitcoin: import json import urllib2 from Tkinter import *

url='http://api.bitcoincharts.com/v1/markets.json' req=urllib2.Request(url) response=urllib2.urlopen(req).read output=json.loads(response)

mtusd_price = output[-10]['avg'] mteur_price = output[31]['avg'] mtgbp_price = output[44]['avg']

MTUSD= 'USD' + ' ' + str(mtusd_price) MTEUR= 'EUR' + ' ' + str(mteur_price) MTGBP = 'GBP' + ' ' + str(mtgbp_price)

root= Tk root.title("Bitcoin Price") root.geometry("250x100")

USD = Label(root, text=MTUSD) EUR = Label(root, text=MTEUR) GBP = Label(root, text=MTGBP) USD.pack EUR.pack GBP.pack mainloop

ctypes
"ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python."
 * -- 15.17. ctypes — A foreign function library for Python — Python v2.7.7 documentation - https://docs.python.org/2/library/ctypes.html

Use "c scope" to build a database of symbols to see what options are available.

---

testlib.c
 * 1) include <stdio.h>

void myprint(void);

void myprint {   printf("hello world\n"); }

Compile: gcc -shared -Wl,-soname,testlib -o testlib.so -fPIC testlib.c

testlibwrapper.py

Python: import ctypes

testlib = ctypes.CDLL('/full/path/to/testlib.so') testlib.myprint

Source: python - ctypes - Beginner - Stack Overflow - http://stackoverflow.com/questions/5081875/ctypes-beginner

---

nzjrs's Gists - https://gist.github.com/nzjrs

---


 * 1) include <stdio.h>

int sum(double *x,int n) { int i; double counter; counter = 0; for(i=0;i<n;i++) {     counter=counter+x[i];

} return counter; }

gcc -c sum.c gcc -shared -o sum.so sum.o

from ctypes import * my_sum=CDLL('sum.so') a=numpy.array(range(10),dtype=float) my_sum.sum(a.ctypes.data_as(c_void_p),int(10))

a=(c_double*10) for i in range(10): a[i]=i my_sum.sum(a,int(10))

Source: Ctypes — Numerical Computing with Sage v6.2 - http://www.sagemath.org/doc/numerical_sage/ctypes.html

---

Warnings
if sys.version_info >= (1, 0, 0): warnings.warn("Danger Will Robinson: trying Phi on untested Python 3.0!")

enum
Python 2: class Enum(set): """ Basic enumerated type   Example        >>> Animals = phi.Enum(["DOG", "CAT", "HORSE"])        >>> Animals.DOG        'DOG'        >>> 'HORSE' in Animals        True    """ def __getattr__(self, name): if name in self: return name raise AttributeError("Missing Enum property: %s" % name) def __setattr__(self, name, value): # this makes it read-only raise AttributeError("Unable to modify Enum property: %s (%s)" % (name, value))

Animals = Enum(["DOG", "CAT", "HORSE"])

print(Animals.DOG) # 'DOG'

--

Alternative: class Stationery: Pen, Pencil, Eraser = range(0, 3)

One liner: class Enum(tuple): __getattr__ = tuple.index

>>> State = Enum(['Unclaimed', 'Claimed']) >>> State.Claimed 1 >>> State[1] 'Claimed' >>> State ('Unclaimed', 'Claimed') >>> range(len(State)) [0, 1] >>> [(k, State[k]) for k in range(len(State))] [(0, 'Unclaimed'), (1, 'Claimed')] >>> [(k, getattr(State, k)) for k in State] [('Unclaimed', 0), ('Claimed', 1)]

Python 3 supports enum natively with the enum.Enum library:

pySerial
https://pythonhosted.org/pyserial/

"This module encapsulates the access for the serial port. It provides backends for Python running on Windows, OSX, Linux, BSD (possibly any POSIX compliant system) and IronPython. The module named “serial” automatically selects the appropriate backend."

https://github.com/pyserial/

Intro - https://pythonhosted.org/pyserial/shortintro.html

---

import serial import time

ser = serial.Serial(   port='COM4',\    baudrate=9600,\    parity=serial.PARITY_NONE,\    stopbits=serial.STOPBITS_ONE,\    bytesize=serial.EIGHTBITS,\        timeout=0)

print("connected to: " + ser.portstr)

seq = [] count = 1
 * 1) this will store the line

while True: for c in ser.read: seq.append(chr(c)) #convert from ANSII joined_seq = ''.join(str(v) for v in seq) #Make a string from array

if chr(c) == '\n': print("Line " + str(count) + ': ' + joined_seq) seq = [] count += 1 break

ser.close

Ref: http://stackoverflow.com/questions/16077912/python-serial-how-to-use-the-read-or-readline-function-to-read-more-than-1-char

---

I modified the code to monitor a COM port: (Windows)

Monitor Serial:

import serial, time, sys

logger = open("serial.txt", "w") timer = time.time

def log(msg, head=False): msg = msg.strip if head: msg = ">>> " + msg logger.write(msg + "\n") sys.stdout.write(msg + "\n") newtime = time.time if newtime > timer + 1: logger.flush sys.stdout.flush

while True:

ser = None try: ser = serial.Serial(           port='COM1',\            baudrate=115200,\            parity=serial.PARITY_NONE,\            stopbits=serial.STOPBITS_ONE,\            bytesize=serial.EIGHTBITS,\                timeout=0) except serial.serialutil.SerialException as e:       log(str(e), True) time.sleep(1) continue

log("Connected to: " + ser.portstr, True) count=1

try: line = [] while True: for c in ser.readline: line.append(c) if c == "\n": log("".join(line)) line = [] except serial.serialutil.SerialException as e:       log(str(e), True) time.sleep(1) ser.close

md5
import md5 md5.new("Nobody inspects the spammish repetition").hexdigest
 * 1) 'bb649c83dd1ea5c9d9dec9a18df0ffe9'

$ echo -n "Nobody inspects the spammish repetition" | md5sum bb649c83dd1ea5c9d9dec9a18df0ffe9 -

execnet
Python Package Index - execnet - https://pypi.python.org/pypi/execnet

Examples - http://codespeak.net/execnet/examples.html
 * basic local and remote communication - http://codespeak.net/execnet/example/test_info.html

Execute source code in subprocess, communicate through a channel: >>> import execnet >>> gw = execnet.makegateway >>> channel = gw.remote_exec("channel.send(channel.receive+1)") >>> channel.send(1) >>> channel.receive 2

Compare current working directories >>> import execnet, os >>> gw = execnet.makegateway >>> ch = gw.remote_exec("import os; channel.send(os.getcwd)") >>> res = ch.receive >>> assert res == os.getcwd

Get information from remote ssh account >>> import execnet, os >>> gw = execnet.makegateway("ssh=codespeak.net") >>> channel = gw.remote_exec(""" ...    import sys, os ...     channel.send((sys.platform, sys.version_info, os.getpid)) ... """) >>> platform, version_info, remote_pid = channel.receive >>> platform 'linux2' >>> version_info (2, 4, 2, 'final', 0)

remote-exec a function (avoiding inlined source part I) import execnet

def multiplier(channel, factor): while not channel.isclosed: param = channel.receive channel.send(param * factor)

gw = execnet.makegateway channel = gw.remote_exec(multiplier, factor=10)

for i in range(5): channel.send(i) result = channel.receive assert result == i * 10

gw.exit

remote-exec a module (avoiding inlined source part II)
 * "You can pass a module object to remote_exec in which case its source code will be sent. No dependencies will be transferred so the module must be self-contained or only use modules that are installed on the “other” side. Module code can detect if it is running in a remote_exec situation by checking for the special __name__ attribute."


 * 1) content of a module remote1.py

if __name__ == '__channelexec__': channel.send('initialization complete')

>>> import execnet, remote1 >>> gw = execnet.makegateway >>> ch = gw.remote_exec(remote1) >>> print (ch.receive) initialization complete

Use a callback instead of receive and wait for completion >>> import execnet >>> gw = execnet.makegateway >>> channel = gw.remote_exec("for i in range(10): channel.send(i)") >>> l = [] >>> channel.setcallback(l.append, endmarker=None) >>> channel.waitclose # waits for closing, i.e. remote exec finish >>> l [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]

a simple command loop pattern import os
 * 1) contents of: remotecmd.py

def simple(arg): return arg + 1

def listdir(path): return os.listdir(path) if __name__ == '__channelexec__': for item in channel: channel.send(eval(item))

>>> import execnet, remotecmd >>> gw = execnet.makegateway >>> ch = gw.remote_exec(remotecmd) >>> ch.send('simple(10)') # execute func-call remotely >>> ch.receive 11

evdev
Install: apt-get install python-setuptools python-dev easy_install pip pip install evdev

Simple read usage: from evdev import InputDevice from evdev import ecodes

DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd"
 * 1) look for a /dev/input/by-id/usb...kbd or similar

dev = InputDevice(DEVICE)

while True: r, w, x = select([dev], [], [], 5) for event in dev.read: if event.type == 1 and event.value == 1: print event.code

List LED states: dev.leds dev.leds(verbose=True)

Inject usage: (does not appear to really work) from evdev import InputDevice from evdev import ecodes from evdev import UInput

DEVICE = "/dev/input/by-id/usb-ORTEK_USB_Keyboard_Hub-event-kbd" dev = InputDevice(DEVICE)

ui = UInput

ui.write(ecodes.EV_KEY, ecodes.KEY_4, 1) ui.write(ecodes.EV_KEY, ecodes.KEY_4, 0) ui.syn

ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 1) ui.write(ecodes.EV_KEY, ecodes.KEY_NUMLOCK, 0) ui.syn

ui.close

Tutorial: References:
 * evdev documentation — python-evdev 0.4.3 documentation - http://pythonhosted.org/evdev/index.html
 * python-evdev/doc/tutorial.rst at master · gvalkov/python-evdev · GitHub - https://github.com/gvalkov/python-evdev/blob/master/doc/tutorial.rst

ldap-python
python-ldap: LDAP client API for Python - http://www.python-ldap.org/

Index of Packages : Python Package Index - https://pypi.python.org/pypi/python-ldap/

yum install python-devel openldap-devel pip install python-ldap

Articles:
 * Python LDAP Applications: Part 1 - Installing and Configuring the Python-LDAP Library and Binding to an LDAP Directory | Packt Publishing - http://www.packtpub.com/article/installing-and-configuring-the-python-ldap-library-and-binding-to-an-ldap-directory

xmmp
PDF:
 * XMPP: The Definitive Guide - Oreilly - http://oriolrius.cat/blog/wp-content/uploads/2009/10/Oreilly.XMPP.The.Definitive.Guide.May.2009.pdf

xmmp is the Jabber protocol

Best options:

MIT libraries.
 * sleekxmpp (was: sleekxmpp) is pretty popular and is used for examples in Peter Saint-Andre's XMPP book from O'Reilly. It reportedly works well, and finally got an email list in April 2010, and has a chat room at sleek@conference.jabber.org.

GPL libraries.
 * xmpppy is used by gajim.

LPGL libraries.
 * pyxmpp is pretty good and uses libxml2 internally for xml parsing.
 * pyxmpp2 is the next version of pyxmpp, runs on python 2.7 and 3.2, and removes the libxml2 requirement. Like many, it requires dnspython.
 * jabberpy is the original and still works for a lot of tasks, but is thoroughly unmaintained.

Other libraries.
 * Wokkel, mentioned in another post. That's a new one for me, based on twisted.

chat - Best python XMPP / Jabber client library? - Stack Overflow - http://stackoverflow.com/questions/1901828/best-python-xmpp-jabber-client-library

xmpppy
xmpppy: the jabber python project - http://xmpppy.sourceforge.net/

"xmpppy is a Python library that is targeted to provide easy scripting with Jabber. Similar projects are Twisted Words and jabber.py.

This library was not designed from scratch. It inherits some code from jabberpy and have very similar API in many places. Though it is separate project since it have almost completely different architecture and primarily aims to work with jabberd2 - the new Open Source Jabber Server.

xmpppy is distributed under the terms of GNU General Public License and can be freely redistributed without any charge."

wget http://sourceforge.net/projects/xmpppy/files/xmpppy/0.4.1/xmpppy-0.4.1.tar.gz/download -O xmpppy-0.4.1.tar.gz tar -zvxf xmpppy-0.4.1.tar.gz pip install pydns

--- basic connection ---

import xmpp user="username@gmail.com" password="password" server="gmail.com" jid = xmpp.JID(user) connection = xmpp.Client(server,debug=[]) connection.connect result = connection.auth(jid.getNode, password,"LFY-client") connection.sendInitPresence while connection.Process(1): pass
 * 1) !/usr/bin/env python

References:
 * Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- basic receive ---

Print messages when received: import xmpp user="username@gmail.com" password="password" server="gmail.com" def message_handler(connect_object, message_node): message = "Welcome to my first Gtalk Bot :)"       connect_object.send( xmpp.Message( message_node.getFrom ,message)) jid = xmpp.JID(user) connection = xmpp.Client(server) connection.connect result = connection.auth(jid.getNode, password, "LFY-client") connection.RegisterHandler('message', message_handler) connection.sendInitPresence while connection.Process(1):    pass
 * 1) !/usr/bin/env python

References:
 * Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- remote-control shell bot ---

Replace the simple bot’s message_handler function with this new one:

def message_handler(connect_object,message_node): command = str(message_node.getBody) process = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE) message = process.stdout.read if message=="": message=process.stderr.read connect_object.send( xmpp.Message( message_node.getFrom ,message))

References:
 * Use XMPP to Create Your Own Google Talk Client - LINUX For You - http://www.linuxforu.com/2012/06/use-xmpp-to-create-your-own-google-talk-client/

--- roster ---

import xmpp import sys

userID  = 'myname@gmail.com' password = 'mypassword' ressource = 'Script'

jid = xmpp.protocol.JID(userID) jabber = xmpp.Client(jid.getDomain, debug=[])

connection = jabber.connect(('talk.google.com',5222)) auth = jabber.auth(jid.getNode, password, ressource)

jabber.sendInitPresence(requestRoster=1) myroster = jabber.getRoster

To get status, you have to wait for the other clients to send you their status: jabber.RegisterHandler('presence', myPresenceHandler)

def myPresenceHandler(con, event): fromjid = event.getFrom.getStripped status = myroster.getStatus(fromjid)

References: python - XMPP chat: accessing contacts' status messages with xmppPy's Roster - Stack Overflow - http://stackoverflow.com/questions/2381597/xmpp-chat-accessing-contacts-status-messages-with-xmpppys-roster

--- full roster script ---

kparal/jabber-roster · GitHub - https://github.com/kparal/jabber-roster

A simple Python tool for listing your Jabber roster contacts. You can use it to easily backup list of your buddies.

This program is maintained, but not further developed. Bugs will be fixed, but no new features will be added. If you want to work on this program, don't hesitate to contact me, I will gladly assign you to the development team.

git clone https://github.com/kparal/jabber-roster python jabber_roster.py

https://pypi.python.org/pypi/jabber-roster

--- vcard ---

This information is not in the roster. You will need to query the clients individually and get their vCard by sending this IQ :

<iq from='stpeter@jabber.org/roundabout' id='v1' type='get'> <vCard xmlns='vcard-temp'/> </iq>

My solution: def vcard(disp, jid): msg = xmpp.protocol.Iq msg.setType('get') msg.setTo(jid) qc = msg.addChild('vCard') qc.setAttr('xmlns', 'vcard-temp') rep = disp.SendAndWaitForResponse(msg) # to see what other fields are available in the XML output: # print rep userid=fname=lname=title=department=region=None for i in rep.getChildren: for j in i.getChildren: if j.getName == "TITLE": title = j.getData.encode('utf-8') for k in j.getChildren: if k.getName == "GIVEN": fname = k.getData.encode('utf-8') if k.getName == "FAMILY": lname = k.getData.encode('utf-8') if k.getName == "ORGUNIT": department = k.getData.encode('utf-8') if k.getName == "REGION": region = k.getData.encode('utf-8') return fname, lname, title, department, region

References:
 * xmpppy - Retrieve gtalk nickname in python xmpp - Stack Overflow - http://stackoverflow.com/questions/4539477/retrieve-gtalk-nickname-in-python-xmpp
 * XEP-0054: vcard-temp - http://xmpp.org/extensions/xep-0054.html

google-api-python-client
Google APIs Client Library for Python, which includes:
 * google-api-python-client: The core Python library for accessing Google APIs.
 * oauth2client: A Python client library for OAuth 2.0.
 * Sample applications using google-api-python-client and oauth2client

-

https://developers.google.com/api-client-library/python/start/installation
 * Calendar API - Python/Command Line

- Download: wget https://code.google.com/p/google-api-python-client/downloads/detail?name=google-api-python-client-1.2.zip wget https://code.google.com/p/google-api-python-client/downloads/detail?name=oauth2client-1.2.zip

Git: git clone https://github.com/google/google-api-python-client git clone https://github.com/google/oauth2client

References:
 * google-api-python-client - Google APIs Client Library for Python - Google Project Hosting - https://code.google.com/p/google-api-python-client/

pydoc
pydoc - http://docs.python.org/library/pydoc.html
 * Documentation generator and online help system

Doxygen and Doxypy
See Python/Doxygen

Sphinx
Sphinx - http://sphinx.pocoo.org/

"Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license.

It was originally created for the new Python documentation, and it has excellent facilities for the documentation of Python projects, but C/C++ is already supported as well, and it is planned to add special support for other languages as well. Of course, this site is also created from reStructuredText sources using Sphinx!"

epydoc
epydoc - http://epydoc.sourceforge.net/

"Epydoc is a tool for generating API documentation for Python modules, based on their docstrings. For an example of epydoc's output, see the API documentation for epydoc itself (html, pdf). A lightweight markup language called epytext can be used to format docstrings, and to add information about specific fields, such as parameters and instance variables. Epydoc also understands docstrings written in reStructuredText, Javadoc, and plaintext. For a more extensive example of epydoc's output, see the API documentation for Python 2.5. "

Python 3 Wall of Superpowers
Python 3 Wall of Superpowers - https://python3wos.appspot.com/

"As listed on PyPI - packages in red don't support python 3, packages in green do. Hopefully one day everything will be greener. "

qualnames
New in 3.3 >>> def f: ...  def g: ...    class H: pass ...    return H ... return g ...

>>> H=f

>>> # 3.3: >>> H <class '__main__.f. .g. .H'> >>> H.__qualname__ 'f. .g. .H'

>>> # 2.6: >>> H <class __main__.H at 0x2b885044b890> >>> H.__qualname__ AttributeError: class H has no attribute '__qualname__'

Python 2 or Python 3
Python2orPython3 - PythonInfo Wiki - http://wiki.python.org/moin/Python2orPython3
 * Short version: Python 2.x is the status quo, Python 3.x is the shiny new thing.
 * At the time of writing (July 4, 2010), the final 2.7 release is out, with a statement of extended support for this end-of-life release. The 2.x branch will see no new major releases after that. 3.x is under active and continued development, with 3.1 already available and 3.2 due for release around the turn of the year.

Python 2.7.1 Release - http://www.python.org/download/releases/2.7.1/

What’s New In Python 3.0 — Python v3.0.1 documentation - http://docs.python.org/release/3.0.1/whatsnew/3.0.html
 * print requires parenthesis
 * raw_input is now input

Python for Windows
Python for Windows - http://www.python.org/download/windows/

PyCharm
http://www.pycharm.com/

Hide Console
fork does not work on windows. So what else can we do?

How to hide console window in python? - http://stackoverflow.com/questions/764631/how-to-hide-console-window-in-python
 * "Simply save it with a .pyw extension. This will prevent the console window from opening. Explanation at the bottom of section "

32bit vs 64bit
IDLE:

32bit: C:\Python27>python Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

64bit: (note: default path is C:\Python27) C:\Python27-64bit>python.exe Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.

Task Manager: python.exe *32

python.exe

Platform: >>> import platform >>> platform.python_compiler 'MSC v.1500 32 bit (Intel)'

>>> import platform >>> platform.python_compiler 'MSC v.1500 64 bit (AMD64)'

Code: import struct def is_python_64bit: return (struct.calcsize("P") == 8)

Python ZIP Application
Python can be run from a single bundled zip file - similar to a Java WAR! Python has been able to (Allow interpreter to execute a zip file http://bugs.python.org/issue1739468)

"Python quietly added a new feature in 2.6 that makes it possible to bundle up a directory full of Python code into a single executable file." 

When a python zip application is executed, it will execute '__main__.py'

mkdir app touch app/__main__.py cd app zip -r ../app.zip * cd .. python app.zip

Load a resource - The easiest way to find and load a program bundle like this is to use the pkg_resources module. - http://packages.python.org/distribute/pkg_resources.html


 * 1) file: app/__main__.py

import pkg_resources

def main: print('The rain in Spain falls mainly in the plain.') print(pkg_resources.resource_string('resources', 'inFrance.txt'))

if __name__ == '__main__': main

Now use a bit of UNIX magic to turn app2.zip into an executable: echo '#!/usr/bin/env python' | cat - app2.zip > app2 chmod +x app2

"The zip file format is designed to allow a small executable program to be inserted at the front (that's how self-extracting zip files are created), so this is kosher and doesn't corrupt the zip file." 

Now you can simply run app2 like any executable. $ ./app2

References:
 * Able Pear Software: Bundling Python files into a stand-alone executable - http://blog.ablepear.com/2012/10/bundling-python-files-into-stand-alone.html
 * PEP 441 -- Improving Python ZIP Application Support - http://www.python.org/dev/peps/pep-0441/
 * Issue 1739468: Allow interpreter to execute a zip file - Python tracker - http://bugs.python.org/issue1739468
 * Package Discovery and Resource Access using pkg_resources — Distribute documentation - http://packages.python.org/distribute/pkg_resources.html

Shell Python Loader
Came before the Python ZIP Application support.

Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe

"This takes your Python application and packages it into a single executable shell script. You application can be made of up multiple python scripts. The executable shell script is also compressed, so the resulting script is smaller than the original. This should work on any system that has Python and a Bourne shell (works on Cygwin too). Note that this does not package the Python interpreter into the executable. This packages only you python code into a zip file, but this still allows you to distribute your application as a single shell script that will run on any system with Python installed. In theory, you could zip the .pyc files, but the byte code is guaranteed to run only on exactly the same version Python interpreter (including minor version). It is safer to just package the .py files."

First you need to zip all your python files: zip main.zip main.py spam.py eggs.py

zipheader.sh: PYTHON=$(which python 2>/dev/null) if [ ! -x "$PYTHON" ] ; then echo "Python not found!" exit 1 fi exec $PYTHON -c " import sys, os if int(sys.version[0])<2:   print 'Python version 2.3 final or greater is required.'    print 'Your version is', sys.version    os._exit(1) major = sys.version_info[0] minor = sys.version_info[1] releaselevel = sys.version_info[3] if (major==2 and minor<3) or (major==2 and minor==3 and releaselevel!='final'):    print 'Python version 2.3 final or greater is required.'    print 'Your version is', sys.version    os._exit(1) sys.path.insert(0, sys.argv[1]) del sys.argv[0:1] import main main.main " $0 $@
 * 1) !/bin/sh
 * 2) This is a self-extracting executable.
 * 3) Execute this like any normal executable.
 * 4) You may need to "chmod a+x" this file.
 * 5) This is a binary ZIP file with a Python loader header.
 * 6) Bourne shell loader:
 * 1) Bourne shell loader:
 * 1) Python loader:
 * 1) Zip file:

Concatinate zipheader.sh with the zip file. cat zipheader.sh main.zip > main

Set main as executable. chmod +x main

"This is an improvement on Joerg Raedler's Python recipe, 215301. The original closes stdin. The herefile in the shell script redirects stdin before python gets a chance to start. This disables raw_input and anything else that reads sys.stdin. Unfortunately, Raedler's boot script closes stdin, which is a fairly big limitation. The herefile in the shell script (END_OF_PYTHON_CODE) redirects stdin before python starts; stdin is closed at the end of the herefile. This disables raw_input and anything else that reads sys.stdin. "

References:
 * Python zip exe - Noah.org - http://www.noah.org/wiki/Python_zip_exe
 * Build a compressed self-extracting executable script on UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/497000-build-a-compressed-self-extracting-executable-scri/
 * binding main skript and modules to one executable with python-2.3 under UNIX « Python recipes « ActiveState Code - http://code.activestate.com/recipes/215301-binding-main-skript-and-modules-to-one-executable-/

py2exe
py2exe - http://www.py2exe.org/
 * "py2exe is a Python Distutils extension which converts Python scripts into executable Windows programs, able to run without requiring a Python installation. "

Tutorial - http://www.py2exe.org/index.cgi/Tutorial

Note: Install the version that matches your version of python

Note: run installer "as administrator"

setup.py from distutils.core import setup import py2exe

setup(console=['hello.py'])

No console window version: from distutils.core import setup import py2exe

setup(windows=['hello.py'])

Build: python setup.py py2exe

pygame is more complex: http://www.pygame.org/wiki/Pygame2exe

Python for Windows Extensions - PyWin32
Python for Windows Extensions - http://starship.python.net/~skippy/win32/
 * There is a build for each python version and architecture (32bit vs 64bit)

Follow event viewer import win32evtlog # install Python for Windows Extensions - http://starship.python.net/~skippy/win32/

server = "localhost" source = "System" handle = win32evtlog.OpenEventLog(server, source)
 * 1) open event viewer

flags = win32evtlog.EVENTLOG_BACKWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ events = win32evtlog.ReadEventLog(handle, flags, 0)
 * 1) seek to end of log

flags = win32evtlog.EVENTLOG_FORWARDS_READ | win32evtlog.EVENTLOG_SEQUENTIAL_READ win32evtlog.ReadEventLog(handle, flags, 0) # ignore initial logs
 * 1) return to normal reading direction

while True: events = win32evtlog.ReadEventLog(handle, flags, 0) if events: for event in events: data = event.StringInserts if data: # Example: 07/03/14 12:34:35 - MYEVENTSOURCE - 0 - 1 - 4 "HELLO" print event.TimeGenerated, '-', event.SourceName, '-', event.EventCategory, '-', event.EventID, '-', event.EventType, '-', '"%s"' % " ".join(data) time.sleep(.5) # yield cpu
 * 1) loop through events

References:
 * PyWin32: Getting Windows Event Logs « The Mouse Vs. The Python - http://www.blog.pythonlibrary.org/2010/07/27/pywin32-getting-windows-event-logs/

PyCrypto for Windows
Download Install: http://www.voidspace.org.uk/python/modules.shtml#pycrypto

Installation: easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win32-py2.7.exe easy_install http://www.voidspace.org.uk/downloads/pycrypto26/pycrypto-2.6.win-amd64-py2.7.exe

Paramiko for Windows

 * 1) Install PyCrypto (above)
 * 2) pip install paramiko

PyGame for Windows
Download - http://www.pygame.org/download.shtml
 * http://pygame.org/ftp/pygame-1.9.1.win32-py2.7.msi

Indentation
Python: Myths about Indentation - http://www.secnetix.de/olli/Python/block_indentation.hawk
 * There are quite some prejudices and myths about Python's indentation rules among people who don't really know Python. I'll try to address a few of these concerns on this page.


 * "Whitespace is significant in Python source code."
 * "Python forces me to use a certain indentation style."
 * "You cannot safely mix tabs and spaces in Python."
 * "I just don't like it."
 * "How does the compiler parse the indentation?"

PEP - Python Enhancement Proposals
Python Enhancement Proposals (PEPs)


 * Index of Python Enhancement Proposals (PEPs) - http://www.python.org/dev/peps/

Convert Tabs to Spaces
Convert tabs to 4 spaces: sed -i 's/\t/   /g'  script.py

Linting

 * pep8 - python style guide checker
 * pychecker - more verbose, imports code, slower
 * pylint - most verbose, configurable, "rates" code
 * pyflakes - least verbose (dead/redundant code)

rpmforge: yum install pyflakes pychecker
 * 1) pylint - not sure which pacakge includes

pep8
See Python/pep8

pychecker
See Python/pychecker

pylint
See Python/pylint

PEP8 Style Guidlines
PEP 8 -- Style Guide for Python Code - http://www.python.org/dev/peps/pep-0008/


 * Use spaces - Use 4 spaces per indentation level.
 * Never mix tabs and spaces.
 * Limit all lines to a maximum of 79 characters.
 * Line wrapping - The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses.
 * Encoding - For Python 3.0 and beyond, UTF-8 is preferred over Latin-1, see PEP 3120.
 * Imports should usually be on separate lines
 * Avoid extraneous whitespace in most situations
 * Always use 'self' for the first argument to instance methods.

PEP8 Style Checker
See

Benefits of Python
What is the return on investment for this group by learning Python? The list of benefits below sums up what the language has to offer:
 * Python is designed from the ground up to encourage maintainable, readable and reusable code. It has great support for particular paradigms such as object-oriented programming.
 * Python code is usually around a quarter of the length of code generated by C++ or Java - less typing, less debugging and less maintenance.
 * Python does not need to be compiled or linked, which allows for a more iterative programming style and a quicker development cycle.
 * Python is extremely portable. Code can be easily exchanged between Windows, Mac OS and Linux with little to no modification.
 * Python comes with a staggering array of standard libraries and mature third-party projects. Web development, scientific computing, operator system access and network scripting are all supported.
 * Python can talk with many other languages easily. It has support for invoking C and C++ code and can itself be called from libraries written in these languages.

Source: http://quantstart.com/articles/Quant-Reading-List-Python-Programming/ - Quant Reading List Python Programming

Benefits of Python:
 * Python is available for most operating systems, including Windows, UNIX, Linux, and Mac OS.
 * Python's clean object-oriented design and extensive support libraries offer two to ten fold the programmer productivity seen with languages like C, C++, C#, Java, VB, and Perl.

Source: Python Development - Benefits of Python - Digital Mesh - http://www.digitalmesh.com/offshore-development-center/python-development/benefits-of-python.html

Install Module
Installing Python Modules — Python v2.7.1 documentation - http://docs.python.org/install/index.html

The new standard: Distutils python setup.py install

Download and extract target package, then: sudo python setup.py install

Note: Running setup.py install builds and installs all modules in one run

Help: python setup.py --help

Incremental method: python setup.py build python setup.py install python setup.py build --help  # help

Force rebuild and install: python setup.py build --force   # if have to repeat python setup.py install --force # if have to repeat

Default installation paths: Platform	Standard installation location		Default value Unix (pure)	prefix/lib/pythonX.Y/site-packages	/usr/local/lib/pythonX.Y/site-packages Unix (non-pure)	exec-prefix/lib/pythonX.Y/site-packages	/usr/local/lib/pythonX.Y/site-packages Windows		prefix					C:\Python

Alternate install path: python setup.py install --home=

Alternate base folders: Type of file			Installation Directory	Override option pure module distribution	home/lib/python		--install-purelib non-pure module distribution	home/lib/python		--install-platlib scripts				home/bin		--install-scripts data				home/share		--install-data

Modifying Python’s Search Path: import sys sys.path.append('/www/python/')

Python Package Index
user-submitted Python packages: http://pypi.python.org/pypi

sudo python setup.py install

Package Site: http://pypi.python.org/pypi/[PACKAGE] http://pypi.python.org/pypi/test

If they are on http://pypi.python.org/: pip install foo

[http://stackoverflow.com/questions/410163/for-python-programmers-is-there-anything-equivalent-to-perls-cpan For Python programmers, is there anything equivalent to Perl's CPAN? - Stack Overflow]

Easy Installation
EasyInstall - The PEAK Developers' Center - http://peak.telecommunity.com/DevCenter/EasyInstall
 * Easy Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages.

CheeseShopTutorial - PythonInfo Wiki - http://wiki.python.org/moin/CheeseShopTutorial
 * EasyInstall (easy_install) gives you a quick and painless way to install packages remotely by connecting to the Package Index or even other websites via HTTP. It is somewhat analogous to the CPAN and PEAR tools for Perl and PHP, respectively.

pip
See Python/pip

Create PIP Package
See Python/pip

Distribute
distribute - Easily download, build, install, upgrade, and uninstall Python packages

distribute 0.6.24 : Python Package Index - http://pypi.python.org/pypi/distribute

Installation: mkdir -p ~/.src ; cd ~/.src wget http://pypi.python.org/packages/source/d/distribute/distribute-0.6.24.tar.gz#md5=17722b22141aba8235787f79800cc452 tar -zvxf distribute-0.6.24.tar.gz cd distribute-0.6.24 sudo python setup.py install

Test: echo "import distribute" | grep python

Named Parameters
http://diveintopython.org/power_of_introspection/optional_arguments.html

Python allows function arguments to have default values; if the function is called without the argument, the argument gets its default value. Futhermore, arguments can be specified in any order by using named arguments. Stored procedures in SQL Server Transact/SQL can do this, so if you're a SQL Server scripting guru, you can skim this part.

Here is an example of info, a function with two optional arguments:

def info(object, spacing=10, collapse=1):

spacing and collapse are optional, because they have default values defined. object is required, because it has no default value. If info is called with only one argument, spacing defaults to 10 and collapse defaults to 1. If info is called with two arguments, collapse still defaults to 1.

Say you want to specify a value for collapse but want to accept the default value for spacing. In most languages, you would be out of luck, because you would need to call the function with three arguments. But in Python, arguments can be specified by name, in any order.

http://www.wellho.net/mouth/1871_Optional-and-named-parameters-in-Python.html

If you add a parameter with two asterixes (**) on the end of a function definition, that parameter is taken as being the name of a dictionary into which all otherwise unidentified parameters are stored.

def getnet(gross, taxrate=17.5, **others): net = gross / (1.0 + taxrate*0.01) for more in others.keys: print "GETNET:",more," - ",others[more] return neto

Python vs other languages
Python is Cool (and Perl is not), Especially for C/C++ Programmers

Experimenting with Memory Remanence
Experimenting with Memory Remanence

#!/usr/bin/env python

# a pirate's favorite chemical element a = "" while 1: a += "ARGON"

sudo strings /dev/mem | less

UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)
See UTOSC 2008 - 90 percent of the Python you need to know (in 90 minutes)

Program Layout Best Practices Presentation
Program layout best practices - Seth House <seth@eseth.com> - Utah Django User Group - 2011-06-23
 * Slides - https://github.com/whiteinge/presentations/tree/master/upyug_2011-06-09_program-layout
 * Video - http://blip.tv/utah-open-source/python-layout-best-practices-for-modules-django-cli-utah-django-user-group-5315808

Module skeleton:
 * 1) !/usr/bin/env python
 * 2) coding: utf-8

"""Module docstring"""

__author__ = ’Mr. Me <me@example.net>’ __version__ = ’1.2.3’

if __name__==’__main__’: main

Executing modules as scripts python -m SimpleHTTPServer
 * 1) PEP 338:

CLI package
cli — command line tools — pyCLI v1.1.1 documentation - http://packages.python.org/pyCLI/

Python IDE
IntegratedDevelopmentEnvironments - PythonInfo Wiki - http://wiki.python.org/moin/IntegratedDevelopmentEnvironments
 * IDEs with introspection-based code completion and integrated debugger
 * IDEs with introspection-based code completion /or/ integrated debugger
 * IDEs with integrated gui builder
 * Editors
 * Reviews

Is there a good, free Python IDE for Windows? - Stack Overflow - http://stackoverflow.com/questions/126753/is-there-a-good-free-python-ide-for-windows
 * Eclipse with PyDev
 * Aptana Studio 3
 * Active State also offers Komodo Edit
 * WingWare - http://www.wingware.com/
 * Eric IDE - http://www.die-offenbachs.de/eric/index.html
 * MMM-Experts - Products - http://mmm-experts.com/Products.aspx?ProductId=4
 * PyScripter is a Python IDE built in Python for Delphi(P4D) components and created with the ambition to become competitive in functionality with commercial Windows-based IDEs available for other languages. Being built in a compiled language is rather snappier than some of the other Python IDEs and provides an extensive blend of features that make it a productive Python development environment.
 * ActiveState Python comes with PythonWin which is reasonably good
 * PyCharm from JetBrains is not free in general, but has free early access program.
 * SPE - Stani's Python Editor, is pretty awesome. See http://pythonide.blogspot.com/.
 * You may want to take a look at the pdb module - http://docs.python.org/lib/module-pdb.html
 * Geany (http://www.geany.org/) is a fairly light text editor that does a lot of IDE-like stuff.
 * SciTE: http://www.scintilla.org/SciTE.html
 * Editra: http://editra.org/
 * Netbeans has support for python development in their Python Early Access version. - http://www.netbeans.org/features/python/
 * I use Notepad++ with the appropriate settings.
 * The Eric Python IDE - http://www.die-offenbachs.de/eric/index.html
 * BOa Constructor www.boa-constructor.sourceforge.net really nice! drag and drop stuff!
 * DrPtyhon is also very nice, and has many plug-ins. http://drpython.sourceforge.net/
 * Vim
 * I will make a plug for my preference of Python development environments: SPE. SPE is an integrated development environment that provides code completion, integrated Python shell, calltips, and various helper tools like notes and todo lists. It also includes wxGlade so you can make wxPython applications easily. Plus, it includes the python debugger WinPdb.
 * http://groups.google.com.tr/group/python_opengl this software is my sample visual python ide.its demo version.
 * I use PyScripter, and find it pretty awesome.
 * There is one Python IDE written in Java: jHepWork: http://jwork.org/jhepwork/ it can detect Python's indentation rules rather nicely (+ detects structure of a code and syntax)
 * gvim built with your python version support and with this plugins:
 * NERDTree (don't read he's blog ;)
 * matchit (it's builtin in new versions just put a symlink or junction)
 * pyflakes
 * pysmell
 * Try Python Tools for Visual Studio. Supports CPython and IronPython. - http://pytools.codeplex.com/

PythonEditors - PythonInfo Wiki - http://wiki.python.org/moin/PythonEditors
 * List of editors

Python Ide - http://c2.com/cgi/wiki?PythonIde
 * There are a lot of Ides for PythonLanguage springing up. Many of them written in Python

Is there a better Python IDE? « Lennart Regebro: Python, Plone, Web - http://regebro.wordpress.com/2010/10/09/is-there-a-better-python-ide/
 * I’m currently a user of WingIDE, but sometimes I feel the need for something better, mainly something that is a bit more stable. WingIDE often hangs when switching projects, sometimes the file listing decides that you must double-click on the expand arrow instead of just click on it, etc. It still doesn’t have macros despite this being a much requested feature (how hard can it be to add, really?) and I never really get the hang of the SVN integration, so I just ignore it. And they keep recommending me to use WingIDE’s separate instance of GTK instead of the System GTK, which is daft. Their designs are ugly, and it should work with the system GTK so it looks the same. Anything else is just silly.

Open Komodo
Open Komodo - http://www.openkomodo.com/
 * Open Komodo is the source code repository for Komodo Edit - a free multi-language editor for dynamic languages, based on the award-winning Komodo IDE. ActiveState created the Open Komodo code repository in August 2007 and it has been used since then for building all Komodo Edit releases.

Eclipse
Build an Eclipse development environment for Perl, Python, and PHP - http://www.eclipse.org/resources/resource.php?id=485
 * This tutorial shows how Eclipse's DLTK makes it possible to build development tools for scripting languages. In particular, it explains how to implement syntax coloring, user preferences, and interpreter integration in a plug-in-based project.

See

EasyEclipse
EasyEclipse - http://www.easyeclipse.org/
 * EasyEclipse packages together Eclipse, the open-source software

development platform, and selected open source plugins.
 * We select, assemble, test, patch, build installers and document

a full IDE, offered as reliable distributions and plugins.
 * EasyEclipse is:


 * Free and open-source,
 * Easy to download and install, and
 * Simple to maintain, without version and dependency issues.

EasyEclipse for Python - http://www.easyeclipse.org/site/distributions/python.html

EclipsePythonIntegration - PythonInfo Wiki - http://wiki.python.org/moin/EclipsePythonIntegration
 * Using Eclipse as a Python editor

PyDev
PyDev - http://pydev.org/
 * PyDev is a Python IDE for Eclipse, which may be used in Python, Jython and IronPython development.

---

Pydev - http://pydev.sourceforge.net/ (old link)

How to Configure Eclipse for Python - http://www.rose-hulman.edu/class/csse/resources/Eclipse/eclipse-python-configuration.htm

Python Development with PyDev and Eclipse - Tutorial - http://www.vogella.de/articles/Python/article.html
 * This article describes how to write and debug Python programs with Eclipse

PyDev Getting Started - http://pydev.org/manual_101_root.html

Running a program - http://pydev.org/manual_101_run.html

---

Supported Task Tags:
 * TODO:
 * TASK:
 * FIXME:

---

Installation:
 * 1) Platform Runtime Binary (download around 45-50 MB) - http://download.eclipse.org/eclipse/downloads/ or http://www.eclipse.org/downloads/
 * 2) * Example: Eclipse Classic 4.2.2
 * 3) Eclipse Software Add - PyDev: http://pydev.org/updates
 * 4) Eclipse Software Add - MercurialEclipse: http://cbes.javaforge.com/update
 * 5) Switch perspectives to PyDev

The Eric Python IDE
The Eric Python IDE - http://eric-ide.python-projects.org/
 * Eric is a full featured Python and Ruby editor and IDE, written in python. It is based on the cross platform Qt gui toolkit, integrating the highly flexible Scintilla editor control. It is designed to be usable as everdays' quick and dirty editor as well as being usable as a professional project management tool integrating many advanced features Python offers the professional coder. eric4 includes a plugin system, which allows easy extension of the IDE functionality with plugins downloadable from the net.

Eric Python IDE - Wikipedia - http://en.wikipedia.org/wiki/Eric_Python_IDE

Wingware Python IDE
The Intelligent Development Environment for Python Programmers - Wingware Python IDE - http://www.wingware.com/
 * Wingware's Python IDE is an Integrated Development Environment designed specifically for the Python programming language. Since 1999, Wingware has focused on Python.

VIM
Python .vimrc: " Kenneth's Preferences

set nobackup           " don't create backup files "set nohlsearch          " don't highlight search results set hlsearch            " do highlight search results set incsearch          " incremental search set ignorecase          " ignore case during search set smartcase          " do case sensitive search when upper case is typed set shiftwidth=4        " shift blocks in 4 space increments set tabstop=4          " the (typewriter) tab stops are every 4 spaces set smarttab            " tab at beginning of line inserts 4 spaces (shiftwidth) set showmatch          " briefly highlight matching bracket set matchtime=1         " highlight match for 0.1 seconds set background=dark    " black background set shiftwidth=4        " shift blocks by 4 spaces set shortmess=at       " use short messages so we don't have to hit enter set expandtab           " expand tabs into spaces set scrolloff=5        " always show 5 lines before & after cursor set shell=bash          " use bash for the command shell set shellcmdflag=-lc   " make bash startup as a login shell for ~/.bashrc "set nowrap              " do not wrap long lines set visualbell          " flash the screen rather than beep set noerrorbells       " don't beep on errors set wildmenu            " enable filename tab completion set ruler              " display the ruler: shows cursor position set cpoptions+=$        " display '$' at end of cw sequence set title              " display filename and vim info in window title bar set modelines=5         " search the first and last 5 lines for "vim:" modeline set backspace=2        " backspace deletes

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

"if v:version > 700 "   set cursorline          " underline the current line "endif

if &t_Co > 2 || has( "gui_running" ) syntax on endif

if has( 'mouse' ) set mouse=a " enable mouse endif

if has( "autocmd" ) " open the file in the last place we were editing   autocmd BufReadPost *        \ if line("'\"") > 1 && line("'\"") <= line("$") |        \     exe "normal! g'\"" | \ endif endif

"folding settings set foldmethod=indent  "fold based on indent set foldnestmax=10     "deepest fold is 10 levels set nofoldenable        "dont fold by default set foldlevel=10       "this is just what i use

" Auto clear trailing white space in Python files func! DeleteTrailingWS   exec "normal mz"    %s/\s\+$//ge    exe "normal `z" endfunc autocmd BufWrite *.py :call DeleteTrailingWS

Older versions: " KENNETH SETTINGS set nocp               " turn off compatibility mode set nomodeline         " do not allow files to dictate vim behavior

set incsearch          " incremental search set ignorecase          " ignore case during search

set visualbell         " flash the screen rather than beep set noerrorbells        " don't beep on errors set background=dark    " black background syntax on               " turn on syntax highlighting set hlsearch           " highligh last search pattern

set nowrap             " do not wrap long lines set scrolloff=5         " always show 5 lines before & after cursor set ruler              " show the cursor position all the time set backspace=2         " allow backspace deletion set bs=indent,eol,start " allow backspacing over everything in insert mode

set shiftwidth=4       " shift blocks in 4 space increments set tabstop=4           " the (typewriter) tab stops are every 4 spaces set expandtab          " expand tab into spaces set smarttab            " tab at beginning of line inserts 4 spaces (shiftwidth) filetype indent on     " turn on auto indent for specific file types

Other options: set number             " show line numbers set numberwidth=1       " remove space before largest line numbers

set smartindent        " ?? - maybe included in 'filetype indent on'

set ai                 " always auto indent

if has( "autocmd" ) " open the file in the last place we were editing   autocmd BufReadPost *        \ if line("'\"") > 1 && line("'\"") <= line("$") |        \     exe "normal! g'\"" | \ endif endif

Kenneth's fio .vimrc:

"source $VIM/mswin.vim

set nobackup           " don't create backup files "set nohlsearch          " don't highlight search results set incsearch           " incremental search set ignorecase         " ignore case during search set smartcase           " do case sensitive search when upper case is typed set shiftwidth=4       " shift blocks in 4 space increments set tabstop=4           " the (typewriter) tab stops are every 4 spaces set smarttab           " tab at beginning of line inserts 4 spaces (shiftwidth) set showmatch           " briefly highlight matching bracket set matchtime=1        " highlight match for 0.1 seconds set background=dark     " black background set shiftwidth=4       " shift blocks by 4 spaces set shortmess=at        " use short messages so we don't have to hit enter set expandtab          " expand tabs into spaces set scrolloff=5         " always show 5 lines before & after cursor set shell=bash         " use bash for the command shell set shellcmdflag=-lc    " make bash startup as a login shell for ~/.bashrc "set nowrap             " do not wrap long lines set visualbell         " flash the screen rather than beep set noerrorbells        " don't beep on errors set wildmenu           " enable filename tab completion set ruler               " display the ruler: shows cursor position set cpoptions+=$       " display '$' at end of cw sequence set title               " display filename and vim info in window title bar set modelines=5        " search the first and last 5 lines for "vim:" modeline

set wildignore=*.o,*.class,*.pyc,*.pyo,*.a,*.so,*.dll,*.exe,core,*.jar,*.zip

"let loaded_matchparen = 1 " don't highlight parenthesis match

filetype indent on

if v:version > 700 set cursorline         " underline the current line endif

if &t_Co > 2 || has( "gui_running" ) syntax on endif

if has( 'mouse' ) set mouse=a " enable mouse endif

if has( "autocmd" ) " open the file in the last place we were editing   autocmd BufReadPost *        \ if line("'\"") > 1 && line("'\"") <= line("$") |        \     exe "normal! g'\"" | \ endif endif

"folding settings set foldmethod=indent  "fold based on indent set foldnestmax=10     "deepest fold is 10 levels set nofoldenable        "dont fold by default set foldlevel=10       "this is just what i use

Interpreter vs Script
Interpreter: >>> import os >>> os.uname ('Linux', 'demo.oeey.com', '2.6.18-194.8.1.el5', '#1 SMP Thu Jul 1 19:07:06 EDT 2010', 'i686')

Script: import os print os.uname
 * 1) !/usr/bin/env python

Web Development
WebProgramming - PythonInfo Wiki - http://wiki.python.org/moin/WebProgramming
 * This topic guide attempts to cover every aspect of programming Web applications (both clients and servers) using Python.

HOWTO Use Python in the web — Python v2.7.2 documentation - http://docs.python.org/howto/webservers.html
 * This document shows how Python fits into the web. It presents some ways to integrate Python with a web server, and general practices useful for developing web sites.

Plug Simple Python Web App
Original Message Subject: 	Re: Writing a simple python web app Date: 	Mon, 25 Oct 2010 19:03:31 -0400 From: 	Roberto Mello <roberto.mello@gmail.com> To: 	Provo Linux Users Group <plug@plug.org>

On Sat, Sep 11, 2010 at 5:40 PM, Dave Smith <dave@thesmithfam.org> wrote: > I'm creating a very simple web application, and I want to use python. My first guess was that mod_python would provide the easiest entry point. Boy, was I wrong. All the mod_python tutorials spend 80% of their time extolling the virtues of mod_python over CGI, but they are quite lean on specific examples. It's quite disappointing. A natural progression then leads me to mod_wsgi, and since that has equally poor documentation, to Django. However, Django is overkill for my app (I don't even have a database).

For something as simple as you have described, and to run on an embedded platform, I would probably stay away from frameworks designed to run on full hardware, and go with something more minimalistic.

Take a look at CherryPy [1] or web.py [2].

[1] http://www.cherrypy.org/ [2] http://webpy.org/

Roberto Mello

---

See also web.py

See also CherryPy

See also Django

Apache CGI
See CGI

Great tutorial: https://www.tutorialspoint.com/python/python_cgi_programming.htm

My favorite option, make all .cgi files executable: <Location /> AddHandler cgi-script .cgi Options +ExecCGI </Location>

To enable on Ubuntu: sudo a2enmod cgi service apache2 restart

Directory index: DirectoryIndex index.py

Or if you really want to go crazy, this will try to run *every* file: # this has problems with / directory being run as a script, instead of index too! #<Location /> #   SetHandler cgi-script #   Options +ExecCGI #</Location>

<Files ~ "."> SetHandler cgi-script Options +ExecCGI </Files>

or: <Files ~ "^.[a-zA-Z]*$"> SetHandler cgi-script Options +ExecCGI </Files>

Or just have index file: <Files "index">

Note: Place within a <VirtualHost> section if needed.

Note: Make sure to 'chmod +x' the files.

Start all HTML output with: print "Content-Type: text/html\n"

If you want just text output: print "Content-Type: text/plain\n"

GET (from Environment): print "&tl;pre>" for env in os.environ: print "%s = %s" % (env, os.environ[env]) print "&tl;/pre>"


 * 1) sample for: http://oeey.com/py/test.py/test?var1=ken&var2=barbie

REQUEST_URI = /py/index.py/test?var1=ken&var2=barbie QUERY_STRING = var1=ken&var2=barbie PATH_INFO = /test REQUEST_METHOD = GET

SCRIPT_NAME = /py/test.py SERVER_NAME = oeey.com REMOTE_ADDR = 216.51.42.66 SERVER_ADDR = 192.168.108.4 DOCUMENT_ROOT = /www/py SCRIPT_FILENAME = /www/py/test.py HTTP_HOST = cgi.oeey.com

POST (from stdin): post = sys.stdin.readline if post: print post

CONTENT_TYPE = application/x-www-form-urlencoded

yin=one+two+three&yang=four+five+six

cgi helper class: import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage

print "fields keys: ", form.keys # list of field keys
 * 1) field keys

if "name" in form: # if form.has_key('name'): print "field 'name' is: ", form["name"].value print "field 'name' is: ", form.getvalue('name')
 * 1) fild values

print "dictionary of headers: ", form.headers print form.type print form.headers['content-type']
 * 1) headers

To prevent XSS escape the characters: print cgi.escape(form.getvalue('name'))

import cgi

print "Content-Type: text/html\n\n"

form = cgi.FieldStorage if (form.has_key("action") and form.has_key("name") \       and form.has_key("age")): if (form["action"].value == "display"): print "%s = %s" % (form["name"].value form["age"].value)

Note: cgi helper class also does sessions and cookies.

References:
 * CGI Tutorial | Web Python - http://webpython.codepoint.net/cgi_tutorial
 * Python CGI: An Interactive Instruction - http://www.cs.virginia.edu/~lab2q/

See also:
 * Web Python - http://webpython.codepoint.net/

---

REST Example:

import os, re def get_rest: if not "PATH_INFO" in os.environ: return # filter for garbage #print os.environ["PATH_INFO"] path_info = re.findall('[/A-Za-z0-9_-]*', os.environ["PATH_INFO"]) if not path_info: return path_info = ''.join(path_info) # collapse empty paths paths = path_info.split('/') final_paths = [] for path in paths: if path: final_paths.append(path) return tuple(final_paths) rest = get_rest;

mod_python
AVOID! AVOID! AVOID! - if you want this simple, just use CGI!

See Apache mod_python

mod_python - http://docs.python.org/howto/webservers.html#mod-python

Install: yum install mod_python

Config: /etc/httpd/conf.d/python.conf

Simple:


 * "People coming from PHP often find it hard to grasp how to use Python in the web. Their first thought is mostly mod_python, because they think that this is the equivalent to mod_php. Actually, there are many differences. What mod_python does is embed the interpreter into the Apache process, thus speeding up requests by not having to start a Python interpreter for each request. On the other hand, it is not “Python intermixed with HTML” in the way that PHP is often intermixed with HTML. The Python equivalent of that is a template engine. mod_python itself is much more powerful and provides more access to Apache internals. It can emulate CGI, work in a “Python Server Pages” mode (similar to JSP) which is “HTML intermingled with Python”, and it has a “Publisher” which designates one file to accept all requests and decide what to do with them.


 * mod_python does have some problems. Unlike the PHP interpreter, the Python interpreter uses caching when executing files, so changes to a file will require the web server to be restarted. Another problem is the basic concept – Apache starts child processes to handle the requests, and unfortunately every child process needs to load the whole Python interpreter even if it does not use it. This makes the whole web server slower. Another problem is that, because mod_python is linked against a specific version of libpython, it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5) without recompiling mod_python. mod_python is also bound to the Apache web server, so programs written for mod_python cannot easily run on other web servers.


 * These are the reasons why mod_python should be avoided when writing new programs. In some circumstances it still might be a good idea to use mod_python for deployment, but WSGI makes it possible to run WSGI programs under mod_python as well."

WSGI
WSGI - http://docs.python.org/howto/webservers.html#mod-wsgi


 * "The Web Server Gateway Interface, or WSGI for short, is defined in PEP 333 and is currently the best way to do Python web programming. While it is great for programmers writing frameworks, a normal web developer does not need to get in direct contact with it. When choosing a framework for web development it is a good idea to choose one which supports WSGI."

See mod_wsgi

Django
See Django

CherryPy
See CherryPy

web.py
web.py: makes web apps

Think about the ideal way to write a web app. Write the code to make it happen.

web.py : Python Package Index - http://pypi.python.org/pypi/web.py

Home Page: http://webpy.org/

"Django lets you write web apps in Django. TurboGears lets you write web apps in TurboGears. Web.py lets you write web apps in Python."
 * — Adam Atlas

NOTE: Can modify source code without having to restart app! :-)

---

PIP Install: pip install web.py

---

Hello World: import web urls = (   '/(.*)', 'hello' ) app = web.application(urls, globals)

class hello: def GET(self, name): if not name: name = 'World' return 'Hello, ' + name + '!'

if __name__ == "__main__": app.run

Note: app.py can be renamed.

Running: python app.py

Note: You can add an IP address/port after the "app.py" bit to control where web.py launches the server. You can also tell it to run a fastcgi or scgi server. python app.py 1234

---

Templates

$def with (name)
 * 1) templates/hello.html

$if name: I just wanted to say hello to $name. $else: Hello, world!

app = ... render = web.template.render('templates/')

index.GET: name = 'Bob' return render.hello(name) # render.[TEMPLATE_NAME](vars)

get variable: (/?name=Joe) i = web.input(name=None) return render.hello(i.name)

variable in url: urls = ('/(.*)', 'hello')

class index: def GET(self, name): return render.hello(name)


 * 1) visit: /Joe

multiple variables in url: urls = ('/(.*)/(.*)', 'hello')

class hello: def GET(self, fname, lname): return render.hello(fname, lname)


 * 1) visit: /Joe/Smith

lpthw.web
lpthw.web: a fork of web.py for Learn Python The Hard Way

Locks web.py at a specific version and removes some magic. Thanks to Aaron Swartz for making it originally.

lpthw.web: a fork of web.py for Learn Python The Hard Way - http://pypi.python.org/pypi/lpthw.web

Home Page: http://learnpythonthehardway.org/

This conflicts with web.py, so only install one.

web
web - Web modules for CGI and WSGI web programming

web : Python Package Index - http://pypi.python.org/pypi/web

Warning: This is an alpha release for those who are interested in   checking progress. No documentation and not for production use.

Auth, session, mail, form and templating tools for writing CGI scripts or WSGI web applications.

This version incompatible with previous versions becuase large portions of the code have been refactored into separate components.

Import DeprecationWarning
Error: /opt/xmpppy-0.4.1/xmpp/auth.py:24: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha,base64,random,dispatcher,re /opt/xmpppy-0.4.1/xmpp/auth.py:26: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5

Hide deprecation warnings with code: import warnings

with warnings.catch_warnings: warnings.filterwarnings("ignore",category=DeprecationWarning) import md5, sha

Hide with header:
 * 1) !/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line: python -W ignore::DeprecationWarning script.py

References:
 * How to ignore deprecation warnings in Python - Stack Overflow - http://stackoverflow.com/questions/879173/how-to-ignore-deprecation-warnings-in-python

Funciton DeprecationWarning
Error: /opt/xmpppy-0.4.1/xmpp/transports.py:307: DeprecationWarning: socket.ssl is deprecated. Use ssl.wrap_socket instead. tcpsock._sslObj   = socket.ssl(tcpsock._sock, None, None)

Hide with header:
 * 1) !/usr/bin/env python -W ignore::DeprecationWarning

Hide from command line: python -W ignore::DeprecationWarning script.py