24 Application Programming Interfaces, Modules, Packages, Syntactic Sugar

We know that objects like lists have methods such as .sort() and .append(). Do they have others? They do, and one way to find them is to run Python in the interactive mode on the command line.^[1]

To do this, we execute the python interpreter without specifying a file to interpret. The result is a Python prompt, >>>, where we can type individual lines of Python code and see the results.

The interactive mode, aside from providing an interface for running quick tests of Python functionality, includes a help system! Simply run the help() function on either a class name (like help(list)) or an instance of the class.

This command opens an interactive viewer in which we can scroll to see all the methods that an object of that type provides. Here’s a sample from the help page:

Browsing through this documentation reveals that Python lists have many methods, including .append(), .count(), and others. Sometimes the documentation isn’t as clear as one would hope. In these cases, some experimentation may be required. Between the description above and some code tests, for example, can you determine what a list’s .count() method does and how to use it effectively?

The set of methods and instance variables belonging to an object or class is known as its API, or Application Programming Interface. The API is the set of functions, methods, or variables provided by an encapsulated collection of code. APIs are an important part of programming because they represent the “user interface” for programming constructs that others have written.

Sometimes it can be useful to determine what class or “type” to which a variable refers, especially for looking up the API for objects of that type. The type() function comes in handy when combined with a print() call—it prints the type (the class) of data referenced by a variable. The result can then be investigated with help().

Modules

Consider this chunk of code from the example in chapter 23, “Objects and Classes,” which makes use of the Chromosome and SNP class definitions:

This segment of code takes a file name and produces a dictionary with nicely organized contents of the file. Because this functionality is succinctly defined, it makes sense to turn this into a function that takes the file name as a parameter and returns the dictionary. The code is almost exactly the same, except for being wrapped up in a function definition.

Later, we can call this function to do all the work of parsing a given file name:

Now, what if this function and the two class definitions were things that we wanted to use in other projects? We may wish to make a module out of them—a file containing Python code (usually related to a single topic, such as a set of functions or class definitions related to a particular kind of data or processing).

We’ve seen a number of modules already, including io, re, and sys. To use a module, we just need to run import modulename. As it turns out, modules are simply files of Python code ending in .py! Thus, when we use import modulename, the interpreter searches for a modulename.py containing the various function and class definitions. Module files may either exist in a system-wide location, or be present in the working directory of the program importing them.^[2]

Importing the module in this way creates a namespace for the module. This namespace categorizes all of the functions and classes contained within that module so that different modules may have functions, classes, and variables with the same name without conflicting.

If we have the above two files in the same directory as our program, source code like the following would print 4, 64, "TAC", and 2.71828.

Unfortunately, Python uses the . operator in multiple similar ways, such as indicating a method belonging to an object (as in a_list.append("ID3")) and to indicate a function, variable, or class definition belonging to a namespace (as above).

Thus, as a “container” for names, the namespace allows different modules to name functions, classes, and variables the same way without conflicting. Just as a class may be documented by its API, so may modules: a module’s API describes the functions, classes, and variables it defines. These may also be accessed through the interactive help menu (and online). Try running import re followed by help(re) to see all the functions provided by the re module.

There are a huge number of modules (and packages) available for download online, and Python also comes with a wide variety of useful modules by default. Aside from re, sys, and io, discussed previously, some of the potentially more interesting modules include:

• string: common string operations (changing case, formatting, etc.)
• time, datetime and calendar: operations on dates and times
• random: generating and working with random numbers
• argparse: user-friendly parsing of complex arguments on the command line
• Tkinter: creating graphical user interfaces (with buttons, scroll bars, etc.)
• unittest: automating the creation and running of unit tests
• turtle: a simple interface for displaying line-based graphics^[3]

Tutorials for these specific modules may be found online, as may lists of many other packages and modules that come installed with Python or that are available for download.

Creating a Module

Let’s create a module called MyVCFModule.py to hold the two class definitions and parsing function from the last example. While we’re doing so, we’ll also convert the comments that we had created for our classes, methods, and functions into “docstrings.” Docstrings are triple-quoted strings that provide similar functionality to comments, but can be parsed later to produce the module’s API. They may span multiple lines, but should occur immediately inside the corresponding module, class, or function definition. Here we leave out the contents of the methods and functions (which are the same as in the previous chapter) to save space:

The program that makes use of this module, which exists in a separate file in the same directory (perhaps called snps_ex2.py), is short and sweet.^[4]

Should other projects need to parse VCF files, this module may again be of use. By the way, because we documented our module with docstrings, its API is readily accessible through the interactive help interface.

Part of the API help page:

Packages

As if Python didn’t already have enough “boxes” for encapsulation—functions, objects, modules, and so on—there are also packages. In Python, a “package” is a directory containing some module files.^[5]

A package directory must also contain a special file called __init__.py, which lets Python know that the directory should be treated as a package from which modules may be imported. (One could put code in this file that would be executed when the import statement is run, but we won’t explore this feature here.)

As an example, suppose that along with our MyVCFModule.py, we also had created a module for parsing gene ontology files called GOParseModule.py. We could put these together into a package (directory) called MyBioParsers.

To use a module contained in a package, the syntax is from packagename import modulename.^[6] Our Python program could live in the same directory in which the MyBioParsers directory was found, and might begin like so:

Later, the module itself can be used just as before.

Parsing a FASTA File

Up until this point, we’ve skipped something that is widely considered a “basic” in computational biology: reading a FASTA file. For the most part, the previous examples needing sequence data read that data from simple row/column formatted files.

Most sequence data, however, appear in so-called FASTA format, where each sequence has a header line starting with > and an ID, and the sequence is broken up over any number of following lines before the next header line appears.

Parsing data in FASTA format isn’t too difficult, but it requires an extra preprocessing step (usually involving a loop and a variable that keeps track of the “current” ID; so long as subsequent lines don’t match the > character, they must be sequence lines and can be appended to a list of strings to be later joined for that ID). While reinventing the FASTA-parsing wheel is good practice for novice coders, the BioPython package provides functions to parse these and many other formats.^[7]

BioPython is a large package with many modules and APIs, which you can read more about at http://biopython.org. For the specific application of parsing a FASTA file, a good place to start is a simple web search for “biopython fasta,” which leads us to a wiki page describing the SeqIO module, complete with an example.

This screenshot shows a few differences from the way we’ve been writing code, for example, the use of open() as opposed to io.open(), and the call to print record.id rather than our usual print(record.id) (the former of which will work only in older versions of Python, whereas the latter will work in all reasonably recent versions of Python). Nevertheless, given our knowledge of packages, modules, and objects, we can deduce the following from this simple code example:

1. The SeqIO module can be imported from the Bio package.
2. A file handle is opened.
3. The SeqIO.parse() function is called, and it takes two parameters: the file handle from which to read data, and a specification string describing the file format.
4. This function returns something iterable (similar to a list, or a regular file handle), so it can be looped over with a for-loop.
5. The loop accesses a series of objects of some kind, and each gets associated with the variable name record.
6. The record.id is printed, which is (apparently) an instance variable belonging to the record object.
7. The file handle is closed (always good practice!).

If we do some more reading on the SeqIO wiki page, we’d find that the record objects actually have the class SeqRecord, and they also have a seq instance variable providing access to the full sequence.

Putting this all together, here’s a short program that converts a FASTA file to the row/column format we’ve been dealing with.

Syntactic Sugar

In Python, nearly everything is an object, even simple integers! By looking at an integer’s API, for example, we discover that they provide a method called .bit_length().

Here’s a portion of the API view:

We can try it like so, to discover that the integer 7 can be represented with three binary bits (as 111):

If you were to try and view the API for an integer as we’ve done, you’d see a number of odd-looking methods, such as .__add__() and .__abs__():

This seems to indicate that we can get the absolute value of an integer by using a method call or by using the standard function call syntax. Similarly, we can apparently add an integer to another by using method call or the standard + operator. These are indeed true, and we can use the standard functions and operators or their method versions:

Operations like addition look like basic, fundamental operations, and Python accomplishes such operations through method calls on objects.^[8] A statement like a = b + c is converted to a = b.__add__(c) behind the scenes. Although we can run such method-oriented operations ourselves, the awkward double-underscore syntax is how the designers let us know that those methods are for internal use, and we should stick with the standard syntax. This automatic syntax conversion is known as syntactic sugar, and it is present in many programming languages so that they can be internally consistent but more pleasant (“sweeter”) for humans to work with.