Friday, June 13, 2014

Python and sendfile

sendfile(2) is a UNIX system call which provides a "zero-copy" way of copying data from one file descriptor (a file) to another (a socket). Because this copying is done entirely within the kernel, sendfile(2) is more efficient than the combination of "file.read()" and "socket.send()", which requires transferring data to and from user space.  This copying of the data twice imposes some performance and resource penalties which sendfile(2) syscall avoids; it also results in a single system call (and thus only one context switch), rather than the series of read(2) / write(2) system calls (each system call requiring a context switch) used internally for the data copying. A more exhaustive explanation of how sendfile(2) works is available here,  but long story short is that sending a file with sendfile() is usually twice as fast than using plain socket.send(). Typical applications which can benefit from using sendfile() are FTP and HTTP servers.

socket.sendfile()

I recently contributed a patch for Python's socket module which adds a high-level socket.sendfile() method (see full discussion at issue 17552). socket.sendfile() will transmit a file until EOF is reached by attempting to use os.sendfile(), if available, else it falls back on using plain socket.send(). Internally, it takes care of handling socket timeouts and provides two optional parameters to move the file offset or to send only a limited amount of bytes. I came up with this idea because getting all of that right is a bit tricky, so a generic wrapper seemed to be convenient to have. socket.sendfile() will make its appearance in Python 3.5.

sendfile and Python

sendfile(2) made its first appearance into the Python stdlib kind of late: Python 3.3. It was contributed by Ross Lagerwall and me in issue 10882. Since the patch didn't make it into python 2.X and I wanted to use sendfile() in pyftpdlib I later decided to release it as a stand alone module working with older (2.5+) Python versions (see pysendfile project). Starting with version 3.5, Python will hopefully start using sendfile() more extensively, in details:
Also, Windows provides something similar to sendfile(2): TransmitFile. Now that socket.sendfile() is in place it seems natural to add support for it as well (see issue 21721).

Backport to Python 2.6 and 2.7

For those of you who are interested in using socket.sendfile() with older Python 2.6 and 2.7 versions here's a backport. It requires pysendfile modules to be installed. Full code including tests is hosted here.

#!/usr/bin/env python

"""
This is a backport of socket.sendfile() for Python 2.6 and 2.7.
socket.sendfile() will be included in Python 3.5:
http://bugs.python.org/issue17552
Usage:

>>> import socket
>>> file = open("somefile.bin", "rb")
>>> sock = socket.create_connection(("localhost", 8021))
>>> sendfile(sock, file)
42319283
>>>
"""

import errno
import io
import os
import select
import socket
try:
    memoryview  # py 2.7 only
except NameError:
    memoryview = lambda x: x

if os.name == 'posix':
    import sendfile as pysendfile  # requires "pip install pysendfile"
else:
    pysendfile = None


_RETRY = frozenset((errno.EAGAIN, errno.EALREADY, errno.EWOULDBLOCK,
                    errno.EINPROGRESS))


class _GiveupOnSendfile(Exception):
    pass


if pysendfile is not None:

    def _sendfile_use_sendfile(sock, file, offset=0, count=None):
        _check_sendfile_params(sock, file, offset, count)
        sockno = sock.fileno()
        try:
            fileno = file.fileno()
        except (AttributeError, io.UnsupportedOperation) as err:
            raise _GiveupOnSendfile(err)  # not a regular file
        try:
            fsize = os.fstat(fileno).st_size
        except OSError:
            raise _GiveupOnSendfile(err)  # not a regular file
        if not fsize:
            return 0  # empty file
        blocksize = fsize if not count else count

        timeout = sock.gettimeout()
        if timeout == 0:
            raise ValueError("non-blocking sockets are not supported")
        # poll/select have the advantage of not requiring any
        # extra file descriptor, contrarily to epoll/kqueue
        # (also, they require a single syscall).
        if hasattr(select, 'poll'):
            if timeout is not None:
                timeout *= 1000
            pollster = select.poll()
            pollster.register(sockno, select.POLLOUT)

            def wait_for_fd():
                if pollster.poll(timeout) == []:
                    raise socket._socket.timeout('timed out')
        else:
            # call select() once in order to solicit ValueError in
            # case we run out of fds
            try:
                select.select([], [sockno], [], 0)
            except ValueError:
                raise _GiveupOnSendfile(err)

            def wait_for_fd():
                fds = select.select([], [sockno], [], timeout)
                if fds == ([], [], []):
                    raise socket._socket.timeout('timed out')

        total_sent = 0
        # localize variable access to minimize overhead
        os_sendfile = pysendfile.sendfile
        try:
            while True:
                if timeout:
                    wait_for_fd()
                if count:
                    blocksize = count - total_sent
                    if blocksize <= 0:
                        break
                try:
                    sent = os_sendfile(sockno, fileno, offset, blocksize)
                except OSError as err:
                    if err.errno in _RETRY:
                        # Block until the socket is ready to send some
                        # data; avoids hogging CPU resources.
                        wait_for_fd()
                    else:
                        if total_sent == 0:
                            # We can get here for different reasons, the main
                            # one being 'file' is not a regular mmap(2)-like
                            # file, in which case we'll fall back on using
                            # plain send().
                            raise _GiveupOnSendfile(err)
                        raise err
                else:
                    if sent == 0:
                        break  # EOF
                    offset += sent
                    total_sent += sent
            return total_sent
        finally:
            if total_sent > 0 and hasattr(file, 'seek'):
                file.seek(offset)
else:
    def _sendfile_use_sendfile(sock, file, offset=0, count=None):
        raise _GiveupOnSendfile(
            "sendfile() not available on this platform")


def _sendfile_use_send(sock, file, offset=0, count=None):
    _check_sendfile_params(sock, file, offset, count)
    if sock.gettimeout() == 0:
        raise ValueError("non-blocking sockets are not supported")
    if offset:
        file.seek(offset)
    blocksize = min(count, 8192) if count else 8192
    total_sent = 0
    # localize variable access to minimize overhead
    file_read = file.read
    sock_send = sock.send
    try:
        while True:
            if count:
                blocksize = min(count - total_sent, blocksize)
                if blocksize <= 0:
                    break
            data = memoryview(file_read(blocksize))
            if not data:
                break  # EOF
            while True:
                try:
                    sent = sock_send(data)
                except OSError as err:
                    if err.errno in _RETRY:
                        continue
                    raise
                else:
                    total_sent += sent
                    if sent < len(data):
                        data = data[sent:]
                    else:
                        break
        return total_sent
    finally:
        if total_sent > 0 and hasattr(file, 'seek'):
            file.seek(offset + total_sent)


def _check_sendfile_params(sock, file, offset, count):
    if 'b' not in getattr(file, 'mode', 'b'):
        raise ValueError("file should be opened in binary mode")
    if not sock.type & socket.SOCK_STREAM:
        raise ValueError("only SOCK_STREAM type sockets are supported")
    if count is not None:
        if not isinstance(count, int):
            raise TypeError(
                "count must be a positive integer (got %s)" % repr(count))
        if count <= 0:
            raise ValueError(
                "count must be a positive integer (got %s)" % repr(count))


def sendfile(sock, file, offset=0, count=None):
    """sendfile(sock, file[, offset[, count]]) -> sent

    Send a *file* over a connected socket *sock* until EOF is
    reached by using high-performance sendfile(2) and return the
    total number of bytes which were sent.
    *file* must be a regular file object opened in binary mode.
    If sendfile() is not available (e.g. Windows) or file is
    not a regular file socket.send() will be used instead.
    *offset* tells from where to start reading the file.
    If specified, *count* is the total number of bytes to transmit
    as opposed to sending the file until EOF is reached.
    File position is updated on return or also in case of error in
    which case file.tell() can be used to figure out the number of
    bytes which were sent.
    The socket must be of SOCK_STREAM type.
    Non-blocking sockets are not supported.
    """
    try:
        return _sendfile_use_sendfile(sock, file, offset, count)
    except _GiveupOnSendfile:
        return _sendfile_use_send(sock, file, offset, count)

Monday, May 26, 2014

Goodbye Google Code, I'm moving to Github

8 years ago I started hosting my first open source project (pyftpdlib) on Google Code and I later ended up also hosting psutil and pysendfile. Back then GC had just been released and similarly to other Google products I appreciated the clean and minimalistic interface, the excellent bug tracker and the freedom to choose between different revision control systems (SVN, GIT and Mercurial, which is my favourite one). Unfortunately as the years passed Google completely lost interest in maintaining GC to the point that now GC can basically be considered an abandoned project.  If you take a look at GC bug tracker you can see literally hundreds of issues which have been open for years, even some apparently easy ones such as #60 and #919. The lack of interest from Google is absolutely astonishing and it is the main reason why I ultimately decided to change. After at least a couple of years of thinking about migrating to github I finally bite the bullet and as of today psutil is now hosted on github (update: now also pyftpdlib and pysendfile).

What I will miss the most about GC

First of all I must say that despite the unfortunate situation of GC I'm also sad for abandoning it. It started as a really great hosting platform, and it still has some peculiar aspects which I know I will be missing. In order of importance:
  • The bug tracker: it is much more powerful than github's, especially for the extremely customizable labeling system which is pure gold, the excellent searching system and the grid view. GC bug tracker seriously kicks some ass so kudos to whoever was behind its design! By comparison github bug tracker is too minimalistic and it has no good way to order issues or list them in a more compact form. I'm totally gonna miss psutil bug tracker.
  • Mercurial: I'm a big fan of Mercurial and I consider it way more pleasant to work with compared to GIT. I don't know exactly why GIT ended up being so much more used than Mercurial (probably because of github?) but I'm sure that many other guys like me who know both systems will agree that Mercurial is simply so much easier to use. Unfortunately once you decide to stick with github you have no other choice. Mercurial, I'm gonna miss you too!
  • GC layout: it is much simpler than github's! Everything is easy to find, even for a non-geek person. The home page alone is perfect to summarize what the project is about and doesn't have tens of icons all over the place. github layout is more complicated and needs some time to get used to, even for a programmer. If these projects (psutil and others) weren't about programming I wouldn't have chosen github because it's "not for the masses".

What I appreciate about github

  • Travis integration: there's this totally awesome free continuos integration service called Travis which given a configuration file like this it will automatically run tests on multiple python versions every time a commit is pushed. They recently added OSX support and Windows support is on the way. This way I will finally be able to quickly test psutil on Linux, OSX and Windows without using virtualized systems except for FreeBSD and Solaris! To me this is like the ultimate Christmas gift and I couldn't ask for any better. Note: as of today Travis only works with github.
  • forks and pull requests: honestly I'm not a big fan of them (yet?), probably because I'm used to the python-dev development workflow consisting in uploading patches on the bug tracker and reviewing them (see this for example). Nevertheless to my understanding most people use pull requests in order to contribute to open source projects so basically this is a service I'm glad to offer to my users who hopefully will be able to contribute back more easily. GC has a cloning system but isn't anywhere near github's and I'm not even sure how it works (never cared).
  • the "social" side of github including the fact that you can "star" developers and receive notifications about their activity was another big incentive for migrating. The personal landing page collecting all your contributions to different projects is absolutely cool. GC had something similar but they stupidly removed it all of the sudden and never reintroduced it back. A lot of people were angry but again, they didn't care. Actually this was the feature I appreciated the most about GC after the bug tracker and that is when I seriously started thinking about flipping off GC for good. 
  • the enormous user base: the fact that github is the most used code hosting platform out there will hopefully help me and my projects have a little more visibility. Also, in many job interviews I've been asked what my github profile was so it seems github also became an active part in getting jobs.
  • gists: gists are "a simple way to share code snippets and pastes with others. All gists are Git repositories, so they are automatically versioned, forkable and usable from Git". Seriously, they are  beautiful. In order to share my code snippets I've always used Active State but I think I will eventually migrate them as well in order to have everything in one place.
  • the fact that if you mention an issue number as part of your commit message that specific issue will automatically be updated. As I said GC bug tracker is superior in basically any aspect but since I always took care of updating issues by mentioning the specific cset which fixed them (see for example here) having this little extra feature will save me some time. 
  • SSH keys: using Mercurial on GC means using password based authentication. Incredibly they still do not support SSH key based auth. Simply "git push"ing without entering any password when I'm not on my laptop is nice, and of course, it is also much more secure.

Migration

For those of you who are interested in knowing how I did it, here goes: as for moving the issue from GC bug tracker to github's I used this tool. I managed to preserve the issue IDs but unfortunately not the real owners nor the real issue dates, which kind of sucks. As for migrating the code from mercurial to git I just used this. The Mercurial -> GIT transition was perfect and I also managed to preserve the original Mercurial named branches and tags, which for me it was crucial. In conclusion, psutil is a 5 years, medium sized  project with hundreds of issues: the transition in this case is definitively possible but not painless so if you plan on migrating, the sooner you do it the better.

Tuesday, April 8, 2014

Reimplementing netstat in C / Python

psutil 2.1.0 is out and with it I finally managed to implement something I've been wanting to have for a long time: netstat-like functionalities (see ticket). Similarly to "netstat -antp" on UNIX you can now list system-wide connections in pure python and also determine what process (PID) is using a particular connection:
>>> import psutil
>>> from pprint import pprint as pp
>>> pp(psutil.net_connections())
[sconn(fd=-1, family=2, type=1, laddr=('127.0.0.1', 587), raddr=(), status='LISTEN', pid=None),
 sconn(fd=-1, family=2, type=1, laddr=('127.0.0.1', 6379), raddr=(), status='LISTEN', pid=None),
 sconn(fd=-1, family=2, type=1, laddr=('127.0.1.1', 53), raddr=(), status='LISTEN', pid=None),
 sconn(fd=-1, family=2, type=1, laddr=('10.0.3.1', 53), raddr=(), status='LISTEN', pid=None),
 sconn(fd=-1, family=2, type=1, laddr=('127.0.0.1', 631), raddr=(), status='LISTEN', pid=None),
 sconn(fd=-1, family=2, type=1, laddr=('127.0.0.1', 25), raddr=(), status='LISTEN', pid=None),
 sconn(fd=-1, family=2, type=1, laddr=('0.0.0.0', 3389), raddr=(), status='LISTEN', pid=None),
 sconn(fd=17, family=2, type=1, laddr=('127.0.0.1', 34785), raddr=(), status='LISTEN', pid=3591),
 sconn(fd=15, family=2, type=1, laddr=('127.0.0.1', 56359), raddr=(), status='LISTEN', pid=3591),
 sconn(fd=-1, family=10, type=2, laddr=('::', 56720), raddr=(), status='NONE', pid=None)]
>>>
This is yet another functionality which can be used for monitoring purposes. For example, say you want to make sure your HTTP server is running on port 80, you can do something like this:
import psutil

def check_listening_port(port):
    """Return True if the given TCP port is busy and in LISTEN mode."""
    for conn in psutil.net_connections(kind='tcp'):
        if conn.laddr[1] == port and conn.status == psutil.CONN_LISTEN:
            return True
    return False

print(check_listening_port(80))

Netstat in pure python

Here it is, in 65 lines of code: netstat.py. Pretty neat right? ;-)

Implementation(s)

As always, each platform required its own, different, implementation. Luckily for some platforms (OSX, Windows) I was able to reuse and customize some code from the existing Process.connections() implementation which was already in place. For those of you who are interested in knowing how this was done here's the source code references:
Hopefully this will help whoever needs to do this into another language. The only platform where this is sort of clunky is OSX, which does not expose anything to list all system-wide sockets in a single shot, so you're forced to query each process. That means you'll need root privileges otherwise you'll get an access denied error. For what it's worth, I took a look at lsof and it has the same limitation and netstat runs with SUID. Well, I guess this is it. I'll leave you with some docs and the list of other bugfixes included in this 2.1.0 release. For the next one I'm planning on working on a couple of other network-related functionalities: "ifconfig" and NIC speeds. But that's for another time...

Monday, March 10, 2014

psutil 2.0

The time has finally come: psutil 2.0 is out! This is a release which took me a considerable amount of effort and careful thinking during the past 4 months as I went through a major rewrite and reorganization of both python and C extension modules. To get a sense of how much has changed you can compare the differences with old 1.2.1 version by running "hg diff -r release-1.2.1:release-2.0.0" which will produce more than 22,000 lines of output! In those 22k lines I tried to nail down all the quirks the project had accumulated since its start 4 years ago and the resulting code base is now cleaner than ever, more manageable and fully compliant with PEP-7 and PEP-8 guidelines.
There were some difficult decisions because many of the changes I introduced are not backward compatible so I was concerned with the pain this may cause existing users. I kind of still am, but I'm sure the transition will be well perceived on the long run as it will result in more manageable user code. OK, enough with the preface and let's see what changed.

API changes

I already wrote a detailed blog post about what changed so I recommend you to use that as the official reference on how to port your code. Long story short:
  • all get_* prefixes for functions and methods are gone, e.g.:
    • psutil.get_boot_time() -> psutil.boot_time()
    • psutil.Process.get_cpu_percent() -> psutil.Process.cpu_percent()  
  • all set_* prefixes for Process methods are gone and were unified in a single method which can be used for both getting and setting, e.g:
    • psutil.Process.set_nice(value) -> psutil.Process.nice(value)
  • Process properties were turned into methods, e.g.:
    • psutil.Process.cmdline -> psutil.Process.cmdline()   
  • module level constants (BOOT_TIME, NUM_CPUS, TOTAL_PHYMEM) were turned into functions (psutil.boot_time(), psutil.cpu_count(), psutil.virtual_memory().total)
  • all the old names are still there but will raise a DeprecationWarning
    • you will have to explicitly enable warnings via "python -Wd foo.py" though
  • the only fully incompatible change is represented by the Process properties which are now methods

RST documentation 

I've never been happy with old doc hosted on Google Code. The markup language provided by Google is pretty limited, plus it's not put under revision control. New doc is more detailed, it uses reStructuredText as the markup language, it lives in the same code repository as psutil's and it is hosted on the excellent readthedocs web site: http://psutil.readthedocs.org/

Physical CPUs count

You're now able to distinguish between logical and physical CPUs:
>>> psutil.cpu_count()  # logical
4
>>> psutil.cpu_count(logical=False)  # physical cores only
2
Full story is in issue 427.

Process instances are hashable

Basically this means process instances can now be checked for equality and can be used with set()s:
>>> p1 = psutil.Process()
>>> p2 = psutil.Process()
>>> p1 == p2
True
>>> set((p1, p2))
set([<psutil.Process(pid=8217, name='python') at 140007043550608>])
Full story is in issue 452.

Speedups 

  • #477: process cpu_percent() is about 30% faster.
  • #478: (Linux) almost all APIs are about 30% faster on Python 3.X.

Other improvements and bugfixes

List of all changes is available here.
OK, that's all folks. I hope you will enjoy this new version and report your feedback.

Saturday, January 11, 2014

psutil 2.0 porting

This my second blog post is going to be about psutil 2.0, a major release in which I decided to reorganize the existing API for the sake of consistency. At the time of writing psutil 2.0 is still under development and the intent of this blog post is to serve as an official reference which describes how you should port your existent code base. In doing so I will also explain why I decided to make these changes. Despite many APIs will still be available as aliases pointing to the newer ones, the overall changes are numerous and many of them are not backward compatible. I'm sure many people will be sorely bitten but I think this is for the better and it needed to be done, hopefully for the first and last time. OK, here we go now.

Module constants turned into functions

What changed

Old name Replacement
psutil.BOOT_TIMEpsutil.boot_time()
psutil.NUM_CPUSpsutil.cpu_count()
psutil.TOTAL_PHYMEMpsutil.virtual_memory().total

Why I did it

I already talked about this more extensively in this blog post. In short: other than introducing unnecessary slowdowns, calculating a module level constant at import time is dangerous because in case of error the whole app will crash. Also, the represented values may be subject to change (think about the system clock) but the constant cannot be updated.
Thanks to this hack accessing the old constants still works and produces a DeprecationWarning.

Module functions renamings

What changed


Old name Replacement
psutil.get_boot_time() psutil.boot_time()
psutil.get_pid_list() psutil.pids()
psutil.get_users() psutil.users()

Why I did it

They were the only module level functions having a "get_" prefix. All others do not.

Process class' methods renamings

What changed

All methods lost their "get_" and "set_" prefixes. A single method can now be used for both getting and setting (if a value is passed). Assuming p = psutil.Process():

Old name Replacement
p.get_children()p.children()
p.get_connections()p.connections()
p.get_cpu_affinity()p.cpu_affinity()
p.get_cpu_percent()p.cpu_percent()
p.get_cpu_times()p.cpu_times()
p.get_io_counters()p.io_counters()
p.get_ionice()p.ionice()
p.get_memory_info()p.memory_info()
p.get_ext_memory_info()p.memory_info_ex()
p.get_memory_maps()p.memory_maps()
p.get_memory_percent()p.memory_percent()
p.get_nice()p.nice()
p.get_num_ctx_switches()p.num_ctx_switches()
p.get_num_fds()p.num_fds()
p.get_num_threads()p.num_threads()
p.get_open_files()p.open_files()
p.get_rlimit()p.rlimit()
p.get_threads()p.threads()
p.getcwd()p.cwd()

...as for set_* methods:

Old name Replacement
p.set_cpu_affinity()p.cpu_affinity(cpus)
p.set_ionice()p.ionice(ioclass, value=None)
p.set_nice()p.nice(value)
p.set_rlimit()p.rlimit(resource, limits=None)

Why I did it

I wanted to be consistent with system-wide module level functions which have no "get_" prefix. After I got rid of "get_" prefixes removing also "set_" seemed natural and helped diminish the number of APIs.

Process properties are now methods

What changed

Assuming p = psutil.Process():

Old name Replacement
p.cmdlinep.cmdline()
p.create_timep.create_time()
p.exep.exe()
p.gidsp.gids()
p.namep.name()
p.parentp.parent()
p.ppidp.ppid()
p.statusp.status()
p.uidsp.uids()
p.usernamep.username()

Why I did it

Different reasons:
  • Having a mixed API which uses both properties and methods for no particular reason is confusing and messy as you don't know whether to use "()" or not (see here). 
  • It is usually expected from a property to not perform many computations internally whereas psutil actually invokes a function every time it is accessed. This has two drawbacks:
    • you may get an exception just by accessing the property (e.g. "p.name" may raise NoSuchProcess or AccessDenied)
    • you may erroneously think properties are cached but this is true only for name, exe, and create_time.

CPU percent intervals

What changed

The timeout parameter of cpu_percent* functions now defaults to 0.0 instead of 0.1. The functions affected are:
psutil.Process.cpu_percent()
psutil.cpu_percent()
psutil.cpu_times_percent()

Why I changed it

I originally set 0.1 as the default timeout because in order to get a meaningful percent value you need to wait some time.
Having an API which "sleeps" by default is risky though, because it's easy to forget it does so. That is particularly problematic when calling cpu_percent() for all processes: it's very easy to forget about specifying timeout=0 resulting in dramatic slowdowns which are hard to spot. Example:
>>> # this will be slow
>>> for p in psutil.process_iter():
...    print(p.cpu_percent())

Migration strategy

Except for Process properties (name, exe, cmdline, etc.) all the old APIs are still available as aliases pointing to the newer names and raising DeprecationWarning. psutil will be very clear on what you should use instead of the deprecated API as long you start the interpreter with the "-Wd" option (this will enable deprecation warnings which were silenced in Python 2.7)
giampaolo@ubuntu:/tmp$ python -Wd
Python 2.7.3 (default, Sep 26 2013, 20:03:06) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.get_pid_list()
__main__:1: DeprecationWarning: psutil.get_pid_list is deprecated; use psutil.pids() instead
[1, 2, 3, 6, 7, 13, ...]
>>>
>>>
>>> p = psutil.Process()
>>> p.get_cpu_times()
__main__:1: DeprecationWarning: get_cpu_times() is deprecated; use cpu_times() instead
pcputimes(user=0.08, system=0.03)
>>> 
If you have a solid test suite you can run tests and fix the warnings one by one.

As for the the Process properties which were turned into methods it's more difficult because whereas psutil 1.2.1 returns the actual value, psutil 2.0.0 will return the bound method:
# psutil 1.2.1
>>> psutil.Process().name
'python'
>>>

# psutil 2.0.0
>>> psutil.Process().name
<bound method Process.name of psutil.Process(pid=19816, name='python') at 139845631328144>
>>>
What I would recommend if you want to drop support with 1.2.1 is to grep for ".name", ".exe" etc. and just replace them with ".exe()" and ".name()" one by one.
If on the other hand you want to write a code which works with both versions I see two possibilities:

#1 check version info, like this:
>>> PSUTIL2 = psutil.version_info >= (2, 0)
>>> p = psutil.Process()
>>> name = p.name() if PSUTIL2 else p.name
>>> exe = p.exe() if PSUTIL2 else p.exe
#2 get rid of all ".name", ".exe" occurrences you have in your code and use as_dict() instead:
>>> p = psutil.Process()
>>> pinfo = p.as_dict(attrs=["name", "exe"])
>>> pinfo
{'exe': '/usr/bin/python2.7', 'name': 'python'}
>>> name = pinfo['name']
>>> exe = pinfo['exe']

New features introduced in 2.0.0

Ok, enough with the bad news. =) psutil 2.0.0 is not only about code breakage. I also had the chance to integrate a bunch of interesting features.
  • issue 427: you're now able to distinguish between the number of logical and physical CPUs:
    >>> psutil.cpu_count()  # logical
    4
    >>> psutil.cpu_count(logical=False)  # physical cores only
    2
  • issue 452: process classes are now hashable and can be checked for equality. That means you can use Process objects with sets (finally!).
  • issue 447: psutil.wait_procs() "timeout" parameter is now optional 
  • issue 461: functions returning namedtuples are now pickle-able
  • issue 459: a Makefile is now available to automatize repetitive tasks such as build, install, running tests etc. There's also a make.bat for Windows.
  • introduced unittest2 module as a requirement for running tests

Friday, December 20, 2013

Making module constants part of your API is evil

One of the initial features which were included in psutil since day one (5 years ago) were system's boot time, number of CPUs and total physical memory. These metrics have one thing in common: they are (apparently) not supposed to change over time. That is why we (me and Jay) decided that exposing them as module constants calculated at import time was the way to go.
>>> import psutil
>>> psutil.NUM_CPUS
2
>>> psutil.BOOT_TIME  # as seconds since the epoch  
1387579049.799092   
>>> psutil.TOTAL_PHYMEM
8374120448
5 years later I regret that decision and I'm going to explain you why you don't want to do the same mistake.

A constant should not change

When we think of  'constants', our expectations are that they should not change over time. It may be obvious, but before thinking about introducing a constant be absolutely sure the value it represents is going to remain the same.
Now, back then we thought these 3 metrics were not supposed to change, at least until the system is rebooted. Well, we were wrong: it turns out 2 of them actually can.
Apparently virtualized systems can change physical installed memory at runtime (see here and here) and system boot time can easily be altered every time you update the system clock.
In both of these cases, of course, the constants will not reflect the updated values.

Doing things at import time is dangerous

That's because import time usually means startup time and if something goes wrong the whole application will crash. In general the only reason for a module to crash at import time is because of a missing dependancy or because it's not supposed to run on that platform in the first place.
Now, here's a couple of bug reports which were collected over time: issue 188issue 313.
The inconvenience was so severe that I had to release different fixed versions ASAP and the fix consisted of a stinky workaround.
That's when I started thinking about getting rid of those constants once and for all and introduce functions instead. But how to do that without breaking everybody's code?

Backward compatibility matters

Now here's the crucial part: every time you deliver a library to someone else you just cannot remove an API all of the sudden, especially if they are 3 and have been around since day one.
It should first be deprecated, possibly turned into an alias pointing to a newer API and finally be removed after 1 or 2 major releases. Also, you want the deprecated API to explicitly raise a DeprecationWarning informing the user he's relying on something which will eventually be removed. With a module constant you cannot do any of that. What you would need is a module property.

Module properties

One of the greatest things about Python is that it's so dynamic that it lets you do all sort of nasty things with objects, including injecting names into modules (which are also objects) and make them behave like actual class properties!
For this I have to thank Josiah Carlson who came up with this very simple yet effective solution:

class ModuleWrapper(object):

    def __repr__(self):
        return repr(self._module)
    __str__ = __repr__

    @property
    def NUM_CPUS(self):
        msg = "NUM_CPUS constant is deprecated; use cpu_count() instead"
        warnings.warn(msg, category=DeprecationWarning, stacklevel=2)
        return cpu_count()

    @property
    def BOOT_TIME(self):
        msg = "BOOT_TIME constant is deprecated; " \
              "use get_boot_time() instead"
        warnings.warn(msg, category=DeprecationWarning, stacklevel=2)
        return get_boot_time()

    @property
    def TOTAL_PHYMEM(self):
        msg = "TOTAL_PHYMEM constant is deprecated; " \
              "use virtual_memory().total instead"
        warnings.warn(msg, category=DeprecationWarning, stacklevel=2)
        return virtual_memory().total

mod = ModuleWrapper()
mod.__dict__ = globals()
mod._module = sys.modules[__name__]
sys.modules[__name__] = mod

You can put this at the bottom of your module and you'll have fully working module constants (tested on Python from 2.4 to 3.4)!

EDIT: the only reason I applied this hack is to turn the old constants into aliases pointing to the newly introduced functions and produce a deprecation warning. That aside I can't think of a case where the use of a module property would be justified.