Tuesday, November 24, 2015

OpenBSD support for psutil

OK, this is a big one: starting from version 3.3.0 (released just now) psutil will officially support OpenBSD platforms. This was contributed by Landry Breuil (thanks dude!) and myself in PR #615. The interesting parts of the code changes are this and this.

Differences with FreeBSD

As expected, OpenBSD implementation is very similar to FreeBSD's (which was already in place), that is why I decided to merge most of it in a single C file (_psutil_bsd.c) and use 2 separate C files for when the two implementations differed too much: freebsd.c and openbsd.c. In terms of functionality here's the differences with FreeBSD. Unless specified, these differences are due to the kernel which does not provide the information natively (meaning we can't do anything about it).
Similarly to FreeBSD, also OpenBSD implementation of Process.open_files() is problematic as it is not able to return file paths (FreeBSD can sometimes). Other than these differences the functionalities are all there and pretty much the same, so overall I'm pretty satisfied with the result. 

Considerations about BSD platforms

psutil has been supporting FreeBSD basically since the beginning (year 2009). At the time it made sense to support FreeBSD instead of other BSD variants because it is the most popular, followed by OpenBSD and NetBSD. Compared to FreeBSD, OpenBSD appears to be more "minimal" both in terms of facilities provided by the kernel and the number of system administration tools available. One thing which I appreciate a lot about FreeBSD is that the source code of all CLI tools installed on the system is available under /usr/bin/src, which was a big help for implementing all psutil APIs. OpenBSD source code is also available but it uses CSV and I am not sure it includes the source code for all CLI tools. There are still two more BSD variants for which it may be worth to add support for: NetBSD and DragonflyBSD (in this order). About a year ago some guy provided a patch for adding basic NetBSD support so it is likely that will happen sooner or later.

Other enhancements available in this release

The only other enhancement is issue #558, which allows specifying a different location of /proc filesystem on Linux.

External discussions


Saturday, June 13, 2015

psutil 3.0

Here we are. It's been a long time since my last blog post and my last psutil release. The reason? I've been travelling! I mean... a lot. I've spent 3 months in Berlin, 3 weeks in Japan and 2 months in New York City. While I was there I finally had the chance to meet my friend Jay Loden in person. Jay and I originally started working on psutil together 7 years ago



Back then I didn't know any C (and I still am a terrible C developer) so he's been crucial to develop the initial psutil skeleton including OSX and Windows support. I'm back home now (but not for long ;-)), so I finally have some time to write this blog post and tell you about the new psutil release. Let's see what happened.

net_if_addrs()

In a few words, we're now able to list network interface addresses similarly to "ifconfig" command on UNIX:
>>> import psutil
>>> psutil.net_if_addrs()
{'eth0': [snic(
            family=<AddressFamily.AF_INET: 2>, 
            address='10.0.0.4', 
            netmask='255.0.0.0', 
            broadcast='10.255.255.255'),
          snic(
            family=<AddressFamily.AF_PACKET: 17>, 
            address='9c:eb:e8:0b:05:1f', 
            netmask=None, 
            broadcast='ff:ff:ff:ff:ff:ff')],
 'lo': [snic(
            family=<AddressFamily.AF_INET: 2>, 
            address='127.0.0.1', 
            netmask='255.0.0.0', 
            broadcast='127.0.0.1'),
        snic(
            family=<AddressFamily.AF_PACKET: 17>, 
            address='00:00:00:00:00:00', 
            netmask=None, 
            broadcast='00:00:00:00:00:00')]}
This is limited to AF_INET (IPv4), AF_INET6 (IPv6) and AF_LINK (ETHERNET) address families. If you want something more poweful (e.g. AF_BLUETOOTH) you can take a look at netifaces extension. And here's the code which does these tricks on POSIX and Windows:
Also, here's some doc.

net_if_stats()

This will return a bunch of information about network interface cards:
>>> import psutil
>>> psutil.net_if_stats()
{'eth0': snicstats(
    isup=True, 
    duplex=<NicDuplex.NIC_DUPLEX_FULL: 2>, 
    speed=100, 
    mtu=1500),
 'lo': snicstats(
    isup=True, 
    duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>, 
    speed=0, 
    mtu=65536)}
Again, here's the code:
...and the doc.

Enums

Enums are a nice new feature introduced in Python 3.4. Very briefly (or at least, this is what I appreciate the most about them), they help you write an API with human-readable constants. If you use Python 2 you'll see something like this:
>>> import psutil
>>> psutil.IOPRIO_CLASS_IDLE
3
On Python 3.4 you'll see a more informative:
>>> import psutil
>>> psutil.IOPRIO_CLASS_IDLE
<IOPriority.IOPRIO_CLASS_IDLE: 3>
They are backward compatible, meaning if you're sending serialized data produced with psutil through the network you can safely use comparison operators and so on. The psutil APIs returning enums (on Python >=3.4) are:
All the other existing constants remained plain strings (STATUS_*) or integers (CONN_*).

Zombie processes

This is a big one. The full story is here but basically the support for zombie processes on UNIX was broken (except on Linux - Windows doesn't have zombie processes). Up until psutil 2.* we could instantiate a zombie process:
>>> pid = create_zombie()
>>> p = psutil.Process(pid)
...but every time we queried it we got a NoSuchProcess exception:
>>> psutil.name()
  File "psutil/__init__.py", line 374, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: no process found with pid 123
That was misleading though because the PID technically still existed:
>>> psutil.pid_exists(p.pid)
True
Furthermore, depending on what platform you were on, certain process stats could still be queried (instead of raising NoSuchProcess):
>>> psutil.cmdline()
['python']
Also process_iter() did not return zombie processes at all. This was probably the worst aspect because being able to identify them is an important use case, as they signal an issue with process: if a parent process spawns a child, terminates it (via kill()), but doesn't wait() for it it will create a zombie. Long story short, the way this changed in psutil 3.0 is that:
  • we now have a new ZombieProcess exception, raised every time we're not able to query a process because it's a zombie
  • it is raised instead of NoSuchProcess (which was incorrect and misleading)
  • it is still backward compatible (meaning you won't have to change your old code) because it inherits from NoSuchProcess
  • process_iter() finally works, meaning you can safely identify zombie processes like this:
import psutil
zombies = []
for p in psutil.process_iter():
    try:
        if p.status() == psutil.STATUS_ZOMBIE:  
            zombies.append(p)
    except NoSuchProcess:
        pass

Removal of deprecated APIs

This is another big one, probably the biggest. In a previous blog post I already talked about deprecated APIs. What I did back then (January 2014) was to rename and officially deprecate different APIs and provide aliases for them so that people wouldn't yell at me because I broke their existent code. The most interesting deprecation was certainly the one affecting module constants and the hack which was used in order to provide "module properties". With this new release I decided to get rid of all those aliases. I'm sure this will cause problems but hey! This is a new major release, right? =). Plus the amount of crap which was removed is impressive (see the commit). Here's the old aliases which are now gone for good (or bad, depending on how much headache they will cause you):

Removed module functions and constants

Already deprecated nameNew name
psutil.BOOT_TIME()psutil.boot_time()
psutil.NUM_CPUS()psutil.cpu_count()
psutil.TOTAL_PHYMEM()psutil.virtual_memory().total
psutil.avail_phymem()psutil.virtual_memory().free
psutil.avail_virtmem()psutil.swap_memory().free
psutil.cached_phymem() (Linux only)psutil.virtual_memory().cached
psutil.get_pid_list()psutil.pids().cached
psutil.get_process_list()-
psutil.get_users()psutil.users()
psutil.network_io_counters()psutil.net_io_counters()
psutil.phymem_buffers() (Linux only)psutil.virtual_memory().buffers
psutil.phymem_usage()psutil.virtual_memory()
psutil.total_virtmem()psutil.swap_memory().total
psutil.used_virtmem()psutil.swap_memory().used
psutil.used_phymem()psutil.virtual_memory().used
psutil.virtmem_usage()psutil.swap_memory()

Process methods (assuming p = psutil.Process()):

Already deprecated nameNew name
p.get_children()p.children()
p.get_connections()p.connections()
p.get_cpu_affinity()p.cpu_affinity()
p.get_cpu_percent()p.cpu_percent()
p.get_cpu_times()p.cpu_times()
p.get_io_counters()p.io_counters()
p.get_ionice()p.ionice()
p.get_memory_info()p.memory_info()
p.get_ext_memory_info()p.memory_info_ex()
p.get_memory_maps()p.memory_maps()
p.get_memory_percent()p.memory_percent()
p.get_nice()p.nice()
p.get_num_ctx_switches()p.num_ctx_switches()
p.get_num_fds()p.num_fds()
p.get_num_threads()p.num_threads()
p.get_open_files()p.open_files()
p.get_rlimit()p.rlimit()
p.get_threads()p.threads()
p.getcwd()p.cwd()
p.set_cpu_affinity()p.cpu_affinity()
p.set_ionice()p.ionice()
p.set_nice()p.nice()
p.set_rlimit()p.rlimit()

If your code suddenly breaks with AttributeError after you upgraded psutil it means you were using one of those deprecated aliases. In that case just take a look at the table above and rename stuff in accordance.

Bug fixes

I fixed a lot of stuff (full list here), but here's the list of things which I think are worth mentioning:

Ease of development

These are not enhancements you will directly benefit from but I put some effort into making my life easier every time I work on psutil.
  • I care about psutil code being fully PEP8 compliant so I added a pre-commit GIT hook which runs flake8 on every commit and rejects it if the coding style is not compliant. The way I install this is via make install-git-hooks.
  • I added a make install-dev-deps command which installs all deps and stuff which is useful for testing (ipdb, coverage, etc).
  • A new make coverage command which runs coverage. With this I discovered some of parts in the code which weren't covered by tests and I fixed that.
  • I started using tox to easily test psutil against all supported Python versions (from 2.6 to 3.4) in one shot.
  • I reorganized tests so that now they can be easily executed with py.test and nose (before, only unittest runner was fully supported)

Final words

I must say I'm pretty satisfied with how psutil is going and the satisfaction I still get every time I work on it. Right now it gets almost 800.000 download a month, which is pretty great for a Python library. As of right now I consider psutil almost "completed" in terms of features, meaning I'm basically running out of ideas on what I should add next (see TODO). From now on the future development will probably focus on adding support for more exotic platforms (OpenBSD, NetBSD, Android). There also have been some discussions on python-ideas mailing list about including psutil into Python stdlib but, assuming that will ever happen, it's still far away in the future as it would require a lot of time which I currently don't have. That should be all. I hope you will all enjoy this new release.