Here we are. It's been a long time since my last blog post and my last
psutil release. The reason? I've been travelling! I mean... a lot. I've spent 3 months in Berlin, 3 weeks in Japan and 2 months in New York City. While I was there I finally had the chance to meet my friend
Jay Loden in person.
Jay and I originally started working on psutil together
7 years ago.
Back then I didn't know any C (and I still am a terrible C developer) so he's been crucial to develop the initial psutil skeleton including OSX and Windows support. I'm back home now (but not for long ;-)), so I finally have some time to write this blog post and tell you about the new psutil release. Let's see what happened.
net_if_addrs()
In a few words, we're now able to list network interface addresses similarly to "ifconfig" command on UNIX:
>>> import psutil
>>> psutil.net_if_addrs()
{'eth0': [snic(
family=<AddressFamily.AF_INET: 2>,
address='10.0.0.4',
netmask='255.0.0.0',
broadcast='10.255.255.255'),
snic(
family=<AddressFamily.AF_PACKET: 17>,
address='9c:eb:e8:0b:05:1f',
netmask=None,
broadcast='ff:ff:ff:ff:ff:ff')],
'lo': [snic(
family=<AddressFamily.AF_INET: 2>,
address='127.0.0.1',
netmask='255.0.0.0',
broadcast='127.0.0.1'),
snic(
family=<AddressFamily.AF_PACKET: 17>,
address='00:00:00:00:00:00',
netmask=None,
broadcast='00:00:00:00:00:00')]}
This is limited to AF_INET (IPv4), AF_INET6 (IPv6) and AF_LINK (ETHERNET) address families. If you want something more poweful (e.g. AF_BLUETOOTH) you can take a look at
netifaces extension. And here's the code which does these tricks on POSIX and Windows:
Also, here's some
doc.
net_if_stats()
This will return a bunch of information about network interface cards:
>>> import psutil
>>> psutil.net_if_stats()
{'eth0': snicstats(
isup=True,
duplex=<NicDuplex.NIC_DUPLEX_FULL: 2>,
speed=100,
mtu=1500),
'lo': snicstats(
isup=True,
duplex=<NicDuplex.NIC_DUPLEX_UNKNOWN: 0>,
speed=0,
mtu=65536)}
Again, here's the code:
...and the
doc.
Enums
Enums are a nice new feature introduced in Python 3.4. Very briefly (or at least, this is what I appreciate the most about them), they help you write an API with human-readable constants. If you use Python 2 you'll see something like this:
>>> import psutil
>>> psutil.IOPRIO_CLASS_IDLE
3
On Python 3.4 you'll see a more informative:
>>> import psutil
>>> psutil.IOPRIO_CLASS_IDLE
<IOPriority.IOPRIO_CLASS_IDLE: 3>
They are backward compatible, meaning if you're sending serialized data produced with psutil through the network you can safely use comparison operators and so on. The psutil APIs returning enums (on Python >=3.4) are:
All the other existing constants remained plain strings (
STATUS_*) or integers (
CONN_*).
Zombie processes
This is a big one. The full story is
here but basically the support for
zombie processes on UNIX was
broken (except on Linux - Windows doesn't have zombie processes). Up until psutil 2.* we could instantiate a zombie process:
>>> pid = create_zombie()
>>> p = psutil.Process(pid)
...but every time we queried it we got a
NoSuchProcess exception:
>>> psutil.name()
File "psutil/__init__.py", line 374, in _init
raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: no process found with pid 123
That was misleading though because the PID technically still existed:
>>> psutil.pid_exists(p.pid)
True
Furthermore, depending on what platform you were on, certain process stats could still be queried (instead of raising
NoSuchProcess):
>>> psutil.cmdline()
['python']
Also
process_iter() did not return zombie processes at all. This was probably the worst aspect because being able to identify them is an important use case, as they signal an issue with process: if a parent process spawns a child, terminates it (via
kill()), but doesn't
wait() for it it will create a zombie. Long story short, the way this
changed in psutil 3.0 is that:
- we now have a new ZombieProcess exception, raised every time we're not able to query a process because it's a zombie
- it is raised instead of NoSuchProcess (which was incorrect and misleading)
- it is still backward compatible (meaning you won't have to change your old code) because it inherits from NoSuchProcess
- process_iter() finally works, meaning you can safely identify zombie processes like this:
import psutil
zombies = []
for p in psutil.process_iter():
try:
if p.status() == psutil.STATUS_ZOMBIE:
zombies.append(p)
except NoSuchProcess:
pass
Removal of deprecated APIs
This is another big one, probably the biggest. In a previous blog post I already talked about
deprecated APIs. What I did back then (January 2014) was to rename and officially deprecate different APIs and provide aliases for them so that people wouldn't yell at me because I broke their existent code. The most interesting deprecation was certainly the one affecting
module constants and the hack which was used in order to provide "module properties". With this new release I decided to get rid of all those aliases. I'm sure this will cause problems but hey! This is a new major release, right? =). Plus the amount of crap which was removed is impressive (see the
commit). Here's the old aliases which are now gone for good (or bad, depending on how much headache they will cause you):
Removed module functions and constants
| Already deprecated name | New name |
|---|
| psutil.BOOT_TIME() | psutil.boot_time() |
| psutil.NUM_CPUS() | psutil.cpu_count() |
| psutil.TOTAL_PHYMEM() | psutil.virtual_memory().total |
| psutil.avail_phymem() | psutil.virtual_memory().free |
| psutil.avail_virtmem() | psutil.swap_memory().free |
| psutil.cached_phymem() (Linux only) | psutil.virtual_memory().cached |
| psutil.get_pid_list() | psutil.pids().cached |
| psutil.get_process_list() | - |
| psutil.get_users() | psutil.users() |
| psutil.network_io_counters() | psutil.net_io_counters() |
| psutil.phymem_buffers() (Linux only) | psutil.virtual_memory().buffers |
| psutil.phymem_usage() | psutil.virtual_memory() |
| psutil.total_virtmem() | psutil.swap_memory().total |
| psutil.used_virtmem() | psutil.swap_memory().used |
| psutil.used_phymem() | psutil.virtual_memory().used |
| psutil.virtmem_usage() | psutil.swap_memory() |
Process methods (assuming p = psutil.Process()):
| Already deprecated name | New name |
|---|
| p.get_children() | p.children() |
| p.get_connections() | p.connections() |
| p.get_cpu_affinity() | p.cpu_affinity() |
| p.get_cpu_percent() | p.cpu_percent() |
| p.get_cpu_times() | p.cpu_times() |
| p.get_io_counters() | p.io_counters() |
| p.get_ionice() | p.ionice() |
| p.get_memory_info() | p.memory_info() |
| p.get_ext_memory_info() | p.memory_info_ex() |
| p.get_memory_maps() | p.memory_maps() |
| p.get_memory_percent() | p.memory_percent() |
| p.get_nice() | p.nice() |
| p.get_num_ctx_switches() | p.num_ctx_switches() |
| p.get_num_fds() | p.num_fds() |
| p.get_num_threads() | p.num_threads() |
| p.get_open_files() | p.open_files() |
| p.get_rlimit() | p.rlimit() |
| p.get_threads() | p.threads() |
| p.getcwd() | p.cwd() |
| p.set_cpu_affinity() | p.cpu_affinity() |
| p.set_ionice() | p.ionice() |
| p.set_nice() | p.nice() |
| p.set_rlimit() | p.rlimit() |
If your code suddenly breaks with
AttributeError after you upgraded psutil it means you were using one of those deprecated aliases. In that case just take a look at the table above and rename stuff in accordance.
Bug fixes
I fixed a lot of stuff (full list
here), but here's the list of things which I think are worth mentioning:
Ease of development
These are not enhancements you will directly benefit from but I put some effort into making my life easier every time I work on psutil.
- I care about psutil code being fully PEP8 compliant so I added a pre-commit GIT hook which runs flake8 on every commit and rejects it if the coding style is not compliant. The way I install this is via make install-git-hooks.
- I added a make install-dev-deps command which installs all deps and stuff which is useful for testing (ipdb, coverage, etc).
- A new make coverage command which runs coverage. With this I discovered some of parts in the code which weren't covered by tests and I fixed that.
- I started using tox to easily test psutil against all supported Python versions (from 2.6 to 3.4) in one shot.
- I reorganized tests so that now they can be easily executed with py.test and nose (before, only unittest runner was fully supported)
Final words
I must say I'm pretty satisfied with how psutil is going and the satisfaction I still get every time I work on it. Right now it gets almost
800.000 download a month, which is pretty great for a Python library. As of right now I consider psutil almost "completed" in terms of features, meaning I'm basically running out of ideas on what I should add next (see
TODO). From now on the future development will probably focus on adding support for more exotic platforms (
OpenBSD,
NetBSD,
Android). There also have been some discussions on python-ideas mailing list about
including psutil into Python stdlib but, assuming that will ever happen, it's still far away in the future as it would require a lot of time which I currently don't have. That should be all. I hope you will all enjoy this new release.
