Jaharmi’s Irreality renewable hyperlinks with 0% transfat since 1999

Irrational Exuberance has An Epic Introduction to PyObjC and Cocoa. Skimming over it, it really does seem to start at the ground level — and that’s what I need. I have high hopes it will help me figure out PyObjC.

[Via PyObjC Development mailing list.]

While trying to troubleshoot what I’d done to mess up the Mercurial repositories managing my Drupal installations last weekend, I really would have liked a way to see what files had changes in specific revisions. Each revision to a Mercurial repository affects some files, of course, but it seems awfully hard to figure what files changed in that check-in.

I have since found a way to do that by customizing the output of Mercurial. To customize output, you can create templates on the command line (with --template) or for more powerful reformatting, create an output style file.

I struggled for a while to figure out how to use style files, and eventually came up with something that works for me so far.

Since I’ve installed Mercurial from Lee Cantey’s standard binary package for Mac OS X Leopard, I created the file “map-cmdline.changedfiles” at the “/Library/Python/2.5/site-packages/mercurial/templates” path. (Where you put the file may vary depending on where Mercurial is installed, and I’m sorry but I don’t know where it gets installed on other systems.) The contents of “map-cmdline.changedfiles” are below, along with my possibly inept description of what each line is doing:

# Get all of the files in the selected revision
# and stringify them, whatever that means
# but do not 'tabindent' or wrap them to 68/76 columns
# Without first setting changeset to the list of files
# you won't get output from subsequent lines
changeset = '{files|stringify}'
# List modified files, one per line
# preceded by M to mimic `hg status`
file = 'M {file}\n'
last_file = 'M {file}\n'
# List added files, one per line
# preceded by A to mimic `hg status`
file_add = 'A {file_add}\n'
last_file_add = 'A {file_add}\n'
# List deleted files, one per line
# preceded by ! to mimic `hg status`
file_del = '! {file_del}\n'
last_file_del = '! {file_del}\n'

I don’t know why the “map-cmdline.” portion of the filename is there, but as long as I have it, I can call the style file from the command line with what follows the period. So, I can call the style with “--style changedfiles” — and that tiny bit of voodoo seems reasonable enough to me. (The other styles in the directory above, many of which end in “.tmpl” extensions, seem related to the Mercurial Web server, hgweb. I tried, but I couldn’t use their names at the command line, with or without their extensions. Plus, their contents looked HTML-ish.)

With the “map-cmdline.changedfiles” style file saved in that location, I can call Mercurial’s “log” command:

$ hg log --style changedfiles -r tip

… which gives me a list of the files changed in the “tip” (or latest revision) of the repository. I could substitute in any revision identifier for “tip.”

I haven’t actually seen the “file_add” and “file_del” keywords in action; every time I’ve used this style file in the manner described, I’ve only seen files marked as “M” — even if I’m looking at a revision where new files were first checked into the repo. I’m confused by that, but I’m not going to let it sour my day at this point.

There might have been an easier way to do this but I didn’t find one last weekend. It took me some time to figure even this bit out, and I hope writing this post saves someone new to Mercurial from future frustration.

I have been struggling with the issue of module availability in Python. While the “batteries included” nature of the standard library is great, there are occasionally times when I need to resort to a module that isn’t included with Python.

There are also times where I’m using modules whose status has changed. I expect that to happen more in the eventual transition to Python 2.6 and 3.0, because I’ve used modules that are being deprecated.

So I wondered how I could conditionally import a module if it was available, without stopping the flow of my scripts — and gracefully handle situations where it is missing. And here’s one basic answer: use a “try” block to catch the “ImportError” exception. For example, if I were concerned that DNSPython wasn’t going to be installed on my target system:

try:
import dns.resolver # Import DNSPython
dnspython_available = True
except ImportError:
dnspython_available = False

The “except ImportError” clause could specify a different module to load, or other workarounds entirely. You could map the namespaces in the “try” bock so the rest of your script doesn’t notice the change in module functions, if you have a way to work around the missing module. Perhaps, you could even try to obtain and install the module, at least for temporary use by your script.

Thanks to authors of the article Python modules – how do they work? for the assist. The information under heading 2.11, “Is my module available?” answered my question and has given me something to think about.

Mac OS X has a type of URL specifically for opening UNIX man pages. For me, using one of these URLs opens a new Terminal window to display the man page. Just put the name of the man page after the “x-man-page://” URL scheme to create one of these URLs.

For example, a link to the rsync man page would look like “x-man-page://rsync” when written out.

This is a handy way to refer to UNIX man pages with other Mac OS X users in e-mail correspondence, on mailing lists, or on the Web. Because of that, I wanted a quicker way to create these man page URLs. I wrote the Mac OS X Service named “Man Page URL for Command” to satisfy that desire. The “Man Page URL for Command” Service is provided “as-is” with no warranty.

The Service was wrapped up with ThisService by Peter Hosey, as with my earlier Mac OS X Service to shorten a URL with Bit.ly.

To install the Service:

1. Unzip the downloaded file.
2. Drag the unzipped Service to /Library/Services (for all users) or ~/Library/Services (for just the current user).

To use it:

1. Select a single word from some text you can highlight in an application. The word should ideally correspond to the name of a general UNIX command (“ls”), a Mac OS X-specific command (“dsmemberutil”), or another term the matches a man page (“sshd_config”). The Service is specifically coded to work only when you select a single word; if you select a text string with any white space in it, the Service will just return your original text. The Service also checks to ensure that the selected word corresponds to either a command in the $PATH or an existing man page, so it will not generate a “x-man-page://” for just any word.
2. Choose “Man Page URL for Command” from the Services submenu of the application menu. As long as you have selected only a single word that matches a command or existing man page, the selection will be replaced with the “x-man-page://” URL.

Thanks to Nigel for the inspiration.

Since the Bit.ly URL-shortening service is all the rage lately, and I hadn’t seen anyone create a Mac OS X Service for it yet, I decided to try my hand at it.

Here’s the result. The core is a relatively simple Python script and requires Mac OS X 10.5 (or Python 2.5 if you have an earlier version of Mac OS X). The Service was wrapped up with ThisService by Peter Hosey. It’s my first attempt at creating a Mac OS X Service — with or without ThisService — and I hope it works for you. However, it is provided “as-is” with no warranty.

That said, if you have comments or suggestions, please feel free to contact me.

Also, I’m not counting this as an endorsement of Bit.ly; I just looked at info on their site and thought I could probably script it and, as you can see, I did.

To install the Service:

1. Unzip the downloaded file.
2. Drag the unzipped Service to /Library/Services (for all users) or ~/Library/Services (for just the current user).

To use it:

1. Select the full text of an “http” or “https” URL that you’d like to shorten. (Don’t just select a site’s domain name. The Service is specifically coded to work on full URLs starting with “http” or “https.” So, “www.jaharmi.com” will not work but “http://www.jaharmi.com” will.)
2. Choose “Shorten URL with Bit.ly” from the Services submenu of the application menu. The selected URL will be replaced with the text of the shortened URL from Bit.ly. If it is not, the original URL will remain.

Of special note, however, is that if you shorten URLs with Bit.ly this way rather than through your browser, you probably won’t see them show up in your history (the most recent 15 URLs you’ve shortened). The script just shortens URLs for you and does so outside of your browser, so whatever cookies or other tracking Bit.ly is doing to generate your history, it doesn’t appear to carry over when using this simple little Service.

Drat! I’ve learned that the commands module for Python, which I use, is deprecated and removed in Python according to PEP 3108. That means I can no longer safely call commands.getstatusoutput() anymore. I’ve frequently used this call in the past because it seemed the sanest, easiest way to call for a shell utility and get both its output and return status (for success or error).

I’ll have to find some other way to perform the same function — preferably one that will work on Python 2.3 from Mac OS X Tiger, Python 2.5.1 from Leopard, and future Pythons. The stated replacement for a number of similar modules (including popen2, which frankly kind-of frightened me off with its name) is the subprocess module from PEP 324, but I don’t know if that will work for my purposes.

There are also a bunch of Mac-specific modules being removed. I don’t use any of them right now, but that doesn’t mean they wouldn’t have been useful.

This kind of thing is spirit-crushing to me for some reason. I’m especially annoyed that Python has been around for so long and it is still reorganizing the ways it calls shell commands. Just settle on something! It seems hard to take it seriously as a system administration scripting language when things like this happen.

On the other hand, I love so much of the Python Standard Library, which has afforded me a lot for system administration …

Under normal circumstances, the latest Radmind tools that communicate with the server report client status updates in the Radmind server’s system log. These standard messages can include ones like:

May 8 03:14:56 RadmindServerHost radmind[7890]: report radmind-client.example.com 192.168.7.42 - - ktcheck No updates needed
May 15 03:15:25 RadmindServerHost radmind[24531]: report radmind-client.example.com 192.168.7.42 - - ktcheck Updates retrieved
May 15 03:21:48 RadmindServerHost radmind[24534]: report radmind-client.example.com 192.168.7.42 - - lapply Changes applied successfully
May 15 03:31:07 RadmindServerHost radmind[24356]: report radmind-client.example.com 192.168.7.42 CertificateCN - lapply Error, changes made

The Radmind repo, or “report,” tool provides the ability to send arbitrary messages to the Radmind server process. But how are these messages formatted and sent?

$ repo -e "Debug" -h radmindserverhost.example.com -w2 "Test message"

… results in the system log message:

May 15 03:31:56 RadmindServerHost radmind[25236]: report radmind-client.example.com 192.168.7.42 CertificateCN - Debug Test message

Here, we can see that an entry created with repo looks like the standard Radmind log messages above. The client hostname and IP address are reported after the “report” text. The CertificateCN for the client — if the highest authorization level is specified (with the -w2 flag) — is also listed; if not, a dash takes its place. I haven’t seen a case where the second dash is substituted, however.

Finally, where the Radmind command/tool used would normally be, the “event” specified by repo will printed. After that, the message text appears.

The value proposition is that if you’re using Radmind, the repo command can help you send arbitrary messages to the server for logging. As bonus, if you've taken the time and effort to build the certificate infrastructure for Radmind, you can send these messages securely between the clients and the server cloaked in SSL.

If you’re using multiple servers, you may want to combine their logs in one location so that you can get all of the clients’ reports in one location. You may also want or need to retain these reports for more time. In either case, determine what policies you should apply to the syslog or Apple System Logger (ASL, for Mac OS X) configuration for your server systems.

Whether or not you use repo, it’s good to know that the tools do some logging. The logging can be followed to try to determine the status of your clients, or whether they are failing their updates.

Unfortunately, the most common client failures I have seen tend to involve the lapply tool, and the default level of detail I’ve seen reported back to the server does not provide an indication of what problem has been encountered. You see only that there was an error. Still, even though you may not get enough detail to remotely resolve the problem, it’s something for you to go by find problems in the first place.

I’m constantly astounded by the breadth of features available in the Python Standard Library. Although the functions I find there are not always easy to grasp, it is almost always worth searching around a bit for a function or method in the standard library before I write my own code to do something someone else has probably had to do before.

Take the “locale” module. It lets you format certain kinds of data based on your locale and its customs. Numbers (including currency) happen to be one of its specialties. Since I had a need to output long numbers whose digits were grouped with commas — which makes them easier to read — and locale.format() does just that. Even better, it’s internationalized and formats them for your system’s own locale.

>>> import locale
>>> a = {'size': 123456789, 'unit': 'bytes'}
>>> print(locale.format("%(size).2f", a, 1))
123456789.00
>>> locale.setlocale(locale.LC_ALL, '') # Set the locale for your system
'en_US.UTF-8'
>>> print(locale.format("%(size).2f", a, 1))
123,456,789.00

In trying to use it in the bundled Python 2.5.1 on Mac OS X Leopard, I noticed that the default for scripts and the interpreter doesn’t format numbers as I expected for my locale. I found that I needed to set a locale to get the expected formatting. To do this, I run locale.setlocale(), as above. I’m not sure if this is required for other Python installations, but it’s worth mentioning.

One difficulty I created for myself was when I tried mixing more into my format string. For example, locale.format() would fail to reformat a number when I added in the string for the unit from my original dictionary:

>>> print(locale.format("%(size).2f %(unit)s", a, 1))
123456789.00 bytes

In retrospect, this makes total sense, but it took Mark looking over my code to discover my error.

I ran a quick test of the Python “title” string method (from Python 2.5.1 in Leopard); it fails the tests on the edge cases mentioned in the Title Case post at Daring Fireball. It really only uppercases the first letter of every word and lowercases everything else.

Not much better than looping through a string with ucfirst() in Perl, except you don't have to do the looping yourself.

Drat. I was expecting better.

The Distutils Python module includes functions to obtain information about the Python installation. This may be useful for system administrators, and it certainly caught my eye when I read about it.

The results below are from Apple’s bundled build of Python 2.5.1 in Mac OS X Leopard. Credit for the comments describing each function comes from the distutils.sysconfig documentation.

>>> import distutils.sysconfig
>>> distutils.sysconfig.get_python_version() # Get the major Python version without patchlevel
'2.5'
>>> distutils.sysconfig.get_python_lib(standard_lib=True) # Return the directory containing the Python library; if 'standard_lib' is true, return the directory containing standard Python library modules
'/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5'
>>> distutils.sysconfig.get_python_lib() # Otherwise, return the directory for site-specific modules
'/Library/Python/2.5/site-packages'
>>> distutils.sysconfig.get_python_lib(plat_specific=True) # Return any platform-specific modules from a non-pure-Python module distribution
'/Library/Python/2.5/site-packages'

You’d install your own modules for system-wide use in the directory returned by distutils.sysconfig.get_python_lib().