Every time I do hashing in Python, I have to look it up. I forget how to do it. That’s probably a bad thing, at least compared to the shell. The shell way isn’t exactly simple, but I ﬁnd it something I can do by rote.
I’m going to write down how I got MD5 and SHA-1 hashes for a ﬁle — which is something I occasionally have to do when posting a download, for example — thereby making it possible to ﬁnd my own perfectly-tailored how-to next time:
>>> import hashlib # hashlib is new in Python 2.5
>>> file_reference=open('/path/to/file', 'rb').read() # open the file for reading, in binary mode
There, that wasn’t so bad. I just have to remember the name of the built-in hashlib module and how to call for a hash of some data with it. You’re missing the twenty other lines I tried which didn’t work, of course, but you don’t really need to see that. Sigh.
Without specifying hexdigest(), the result is a hash object rather than the hash value.
<md5 HASH object @ 0x639c0>
I compared the Python hashlib results above with the following output from OpenSSL, and they are the same:
$ openssl md5 /path/to/file
$ openssl sha1 /path/to/file
On balance, I think I’d still like comparing that hash against another string better in Python, but getting the hashes was quite a bit more confusing to me. It was enough to interrupt my ﬂow.
Wow, Apple bought the Common UNIX Printing System (CUPS) back in February, and the announcement has just come out. (I have to wonder why the delay … perhaps it has something to do with Leopard?) The software continues to be licensed under its regular terms.
Michael Sweet, one of the principals behind Easy Software Products and developer of CUPS, is now an Apple employee.
I’m sure this all means something.
In reviewing Jesper’s list of requirements for The Email Client That Doesn’t Suck, I was somewhat surprised how many of his points are already handled by Microsoft Entourage 2004.
I’d give it 19 out of 26 points. There are some places where I’m being charitable towards Entourage, partly because it can support the requirement with a little work (which does not always mean scripting — and it should be noted that Entourage is very scriptable) or I didn’t understand what Jesper meant by the requirement.
Many Mac users discount Entourage. There are a couple of reasons that may be cited:
That said, there are many valid concerns about Entourage. I voice many through the Microsoft feedback channels available to me.
However, I think there is a huge impediment to creating a new e-mail client today, simply because of how connected this kind of product is to your whole computing experience. Any developer should take that into consideration, and realize that it’s probably an unending eﬀort.
Mentioning Plugin <em></em> Not Found reminded me that I’d never tooted my own horn here regarding another press placement. Why? I was interviewed by Lisa Nadile for CIO Magazine back in spring, and the article appeared quite some time ago (after I’d almost forgotten about it or thought I’d missed it).
So, read below the fold to ﬁnd my quoted moments — not necessarily quotable moments, mind you — in the article: Hi, I’m a Mac and I’m your Enterprise Computer.
The small corner of the ‘net I inhabit is already abuzz with things that have broken with the Mac OS X 10.4.10 update. Software seems to be breaking not because of signiﬁcant changes in this update, but because of poor version checking routines.
Awesome — isn’t it?
Anyway, here are some reports I’ve heard:
Yeah, I know … I guess I’m becoming a Python fanboy. Apologies.
From Python 2.3.5, installed by default in Mac OS X Tiger:
>>> import sys
['', '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python23.zip', '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3', '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.3/Extras/lib/python']
>>> for z in sys.path:
... if '2.3' not in z:
>>> import sys
['', '/Library/Frameworks/Python.framework/Versions/2.5/lib/python25.zip', '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5', '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-darwin', '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac', '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk', '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload', '/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages']
>>> for z in sys.path:
... if '2.5' not in z:
This simply means there are no directories in common in sys.path — the list of directories where modules can be installed — between these two versions of Python. I ﬁnd that a bit annoying, since I can’t by default count on deploying one module to a single location that will work in both the default and upgraded versions of Python.
That lack of a default common location also has an impact if you’re managing the ﬁlesystem with Radmind.
Yesterday, I thought I’d came across a potential solution to a problem I’d been having with restoring disk images to target drives using Disk Utility. I keep getting error 22, “invalid argument,” when restoring images over HTTP, using Disk Utility on the original Tiger version of the Mac OS X Install DVD.
While there could be some form of image ﬁle corruption, restoring the same images locally — over FireWire target disk mode, for example — appears to succeed reliably and repeatedly. (See also this+Shirt+Pocket+Software+thread, where the core error was also not resolved.)
Let’s say that you have your images stored on a Web server — preferably one with some access controls, because you don’t want your system images open to just anyone. That should let you use Apple Software Restore’s HTTP-based image restore feature. You have already prepped the images for ASR, creating them appropriately (possibly through the use of shadow ﬁles) and applying the volume-level checksum that is required for block-level restores.
However, let’s assume that your naming convention for system images includes more than one period. After all, you’re creating images for Mac OS X versions like 10.3.9 and 10.4.8 … it’s natural to want to use those version numbers somewhere in your system image names. There’s no sense in changing those periods to some other character, right?
Well, I surmised there might have been a reason to change or remove those periods. I expected that removing — or encoding with %2e (as noted in this URL encoding reference) — the periods from a test image’s name would have a beneﬁcial eﬀect, to let me restore it successfully via HTTP. However, it was still a no-go with the encoded characters, so I’m now back where I started. The image restoration works until near the end of the process, and I get error 22 again. Grr.
I’m wondering out loud now whether there’s a problem with automount, since I’ve seen some search results that seem to mention error 22 in conjunction with it. I have no clue why it would fail only for HTTP-based ASR restores using a Tiger install DVD, though.
It seems I can recall the restore-via-HTTP feature working at some point in the past, but it certainly hasn’t done so to my recollection under Tiger.
When deploying system software with disk images, it is helpful to have various checkpoint images that you can revert to while you’re building up a fully-ﬂedged template computer. This is something they teach you in school (really, I was taught it in a systems administration class) and it’s more or less encoded in the solution accelerator documentation for Microsoft’s Business Desktop Deployment 2007 for Windows.
However, if you’re updating images, keeping the base and intermediate images can strain your storage capacity. Mac OS X lacks the compelling live editing features of Microsoft’s new WIM image format — which if it had appeared ﬁrst on the Mac, I’d be trumpeting loudly, so I feel compelled to at least give a nod to Microsoft here.
Since I’m always struggling with storage capacity and I prefer having an up-to-date base image, I thought about this problem a bit in the context of Mac OS X imaging and have come upon what seems to be a unique solution: the use of shadow ﬁles.
Here’s the basic idea:
Congratulate yourself on this use of shadow ﬁles, because you’ve saved at least one intermediate step and the space required for a full read-write disk image — or worse, an extra local partition needed only for restoration and updating the base image.
Unmount that volume and throw away the shadow ﬁle at this point if you want, because you’ve now got two system images ready for deployment. One has the base system software, and serves as a checkpoint that you can return to later; it’s the base for all future updates of that major revision of Mac OS X. The other image has the latest version of Mac OS X. If you’re deploying that image with ASR, the result will be a more secure system because it’s closer to being fully patched — and it should take less time to update it with the additional security updates and application installs — whether you use installers or Radmind or another solution — because you’ve got the bulk of the operating system done.
Unfortunately, many updates can only be installed on the startup disk and thus cannot be included in the updated base image. Beyond the combo operating system updates, few of Apple’s other installers will work on a non-startup volume. But you can install them after deploying the updated base image, using your tool of choice. For reasons like this, Geoﬀ doesn’t see updating the base image as valuable, but in some IT environments, it may be very worthwhile.
My next step is to script this process and tie it to a watched folder. Imagine dropping a combo update into a watched folder … and letting a script generate the new, updated image for you.
I wanted to determine how much space I’d save—and what the compression time tradeoff might be—to compress Mac OS X system images with gzip versus bzip2. Both are options in Mac OS X Tiger, but the bzip2 compression is new in that release (and thus not backward compatible with Panther and earlier). Given that Apple’s own disk imaging is the bedrock of almost all imaging solutions from Mac OS X, including those from third party computer management solutions providers, this information is valuable and widely applicable on the Mac platform.
If the time were the same but I saved some disk space, I’d count that as a win for one compression choice or the other. For significant space savings, I’d even sacrifice time to a longer conversion operation because disk image compression and ASR prep can be handled in the background during downtime or operational lulls. A Folder Action or launchd task might be ideal for that, and might even be a useful option if you are just archiving images.
The two disk image formats I tested are UDZO and UDBZ, as defined by the
hdiutil command and its man page.
For an UDZO version of the image (gzip compression at level 9—or best),
I got the following results from
Elapsed Time: 1h 1m 50.751s
File size: 4226869141 bytes, Checksum: CRC32 $8CF3F127
Sectors processed: 20184610, 19980489 compressed
I ran the
hdiutil command with
time (a handy utility if you haven’t used it), to get these figures:
For the same image converted to UDBZ (bzip2 compression):
Elapsed Time: 2h 16m 28.057s
File size: 3956667545 bytes, Checksum: CRC32 $8CF3F127
Sectors processed: 20184610, 19980489 compressed
The time results for that were:
You may draw your own conclusions, as appropriate to the data and your situation. The source image was the same size, nearly 10 GB, which included Mac OS X Tiger and a suite of applications with a small number of user accounts. The source and converted files were both stored on an external FireWire 400 hard disk, and converted with a 1.5-GHz 12-inch PowerBook G4.
Although the ASR-prepped (i.e.
asr --imagescan) gzip and bzip2 images were not that different in the end, the bzip2 version definitely came out smaller. It has in ever test I’ve ever run. However, it did take twice as long to create the image. So, if you’re strapped for disk space to collect your images and CPU time means less to you than storage (which is often the case for me, given that I have access to a lot of G4-class Mac hardware), I would consider bzip2. However, gzip at level 9 is perfectly acceptable (still providing almost 60% savings in this case), is faster to create, and has the benefit of working on older versions of Mac OS X, as well.
I should try gzip at other settings (like levels 1 and 6, for example), because you can’t choose level 9 in the Disk Utility interface; you must use the command line -imagekey flag. (You can get bzip2 from the GUI if you enable the hidden advanced disk image formats, but I still don’t see a way to choose a higher gzip level.)
The use of the
defaults command in Mac OS X is something of a black art. For one thing, you need to delve into property lists to determine what property you want to change.
Therefore, I didn’t want to resort to it to set Safari’s “open safe downloads” preference, but it looks like that particular setting is regrettably not one that you can control with Workgroup Manager (or the Apple MCX schema for managed clients).
So, to make that settings change at the command line or through a script, you can use the following tips. (If you have to do something much more advanced with plists than this, you may want to consider using PlistBuddy, a propertly list manipulation tool which Apple includes in their installers.)
defaults read, you can read the AutoOpenSafeDownloads property in the user’s domain:
% defaults read com.apple.Safari AutoOpenSafeDownloads
That will return 0 or 1, the former meaning the “off” or “no” and the latter meaning “on” or “yes.”
To turn AutoOpenSafeDownloads off, use the
defaults write command with the -bool option at the command prompt:
% defaults write com.apple.Safari AutoOpenSafeDownloads -bool NO
To turn it on, change the boolean option to YES:
% defaults write com.apple.Safari AutoOpenSafeDownloads -bool YES