Shell

Compiling a five-way Universal Binary of Rsync 3 with LLVM-GCC on Leopard

The Xcode 3.1 Developer Tools install both the standard GNU compiler, gcc, and the LLVM compiler, llvm-gcc. I thought I’d try my hand at compiling something with LLVM, so naturally I decided to update my instructions for compiling Rsync 3 as a Universal Binary for Leopard. Let’s try Rsync 3.0.3!

The following set of instructions, of course, requires that you have already installed the Xcode 3.1, available as a download from Apple. To choose LLVM as your C compiler temporarily for this compile, specify the CC=“/Developer/usr/bin/llvm-gcc” variable (as explained in the LLVM FAQ).

$ cd /tmp
$ curl -O curl -O http://rsync.samba.org/ftp/rsync/rsync-3.0.3.tar.gz
$ tar -xzvf rsync-3.0.3.tar.gz
$ rm rsync-3.0.3.tar.gz
$ curl -O http://rsync.samba.org/ftp/rsync/rsync-patches-3.0.3.tar.gz
$ tar -xzvf rsync-patches-3.0.3.tar.gz
$ rm rsync-patches-3.0.3.tar.gz
$ cd rsync-3.0.3
$ patch -p1 <patches/fileflags.diff
$ patch -p1 <patches/crtimes.diff
$ ./prepare-source
$ env CFLAGS="-O -g -isysroot /Developer/SDKs/MacOSX10.5.sdk -arch i386 \
-arch ppc7400 -arch ppc970 -arch ppc64 -arch x86_64" LDFLAGS="-arch i386 \
-arch ppc7400 -arch ppc970 -arch ppc64 -arch x86_64" \
CC="/Developer/usr/bin/llvm-gcc" \
./configure --prefix=/usr/local --disable-dependency-tracking

$ make
$ ./rsync --version
rsync version 3.0.3 protocol version 30
Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 32-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, no iconv, symtimes, file-flags
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
$ file ./rsync
./rsync: Mach-O universal binary with 5 architectures
./rsync (for architecture i386): Mach-O executable i386
./rsync (for architecture ppc7400): Mach-O executable ppc
./rsync (for architecture ppc970): Mach-O executable ppc
./rsync (for architecture ppc64): Mach-O 64-bit executable ppc64
./rsync (for architecture x86_64): Mach-O 64-bit executable x86_64
$ lipo -info ./rsync
Architectures in the fat file: ./rsync are: i386 ppc7400 ppc970 ppc64 x86_64
$ sudo make install

Warning: I have not tested Rsync 3 compiled with LLVM to ensure that it operates at all or is compatible with Rsync compiled by GCC. I haven’t compared the compilation speed against GCC’s performance, determined whether the files are larger/smaller, or have produced a more optimized binary. You use the software compiled with these instructions at your own risk.

Most of the original instructions were helpfully documented by Mike Bombich. My contributions are simply:

  • Universal Binary support: four- or five-way, depending on whether you want to use “-arch i386 -arch ppc7400 -arch ppc970 -arch ppc64 -arch x86_64” (which gives specific optimized binaries for PowerPC G4 and G5 systems in 32-bit mode) or “-arch i386 -arch ppc -arch ppc64 -arch x86_64” (for more general 32-bit PowerPC support, as I did in my previous article).
  • LLVM compilation: use of the llvm-gcc front end, which hopefully produces a working Rsync 3.

Start Directory Service debug logging automatically after the next restart on Mac OS X

There can be times when you need to troubleshoot Directory Services in Mac OS X during system starts up. Directory Services apparently sits above and below the kernel — depending on what each of those components needs to do — so having debugging capability early on in the boot process can help you hone in on problems.

To enable debug logging at startup, run the following at the command line:

$ sudo touch /Library/Preferences/DirectoryService/.DSLogDebugAtStart

… and then, on the next restart, debug logging will be enabled. This is equivalent to running:

$ sudo killall -USR1 DirectoryService

… but the advantage is is that debug logging begins at startup, rather than whenever you can log in and start it manually.

Remove the file from the path above when you want to disable the logging for the subsequent restart. As a file whose name starts with a dot, you won’t see it listed by default in the Finder. (The Finder hides dot files by default.)

$ sudo rm /Library/Preferences/DirectoryService/.DSLogDebugAtStart

For more details, see the Apple article on Mac OS X Server v10.5, 10.6: Enabling Directory Service debug logging. (Note that directory service logging works on the workstation version of Mac OS X, not just Mac OS X Server.)

Is that site running Drupal, curl edition

The Lullabot Web site has a clever way to help answer the question of “Is that site running Drupal?” Angie Byron mentions that the HTTP “Expires” header returned by Drupal corresponds to a specific default date. Look for that date in the HTTP headers, and you can make a reasonable guess that a site is a Drupal site — or at least one that hasn’t modified one of the core files.

Some commenters posted notes about how to do the same thing with wget and then curl. I expanded on the curl instructions to make them a little more robust (especially in the case of redirects or URL rewriting), and here’s the result:

$ curl -fsIL http://jaharmi.com/ 2>&1 | grep -q -m 1 "Expires: Sun, 19 Nov 1978 05:00:00 GMT" && echo "Yes, this appears to be a Drupal site." || echo "No, this does not appear to be a Drupal site."
Yes, this appears to be a Drupal site.

Navigating through files with less

It’s taken about a decade but I finally read the man page for “less.” Here are some important keyboard commands for navigating through files with it that I wanted to note for my own future use. If someone else gets a benefit from this, great, but it’s already helping me jump through man pages with more ease. (And it gave me an opportunity to write a post with a really ambiguous title.)

Key Function
f Move forward one screen
b Move backward one screen
j Move forward one line
y Move backward one line
= Show the number of lines and your progress through the file
< Move to the beginning of the file
> Move to the end of the file
% Scroll to the position in the file represented by the number before the percent sign

These commands had eluded me for a long time, even though it would have been great to know them. After all, less is the default pager in Mac OS X. Thanks to the wonders of air travel, I had some time to read some off-line documentation.

Of course, I’ve long used “/” followed by some text to search through man pages, and that’s worth knowing if you don’t already. One useful trick I picked up was the search “/^EXAM” (props to Adam for mentioning the caret and making my recipe one character longer, but more specific) to search for the examples section.

Python to remove commands module and some Mac-specific modules

Drat! I’ve learned that the commands module for Python, which I use, is deprecated and removed in Python according to PEP 3108. That means I can no longer safely call commands.getstatusoutput() anymore. I’ve frequently used this call in the past because it seemed the sanest, easiest way to call for a shell utility and get both its output and return status (for success or error).

I’ll have to find some other way to perform the same function — preferably one that will work on Python 2.3 from Mac OS X Tiger, Python 2.5.1 from Leopard, and future Pythons. The stated replacement for a number of similar modules (including popen2, which frankly kind-of frightened me off with its name) is the subprocess module from PEP 324, but I don’t know if that will work for my purposes.

There are also a bunch of Mac-specific modules being removed. I don’t use any of them right now, but that doesn’t mean they wouldn’t have been useful.

This kind of thing is spirit-crushing to me for some reason. I’m especially annoyed that Python has been around for so long and it is still reorganizing the ways it calls shell commands. Just settle on something! It seems hard to take it seriously as a system administration scripting language when things like this happen.

On the other hand, I love so much of the Python Standard Library, which has afforded me a lot for system administration …

Radmind server logging and the repo command

Under normal circumstances, the latest Radmind tools that communicate with the server report client status updates in the Radmind server’s system log. These standard messages can include ones like:

May 8 03:14:56 RadmindServerHost radmind[7890]: report radmind-client.example.com 192.168.7.42 - - ktcheck No updates needed
May 15 03:15:25 RadmindServerHost radmind[24531]: report radmind-client.example.com 192.168.7.42 - - ktcheck Updates retrieved
May 15 03:21:48 RadmindServerHost radmind[24534]: report radmind-client.example.com 192.168.7.42 - - lapply Changes applied successfully
May 15 03:31:07 RadmindServerHost radmind[24356]: report radmind-client.example.com 192.168.7.42 CertificateCN - lapply Error, changes made

The Radmind repo, or “report,” tool provides the ability to send arbitrary messages to the Radmind server process. But how are these messages formatted and sent?

$ repo -e "Debug" -h radmindserverhost.example.com -w2 "Test message"

… results in the system log message:

May 15 03:31:56 RadmindServerHost radmind[25236]: report radmind-client.example.com 192.168.7.42 CertificateCN - Debug Test message

Here, we can see that an entry created with repo looks like the standard Radmind log messages above. The client hostname and IP address are reported after the “report” text. The CertificateCN for the client — if the highest authorization level is specified (with the -w2 flag) — is also listed; if not, a dash takes its place. I haven’t seen a case where the second dash is substituted, however.

Finally, where the Radmind command/tool used would normally be, the “event” specified by repo will printed. After that, the message text appears.

The value proposition is that if you’re using Radmind, the repo command can help you send arbitrary messages to the server for logging. As bonus, if you’ve taken the time and effort to build the certificate infrastructure for Radmind, you can send these messages securely between the clients and the server cloaked in SSL.

If you’re using multiple servers, you may want to combine their logs in one location so that you can get all of the clients’ reports in one location. You may also want or need to retain these reports for more time. In either case, determine what policies you should apply to the syslog or Apple System Logger (ASL, for Mac OS X) configuration for your server systems.

Whether or not you use repo, it’s good to know that the tools do some logging. The logging can be followed to try to determine the status of your clients, or whether they are failing their updates.

Unfortunately, the most common client failures I have seen tend to involve the lapply tool, and the default level of detail I’ve seen reported back to the server does not provide an indication of what problem has been encountered. You see only that there was an error. Still, even though you may not get enough detail to remotely resolve the problem, it’s something for you to go by find problems in the first place.

Python string method for title case, versus the Daring Fireball script

I ran a quick test of the Python “title” string method (from Python 2.5.1 in Leopard); it fails the tests on the edge cases mentioned in the Title Case post at Daring Fireball. It really only uppercases the first letter of every word and lowercases everything else.

Not much better than looping through a string with ucfirst() in Perl, except you don’t have to do the looping yourself.

Drat. I was expecting better.

Computational groups for Leopard directory services

There are occasionally times in system administration where I feel as if I’m on the bleeding edge. They are not frequent … which probably puts me somewhere much closer to the lunatic fringe. These are also the times when no one else probably cares about the same things — but no matter, I generally have my own pragmatic reasons for them. This is one of those times, and I have waited for this particular feature for a while. I’m happy to have discovered it.

Let’s set up a scenario: you have a Mac OS X Tiger system bound to an Active Directory. All of the thousands of Active Directory users are available on that Tiger system; any one of them could log in to that computer. This isn’t bad if you only allow logins at the login window. However, if you enable any sharing services, the game changes and your exposure widens.

What if you don’t want everyone in your enterprise directory to log in? This gets a little scary to think about. What if you want to limit those who can log in, particularly for services like SSH? How do you let some users in but keep others out — even going to the point of denying access to whole classes of users?

There are a number of things you can do, some at the client and some in the directory. Unfortunately, some kinds of configuration must be done at the client. There isn’t a great way I’m aware of to distinguish between local and network accounts, complicates the situation for the administrator. With Open Directory in Tiger, any user from any directory service was as valid as any other. There will probably be some manual effort — ongoing effort at that — required to restrict access.

Enter Mac OS X Leopard. It now has “computational groups” in its Open Directory client. They are like smart groups in your local node. The memberships of these groups are calculated, rather than static. They are more like query-based groups.

For the scenario above, the two computational groups that provide the most utility are probably “localaccounts” and “netaccounts.” They allow you to distinguish users who are in the DSLocal database from those in Active Directory. This magic gives the glue we need to maintain finer-grained access controls automatically over time; you could use these groups for the sshd server configuration or in the authorization database at /private/etc/authorization, as examples.

Let’s try the groups out with dsmemberutil, with which we can determine whether or not a user is a member of a specific group. What about the first local user, typically created by the Mac OS X Setup Assistant?

$ dsmemberutil checkmembership -u 501 -G localaccounts
user is a member of the group
$ dsmemberutil checkmembership -u 501 -G netaccounts
user is not a member of the group

Digging further, all of the computational groups I’ve found had the same comment. We can search for that to find the rest, to find all of the computational groups that ship with a Leopard client.

$ dscl . -search /Groups Comment "Group membership calculated by system" | awk '/user|account/ { print $1 }'
authedusers
consoleusers
interactusers
localaccounts
netaccounts
netusers

From that list, we can determine more information about the groups:

Short name Real name GID Notes
authedusers Authenticated Users 50
consoleusers Terminal Server User 53
interactusers Interactive 51
netaccounts Network Accounts 61 Accounts from a non-local database
localaccounts Local Accounts 62 Accounts residing only in one of the local databases
netusers Network 52

Unfortunately, I couldn’t find any other documentation about the computational groups. If you do find some, I’d love to hear about it.

Get the UUID for a user quickly with dsmemberutil on Leopard

Never let it be said that Apple’s Open Directory command lines tools make complete sense. After all, why would you think to use the new dsmemberutil tool in Leopard to find the Universally Unique Identifier (UUID) for a user account?

You would have if you had read the first line of its man page, maybe, where it outlines exactly how miscellaneous — and I say that in the kindest possible way — the command is: “dsmemberutil — various operations for the membership APIs.”

Enough lead in; the UUID for a user (as well as a group) can be found with this tool:

$ dsmemberutil getuuid -u 501
E4BDC9C9-AD89-4CD6-BE25-16B197669B47

Choose a user with either -u with the userid, or -U with the short username.

One way this can be exceedingly useful is if you want to track down the shadow password file corresponding to a specific user account:

$ sudo ls -1 /private/var/db/shadow/hash | grep `dsmemberutil getuuid -u 501`
E4BDC9C9-AD89-4CD6-BE25-16B197669B47
E4BDC9C9-AD89-4CD6-BE25-16B197669B47.state

Hey, look, the UUIDs match, saving a trip to dscl!

Dissecting group membership with dsmemberutil in Leopard

There are times when you need to determine whether a user is a member of a group or not. Knowing the membership of a group can have an impacts on security or operation of system administration scripts. The dsmemberutil tool in Mac OS X Leopard lets you perform checks like this on the command line or in scripts.

You feed it a group and a user, and the tool tells you whether or not the user is a member of the group. This works with group nesting, thanks to the wonder of the memberd functionality first available in Mac OS X Tiger.

$ dsmemberutil checkmembership -u 501 -G admin
user is a member of the group
$ dsmemberutil checkmembership -u 502 -G admin
user is not a member of the group

Unfortunately, you must parse the output, which hinges on whether “not” appears in the text. The exit code for both commands above is “0,” for success.

Syndicate content