I was trying to install Dulwich on a Mac OS X Lion system this week and ran into difficulty. I kept getting installation failures that included a missing “python.h” and, eventually, llvm-gcc-4.2 failed to compile the module.
I found the situation frustrating, partly because I pretty much own the top search hits about how to install Dulwich and Hg-Git on Mac OS X Lion, thanks to some earlier article.
It turns out that I had reinstalled Lion about two weeks ago, and had not reinstalled Xcode 4. So, I updated to Xcode 4.2 and this completely eliminated my problem. Presumably, it would also work for you — and for future me, since I’m likely to repeat this — even if Lion hadn’t been reinstalled in between.
Things that didn’t work included but were not limited to:
In previous articles, I described how to Install Mercurial 1.9, Dulwich, and Hg-Git on Mac OS X Snow Leopard and Install Mercurial 1.9, Dulwich, and Hg-Git on Mac OS X Lion. For either of those operating systems, we can further extend Mercurial with support for connecting to Subversion repositories. While we could enable the bundled Convert extension, the Hgsubversion extension adds live access to remote networked Subversion repos, so it sounds more interesting.
My personal goal in doing this is to be able to work with the InstaDMG repository. It is hosted on Google Code, uses a Subversion repo, and depends upon Subversion keyword substitution. It presents an interesting challenge.
So, let’s install Hgsubversion to add Subversion support to the Git support we’ve previously set up. We need to take the following additional steps, going beyond what is done in the previous articles. To continue, you must already have installed:
Hgsubversion also needs Subversion to be installed. The Mercurial extension then has two different ways of working with Subversion. The Hgsubversion docs I read indicated that Subvertpy is preferred over the other method — using Subversion’s SWIG bindings — so we’ll install this additional Python module first. The docs also recommend running tests on Hgsubversion before using it on projects, so we will try that, as well.
Start with installing Subvertpy:
easy_install command in Terminal.easy_install again to get hgsubversion. However, version 1.2.1 in PyPi didn’t work for me. So, to get a more current working version, I went to the source repository to get the latest software.That’s it! Hgsubversion has been installed and enabled. Even though we got errors from the tests, we can proceed.
kwdemo. You must be in the directory for the cloned repository and have enabled the Keyword extension.Notice that “Revision: 425” is returned when keywords are expanded, replacing the “$Revision” keyword. This is precisely what we need with InstaDMG, where the revision information is reported when the instadmg.bash script is run.
Hg-Git is the Mercurial extension to use if you want to connect to local or remote Git repositories. I exclusively use Mercurial and Hg-Git for all of my Github transactions, so I can personally vouch that it works.
Now that Hg-Git has been updated to better support Mercurial 1.9, let’s see if we can get an Hg toolchain working on Lion. Since I did that on Snow Leopard a few days ago, Hg-Git has made it into PyPI. The installation instructions this time are a bit more streamlined, because we can now use easy_install to get Hg-Git and its dependencies.
To get the toolchain set up, we’ll need Xcode. The Xcode suite includes tools we’ll need to make Python easy_install work, along with Subversion (a prerequisite for Hgsubversion, which I’ll talk about in a later article) and other useful tools.
The Xcode installation is a multi-step install process. Both current download methods — the developer download through connect.apple.com (if you have a paid Mac Developer Account) and the Mac App Store — give you an “Install Xcode” application. That application runs a second, real installer that you have to finish before you actually have the Xcode tools available in a ready-to-use state. This is very similar to the situation for Mac OS X Lion, so you may be developing a sense of familiarity with the situation.
To install Mercurial:
To add Hg-Git to Mercurial on Lion:
That’s it! Once Mercurial 1.9 plus Hg-Git 0.3.1 or later are installed and you’ve enabled Hg-Git in your ~/.hgrc, you are ready to use Mercurial with local and remote Git repositories.
Here is my current installation method for Mercurial 1.9 and Hg-Git on Mac OS X Snow Leopard. I use Hg-Git on a sporadic basis to work with projects on Github. That activity is frequent enough that it’s really helpful to be able to get Hg-Git set up and working quickly.
I had a rough time with Hg-Git after I upgraded to Mercurial 1.9 a few weeks ago. It was notable because that was the first time I can remember having incompatibilities between this stack of software. Since then, Hg-Git has changes available that allow it to install and work with newer versions of Mercurial, including version 1.9, and the most recent Dulwich release. These changes weren’t as readily available at the time I upgraded, so I thought it was worth writing this to remind me how to put the right pieces together.
I don’t yet have a working installation method for all of this software on Mac OS X Lion. I ran into a stumbling block getting Dulwich installed there, and Dulwich is a necessity for Hg-Git. Update: The situation changed a few days later, and I have instructions for Mac OS X Lion available now.
This may have been a problem isolated to the particular Lion system I was using. I haven’t had the chance to duplicate it elsewhere or troubleshoot the problem on the system where I encountered it.
Update: Since I wrote this, Hg-Git 0.3.1 became available through PyPI, which means you can install it via easy_install. You no longer need to clone the Hg-Git repository and run the setup process manually. Instead, you can run the following single command to get Hg-Git 0.3.1 and its dependencies (Dulwich 0.8.0 or later):
This simplifies the instructions quite a bit, but I leave the original steps above for reference.
I’ve had occasion to create some installer packages lately, and the topic of assigning a version number to them has rattled around in my head. The version number of an installer package is shown in the Finder’s Get Info windows, and in the preview pane of column view, so it has some usefulness just for telling one installer apart from another. And, importantly, the version number provides a way for Installer to determine whether a package has previously been installed and simply needs to be upgraded.
At first, I was just going to assign the date in YYYYMMDD format as the version, just to “keep things simple” by using:
Then I was rebuilding the packages several times a day, and wanted to distinguish between the builds, so I switched to YYYYMMDD.hhmm.
This seemed ridiculous after a day. Although I could distinguish between builds, the installers I was building weren’t really changing per se. They were the same payload and scripts and resources, just rebuilt by the command line packagemaker tool using The Luggage.
I crowdsourced my question about versioning, and Steve Losh helpfully responded with the idea to use Mercurial tags along with Semantic Versioning.
My package sources are already checked into Mercurial repositories, so this made a good deal of sense. It meant that I’d have to use tags to identify a version number — it would no longer be as automatic as using the current date. However, certain parts of the version string would be automatic, and the only manual part would be tagging revisions in Hg.
I had already bookmarked Semantic Versioning a while back, but had forgotten about it and never done much of anything with it. If I followed the Semantic Versioning standard, I’d create a “semver” tag to set up the repository, and then a new “vX.Y.Z” tag for actual releases thereafter. Steve further suggested the use of the “latest tag distance” and “short node hash” for the Hg revision, and that information is automatically available from Hg.
Semantic Versioning states, “when tagging releases in a version control system, the tag for a version MUST be ’vX.Y.Z’ e.g. ’v3.1.0’.” I felt I would have to strip the “v” from the version tag to use it cleanly as a package version. This results in commands like the following, where I’ve used sed to remove that character:
This is something I can build into makefiles for use with The Luggage, so I’m going to try it out. On first glance, I think it even fits in nicely with the concept of The Luggage. It automates much of the version-assignment process. And, it takes advantage of version control for both repeatable results and peer review.
After using Acquia Drupal for a while, I took advantage of a trial subscription to the Acquia Network. The network’s services showed me that I had files present in my install that the agent could not account for.
I suspected this was happening because of the way I manage my Acquia Drupal installation with Mercurial. So, I’ve modified my previous process (and updated my instructions) to extract the downloaded tar archive with the —recursive-unlink option. This option appears to successfully remove the contents of every directory before putting new files back into them.
When the archive is extracted in this way, my repository’s working directory shows modified, unknown, and deleted files. This allows me to treat each category of files individually before I commit the changes for a Drupal update as a revision.
$ hg status
The modified files will be tracked normally because they’ve already been added to the Mercurial repository, so I don’t need to do anything special for them.
The unknown files are ones that are completely new, and have not appeared in the same position in a previous revision. They have yet to be tracked by Mercurial, so I have to add them to the repository. To add just those unknown files, then, I have to pick them out from the status listing:
$ hg status --unknown
In order to operate just on those files to add them to the repository, I run a for loop:
This changes the “?” status to “A,” because the files were successfully being tracked by Mercurial.
I use the “—no-status” flag on the “status” command so that just the file paths are printed; the actual status code is not, which is appropriate for the target of the “add” command in the loop.
I do the same basic steps with deleted files. These are files that were in the previous revisions but have been deleted by the —recursive-unlink option from the tar extraction and not replaced with the extraction of the new Acquia Drupal tar archive. If the deleted files had been replaced by the tar extraction, they would either be unchanged (which would not show up in the “status” output) or marked as modified.
To remove the files that are marked as deleted from the repository’s working directory:
However, that may be the same as simply using the following, which I have to explore further:
$ hg remove --after
So, to follow all of these changes in the repository, I run the loop for the uknown files and the loop for the deleted files. The modified files are already tracked, so I don’t need to do anything additional for them. After that, a “commit” will record all of the changes — modifications, additions, and deletions — in the repo.
These commands are based on my current understanding of Mercurial, and they do work for me right now. There could certainly be another better way to do this in one fell swoop — or at least fewer steps. I would welcome that, so if you’re aware of a way, feel free to comment or contact me.
Update: I found that the “hg addremove” command cleanly replaces all of the shell loops I mentioned above. Therefore, I recommend using it instead of the “for” loops I described.
Here is a sequence of commands and output that show how I keep the Acquia Drupal open source content management system up to date with Mercurial, the open source distributed version control system.
In the example below, my Mercurial repositories for Drupal are located in the “drupal” subdirectory of my “repo” folder. Once I’ve moved into that directory, I download the Acquia Drupal distribution with curl and then extract it into my previously-created Mercurial working directory, “acquia_drupal,” using tar.
(Update: I added the —recursive-unlink option after I noticed that the Acquia Network control panel keeps track of extra — possibly unneeded — files and folders you have in your install. The recursive unlink option seems to avoid having stray files from old versions of modules hanging around in your repository after you install updates.)
After extracting Acquia Drupal my Mercurial working directory, I get the status of the repository. It shows there are changes from the last version I checked in — and this includes new files, denoted by a “?” at the beginning of their line.
Since there are new files, I have to add them so they’ll be tracked by the repository. I only need to add in the parent directory for any changed files, and any new files within it will also be added for tracking.
Excellent; the new files have been added. After this, I just need to accommodate the deleted files that no longer need to be tracked (created when using the “—recursive-unlink” option on tar). For that, see my newer instructions.
Now that the right files are being tracked, I need to commit the changes — modified, added, and deleted files — to the repository. This will create a new revision in the repository’s history, which I’ll tag with the text “Acquia Drupal 1.2.0.”
Once this revision is checked in, I can use it to propagate changes to other repositories. I keep the main Acquia Drupal distribution in its own repository, and then use the “hg fetch” command to pull its changes into one where I track contributed modules. That second repository is then pulled into a third repository which stores just the changes for my production Web site. The use of three repositories in this way modularizes and isolates the updates.
While trying to troubleshoot what I’d done to mess up the Mercurial repositories managing my Drupal installations last weekend, I really would have liked a way to see what files had changes in specific revisions. Each revision to a Mercurial repository affects some files, of course, but it seems awfully hard to figure what files changed in that check-in.
I have since found a way to do that by customizing the output of Mercurial. To customize output, you can create templates on the command line (with —template) or for more powerful reformatting, create an output style file.
I struggled for a while to figure out how to use style files, and eventually came up with something that works for me so far.
Since I’ve installed Mercurial from Lee Cantey’s standard binary package for Mac OS X Leopard, I created the file “map-cmdline.changedfiles” at the “/Library/Python/2.5/site-packages/mercurial/templates” path. (Where you put the file may vary depending on where Mercurial is installed, and I’m sorry but I don’t know where it gets installed on other systems.) The contents of “map-cmdline.changedfiles” are below, along with my possibly inept description of what each line is doing:
# Get all of the files in the selected revision
# and stringify them, whatever that means
# but do not 'tabindent' or wrap them to 68/76 columns
# Without first setting changeset to the list of files
# you won't get output from subsequent lines
changeset = '{files|stringify}'
# List modified files, one per line
# preceded by M to mimic `hg status`
file = 'M {file}\n'
last_file = 'M {file}\n'
# List added files, one per line
# preceded by A to mimic `hg status`
file_add = 'A {file_add}\n'
last_file_add = 'A {file_add}\n'
# List deleted files, one per line
# preceded by ! to mimic `hg status`
file_del = '! {file_del}\n'
last_file_del = '! {file_del}\n'
I don’t know why the “map-cmdline.” portion of the filename is there, but as long as I have it, I can call the style file from the command line with what follows the period. So, I can call the style with “—style changedfiles” — and that tiny bit of voodoo seems reasonable enough to me. (The other styles in the directory above, many of which end in “.tmpl” extensions, seem related to the Mercurial Web server, hgweb. I tried, but I couldn’t use their names at the command line, with or without their extensions. Plus, their contents looked HTML-ish.)
With the “map-cmdline.changedfiles” style file saved in that location, I can call Mercurial’s “log” command:
$ hg log --style changedfiles -r tip
… which gives me a list of the files changed in the “tip” (or latest revision) of the repository. I could substitute in any revision identifier for “tip.”
I haven’t actually seen the “file_add” and “file_del” keywords in action; every time I’ve used this style file in the manner described, I’ve only seen files marked as “M” — even if I’m looking at a revision where new files were first checked into the repo. I’m confused by that, but I’m not going to let it sour my day at this point.
There might have been an easier way to do this but I didn’t find one last weekend. It took me some time to figure even this bit out, and I hope writing this post saves someone new to Mercurial from future frustration.
In my Mercurial-based workflow for updating Drupal sites, there is a sequence of commands I need whenever a new version of Drupal comes out. I have a hard time remembering the options for “tar” in this sequence — and my original source for the instructions differs from what I need to do on my Web host — so I need to help my memory. The tar command, as constructed below, places its output into the specified destination directory.
Here it is, with tar’s “—strip-path=1” and “-C” options:
$ cd path/to/repository/parent/directory
$ curl -O http://ftp.drupal.org/files/projects/drupal-5.12.tar.gz
$ tar --strip-path=1 -C drupal_source -zxv -f drupal-5.12.tar.gz
I’ve been using zsh for a while as my preferred shell. I have a hacked-together zshrc file, and yet really wanted to use it across multiple systems. Some of those systems are running Mac OS X, others Solaris, and still others Linux. Executables are in a different locations and even have different switches across this range of systems, so my cobbled zshrc was not helping me.
As I was about to fall asleep last night, it finally hit me that fixing my zshrc would be a good thing to do. I jotted down some notes about an idea to reorganize it, and did something about it today.
Of course, since I’ve checked my zshrc and other dotfiles into a Mercurial repository, I could experiment without fear.
I created three top-level functions, with one “case” statement in each. Case statements may be evil in some fashion, but they are one of the things I like about shell scripting. These statements do allow the script to make choices based on the host, operating system, or shell that it was running in. (Yeah, it’s a zshrc, but I sometimes do stupid things — like sourcing it in bash on the one Linux system that won’t let me switch to zsh. Site5, I’m looking at you.)
I separated all the important sections of my zshrc into their own individual function calls. Each of those function calls was placed into one of the applicable case statements.
The case statement functions figure out the conditions the zshrc is running in, and then run the other functions to set up my environment.
The changes tested well from first try across the various platforms and hosts I log in to. I did have a minor problem with the `hostname` command, because Solaris doesn’t have a “-s” flag for it. Eventually, I solved that — and the odd “uname: error in setting name: Not owner” error I got, even though I wasn’t directly running `uname` there — by replacing `hostname` with `uname`.
Thankfully, it works for me, and it should be a little easier to manage changes in the future.