Lubberly & Unnavylike

I was running into the error described here,

ERROR: index 'model_name_core': sql_range_query: ERROR: integer out of range

so I re-compiled Postgres with 64 bit support and upgraded to a version of Thinking Sphinx that incorporated the patch (1.4.6), but still no dice. After manually re-running the SQL query Sphinx was using many times, I finally narrowed the problem down to a row with a date set in the year 8, as in 0008. TS generates a SQL statement that extracts the epoch from date column and casts them as INTs, but this should have been cast as something bigger. This was fixed in d9f3df980a6f8441a17b with the use_64_bit config option, but that’s fix isn’t available in the 1.x line of TS, and the 2.x line doesn’t work with Rails 2.x, so I lose.

The problematic row was erroneous, so I fixed it and that repaired indexing, but I’m still running a potentially buggy version of TS. Ugh.

However, I did learn that \x enables vertical output in psql, much like \G does in mysql. Yay.

Just installed Mapnik on the iNat slice at Slicehost, which runs Ubuntu Hardy (8.04). Since the process was not exactly trivial, here’s what I ended up doing:

# Get up to date and install deps
# Docs: http://trac.mapnik.org/wiki/UbuntuInstallationOld
aptitude update
aptitude install binutils cpp-3.3 g++-3.3 gcc-3.3 gcc-3.3-base libboost-dev libboost-filesystem-dev libboost-filesystem1.34.1 libboost-iostreams-dev libboost-iostreams1.34.1 libboost-program-options-dev libboost-program-options1.34.1 libboost-python-dev libboost-python1.34.1 libboost-regex-dev libboost-regex1.34.1 libboost-serialization-dev libboost-serialization1.34.1 libboost-thread-dev libboost-thread1.34.1 libicu-dev libicu38 libstdc++5 libstdc++5-3.3-dev python2.5-dev
aptitude install libfreetype6 libfreetype6-dev libjpeg62 libjpeg62-dev libltdl3 libltdl3-dev libpng12-0 libpng12-dev libtiff4 libtiff4-dev libtiffxx0c2 python-imaging python-imaging-dbg proj
aptitude install libcairo2 libcairo2-dev python-cairo python-cairo-dev libcairomm-1.0-1 libcairomm-1.0-dev libglib2.0-0 libpixman-1-0 libpixman-1-dev libpthread-stubs0 libpthread-stubs0-dev ttf-dejavu ttf-dejavu-core ttf-dejavu-extra
aptitude install libgdal-dev python2.5-gdal
aptitude install libxslt1.1 libxslt1-dev libxml2-dev libxml2
easy_install jonpy
easy_install lxml
 
# Configure and instlall mapnik from source
# Docs: http://trac.mapnik.org/wiki/UsingScons
python scons/scons.py configure INPUT_PLUGINS=shape,gdal,ogr,sqlite
python scons/scons.py
python scons/scons.py install
 
# Make sure ld knows where to find the new libs
# Thanks to http://www.drazzib.com/projets:openstreetmap:postgis_mapnik_tile_server
echo "/usr/local/lib64" >> /etc/ld.so.conf.d/local.conf
ldconfig

My main mistake was thinking that the python-mapnik package in Aptitude was going to cut mustard. Didn’t even include OGR support! Luckily the most recent Mapnik release (0.6.1) built just fine. Still easier that installing under OS X.

I just put iNaturalist.org on Ruby Enterprise Edition. Unfortunately, I’m really not seeing any savings in memory consumption, aside from the Passenger ApplicationSpawner taking up a few less MB. Kind of dissapointed.

However, installing REE on my Mac was slightly less trivial than I was led to believe. Suffice it to say the REE installer assumes you have dependent libs in /usr or /usr/local, and if you don’t, there aren’t any flags you can pass in to change that. However, you can set ENV vars, so I was able to get the installer to work like this:

export CC=gcc && export CXX=g++ && ./installer

Basically, the REE installer checks deps by using Ruby to write and compile a C file, but it’s default path to gcc was not the same as the gcc I normally use.

After that, I had to make sure all my gems were in place. I ended up installing rails, Hpricot, and ruby-debug (the latter 2 require compiled C) using REE’s own copy of rubygems, and I made sure the rest of our gems were vendorized. REE’s rubygems is just like normal:

/path/to/ree/bin/gem install rails ruby-debug hpricot

If you run into problems, make sure you’re tailing Apache’s error log and not just the Rails log, b/c that’s where Passenger will moan about its problems.

As I said, memory gains seem non-existent on our Ubuntu production machine, which is weird because I saw a 25% reduction in memory usage on my Macbook.

With the gracious help of Dane Springmeyer, I recently got Mapnik up and running on my Mac, so naturally the first thing I wanted to do was to plot iNaturalist observations. Mapnik will talk with a PostGIS database just fine, but it wont’t pull points from a MySQL database table, at least not directly. What it does support, though, is the all-powerful OGR. Back when I used to use Mapserver, I had used OGR a few times to act as a bridge between simple lat/lon columns in a MySQL table and Mapserver (great description here), so I figured the same would be possible in Mapnik.

Read the rest of this entry »

So I recently realized I was foolishly not backing up the database dumps for a project, and figured a good stopgap / semi-permanent solution would be to just rsync the backups that were stored on the server on a regular basis. Of course, the way to run periodic scripts in OS X is launchd, not cron, so I figured I’d use this as an excuse to try that route.

Read the rest of this entry »

You need to have svn installed, of course. If you want to improve it, here’s the gist.

#!/bin/bash
tmsupportpath="/Library/Application Support/TextMate"
bundlepath="$tmsupportpath/Bundles"
if [ ! -d "$bundlepath" ]
  then
  echo "First time, eh?  Making $bundlepath..."
  mkdir -p "$bundlepath"
fi
 
if [ ! -d "$tmsupportpath/Support" ]
  then 
  cd "$tmsupportpath"
  echo "Checking out the most recent Support folder..."
  svn co http://macromates.com/svn/Bundles/trunk/Support
fi
 
cd "$bundlepath"
svn co http://macromates.com/svn/Bundles/trunk/Bundles/$1.tmbundle
svn up *.tmbundle
osascript -e 'tell app "TextMate" to reload bundles'
 
cd "$tmsupportpath/Support"
svn up

Andrew and I were recalling BBC’s The Box today when I wondered if people have tried living in shipping containers.  I know there have been many efforts to sell prefab homes based on shipping containers as low-cost, eco-friendly housing solutions, but I’ve never heard of someone living in one while it was shipped around the world.

I thought this might be a cool idea for a story: who lives in shipping container communities on boats and at port?  What kind of relationships and norms form under such fleeting conditions?  Could this ever be economically feasible if shipping a container from China to the US costs $8000?  Any ship carrying house containers could carry freight beneath them, but I guess the only way it could work is if passengers were willing to pay as much or more than their weight in freight was worth.

I also started thinking about how this might work in space.  Rotating container bays for simulated gravity?  Completely self-sufficient shipping container space homes?

Here’s a silly bash trick that will delete all unversioned SVN files

svn st | grep "^?" | sed s/?[[:space:]]*// | xargs rm -rf

Use with care, obviously. If there’s a more concise method, please let me know.

And here’s how you might use similar commands to apply fmresolve to all conflicted files, and then resolve them.

for f in $(svn st | grep "^C" | sed s/C[[:space:]]*//); do fmresolve $f; done
svn st | grep "^C" | sed s/C[[:space:]]*// | xargs svn resolved

I recently refactored some code in iNaturalist that fetches taxon names from external name providers like uBio and the Catalogue of Life. They return names and classification data that are similar but (of course) not identical to ActiveRecord models we use in iNat, so I figured I’d write adapters for them, and I thought a really smart solution would be to subclass the models themselves, simply overriding the attributes with getters that mined a private instance variable holding an XML response from one of the web services. Big mistake. ActiveRecord mixes in all kinds of magic into the models that doesn’t necessarily get passed on to child classes. I ran into all sorts of fun errors and problems until I remembered the Adapter implementation described here, which takes a much more sensible approach: don’t sublcass, and instead hold an internal copy of the adaptee, passing calls to anything you don’t want to override to the adaptee.

Read the rest of this entry »