ERROR: index 'model_name_core': sql_range_query: ERROR: integer out of range

so I re-compiled Postgres with 64 bit support and upgraded to a version of Thinking Sphinx that incorporated the patch (1.4.6), but still no dice. After manually re-running the SQL query Sphinx was using many times, I finally narrowed the problem down to a row with a date set in the year 8, as in 0008. TS generates a SQL statement that extracts the epoch from date column and casts them as INTs, but this should have been cast as something bigger. This was fixed in d9f3df980a6f8441a17b with the use_64_bit config option, but that’s fix isn’t available in the 1.x line of TS, and the 2.x line doesn’t work with Rails 2.x, so I lose.

The problematic row was erroneous, so I fixed it and that repaired indexing, but I’m still running a potentially buggy version of TS. Ugh.

However, I did learn that \x enables vertical output in psql, much like \G does in mysql. Yay.

# Get up to date and install deps
# Docs: http://trac.mapnik.org/wiki/UbuntuInstallationOld
aptitude update
aptitude install binutils cpp-3.3 g++-3.3 gcc-3.3 gcc-3.3-base libboost-dev libboost-filesystem-dev libboost-filesystem1.34.1 libboost-iostreams-dev libboost-iostreams1.34.1 libboost-program-options-dev libboost-program-options1.34.1 libboost-python-dev libboost-python1.34.1 libboost-regex-dev libboost-regex1.34.1 libboost-serialization-dev libboost-serialization1.34.1 libboost-thread-dev libboost-thread1.34.1 libicu-dev libicu38 libstdc++5 libstdc++5-3.3-dev python2.5-dev
aptitude install libfreetype6 libfreetype6-dev libjpeg62 libjpeg62-dev libltdl3 libltdl3-dev libpng12-0 libpng12-dev libtiff4 libtiff4-dev libtiffxx0c2 python-imaging python-imaging-dbg proj
aptitude install libcairo2 libcairo2-dev python-cairo python-cairo-dev libcairomm-1.0-1 libcairomm-1.0-dev libglib2.0-0 libpixman-1-0 libpixman-1-dev libpthread-stubs0 libpthread-stubs0-dev ttf-dejavu ttf-dejavu-core ttf-dejavu-extra
aptitude install libgdal-dev python2.5-gdal
aptitude install libxslt1.1 libxslt1-dev libxml2-dev libxml2
easy_install jonpy
easy_install lxml
 
# Configure and instlall mapnik from source
# Docs: http://trac.mapnik.org/wiki/UsingScons
python scons/scons.py configure INPUT_PLUGINS=shape,gdal,ogr,sqlite
python scons/scons.py
python scons/scons.py install
 
# Make sure ld knows where to find the new libs
# Thanks to http://www.drazzib.com/projets:openstreetmap:postgis_mapnik_tile_server
echo "/usr/local/lib64" >> /etc/ld.so.conf.d/local.conf
ldconfig

My main mistake was thinking that the python-mapnik package in Aptitude was going to cut mustard. Didn’t even include OGR support! Luckily the most recent Mapnik release (0.6.1) built just fine. Still easier that installing under OS X.

However, installing REE on my Mac was slightly less trivial than I was led to believe. Suffice it to say the REE installer assumes you have dependent libs in /usr or /usr/local, and if you don’t, there aren’t any flags you can pass in to change that. However, you can set ENV vars, so I was able to get the installer to work like this:

export CC=gcc && export CXX=g++ && ./installer

Basically, the REE installer checks deps by using Ruby to write and compile a C file, but it’s default path to gcc was not the same as the gcc I normally use.

After that, I had to make sure all my gems were in place. I ended up installing rails, Hpricot, and ruby-debug (the latter 2 require compiled C) using REE’s own copy of rubygems, and I made sure the rest of our gems were vendorized. REE’s rubygems is just like normal:

/path/to/ree/bin/gem install rails ruby-debug hpricot

If you run into problems, make sure you’re tailing Apache’s error log and not just the Rails log, b/c that’s where Passenger will moan about its problems.

As I said, memory gains seem non-existent on our Ubuntu production machine, which is weird because I saw a 25% reduction in memory usage on my Macbook.

First Hurdle: Getting Mapnik to Show Anything At All

First I made an OVF file so OGR would know how to talk to my database:

<OGRVRTDataSource>
  <OGRVRTLayer name="observations">
    <SrcDataSource>MYSQL:inaturalist_development,user=root,tables=observations</SrcDataSource> 
    <SrcSQL>SELECT id, latitude, longitude, iconic_taxon_id FROM observations</SrcSQL> 
    <GeometryType>wkbPoint</GeometryType>
    <GeometryField encoding="PointFromColumns" x="longitude" y="latitude"/> 
  </OGRVRTLayer>
</OGRVRTDataSource>

and made sure it was working:

mapnik-inat > ogrinfo inat.ovf
ERROR 4: Update access not supported for VRT datasources.
Had to open data source read-only.
INFO: Open of `inat.ovf'
      using driver `VRT' successful.
1: observations (Point)

Then I basically modified the Mapnik tutorial to use the OGR virtual data source:

import mapnik
 
# Setup the datasource and layer
ds = mapnik.Ogr(file="inat.ovf", layer="observations")
lyr = mapnik.Layer('test lyr')
lyr.datasource = ds
 
# Style the points using a simple PNG
m = mapnik.Map(600, 300, "+proj=latlong +datum=WGS84")
s = mapnik.Style()
r = mapnik.Rule()
symbolizer = mapnik.PointSymbolizer('mm_8_stemless_iNatGreen.png', 'png', 8, 8)
symbolizer.allow_overlap = True
r.symbols.append(symbolizer)
s.rules.append(r)
m.append_style('My Style', s)
lyr.styles.append('My Style')
m.layers.append(lyr)
 
# Zoom to the extent of the data and write the map to a PNG
m.zoom_to_box(lyr.envelope())
mapnik.render_to_file(m, 'test.png', 'png')

And then I got a map!

Ok, a kind of crappy map of test data, but actual, real, visible pixels! So much better than arcane error messages!

Second Hurdle: Google

Of course, it’s just a map of some test points, but still, it worked. Yay. Next thing I wanted to do was to see how easy it would be to lay these points over a Google map, since one of the things I’d like to do with Mapnik is have a layer showing ALL observations on the iNat observation browser. Turns out this was a piece of cake using tilelite.

First I put my map configuration in an XML file, adding a bit of complexity using SQL filters to show different markers for different kinds of points (here corresponding to the IDs for different iconic taxa, e.g. plants, animals, fungi, etc.).

<Map srs="+proj=latlong +datum=WGS84">
  <Style name="observationsStyle">
    <Rule>
      <Filter>[iconic_taxon_id] = 47126</Filter>
      <PointSymbolizer file="mm_8_stemless_iNatGreen.png" type="png" width="8" height="8" />
    </Rule>
    <Rule>
      <Filter>[iconic_taxon_id] = 1 or [iconic_taxon_id] = 47178 or [iconic_taxon_id] = 3 or [iconic_taxon_id] = 20978 or [iconic_taxon_id] = 26036 or [iconic_taxon_id] = 40151</Filter>
      <PointSymbolizer file="mm_8_stemless_DodgerBlue.png" type="png" width="8" height="8" />
    </Rule>
    <Rule>
      <Filter>[iconic_taxon_id] = 47115 or [iconic_taxon_id] = 47119 or [iconic_taxon_id] = 47158</Filter>
      <PointSymbolizer file="mm_8_stemless_OrangeRed.png" type="png" width="8" height="8" />
    </Rule>
    <Rule>
      <Filter>[iconic_taxon_id] = 47170</Filter>
      <PointSymbolizer file="mm_8_stemless_DeepPink.png" type="png" width="8" height="8" />
    </Rule>
    <Rule>
      <ElseFilter/>
      <PointSymbolizer file="mm_8_stemless_unknown.png" type="png" width="8" height="8" />
    </Rule>
  </Style>
 
  <Layer name="observationsLayer" srs="+proj=latlong +datum=WGS84">
    <StyleName>observationsStyle</StyleName>
    <Datasource>
      <Parameter name="type">ogr</Parameter>
      <Parameter name="file">inat.ovf</Parameter>
      <Parameter name="layer">observations</Parameter>
    </Datasource>
  </Layer>
</Map>

Then I fired up tilelite:

tilelite > python liteserv.py ~/projects/mapnik-inat/inatmap.xml
Using mapfile: '/Users/kueda/projects/mapnik-inat/inatmap.xml'
Listening on port 8000...

and added a new GTileLayer to my map:

var myCopyright = new GCopyrightCollection();
var allObsLyr = new GTileLayer(myCopyright, 0, 12, {
  isPNG: true,
  tileUrlTemplate: 'http://localhost:8000/{Z}/{X}/{Y}.png'
});
var allObsOverlay = new GTileLayerOverlay(allObsLyr);
map.addOverlay(window.allObsOverlay);

et voila:

I have no idea how scalable this kind of solution is, but I just wanted to show that it works. It was my first real experiment with Mapnik, and I’m pretty happy with how simple things were after the decidedly unsimple installation process.

I had everything seemingly set up, but I kept getting permission denied errors from rsync, even when exactly the same rsync command run manually from the command line worked fine. The problem turned out to be that my private SSH key was password-protected. I had forgotten about this b/c Terminal and Keychain had been magically handling that password for some time. So I deleted that entry in Keychain and made a new public/private key pair. Once that was done, all went smoothly.

Here’s a summary of what I did.

1. Set up passwordless SSH without a password on the private key.
2. Added the following plist to ~/Library/LaunchAgents/

   <?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE plist PUBLIC 
     "-//Apple Computer//DTD PLIST 1.0//EN"  
     "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
   <plist version="1.0">
   <dict>  
     <key>Label</key>
     <string>org.projectname.backup</string>
     <key>ProgramArguments</key>
     <array>
       <string>/opt/local/bin/rsync</string>
       <string>-avz</string>
       <string>projectname.org:/path/to/backup/</string>
       <string>/path/to/local/backup/</string>
     </array>
     <key>LowPriorityIO</key>
     <true/>
     <key>Nice</key>
     <integer>1</integer>
     <key>StartCalendarInterval</key>
     <dict>  
       <key>Hour</key>
       <integer>22</integer>
       <key>Minute</key>
       <integer>00</integer>
     </dict>
   </dict>
   </plist>

3. Loaded the daemon script into launchd:

   launchctl load ~/Library/LaunchAgents/org.projectname.backup

4. Tested it and watched the log in Console:

   launchctl start org.projectname.backup

Now it should run every day at 10pm.

Here are some resources I found useful:

• Getting Started with launchd
• man page for the launchd plist format
• This Apple Support discussion lead me on the path to my problem

I thought this might be a cool idea for a story: who lives in shipping container communities on boats and at port?  What kind of relationships and norms form under such fleeting conditions?  Could this ever be economically feasible if shipping a container from China to the US costs $8000?  Any ship carrying house containers could carry freight beneath them, but I guess the only way it could work is if passengers were willing to pay as much or more than their weight in freight was worth.

I also started thinking about how this might work in space.  Rotating container bays for simulated gravity?  Completely self-sufficient shipping container space homes?

svn st | grep "^?" | sed s/?[[:space:]]*// | xargs rm -rf

Use with care, obviously. If there’s a more concise method, please let me know.

And here’s how you might use similar commands to apply fmresolve to all conflicted files, and then resolve them.

for f in $(svn st | grep "^C" | sed s/C[[:space:]]*//); do fmresolve $f; done
svn st | grep "^C" | sed s/C[[:space:]]*// | xargs svn resolved