ridiculous rails boot times

I recently ran safe-upgrade on Ubuntu, which involved updates to a bunch of stuff, including Linux headers and postgres, but now my Rails boot time is now 3x longer. I still don’t know why, which is frustrating, but I did learn about a few things along the way.

The first is Bumbler, a tool for inspecting gem load times, among other things.

The second is Passenger’s passenger_start_timeout setting, which is how I’m addressing my problem without really addressing my problem.

The third is that Rackspace now has a “Performance” VPS product that seems to be both faster and cheaper than their old VPSs. Unfortunately transitioning is non-trivial, since you can’t do it for 1st gen cloud servers and if you want to create a Performance Cloud Server from an Next Gen image you can only do it from a 1GB Next Gen server.

# reproject into WGS84 lat/lon
gdalwarp -t_srs EPSG:4326 -dstnodata 0 input.tif output.tif
 
# https://developers.google.com/kml/articles/raster#translate
gdal_translate -of vrt -expand rgba output.tif output.vrt
gdal2tiles.py -p geodetic -k output.vrt

This mostly works, but the nodata from the original GeoTIFF doesn’t get preserved as a PNG alpha channel in the KMZ tiles. Still need to figure that out.

visualizing a rails schema

Really didn’t find a perfect solution, which would allow me to optionally specify a model or a set of models and show all the attributes and relationships for only those models. railroady comes pretty close though, especially when using graphviz output and OmniGraffle‘s layout engines. rails-erd made a decent full-model diagram too.

RAILS

  gem "rails-erd"
  gem "railroady"

RAILROADY

railroady -M --hide-through -i \
  -s app/models/*observation*,app/models/photo.rb,app/models/sound.rb,app/models/taxon.rb \
  -o models-observation.dot

OMNIGRAFFLE

Hierarchical layouts seemed to work best, with some tweaking.

iNat observation model and some associated models.

I’m sure this is all irrelevant if you use cocoapods, but I don’t (yet). My approach to installing SSZipArchive was to add it as a git submodule and drag the folder including SSZipArchive.h and minizip into my project, adding it by reference. This caused RestKit compilation to barf, mostly a lot of parse issues in RKURL.h like Expected identifier or '(', which is ridiculous b/c nothing changed in RestKit.

I eventually found my solution at http://stackoverflow.com/questions/9398409/adding-minizip-breaks-objective-c/18153812#18153812: change the file type of the minizip .c files to “Objective-C Source.”

Screen Shot 2013-09-24 at 3.15.51 PM

Now my project builds and everything works normally. Why this solution works is beyond my extremely limited knowledge of C and Xcode. These files look like C and/or C++, so why does telling Xcode to treat them like Objective-C even work?

I have a bunch of iOS apps that use the same Google API identity with different client IDs for sign-in through Google. I just made another one but kept running into this problem where auth would work, but when the browser redirect occurred, I’d get a message reading “Cannot Open Page: Safari cannot open the page because the address is invalid.”

Photo Sep 18, 9 15 00 PM

That didn’t seem right b/c it was redirecting to an internal URL based on the bundle ID I specified when generating the client ID, and the bundle ID for the app was right.

Screen Shot 2013-09-18 at 9.12.49 PM

Turned out I was forgetting that you can specify a URL identifier and URL scheme in the *-Info.plist file. Changing that to match the bundle ID fixed the problem… after suffering hours of madness and having my life expectancy shortened by several years.

AREL where() to sql

Sometimes you just want the SQL for a WHERE clause:

 > Post.send(:sanitize_sql, :title => "foo", :body => "bar")
 => "\"posts\".\"title\" = 'foo' AND \"posts\".\"body\" = 'bar'"

iNat started crashing periodically this weekend, and after numerous attempts to figure out what was wrong, I finally found the culprit with some new-to-me tools: gdb and gdb.rb. For each crash load average was huge (like 10), and it was clearly the rack processes that were eating all the CPU (memory and swap were fine). Restarting nginx and/or killing the rack processes brought things back up, so clearly it was those hung processes that were killing things. Strangely, the rails logs seemed to be humming along normally (probably b/c not all procs were hung). There was some uptick in traffic around each crash, but not like a DOS-level uptick. After ruling out memory leak, DOS, and Rackspace issues (they claimed everything was fine), I figured there was an app-level issue, but none of the changes immediately preceding the crashes looked like they could be causing infinite loops or the like, sooooo I was at a loss.

I talked to n8 and he got me thinking about tools that look directly at the current state of the server processes, strace, gdb, and the like. gdb, which lets you look at what a live process is doing (if it was written in c, c++, or a number of other languages) ended up being the most useful, when combined with gdb.rb, which actually lets you execute ruby commands within the context of a live ruby process. I suck at interpreting C code from the interpreter, but with ruby I’m home. Inspection went something like this:

# install deps on Ubuntu and gdb.rb gem
sudo apt-get install gdb python-dev ncurses-dev
gem install gdb.rb
 
# wait for crash to happen again... and wait... and wait...
 
# CRASH! Attach to hung process by PID (use rvmsudo if you use rvm)
rvmsudo gdb.rb PID
 
# in gdb get a ruby stacktrace with file names and line numbers
# here I'm filtering by files that are actually in my app dir
(gdb) ruby eval caller.select{|l| l =~ /app\//}

That outed some infinite recursion from a totally unexpected place that I was able to fix in a few minutes. Unfortunately running ruby eval in gdb occasionally crashed the process, I think when the proc received a signal while gdb was attached, but stubborn repetition eventually got me my trace. Also, not all of gdb.rb’s commands seemed to be available, like ruby trace and ruby threads. I basically only had ruby objects and ruby eval, not sure why, but ruby eval was all I needed.

Anyway, there are tons of posts about using gdb to inspect ruby procs out there, but almost all of them are really old, and it took me a while to find the right tool for this job, so maybe this post will help someone. The best overview I found was this post from Big Nerd Ranch. This gist by tmm1 (author of gdb.rb) also collects a bunch of useful information about a hung process by combining output from multiple tools.

Recently started using RGeo a bit (I know, way late, but hey, it’s pretty awesome!), and ran into a production problem when using contains?

RGeo::Error::UnsupportedOperation: Method Geometry#contains? not defined.

RGeo depends on some kind of underlying geometry library, usually GEOS. In Ubuntu, the headers and source files for GEOS don’t get installed by default when you install geos, so the solution is to install them, as it is with so many of these issues:

sudo aptitude install geos-dev
bundle exec gem uninstall rgeo
bundle update rgeo

bundler cheat

Dependency management that barely solves more problems than it creates.

# reinstall a single gem (why can't this be gem reinstall gemname?!)
bunde exec gem uninstall gemname
bunde update gemname

capistrano cheatsheat

# run task on particular host / server
cap HOSTS=foo.bar.com,baz.bar.com deploy
 
# run task on particular host / server AND respect roles
cap HOSTFILTER=foo.bar.com,baz.bar.com deploy
 
# run tasks for a particular role
cap ROLES=db deploy
 
# checkout a branch for only one host / server
cap HOSTFILTER=foo.bar.com -s branch=i18n deploy