simplest way to benchmark a URL

Since I will forget this:

ab -c 10 -n 100 http://127.0.0.1:3000/observations/1602.json

ab is a simple benchmarking tool from Apache. -c 10 means 10 concurrent connections, -n 100 means 100 requests. Note that it gave me Benchmarking localhost (be patient)...apr_socket_recv: Connection refused (61) when I tried it on http://localhost:3000/, but that seems to be an IPv6 thing and replacing that with 127.0.0.1 seemed to work.

You can also do this with POST requests if you have HTTP Basic auth:

echo "observation[latitude]=38&observation[longitude]=-121" > post.txt && \
ab -c 10 -n 100 \
  -A username:password \
  -T application/x-www-form-urlencoded \
  -p post.txt \
  "http://127.0.0.1:3000/observations" && \
rm post.txt

Haven’t found a way to just give it POST params without creating a file, but whatever, this works.

single day from GPX

I often carry around a GPS with me as I’m exploring, and I don’t like to remember to clear the track log, so every time I pull a GPX file off the device it’s enormous and has a bunch of extraneous data I don’t want. What I generally want is the track from the previous day. I haven’t found a completely satisfactory way to do this yet, but this comes close:

gpsbabel -t -i gpx -f in.gpx \
  -x nuketypes,waypoints,routes \
  -x track,start=20150729,stop=20150730 \
  -o gpx -F out.gpx

or you can do it directly from a garmin GPS unit plugged in via USB:

gpsbabel -t -i garmin -f usb: \
  -x nuketypes,waypoints,routes \
  -x track,start=20160525,stop=20160531 \
  -o gpx -F ~/gps/201605-rogue-river.gpx

The nuketypes filter removes all the waypoints and routes and the start and stop filters filter the GPX data by timestamp (whatever’s being used in the GPX file, which for me is UTC). What’s annoying here is that inevitably a few track points from the day will be leftovers from the previous day. I can leave them in there, but usually I’m only filtering my tracklog like this to share it with other people, so I don’t want a couple extraneous points. I can view the data, find the earliest track point that is actually correct, and use that for the time filter, but that requires an additional step using a GUI.

FWIW, taking a quick look at the file is as easy as converting to KML and using Google Earth:

ogr2ogr -f KML gpx.kml out.gpx

I’m sure this is all irrelevant if you use cocoapods, but I don’t (yet). My approach to installing SSZipArchive was to add it as a git submodule and drag the folder including SSZipArchive.h and minizip into my project, adding it by reference. This caused RestKit compilation to barf, mostly a lot of parse issues in RKURL.h like Expected identifier or '(', which is ridiculous b/c nothing changed in RestKit.

I eventually found my solution at http://stackoverflow.com/questions/9398409/adding-minizip-breaks-objective-c/18153812#18153812: change the file type of the minizip .c files to “Objective-C Source.”

Screen Shot 2013-09-24 at 3.15.51 PM

Now my project builds and everything works normally. Why this solution works is beyond my extremely limited knowledge of C and Xcode. These files look like C and/or C++, so why does telling Xcode to treat them like Objective-C even work?

postgres cheatsheet

psql

# copy data to csv
psql yourdb -c "COPY (SELECT * FROM tablename) TO STDOUT WITH CSV HEADER" > file.csv
 
# copy data out, load it back in
psql yourdb -c "COPY (SELECT * FROM tablename) TO STDOUT WITH CSV" > file.csv
psql yourdb -c "\copy tablename FROM 'file.csv' WITH CSV"

Stats

View more about stats tables at http://www.postgresql.org/docs/9.2/static/monitoring-stats.html

-- view currently running queries
SELECT client_addr, query_start, current_query FROM pg_stat_activity;

Admin

The first step is always to have a relaxing beverage in hand…

Connecting multiple servers

http://www.cyberciti.biz/tips/postgres-allow-remote-access-tcp-connection.html is a good guide. The most salient points in addition to editing pg_hba.conf are

  1. both client and server firewalls must allow TCP connections from each other on port 5432
  2. you have to alter listen_addresses in postgresql.conf

Some useful debugging commands:

# Check that the port is open from the client. Make sure your firewalls are
# configured correctly on both ends and that SSL has been set up and
# configured
nmap -v -sV remote_host -p 5432
 
# Both of these will tell you if pg is up and listening on the server machine
netstat -plnt on localhost
lsof -i:5432

Setting Up SSL

http://www.howtoforge.com/postgresql-ssl-certificates is a good step-by-step. Note that in ubuntu pg’s data dir seems to live somewhere like /var/lib/postgresql/9.1/main/, and that sever.crt and server.key are already there as symlinks to a self-signed certificate. Seems like you still have to copy server.crt to root.crt, though, and of course you need to make your client certs. Setting the -subj flag to something simple like '/CN=inaturalist' seemed to work for me.

Deleting Unversioned SVN Files

Here’s a silly bash trick that will delete all unversioned SVN files

svn st | grep "^?" | sed s/?[[:space:]]*// | xargs rm -rf

Use with care, obviously. If there’s a more concise method, please let me know.

And here’s how you might use similar commands to apply fmresolve to all conflicted files, and then resolve them.

for f in $(svn st | grep "^C" | sed s/C[[:space:]]*//); do fmresolve $f; done
svn st | grep "^C" | sed s/C[[:space:]]*// | xargs svn resolved

Active Record Model Adapters

I recently refactored some code in iNaturalist that fetches taxon names from external name providers like uBio and the Catalogue of Life. They return names and classification data that are similar but (of course) not identical to ActiveRecord models we use in iNat, so I figured I’d write adapters for them, and I thought a really smart solution would be to subclass the models themselves, simply overriding the attributes with getters that mined a private instance variable holding an XML response from one of the web services. Big mistake. ActiveRecord mixes in all kinds of magic into the models that doesn’t necessarily get passed on to child classes. I ran into all sorts of fun errors and problems until I remembered the Adapter implementation described here, which takes a much more sensible approach: don’t sublcass, and instead hold an internal copy of the adaptee, passing calls to anything you don’t want to override to the adaptee.

Read the rest of this entry »

Git Cheatsheet

It’s small, fast, and github is rad. So I am trying to learn it.

# Setup
git config --global --list
git config --global user.name "Ken-ichi"
git config --global user.email "fortinbras@norway.net"
 
# The colors, children.  Mm-hey
git config --global color.diff auto
git config --global color.status auto
git config --global color.branch auto
 
# Getting Info
git remote -v # Display the remote repository URL
git log # duh
git log --stat --author=Ken-ichi # show all commits by me with change stats
git show COMMIT # equivalent of svn log -r REV_NUM, but with diffs
 
# Diffing
git diff COMMIT1..COMMIT2 # duh
git whatchanged --since="1 day ago" -p # show changes in date range, commit by commit
git diff $(git rev-list -n1 --before="1 day ago" master) # same as above by by file
git log origin/master..master # show changes between remote and local
 
# Committing
git add .
git commit -a
 
# Amending
git commit -a --amend
 
# Reverting
git reset --hard HEAD # revert everything to HEAD
git checkout path/to/file # reset a single file
git checkout HEAD path/to/file # reset a file you just deleted with git rm
git reset --soft HEAD^ # unstage last commit if you forgot something
git fetch --all && git reset --hard origin/master # reset and overwrite conflicting untracked files
git revert COMMIT # undo the changes in a commit in a new commit
git reset HEAD # unstage all staged files. Esp. useful when popping out from stash
 
# Checking out previous states
git checkout COMMIT
git checkout master # to get back
# Go back to a previous date
git checkout 'master@{2014-12-01}'
get checkout master # to get back
 
# Stashing
git stash # stash changes on this branch so you can switch and work on something else
git stash pop # bring back the stashed changes
 
 
# Pushing to another repos
# could be local like /path/to/somewhere 
# or ssh://me@there.net/path/to/repos
git push path/to/repos
git push origin master # if you cloned from a remote named origin
 
# to switch origins
git remote rm origin
git remote add origin 'path/to/a/bare/repo'
 
# note that when sharing a repo with others, you will all want to push/pull from a *bare* repo made with 
git clone --bare path/to/repo
# See http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#public-repositories
 
# Branching and merging
git branch new_branch && git checkout new_branch # make a new branch and switch to it
git checkout -b new_branch # same as above
git diff master..new_branch # diff two branches
git merge other_branch
git mergetool -t opendiff # resolve conflicts in a merge with FileMerge
git branch --track localbranchname origin/remotebranchname # track a remote branch.  you might need to git fetch before this will work
git push origin local_branch # push local_branch to a remote repo.  Will create new local_branch in remote if not there already
git checkout --ours path/to/file # just choose our version
git checkout --theirs path/to/file # just choose their version
# view all branches by date of last update, via http://stackoverflow.com/questions/9236219/git-list-git-branches-sort-by-and-show-date
git for-each-ref --sort=committerdate refs/heads/ --format='%(committerdate:short) %(refname:short)'
 
# Collaboration - add an origin, tracking branch, cherry pick
git remote add -f originname git://github.com/username/repo.git
git branch --track localbranch originname/remotebranch # or whatever branch you want
git fetch # pull in updates
git checkout master && git cherry-pick [commit]
 
# Integrating a range of commits from a branch.
# using cherry-pick
git cherry-pick [first commit]..[last commit
# using rebase --onto
git checkout -b integration # create a branch to integrate changes (optional)
git branch -f integration [last commit] # set integration branch to last commit
git checkout -b tmp # create a tmp branch, not sure why this is necessary
git rebase --onto tmp [first commit]~1 integration # replay from 1 before first commit to last commit
# the above will leave you in integration with the range of commits applied. 
# You can then remove tmp and merge/rebase into a more stable branch
 
# note that in a hierarchical repo network, e.g. there is one authoritative
# repo and several developers have their own forks, it's usually good practice
# to *merge* from the forks into the authoritative repo, and *rebase* from the
# authoritative repo into the forks. Same applies to dev branches / master
# branch.  I think it has something to do with resolving conflicts, though I
# haven't really hit a major issue with it yet.
 
# Adding a submodule
git submodule init
git submodule add path/to/remote/repo path/to/local/checkout
git submodule update
git commit -m "I just added a submodule"
 
# Updating a submodule
cd path/to/local/checkout
git pull origin master # or wherever you're pulling from
cd ../back/to/main/repo
git submodule update
git commit -m "I just updated a submodule"
 
# Clean out unwanted files
git clean -nx *.orig # dry run...
git clean -fx *.orig # actually do it

I’m just getting started with git.  Turns out if your git binaries are in a path specified in something like .bash_profile on your remote machine, something like git clone ssh://you.com/~you/repos.git will fail with an error like bash: git-upload-pack: command not found because sshd ignores .bash_profile, so git won’t find its binaries. So you need to alter the PATH env var in .bashrc instead. Fun.  The git-clone-pack man page is worth a look.

Django 1.0 is coming, and it’s bringing needed but backwards-incompatible changes to the admin site, ostensibly the best thing about Django.  One problem I’ve run into while migrating to the new version is inline editing of related classes in existing classes.  For example, it’s pretty common (and recommended) to “extend” Django’s built-in User model by creating your own, related Profile model.  For the old admin site, you specified the ability to edit a relate Profile in the User form in the Profile model:

class Profile(models.Model):
    user = models.ForeignKey(User, unique=True, edit_inline=True)

That was simple, but kinda quirky. Changing the way the User form looks by changing the related Profile model? Weird. BUT, it did let you make changes to the User model w/o mucking with the built in Django code.

The new way of doing things is to specify all your admin site options in a separate file, and inline editing is now a little more reasonable: if you want to edit related objects in a model’s form, you specify that in that model’s admin options. But this means you need to alter Django’s built-in User model’s admin options, which could mean muckiness. So here’s what I did in my myapp/admin.py:

from django.contrib import admin
from django.contrib.auth.models import User
from myapp.models import *
 
class ProfileAdmin(admin.ModelAdmin):
    # just some stuff for my profile forms
    filter_horizontal = ('teams',)
    radio_fields = {'appointment': admin.HORIZONTAL}
    list_display = ('user', 'picture', 'appointment', 'affiliation')
admin.site.register(Profile, ProfileAdmin)
 
# need to make an inline class...  ick
class ProfileInline(admin.StackedInline):
    model = Profile
    max_num = 1
 
# the meat: grab the existing UserAdmin class, modify it, unregister User,
# and re-register User with the modified UserAdmin
UserAdmin = admin.site._registry[User].__class__
UserAdmin.inlines = [ProfileInline,]
admin.site.unregister(User)
admin.site.register(User, UserAdmin)

Took me a while to realize admin.site._registry held instances of the ModelAdmin classes, not the classes themselves. This technique also assumes that User gets registered with the admin site before you run this code. I’m guessing the order in which admin.autodiscover() registers models is the same as the INSTALLED_APPS setting in settings.py.

In case you’re working through Developing Facebook Platform Applications with Rails, you might have run into problems with authenticity tokens and FBML forms.  Quick dumb fix to save you 2 seconds:

<fb:fbml>
  <fb:request-form
    action="<%= new_invitation_path %>"
    method="POST"
    invite="true"
    type="Karate Poke"
    content="YAYS I added this facebbok app">
    <fb:multi-friend-selector
      showborder="false"
      actiontext="Invite your friends to use Karate Poke"
    />
    <%= hidden_field_tag :authenticity_token, form_authenticity_token %>
  </fb:request-form>
</fb:fbml>