I did a bit more messing around with this.
I got gfxCardStatus to see which GPU was being used. The discrete GPU (the NVIDIA) was being used. So I quit Quartz and set gfxCardStatus to make the computer use only the integrated GPU (Intel Iris). That almost doubled my frame rate (to 15fps). If I could double it again, that’d be fast enough for my purposes.
So it seems like maybe it’s something about the NVIDIA GPU that isn’t dealing with this well.