Your diagnosis seems correct.
Avoid readback to main memory entirely if at all possible (what are you doing with these buffers?).
If you must do it, don’t use CIImage, do it all directly in OpenGL.
Use a pair of PBOs and do buffered readback. The only gotcha is that you can’t usefully glGetTexImage2D a texture from Syphon so you have to draw it into an FBO and then glGetTexImage2D the texture backing the FBO.