float textures, cpu buffer support

[Brian Chasalow]

float32 texture support, and cpu-side per-pixel access of a texture.
These are two features i could really use for some application-specific stuff. I mentioned these awhile back, but now that some time has passed I was curious if you guys had any thoughts on the viability of such features, and whether they makes sense. (Or, are they already implemented in some hackerly fashion I was not aware of? )
Brian

[bangnoise]

Float texture support is partially implemented in trunk on svn – pass in

[NSDictionary dictionaryWithObject:SyphonImageFormatRGBA32 forKey:SyphonServerOptionImageFormat]

as the options to SyphonServer (see SyphonServer.h).

What isn’t finished is the fallback to something that works if a client or server is in a GL context which doesn’t support float textures – otherwise it should work, and I’d be interested to hear how you get on with it.

We’ve discussed CPU-side pixel access a lot. The main concern is that direct access may open up an opportunity for a client app to block a remote server app indefinitely. We’d prefer people do buffered read-back in OpenGL – draw into FBO, download to PBO(s) – which is what we do in Syphon Recorder.

Cheers, Tom

[Brian Chasalow]

oo, interesting! Will poke at float textures in a bit. do you have an example of FBO -> PBO readback that I could take a peek at / hack apart? can go to email if necessary. thanks!
B

[bangnoise]

The exact way you do it depends on usage, and how much state you can preserve between calls, but here are the basics.

Setup

// Set-up: usually do this once and re-use these resources, however you may
// have to recreate them if the dimensions change

GLuint tex, fbo, pbo;
GLint prevFBO, prevReadFBO, prevDrawFBO;

// Store previous state
glGetIntegerv(GL_FRAMEBUFFER_BINDING_EXT, &prevFBO);
glGetIntegerv(GL_READ_FRAMEBUFFER_BINDING_EXT, &prevReadFBO);
glGetIntegerv(GL_DRAW_FRAMEBUFFER_BINDING_EXT, &prevDrawFBO);

// Push attribs
glPushAttrib(GL_ALL_ATTRIB_BITS);

glEnable(GL_TEXTURE_RECTANGLE_ARB);

// Create the texture we draw into
glGenTextures(1, &tex);
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, tex);
glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, GL_RGBA8, width, height, 0, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, NULL);		

// Create the FBO
glGenFramebuffersEXT(1, &fbo);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, fbo);

// Test that binding works
glFramebufferTexture2DEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_TEXTURE_RECTANGLE_ARB, tex, 0);
GLenum status = glCheckFramebufferStatusEXT(GL_FRAMEBUFFER_EXT);
if (status != GL_FRAMEBUFFER_COMPLETE_EXT)
{
		// Deal with this error - you won't be able to draw into the FBO
}

// Restore state we're done with thus-far
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, prevFBO);
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, prevReadFBO);
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, prevDrawFBO);

// Save PBO state
GLint prevPBO;
glGetIntegerv(GL_PIXEL_PACK_BUFFER_BINDING, &prevPBO);

// Create our PBO and request storage for it
glGenBuffers(1, &pbo);
glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo);
glBufferData(GL_PIXEL_PACK_BUFFER, width * height * 4, NULL, GL_DYNAMIC_READ);
if (glGetError() != GL_NO_ERROR)
{
		// Storage for the PBO couldn't be allocated, deal with it here
}

// Restore state
glBindBuffer(GL_PIXEL_PACK_BUFFER, prevPBO);
glPopAttrib();

Per-frame

// Save state as above, skipped for brevity
// ...

// The first thing we do is read data from the previous render-cycle
// This means the GPU has had a full frame's time to perform the download to PBO
// Skip this the first frame
if (thisIsNotTheFirstFrame)
{
	glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo);
	void *pixelData = glMapBuffer(GL_PIXEL_PACK_BUFFER, GL_READ_ONLY);
	// Do something with the pixel data
}

// Now start the current frame downloading

// We have a SyphonImage from somewhere, e.g.
SyphonImage *image = [myClient newFrameImageForContext:cgl_ctx];

// Attach the FBO
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, fbo);
glFramebufferTexture2DEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_TEXTURE_RECTANGLE_ARB, tex, 0);

// Set up required state
glViewport(0, 0,  width, height);
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();

glOrtho(0.0, width,  0.0,  height, -1, 1);		

glMatrixMode(GL_MODELVIEW);
glPushMatrix();
glLoadIdentity();

// Clear
glClearColor(0.0, 0.0, 0.0, 0.0);
glClear(GL_COLOR_BUFFER_BIT);

// Bind the texture
glEnable(GL_TEXTURE_RECTANGLE_ARB);
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_RECTANGLE_EXT, image.textureName);

// Configure texturing as we want it
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_WRAP_T, GL_CLAMP);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE);

glColor4f(1.0, 1.0, 1.0, 1.0);

// Draw it
// These coords flip the texture vertically because often you'll want to do that
GLfloat texCoords[] =
{
	0.0, height,
	width, height,
	width, 0.0,
	0.0, 0.0
};

GLfloat verts[] =
{
	0.0, 0.0,
	width, 0.0,
	width, height,
	0.0, height
};

glEnableClientState( GL_TEXTURE_COORD_ARRAY );
glTexCoordPointer(2, GL_FLOAT, 0, texCoords );
glEnableClientState(GL_VERTEX_ARRAY);
glVertexPointer(2, GL_FLOAT, 0, verts);
glDrawArrays( GL_TRIANGLE_FAN, 0, 4 );

// Now perform the download into the PBO

glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo);
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, tex);

// This is a minimal setup of pixel storage - if anything else might have touched it
// be more explicit
glPixelStorei(GL_PACK_ROW_LENGTH, width);

// Start the download to PBO
glGetTexImage(GL_TEXTURE_RECTANGLE_ARB, 0, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, (GLvoid *)0);

// Restore state, skipped for brevity, see set-up section
// ...

[bangnoise]

Or for less code, see this thread which does the same thing using CVOpenGLBuffers.

[Brian Chasalow]

awesome. thanks tom. will take a look at it in a bit. B

[Andrea Cremaschi]

@bangnoise: thanks for the code!

Actually, it does something different in respect of the one i posted: you are drawing in a -single- PBO, while the CVOpenGLBufferPool code manages -a pool- of frame buffers, using Core Video framework.
This helps when a new frame may be available in Syphon server before the old one has been entirely downloaded in CPU memory (i.e. in case of heavy GPU image manipulation retarding the frame download). This is also a bit more time consuming, since a new buffer must be attached to the openGL context for every new available Syphon image (ie the “CVOpenGLBufferAttach” method call). But with a slightly more complex code than the one I posted (in a multithread app), it is possible to avoid this delay, attaching the buffer as soon as the new Syphon image has been copied to the opengl buffer.

So, always speaking to the ones who want to copy syphon images to CPU memory: who just wants speed, doesn’t want to mess with multithreading and doesn’t need to do great image manipulation on the new Syphon image can go with the “opengl PBO method”. Who needs to do heavy GPU image manipulation, or need to process ALL the frames should go with the “CVOpenGLBufferPool method”..

ciao 🙂

ac

[bangnoise]

The above is intended as a solid start rather than a finished implementation.

You probably would want to use more than one PBO, although the way you do it (a low fixed number, or a pool) would depend on what you were doing with the buffers subsequently.

CoreVideo is likely doing something very similar to the above behind the scenes – its advantage is simply lightness of code.

[rsodre]

Hello!

Any news on the float texture support?

I saw some 1-year old comment on the old google code roadmap, I will need it pretty soon, just want to know if it’s already implemented or I need to hack my way.

Thanks!

[bangnoise]

Yo!

The GitHub framework still has the float branch, but it is considerably behind master and will be some work to merge. It still needs work to cope with a server and client running in contexts with different levels of support for float textures, and I can’t remember how it interacts with clients using the current framework – may need work there too.

If you’re interested/keen, we’d gladly consider any pull requests to the float branch – if you’re making sweeping changes, probably best to discuss them here or on GitHub before starting.

[rsodre]

I took a look in the float branch, not sure if I can help to properly finish it.
For my project, I can have server and client using the same version, so I’ll try to compile and use this branch, I’ll report what happens when I do it.
Thanks!

[Author]

Posts