Syphon image in a PBO-backed CIImage

[Andrea Cremaschi]

hi
since I’ve read in the forum that I am not the only one who need to have Syphon images in a Pixel buffer instead of in a texture (ie to do heavy processing, that take longer than Syphon frame rate), I decided to share some code.
Every time that a new frame is available, I draw it in a pixel buffer, using a Core Video pixel buffer pool.
happy coding!
a.c.

@bangnoise, @vade: maybe there is a better way I don’t know?

// ### SETUP METHODS

// 1. openGLContext

- (bool)initOpenGLContextWithSharedContext: (NSOpenGLContext*)sharedContext error: (NSError **)error {

	NSOpenGLPixelFormatAttribute	attributes[] = {
		NSOpenGLPFAPixelBuffer,
		NSOpenGLPFADepthSize, 32,
		(NSOpenGLPixelFormatAttribute) 0
	};
	NSOpenGLPixelFormat*	newPixelFormat = [[[NSOpenGLPixelFormat alloc] initWithAttributes:attributes] autorelease];

	NSOpenGLContext *newOpenGLContext = [[[NSOpenGLContext alloc] initWithFormat:newPixelFormat
												  shareContext:sharedContext] autorelease];
	if(newOpenGLContext == nil) {
		return false;
	}

	openGLContext = [newOpenGLContext retain];
	pixelFormat = [newPixelFormat retain];

	return true;

}

// 2. CVOpenGLBufferPool
- (bool)initCVOpenGLBufferPoolWithSize: (NSSize) size
								 error: (NSError **)error {

	CVReturn						theError;

	//Create buffer pool
	NSMutableDictionary *attributes = [NSMutableDictionary dictionary];

	[attributes setValue:[NSNumber numberWithInt:size.width] forKey:(NSString*)kCVOpenGLBufferWidth];
	[attributes setValue:[NSNumber numberWithInt:size.height] forKey:(NSString*)kCVOpenGLBufferHeight];

	theError = CVOpenGLBufferPoolCreate(kCFAllocatorDefault, NULL, (CFDictionaryRef)attributes, &_bufferPool);
	if(theError) {
		NSLog(@"CVPixelBufferPoolCreate() failed with error %i", theError);
		[self release];
		return false;
	}
	CVOpenGLBufferPoolRetain(_bufferPool);
	return (theError == kCVReturnSuccess);

}

// ### NEW FRAME METHODS
// didReceiveNewFrame: is the frame handler defined in Syphon init method,
// as in the Syphon example by bangnoise

- (CIImage *)createImageForDelegate: (id) thisDelegate {

	// cgl_ctx???
	if ((nil== thisDelegate) || !([thisDelegate respondsToSelector: @selector(openGLContext)]))
		return nil;

	CIImage *ciImage		= nil;
	CGLContextObj cgl_ctx	= [openGLContext CGLContextObj];
	NSSize imageSize;

	if (cgl_ctx == nil)
		return nil;

	GLuint texture;

	// create image texture for current openGL context
	CGLLockContext(cgl_ctx);
	{
		// get the new Syphon image out of the server
		SyphonImage *image = [[syClient newFrameImageForContext:cgl_ctx] autorelease];

		texture = [image textureName];
		imageSize = [image textureSize];

	BOOL changed = NO;
	if ((_lastSyphonImageSize.width != imageSize.width) ||
		(_lastSyphonImageSize.height != imageSize.height))
	{
		changed = YES;
		_lastSyphonImageSize.width = imageSize.width;
		_lastSyphonImageSize.height = imageSize.height;
		if(_bufferPool)
			CVOpenGLBufferPoolRelease(_bufferPool);
		[self initCVOpenGLBufferPoolWithSize: imageSize error: nil];
	}

	//Get pixel buffer from pool
	CVOpenGLBufferRef	pixelBuffer;
	CVReturn theError = CVOpenGLBufferPoolCreateOpenGLBuffer (kCFAllocatorDefault, _bufferPool, &pixelBuffer);
	if(theError) {
		NSLog(@"CVOpenGLBufferPoolCreateOpenGLBuffer() failed with error %i", theError);
		return NULL;
	}

		if (changed)
		{
			glViewport(0, 0, imageSize.width, imageSize.height);

			glMatrixMode(GL_MODELVIEW);    // select the modelview matrix
			glLoadIdentity();              // reset it

			glMatrixMode(GL_PROJECTION);   // select the projection matrix
			glLoadIdentity();              // reset it

			glOrtho(0, 0, imageSize.width, imageSize.height, -1.0, 1.0);// define a 2-D orthographic projection matrix
		}

		theError = CVOpenGLBufferAttach(pixelBuffer,
										[openGLContext CGLContextObj],
										0, 0, [openGLContext currentVirtualScreen]);
		if (theError)	{
			NSLog(@"CVOpenGLBufferAttach() failed with error %i", theError);
			return NULL;
		}		

		//Use 'texture' to get texture target/id, texture bind, render to quad etc..
		GLenum target = GL_TEXTURE_RECTANGLE_ARB;
		GLint name = texture;

		{
			glEnable(target);
			glBindTexture(target, name);

			// if you need to clear, here is the place to do it!
			// glClearColor( 0.0f, 0.0f, 0.0f, 1.0f );
			// glClear( GL_COLOR_BUFFER_BIT );
			// glClearDepth( 1.0f );
			// glClear( GL_DEPTH_BUFFER_BIT );

			glBegin(GL_QUADS);
			{
				glTexCoord2f( imageSize.width, 0.0f );				glVertex2f(  1.0f, -1.0f );
				glTexCoord2f( 0.0f, 0.0f );							glVertex2f( -1.0f, -1.0f );
				glTexCoord2f( 0.0f, imageSize.height );				glVertex2f( -1.0f, 1.0f );
				glTexCoord2f( imageSize.width, imageSize.height );	glVertex2f(  1.0f, 1.0f );
			}
			glEnd();

			glFlush();
			glDisable(target);

			ciImage = [CIImage imageWithCVImageBuffer: pixelBuffer];
			CVOpenGLBufferRelease(pixelBuffer);

		}

	}
	CGLUnlockContext(cgl_ctx);
	return ciImage;

}

- (void) didReceiveNewFrame{
	CIImage *newImage = [self createImageForDelegate: delegate];

	[_frameQueue enqueue: newImage];

}

[bangnoise]

Thanks for sharing!

I haven’t tested the performance of CVOpenGLBuffers. The “preferred” way is using GL frame-buffer objects paired with pixel-buffer objects – but this is a lot less code. Is it working at a reasonable rate for you?

[Andrea Cremaschi]

well, I haven’t done a in-depth profiling yet, but everything is smooth right now and I am quite happy with it. CoreVideo is supposed to take care of of PBOs’ creation and mantainment, but it is difficult to say what is going on under the hood (ie how many buffers are alive ecc…).
Anyway I will give you some performance feedback as soon as I’ve done some profiling ok?
ciao!
a.c.

[Andrea Cremaschi]

ok, here i am
the method createImageForDelegate: takes about 6-7ms to complete on a i5-Geforce 330M
That’s not bad.
What about FBO-PBO method? Who wants to try ? 🙂
a.

[bangnoise]

Ha great (though you don’t mention what dimensions your frames have).

We use FBO/PBO in Syphon Recorder – I’ll do a comparison when I get a moment.

Seems this is plenty good enough for whatever you’re doing though. Something exciting?

[Andrea Cremaschi]

Frames were 640×400 (webcam input).
I’m developing an application for basic computer vision tasks (ie presence/motion detection) to use in theater shows, live performances ecc.. Of course syphon-enabled! The code-name is kineto.
I will let you know as soon as I release a first beta.

a.c.

[bangnoise]

The FBO/PBO route for comparison.

[Author]

Posts