Syphon image in a PBO-backed CIImage

Home Forums Syphon Syphon Development – Developer Syphon image in a PBO-backed CIImage

Viewing 7 posts - 1 through 7 (of 7 total)
  • Author
  • #5176
    Andrea Cremaschi

    since I’ve read in the forum that I am not the only one who need to have Syphon images in a Pixel buffer instead of in a texture (ie to do heavy processing, that take longer than Syphon frame rate), I decided to share some code.
    Every time that a new frame is available, I draw it in a pixel buffer, using a Core Video pixel buffer pool.
    happy coding!

    @bangnoise, @vade: maybe there is a better way I don’t know?

    // ### SETUP METHODS
    // 1. openGLContext
    - (bool)initOpenGLContextWithSharedContext: (NSOpenGLContext*)sharedContext error: (NSError **)error {
    	NSOpenGLPixelFormatAttribute	attributes[] = {
    		NSOpenGLPFADepthSize, 32,
    		(NSOpenGLPixelFormatAttribute) 0
    	NSOpenGLPixelFormat*	newPixelFormat = [[[NSOpenGLPixelFormat alloc] initWithAttributes:attributes] autorelease];
    	NSOpenGLContext *newOpenGLContext = [[[NSOpenGLContext alloc] initWithFormat:newPixelFormat
    												  shareContext:sharedContext] autorelease];
    	if(newOpenGLContext == nil) {
    		return false;
    	openGLContext = [newOpenGLContext retain];
    	pixelFormat = [newPixelFormat retain];
    	return true;
    // 2. CVOpenGLBufferPool
    - (bool)initCVOpenGLBufferPoolWithSize: (NSSize) size
    								 error: (NSError **)error {
    	CVReturn						theError;
    	//Create buffer pool
    	NSMutableDictionary *attributes = [NSMutableDictionary dictionary];
    	[attributes setValue:[NSNumber numberWithInt:size.width] forKey:(NSString*)kCVOpenGLBufferWidth];
    	[attributes setValue:[NSNumber numberWithInt:size.height] forKey:(NSString*)kCVOpenGLBufferHeight];
    	theError = CVOpenGLBufferPoolCreate(kCFAllocatorDefault, NULL, (CFDictionaryRef)attributes, &_bufferPool);
    	if(theError) {
    		NSLog(@"CVPixelBufferPoolCreate() failed with error %i", theError);
    		[self release];
    		return false;
    	return (theError == kCVReturnSuccess);
    // didReceiveNewFrame: is the frame handler defined in Syphon init method,
    // as in the Syphon example by bangnoise
    - (CIImage *)createImageForDelegate: (id) thisDelegate {
    	// cgl_ctx???
    	if ((nil== thisDelegate) || !([thisDelegate respondsToSelector: @selector(openGLContext)]))
    		return nil;
    	CIImage *ciImage		= nil;
    	CGLContextObj cgl_ctx	= [openGLContext CGLContextObj];
    	NSSize imageSize;
    	if (cgl_ctx == nil)
    		return nil;
    	GLuint texture;
    	// create image texture for current openGL context
    		// get the new Syphon image out of the server
    		SyphonImage *image = [[syClient newFrameImageForContext:cgl_ctx] autorelease];
    		texture = [image textureName];
    		imageSize = [image textureSize];
    	BOOL changed = NO;
    	if ((_lastSyphonImageSize.width != imageSize.width) ||
    		(_lastSyphonImageSize.height != imageSize.height))
    		changed = YES;
    		_lastSyphonImageSize.width = imageSize.width;
    		_lastSyphonImageSize.height = imageSize.height;
    		[self initCVOpenGLBufferPoolWithSize: imageSize error: nil];
    	//Get pixel buffer from pool
    	CVOpenGLBufferRef	pixelBuffer;
    	CVReturn theError = CVOpenGLBufferPoolCreateOpenGLBuffer (kCFAllocatorDefault, _bufferPool, &pixelBuffer);
    	if(theError) {
    		NSLog(@"CVOpenGLBufferPoolCreateOpenGLBuffer() failed with error %i", theError);
    		return NULL;
    		if (changed)
    			glViewport(0, 0, imageSize.width, imageSize.height);
    			glMatrixMode(GL_MODELVIEW);    // select the modelview matrix
    			glLoadIdentity();              // reset it
    			glMatrixMode(GL_PROJECTION);   // select the projection matrix
    			glLoadIdentity();              // reset it
    			glOrtho(0, 0, imageSize.width, imageSize.height, -1.0, 1.0);// define a 2-D orthographic projection matrix
    		theError = CVOpenGLBufferAttach(pixelBuffer,
    										[openGLContext CGLContextObj],
    										0, 0, [openGLContext currentVirtualScreen]);
    		if (theError)	{
    			NSLog(@"CVOpenGLBufferAttach() failed with error %i", theError);
    			return NULL;
    		//Use 'texture' to get texture target/id, texture bind, render to quad etc..
    		GLenum target = GL_TEXTURE_RECTANGLE_ARB;
    		GLint name = texture;
    			glBindTexture(target, name);
    			// if you need to clear, here is the place to do it!
    			// glClearColor( 0.0f, 0.0f, 0.0f, 1.0f );
    			// glClear( GL_COLOR_BUFFER_BIT );
    			// glClearDepth( 1.0f );
    			// glClear( GL_DEPTH_BUFFER_BIT );
    				glTexCoord2f( imageSize.width, 0.0f );				glVertex2f(  1.0f, -1.0f );
    				glTexCoord2f( 0.0f, 0.0f );							glVertex2f( -1.0f, -1.0f );
    				glTexCoord2f( 0.0f, imageSize.height );				glVertex2f( -1.0f, 1.0f );
    				glTexCoord2f( imageSize.width, imageSize.height );	glVertex2f(  1.0f, 1.0f );
    			ciImage = [CIImage imageWithCVImageBuffer: pixelBuffer];
    	return ciImage;
    - (void) didReceiveNewFrame{
    	CIImage *newImage = [self createImageForDelegate: delegate];
    	[_frameQueue enqueue: newImage];

    Thanks for sharing!

    I haven’t tested the performance of CVOpenGLBuffers. The “preferred” way is using GL frame-buffer objects paired with pixel-buffer objects – but this is a lot less code. Is it working at a reasonable rate for you?

    Andrea Cremaschi

    well, I haven’t done a in-depth profiling yet, but everything is smooth right now and I am quite happy with it. CoreVideo is supposed to take care of of PBOs’ creation and mantainment, but it is difficult to say what is going on under the hood (ie how many buffers are alive ecc…).
    Anyway I will give you some performance feedback as soon as I’ve done some profiling ok?

    Andrea Cremaschi

    ok, here i am
    the method createImageForDelegate: takes about 6-7ms to complete on a i5-Geforce 330M
    That’s not bad.
    What about FBO-PBO method? Who wants to try ? 🙂


    Ha great (though you don’t mention what dimensions your frames have).

    We use FBO/PBO in Syphon Recorder – I’ll do a comparison when I get a moment.

    Seems this is plenty good enough for whatever you’re doing though. Something exciting?

    Andrea Cremaschi

    Frames were 640×400 (webcam input).
    I’m developing an application for basic computer vision tasks (ie presence/motion detection) to use in theater shows, live performances ecc.. Of course syphon-enabled! The code-name is kineto.
    I will let you know as soon as I release a first beta.



    The FBO/PBO route for comparison.

Viewing 7 posts - 1 through 7 (of 7 total)
  • You must be logged in to reply to this topic.