Syphon image in a PBO-backed CIImage

Home Forums Syphon Syphon Development – Developer Syphon image in a PBO-backed CIImage

Viewing 7 posts - 1 through 7 (of 7 total)
  • Author
    Posts
  • #5176
    Andrea Cremaschi
    Participant

    hi
    since I’ve read in the forum that I am not the only one who need to have Syphon images in a Pixel buffer instead of in a texture (ie to do heavy processing, that take longer than Syphon frame rate), I decided to share some code.
    Every time that a new frame is available, I draw it in a pixel buffer, using a Core Video pixel buffer pool.
    happy coding!
    a.c.

    @bangnoise, @vade: maybe there is a better way I don’t know?

    // ### SETUP METHODS
    
    // 1. openGLContext
    
    - (bool)initOpenGLContextWithSharedContext: (NSOpenGLContext*)sharedContext error: (NSError **)error {
    
    	NSOpenGLPixelFormatAttribute	attributes[] = {
    		NSOpenGLPFAPixelBuffer,
    		NSOpenGLPFADepthSize, 32,
    		(NSOpenGLPixelFormatAttribute) 0
    	};
    	NSOpenGLPixelFormat*	newPixelFormat = [[[NSOpenGLPixelFormat alloc] initWithAttributes:attributes] autorelease];
    
    	NSOpenGLContext *newOpenGLContext = [[[NSOpenGLContext alloc] initWithFormat:newPixelFormat
    												  shareContext:sharedContext] autorelease];
    	if(newOpenGLContext == nil) {
    		return false;
    	}
    
    	openGLContext = [newOpenGLContext retain];
    	pixelFormat = [newPixelFormat retain];
    
    	return true;
    
    }
    
    // 2. CVOpenGLBufferPool
    - (bool)initCVOpenGLBufferPoolWithSize: (NSSize) size
    								 error: (NSError **)error {
    
    	CVReturn						theError;
    
    	//Create buffer pool
    	NSMutableDictionary *attributes = [NSMutableDictionary dictionary];
    
    	[attributes setValue:[NSNumber numberWithInt:size.width] forKey:(NSString*)kCVOpenGLBufferWidth];
    	[attributes setValue:[NSNumber numberWithInt:size.height] forKey:(NSString*)kCVOpenGLBufferHeight];
    
    	theError = CVOpenGLBufferPoolCreate(kCFAllocatorDefault, NULL, (CFDictionaryRef)attributes, &_bufferPool);
    	if(theError) {
    		NSLog(@"CVPixelBufferPoolCreate() failed with error %i", theError);
    		[self release];
    		return false;
    	}
    	CVOpenGLBufferPoolRetain(_bufferPool);
    	return (theError == kCVReturnSuccess);
    
    }
    
    // ### NEW FRAME METHODS
    // didReceiveNewFrame: is the frame handler defined in Syphon init method,
    // as in the Syphon example by bangnoise
    
    - (CIImage *)createImageForDelegate: (id) thisDelegate {
    
    	// cgl_ctx???
    	if ((nil== thisDelegate) || !([thisDelegate respondsToSelector: @selector(openGLContext)]))
    		return nil;
    
    	CIImage *ciImage		= nil;
    	CGLContextObj cgl_ctx	= [openGLContext CGLContextObj];
    	NSSize imageSize;
    
    	if (cgl_ctx == nil)
    		return nil;
    
    	GLuint texture;
    
    	// create image texture for current openGL context
    	CGLLockContext(cgl_ctx);
    	{
    		// get the new Syphon image out of the server
    		SyphonImage *image = [[syClient newFrameImageForContext:cgl_ctx] autorelease];
    
    		texture = [image textureName];
    		imageSize = [image textureSize];
    
    	BOOL changed = NO;
    	if ((_lastSyphonImageSize.width != imageSize.width) ||
    		(_lastSyphonImageSize.height != imageSize.height))
    	{
    		changed = YES;
    		_lastSyphonImageSize.width = imageSize.width;
    		_lastSyphonImageSize.height = imageSize.height;
    		if(_bufferPool)
    			CVOpenGLBufferPoolRelease(_bufferPool);
    		[self initCVOpenGLBufferPoolWithSize: imageSize error: nil];
    	}
    
    	//Get pixel buffer from pool
    	CVOpenGLBufferRef	pixelBuffer;
    	CVReturn theError = CVOpenGLBufferPoolCreateOpenGLBuffer (kCFAllocatorDefault, _bufferPool, &pixelBuffer);
    	if(theError) {
    		NSLog(@"CVOpenGLBufferPoolCreateOpenGLBuffer() failed with error %i", theError);
    		return NULL;
    	}
    
    		if (changed)
    		{
    			glViewport(0, 0, imageSize.width, imageSize.height);
    
    			glMatrixMode(GL_MODELVIEW);    // select the modelview matrix
    			glLoadIdentity();              // reset it
    
    			glMatrixMode(GL_PROJECTION);   // select the projection matrix
    			glLoadIdentity();              // reset it
    
    			glOrtho(0, 0, imageSize.width, imageSize.height, -1.0, 1.0);// define a 2-D orthographic projection matrix
    		}
    
    		theError = CVOpenGLBufferAttach(pixelBuffer,
    										[openGLContext CGLContextObj],
    										0, 0, [openGLContext currentVirtualScreen]);
    		if (theError)	{
    			NSLog(@"CVOpenGLBufferAttach() failed with error %i", theError);
    			return NULL;
    		}		
    
    		//Use 'texture' to get texture target/id, texture bind, render to quad etc..
    		GLenum target = GL_TEXTURE_RECTANGLE_ARB;
    		GLint name = texture;
    
    		{
    			glEnable(target);
    			glBindTexture(target, name);
    
    			// if you need to clear, here is the place to do it!
    			// glClearColor( 0.0f, 0.0f, 0.0f, 1.0f );
    			// glClear( GL_COLOR_BUFFER_BIT );
    			// glClearDepth( 1.0f );
    			// glClear( GL_DEPTH_BUFFER_BIT );
    
    			glBegin(GL_QUADS);
    			{
    				glTexCoord2f( imageSize.width, 0.0f );				glVertex2f(  1.0f, -1.0f );
    				glTexCoord2f( 0.0f, 0.0f );							glVertex2f( -1.0f, -1.0f );
    				glTexCoord2f( 0.0f, imageSize.height );				glVertex2f( -1.0f, 1.0f );
    				glTexCoord2f( imageSize.width, imageSize.height );	glVertex2f(  1.0f, 1.0f );
    			}
    			glEnd();
    
    			glFlush();
    			glDisable(target);
    
    			ciImage = [CIImage imageWithCVImageBuffer: pixelBuffer];
    			CVOpenGLBufferRelease(pixelBuffer);
    
    		}
    
    	}
    	CGLUnlockContext(cgl_ctx);
    	return ciImage;
    
    }
    
    - (void) didReceiveNewFrame{
    	CIImage *newImage = [self createImageForDelegate: delegate];
    
    	[_frameQueue enqueue: newImage];
    
    }
    #5177
    bangnoise
    Keymaster

    Thanks for sharing!

    I haven’t tested the performance of CVOpenGLBuffers. The “preferred” way is using GL frame-buffer objects paired with pixel-buffer objects – but this is a lot less code. Is it working at a reasonable rate for you?

    #5178
    Andrea Cremaschi
    Participant

    well, I haven’t done a in-depth profiling yet, but everything is smooth right now and I am quite happy with it. CoreVideo is supposed to take care of of PBOs’ creation and mantainment, but it is difficult to say what is going on under the hood (ie how many buffers are alive ecc…).
    Anyway I will give you some performance feedback as soon as I’ve done some profiling ok?
    ciao!
    a.c.

    #5179
    Andrea Cremaschi
    Participant

    ok, here i am
    the method createImageForDelegate: takes about 6-7ms to complete on a i5-Geforce 330M
    That’s not bad.
    What about FBO-PBO method? Who wants to try ? 🙂
    a.

    #5180
    bangnoise
    Keymaster

    Ha great (though you don’t mention what dimensions your frames have).

    We use FBO/PBO in Syphon Recorder – I’ll do a comparison when I get a moment.

    Seems this is plenty good enough for whatever you’re doing though. Something exciting?

    #5181
    Andrea Cremaschi
    Participant

    Frames were 640×400 (webcam input).
    I’m developing an application for basic computer vision tasks (ie presence/motion detection) to use in theater shows, live performances ecc.. Of course syphon-enabled! The code-name is kineto.
    I will let you know as soon as I release a first beta.

    a.c.

    #5182
    bangnoise
    Keymaster

    The FBO/PBO route for comparison.

Viewing 7 posts - 1 through 7 (of 7 total)
  • You must be logged in to reply to this topic.