Home › Forums › Syphon › Syphon Development – Developer › Syphon image in a PBO-backed CIImage
- This topic has 6 replies, 2 voices, and was last updated 9 years, 7 months ago by
bangnoise.
-
AuthorPosts
-
July 18, 2011 at 4:43 am #5176
Andrea Cremaschi
Participanthi
since I’ve read in the forum that I am not the only one who need to have Syphon images in a Pixel buffer instead of in a texture (ie to do heavy processing, that take longer than Syphon frame rate), I decided to share some code.
Every time that a new frame is available, I draw it in a pixel buffer, using a Core Video pixel buffer pool.
happy coding!
a.c.@bangnoise, @vade: maybe there is a better way I don’t know?
// ### SETUP METHODS // 1. openGLContext - (bool)initOpenGLContextWithSharedContext: (NSOpenGLContext*)sharedContext error: (NSError **)error { NSOpenGLPixelFormatAttribute attributes[] = { NSOpenGLPFAPixelBuffer, NSOpenGLPFADepthSize, 32, (NSOpenGLPixelFormatAttribute) 0 }; NSOpenGLPixelFormat* newPixelFormat = [[[NSOpenGLPixelFormat alloc] initWithAttributes:attributes] autorelease]; NSOpenGLContext *newOpenGLContext = [[[NSOpenGLContext alloc] initWithFormat:newPixelFormat shareContext:sharedContext] autorelease]; if(newOpenGLContext == nil) { return false; } openGLContext = [newOpenGLContext retain]; pixelFormat = [newPixelFormat retain]; return true; } // 2. CVOpenGLBufferPool - (bool)initCVOpenGLBufferPoolWithSize: (NSSize) size error: (NSError **)error { CVReturn theError; //Create buffer pool NSMutableDictionary *attributes = [NSMutableDictionary dictionary]; [attributes setValue:[NSNumber numberWithInt:size.width] forKey:(NSString*)kCVOpenGLBufferWidth]; [attributes setValue:[NSNumber numberWithInt:size.height] forKey:(NSString*)kCVOpenGLBufferHeight]; theError = CVOpenGLBufferPoolCreate(kCFAllocatorDefault, NULL, (CFDictionaryRef)attributes, &_bufferPool); if(theError) { NSLog(@"CVPixelBufferPoolCreate() failed with error %i", theError); [self release]; return false; } CVOpenGLBufferPoolRetain(_bufferPool); return (theError == kCVReturnSuccess); } // ### NEW FRAME METHODS // didReceiveNewFrame: is the frame handler defined in Syphon init method, // as in the Syphon example by bangnoise - (CIImage *)createImageForDelegate: (id) thisDelegate { // cgl_ctx??? if ((nil== thisDelegate) || !([thisDelegate respondsToSelector: @selector(openGLContext)])) return nil; CIImage *ciImage = nil; CGLContextObj cgl_ctx = [openGLContext CGLContextObj]; NSSize imageSize; if (cgl_ctx == nil) return nil; GLuint texture; // create image texture for current openGL context CGLLockContext(cgl_ctx); { // get the new Syphon image out of the server SyphonImage *image = [[syClient newFrameImageForContext:cgl_ctx] autorelease]; texture = [image textureName]; imageSize = [image textureSize]; BOOL changed = NO; if ((_lastSyphonImageSize.width != imageSize.width) || (_lastSyphonImageSize.height != imageSize.height)) { changed = YES; _lastSyphonImageSize.width = imageSize.width; _lastSyphonImageSize.height = imageSize.height; if(_bufferPool) CVOpenGLBufferPoolRelease(_bufferPool); [self initCVOpenGLBufferPoolWithSize: imageSize error: nil]; } //Get pixel buffer from pool CVOpenGLBufferRef pixelBuffer; CVReturn theError = CVOpenGLBufferPoolCreateOpenGLBuffer (kCFAllocatorDefault, _bufferPool, &pixelBuffer); if(theError) { NSLog(@"CVOpenGLBufferPoolCreateOpenGLBuffer() failed with error %i", theError); return NULL; } if (changed) { glViewport(0, 0, imageSize.width, imageSize.height); glMatrixMode(GL_MODELVIEW); // select the modelview matrix glLoadIdentity(); // reset it glMatrixMode(GL_PROJECTION); // select the projection matrix glLoadIdentity(); // reset it glOrtho(0, 0, imageSize.width, imageSize.height, -1.0, 1.0);// define a 2-D orthographic projection matrix } theError = CVOpenGLBufferAttach(pixelBuffer, [openGLContext CGLContextObj], 0, 0, [openGLContext currentVirtualScreen]); if (theError) { NSLog(@"CVOpenGLBufferAttach() failed with error %i", theError); return NULL; } //Use 'texture' to get texture target/id, texture bind, render to quad etc.. GLenum target = GL_TEXTURE_RECTANGLE_ARB; GLint name = texture; { glEnable(target); glBindTexture(target, name); // if you need to clear, here is the place to do it! // glClearColor( 0.0f, 0.0f, 0.0f, 1.0f ); // glClear( GL_COLOR_BUFFER_BIT ); // glClearDepth( 1.0f ); // glClear( GL_DEPTH_BUFFER_BIT ); glBegin(GL_QUADS); { glTexCoord2f( imageSize.width, 0.0f ); glVertex2f( 1.0f, -1.0f ); glTexCoord2f( 0.0f, 0.0f ); glVertex2f( -1.0f, -1.0f ); glTexCoord2f( 0.0f, imageSize.height ); glVertex2f( -1.0f, 1.0f ); glTexCoord2f( imageSize.width, imageSize.height ); glVertex2f( 1.0f, 1.0f ); } glEnd(); glFlush(); glDisable(target); ciImage = [CIImage imageWithCVImageBuffer: pixelBuffer]; CVOpenGLBufferRelease(pixelBuffer); } } CGLUnlockContext(cgl_ctx); return ciImage; } - (void) didReceiveNewFrame{ CIImage *newImage = [self createImageForDelegate: delegate]; [_frameQueue enqueue: newImage]; }
July 18, 2011 at 2:45 pm #5177bangnoise
KeymasterThanks for sharing!
I haven’t tested the performance of CVOpenGLBuffers. The “preferred” way is using GL frame-buffer objects paired with pixel-buffer objects – but this is a lot less code. Is it working at a reasonable rate for you?
July 19, 2011 at 7:27 am #5178Andrea Cremaschi
Participantwell, I haven’t done a in-depth profiling yet, but everything is smooth right now and I am quite happy with it. CoreVideo is supposed to take care of of PBOs’ creation and mantainment, but it is difficult to say what is going on under the hood (ie how many buffers are alive ecc…).
Anyway I will give you some performance feedback as soon as I’ve done some profiling ok?
ciao!
a.c.July 22, 2011 at 1:43 am #5179Andrea Cremaschi
Participantok, here i am
the method createImageForDelegate: takes about 6-7ms to complete on a i5-Geforce 330M
That’s not bad.
What about FBO-PBO method? Who wants to try ? 🙂
a.July 22, 2011 at 3:50 am #5180bangnoise
KeymasterHa great (though you don’t mention what dimensions your frames have).
We use FBO/PBO in Syphon Recorder – I’ll do a comparison when I get a moment.
Seems this is plenty good enough for whatever you’re doing though. Something exciting?
July 22, 2011 at 10:47 am #5181Andrea Cremaschi
ParticipantFrames were 640×400 (webcam input).
I’m developing an application for basic computer vision tasks (ie presence/motion detection) to use in theater shows, live performances ecc.. Of course syphon-enabled! The code-name is kineto.
I will let you know as soon as I release a first beta.a.c.
July 27, 2011 at 3:59 pm #5182bangnoise
KeymasterThe FBO/PBO route for comparison.
-
AuthorPosts
- You must be logged in to reply to this topic.