[bangnoise]

The above is intended as a solid start rather than a finished implementation.

You probably would want to use more than one PBO, although the way you do it (a low fixed number, or a pool) would depend on what you were doing with the buffers subsequently.

CoreVideo is likely doing something very similar to the above behind the scenes – its advantage is simply lightness of code.

[bangnoise]

@Vj KayCee

Do you have the Syphon Plug-In for QC installed? It’s available on the Syphon site.

You’ll need that, as well as the effect linked above.

[bangnoise]

http://syphon.v002.info/faq.php

[bangnoise]

The FBO/PBO route for comparison.

[bangnoise]

Or for less code, see this thread which does the same thing using CVOpenGLBuffers.

[bangnoise]

The exact way you do it depends on usage, and how much state you can preserve between calls, but here are the basics.

Setup

// Set-up: usually do this once and re-use these resources, however you may
// have to recreate them if the dimensions change

GLuint tex, fbo, pbo;
GLint prevFBO, prevReadFBO, prevDrawFBO;

// Store previous state
glGetIntegerv(GL_FRAMEBUFFER_BINDING_EXT, &prevFBO);
glGetIntegerv(GL_READ_FRAMEBUFFER_BINDING_EXT, &prevReadFBO);
glGetIntegerv(GL_DRAW_FRAMEBUFFER_BINDING_EXT, &prevDrawFBO);

// Push attribs
glPushAttrib(GL_ALL_ATTRIB_BITS);

glEnable(GL_TEXTURE_RECTANGLE_ARB);

// Create the texture we draw into
glGenTextures(1, &tex);
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, tex);
glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, GL_RGBA8, width, height, 0, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, NULL);		

// Create the FBO
glGenFramebuffersEXT(1, &fbo);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, fbo);

// Test that binding works
glFramebufferTexture2DEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_TEXTURE_RECTANGLE_ARB, tex, 0);
GLenum status = glCheckFramebufferStatusEXT(GL_FRAMEBUFFER_EXT);
if (status != GL_FRAMEBUFFER_COMPLETE_EXT)
{
		// Deal with this error - you won't be able to draw into the FBO
}

// Restore state we're done with thus-far
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, prevFBO);
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, prevReadFBO);
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, prevDrawFBO);

// Save PBO state
GLint prevPBO;
glGetIntegerv(GL_PIXEL_PACK_BUFFER_BINDING, &prevPBO);

// Create our PBO and request storage for it
glGenBuffers(1, &pbo);
glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo);
glBufferData(GL_PIXEL_PACK_BUFFER, width * height * 4, NULL, GL_DYNAMIC_READ);
if (glGetError() != GL_NO_ERROR)
{
		// Storage for the PBO couldn't be allocated, deal with it here
}

// Restore state
glBindBuffer(GL_PIXEL_PACK_BUFFER, prevPBO);
glPopAttrib();

Per-frame

// Save state as above, skipped for brevity
// ...

// The first thing we do is read data from the previous render-cycle
// This means the GPU has had a full frame's time to perform the download to PBO
// Skip this the first frame
if (thisIsNotTheFirstFrame)
{
	glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo);
	void *pixelData = glMapBuffer(GL_PIXEL_PACK_BUFFER, GL_READ_ONLY);
	// Do something with the pixel data
}

// Now start the current frame downloading

// We have a SyphonImage from somewhere, e.g.
SyphonImage *image = [myClient newFrameImageForContext:cgl_ctx];

// Attach the FBO
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, fbo);
glFramebufferTexture2DEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_TEXTURE_RECTANGLE_ARB, tex, 0);

// Set up required state
glViewport(0, 0,  width, height);
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();

glOrtho(0.0, width,  0.0,  height, -1, 1);		

glMatrixMode(GL_MODELVIEW);
glPushMatrix();
glLoadIdentity();

// Clear
glClearColor(0.0, 0.0, 0.0, 0.0);
glClear(GL_COLOR_BUFFER_BIT);

// Bind the texture
glEnable(GL_TEXTURE_RECTANGLE_ARB);
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_RECTANGLE_EXT, image.textureName);

// Configure texturing as we want it
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_WRAP_T, GL_CLAMP);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_RECTANGLE_EXT, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE);

glColor4f(1.0, 1.0, 1.0, 1.0);

// Draw it
// These coords flip the texture vertically because often you'll want to do that
GLfloat texCoords[] =
{
	0.0, height,
	width, height,
	width, 0.0,
	0.0, 0.0
};

GLfloat verts[] =
{
	0.0, 0.0,
	width, 0.0,
	width, height,
	0.0, height
};

glEnableClientState( GL_TEXTURE_COORD_ARRAY );
glTexCoordPointer(2, GL_FLOAT, 0, texCoords );
glEnableClientState(GL_VERTEX_ARRAY);
glVertexPointer(2, GL_FLOAT, 0, verts);
glDrawArrays( GL_TRIANGLE_FAN, 0, 4 );

// Now perform the download into the PBO

glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo);
glBindTexture(GL_TEXTURE_RECTANGLE_ARB, tex);

// This is a minimal setup of pixel storage - if anything else might have touched it
// be more explicit
glPixelStorei(GL_PACK_ROW_LENGTH, width);

// Start the download to PBO
glGetTexImage(GL_TEXTURE_RECTANGLE_ARB, 0, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, (GLvoid *)0);

// Restore state, skipped for brevity, see set-up section
// ...

[bangnoise]

Float texture support is partially implemented in trunk on svn – pass in

[NSDictionary dictionaryWithObject:SyphonImageFormatRGBA32 forKey:SyphonServerOptionImageFormat]

as the options to SyphonServer (see SyphonServer.h).

What isn’t finished is the fallback to something that works if a client or server is in a GL context which doesn’t support float textures – otherwise it should work, and I’d be interested to hear how you get on with it.

We’ve discussed CPU-side pixel access a lot. The main concern is that direct access may open up an opportunity for a client app to block a remote server app indefinitely. We’d prefer people do buffered read-back in OpenGL – draw into FBO, download to PBO(s) – which is what we do in Syphon Recorder.

Cheers, Tom

[bangnoise]

Where does the Quartz Composer plugin come in? VDMX has native Syphon support, there’s no need for QC. I’d try filing a bug report from within VDMX and include the project file. Vidvox are usually very helpful and may be able to spot the problem faster than our guessing 🙂

[bangnoise]

Ha great (though you don’t mention what dimensions your frames have).

We use FBO/PBO in Syphon Recorder – I’ll do a comparison when I get a moment.

Seems this is plenty good enough for whatever you’re doing though. Something exciting?

[bangnoise]

Yo hi

That should be simple to do – I’d ask the author of Graffiti Analysis about adding it. It looks like it’s made using OpenFramworks – we have an OF addon available on svn – it’s a bit in-progress but people are already using it successfully.

Cheers – Tom

[bangnoise]

The border sounds like the canvas dimensions in VDMX don’t match the Syphon layer’s dimensions – check that?

What are the dimensions of the scene you’re sending from Unity? If you send a similarly dimensioned scene from another app (eg Simple Server or QC) do you get the same slowdown going fullscreen?

[bangnoise]

New download is up – Syphon for Unity Public Beta 2 r2

[bangnoise]

Thanks for sharing!

I haven’t tested the performance of CVOpenGLBuffers. The “preferred” way is using GL frame-buffer objects paired with pixel-buffer objects – but this is a lot less code. Is it working at a reasonable rate for you?

[bangnoise]

Are you doing this on a machine with multiple graphics cards, and is the fullscreen display connected to a different card than the one you’re running windowed on? Or, more generally, what is your hardware setup..?

Do you see the black borders in Simple Client ( http://syphon.v002.info/#Apps ) or only VDMX?

[bangnoise]

Can’t you just build up your layers inside Modul8 without using Quartz Composer?

Without knowing what you are doing thus-far, it’s hard to suggest what an easier way might be…

[bangnoise]

Just to be a voice of dissent, I’m a -1 vote for using SyphonNameboundClient – it is intended for things such as plugins where the plugin API dictates that the name and app name parameter are not allowed to change, whereas in a programming environment like Processing, it’s perfectly fine to have properties of the object change over time.

However vade suggested it because you could instantiate a client from a name/app-name pair, which I think is a useful feature for a client object, as he says, to avoid passing the dictionaries the framework uses. Wrapping those as some sort of token object would also be acceptable though. If you want to instantiate clients from name/app-name pairs then SyphonNameboundClient may have useful code to rip out for finding matching servers, just don’t use the whole class as-is.

I think SyphonServerDirectory (to mirror the framework’s naming) would be a useful addition. Hopefully it could deliver app and server name pairs rather than just server names (some servers may have no name).

Most of all though, thanks for giving this thought and time!

[bangnoise]

Is there a way for the screen capture to “Confine to Application WIndow” like Camtwist can?
This is something that I didn’t like about the v002 screen capture plug-in too.

Thanks for the feature request. Noted.

Screen capture should generally be a last resort – perhaps what you’re doing can be done in an app which can use Syphon natively?

For VJs to be able to share their software outputs to other computers would
be so helpful!

There are various ways to do this already. See the FAQ.

Cheers – Tom

[bangnoise]

Great – glad you’re on the right path.

I’d really recommend using PBOs for your GL path – you’ll see a big performance improvement from the async read-back.

[bangnoise]

Sounds a nice idea. I expect a custom plugin would have the same processor-pig issues. We will add it to the list…

[bangnoise]

Your diagnosis seems correct.

Avoid readback to main memory entirely if at all possible (what are you doing with these buffers?).

If you must do it, don’t use CIImage, do it all directly in OpenGL.

Use a pair of PBOs and do buffered readback. The only gotcha is that you can’t usefully glGetTexImage2D a texture from Syphon so you have to draw it into an FBO and then glGetTexImage2D the texture backing the FBO.

[Author]

Posts