[WeivCo]

Thanks for the info vade. You are truly an expert in this area. 🙂

With regards to the high res, my latest round of testing for the last couple days has been with 1024×768, but I’ve noticed this issue the entire time in 640×480 resolution as well. And like I also mentioned before, it is rather odd, but the effect pretty much goes away when we set Unity’s timeScale to less than 1.

FWIW, I just built a custom version of Simple Client with NSOpenGLCPSwapInterval off and it is noticeably better displaying this Weiv scene. I still get occasional jitters, but qualitatively speaking it is definitely less noticeable. Now time spent in CGLFlushDrawable averages 200-300 microseconds much more consistently as opposed to regularly jumping up to 3000 or much higher like it did before during the “grinding.”

Lastly, as a control, I enlarged the Simple Server to ~800×600 or so and looked at the vsync-ed Simple Client. I can notice some occasional jitter from that, too, although it’s subtle. It looks more like our scene through the non-sync-ed Simple Client.

And thanks for the final suggestion. I’ll look into it and talk to my partner John (you would know him by AIResearcher here). Maybe I can get that going and see what kind of effect it has on our performance. We do need to display the texture in our “live view” in Unity, but the more duplication we can cut out, the better.

[WeivCo]

Sorry to add another message here. Above I say that I attach OpenGL Profiler to “our app” but that is misleading. I meant to say that I attach it to the Simple Client and our own custom client which also runs as a separate process.

[WeivCo]

Just found this:
http://fedora.cis.cau.edu/~pmolnar/CIS200P11/Pong/OpenGL.framework/Headers/CGLIOSurface.h

Does “changing IOSurface on the CPU” only mean CPU rendering? Cuz I guess there’s a IOSurfaceLock and unlock. Not sure if that would help sync things when it comes to weird issues like this??

EDIT: Oh you seem to already know about it. Quite a frequenter on the Apple lists 😛

[WeivCo]

I’ve done quite a bit more research here, including testing just on my native display with a Simple Client where I know for sure that I’m using just my AMD card. I still get the issue.

I’ve managed to fire up OpenGL Profiler and attach it to our app to trace calls. It renders very fast normally, and only when I get a “grinding” type of effect I notice a 10x increase in the amount of time spent in CGLFlushDrawable, i.e. copying from the back buffer to the front. Based on my research, I wonder if this has to do with mixing single- and double-buffering?

From what I understand using glFlushRenderAPPLE is for single-buffering, and I don’t see a usage of fence, which means Unity -> Syphon server -> Syphon client would mean double -> single -> double buffering.

I found on one post:

Calling flushBuffer more than once within the refresh period should actually block your application (lots of time spent in CGLFlushDrawable appearing in a Shark profile).

If the rendering process it “out of phase” (as you call it in a Nov 2009 quartz-dev thread), maybe a flush is effectively being called too many times in a refresh period?

[WeivCo]

I’m on a Rev A quad core Macbook Pro i7. Two GPUs, AMD/ATI 6750M and the HD 3000. But I’m using gfxCardStatus to force Discrete Only. Earlier Thunderbolt drivers would not allow output at all while using the HD 3000, so I’m assuming that the AMD is driving both. But there was a driver update recently, so I’m not sure if that has changed or not.

I did notice for the clients that they run much better on the primary display. Do you think it’s one of those CVDisplayLink-set-to-primary issues or that this is actually using the HD 3000 for the secondary display? Because that would be unfortunate in either case…

[WeivCo]

Yeah, sorry, I guess I could have added more “in thisFunction” to the output. But my conclusion is the same. It looks pretty normal to me.

The only thing I’m confused about is the “NEW server description” happening twice. I would have thought that the second time it would have updated instead of making a new one again. But there could be a good reason for that, I don’t have that great of an understanding of everything yet. Still, nothing seems to be getting destroyed and rebuilt every couple seconds or anything. So next up is digging through the client I suppose.

[WeivCo]

Here’s a run-through where I load a scene that creates a server, let it sit there for a few seconds, then open a client, then let it sit there a while longer, then quit:

19:35:27.771 Weiv 1.0 Demo[14489:207] creating Syphon Server
19:35:27.778 Weiv 1.0 Demo[14489:207] pimpedVersionForSyphon didn't find an icon key
19:35:27.793 Weiv 1.0 Demo[14489:207] Made a new appImage
19:35:27.793 Weiv 1.0 Demo[14489:207] NEW server description in handleServerAnnounce
19:35:27.794 Weiv 1.0 Demo[14489:207] uuid found in indexOfDescriptionForSyphonServerUUID
19:35:27.794 Weiv 1.0 Demo[14489:207] index not found in handleServerAnnounce, but didChange
19:35:27.798 Weiv 1.0 Demo[14489:207] Sizes didn't match, rebuilding surface from 0.000000, 0.000000 to 2400.000000, 600.000000
19:35:27.799 Weiv 1.0 Demo[14489:207] Destroying IOSurface now
19:35:27.799 Weiv 1.0 Demo[14489:207] Setting up IOSurface
19:35:27.802 Weiv 1.0 Demo[14489:207] Initializing
19:35:27.803 Weiv 1.0 Demo[14489:207] _pushPending, so glFlush-ing...kawooooossshhhh

Attempting to open second display.

Display showing!

19:36:06.871 Weiv 1.0 Demo[14489:207] pimpedVersionForSyphon didn't find an icon key
19:36:06.874 Weiv 1.0 Demo[14489:207] Made a new appImage
19:36:06.874 Weiv 1.0 Demo[14489:207] NEW server description in handleServerAnnounce
19:36:06.874 Weiv 1.0 Demo[14489:207] uuid found in indexOfDescriptionForSyphonServerUUID
19:36:06.874 Weiv 1.0 Demo[14489:207] index found in handleServerAnnounce
19:36:08.569 Weiv 1.0 Demo[14489:1a903] No response to pings
19:36:08.570 Weiv 1.0 Demo[14489:1a903] All servers responded to announce request.

Bluetooth Adapter looking stopped.

19:36:38.760 Weiv 1.0 Demo[14489:1aa03] Draining queue
19:36:38.760 Weiv 1.0 Demo[14489:207] Destroying IOSurface now
19:36:38.760 Weiv 1.0 Demo[14489:207] Deleting texture
19:36:38.761 Weiv 1.0 Demo[14489:207] Releasing surface and context
19:36:38.761 Weiv 1.0 Demo[14489:207] Destroying IOSurface now
19:36:38.762 Weiv 1.0 Demo[14489:207] destroying Syphon Server

[WeivCo]

Thanks guys, good to know more about what’s going on. I’ll keep digging and try to get some more information.

[WeivCo]

All right, well I came out alive jumping into the deep end of XCode without really knowing that or Obj-C, so that’s good 😛

I added NSLog lines in the Unity plug-in when publishing and in Syphon in bindToDrawFrameOfSize and got these first few lines (2400×600 is just the current test res, my problem is occuring in 640×480 as well):

2011-10-11 19:06:25.031 Weiv 1.0 Demo[6692:207] creating Syphon Server
2011-10-11 19:06:25.032 Weiv 1.0 Demo[6692:207] Publishing to texture 2400, 600
2011-10-11 19:06:25.033 Weiv 1.0 Demo[6692:207] Rebuilding surface from 0.000000, 0.000000 to 2400.000000, 600.000000
2011-10-11 19:06:25.064 Weiv 1.0 Demo[6692:207] Publishing to texture 2400, 600

And I keep publishing till the server is destroyed. So the surface is only being rebuilt once it seems, which is good. Though I publish once first before rebuilding?

Yeah that line of code seems to fix some kind of weird UV issue or something. The effect seems to be that one pixel in the image is being sampled and drawn for the entire screen. A couple scenes we had triggered it, one of them when a camera was in a specific place and/or some sprites were being instantiated at a specific moment. Other scenes seemed to have it all the time. Changing to VertexLit rendering was a workaround, but I looked at your example script and reduced the fix down to that one line while maintaining a Unity-friendly OnRenderImage. I’m assuming that’s why the surface is being rebuilt from 0,0?

BTW our destroying happens here:

using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;

public class SyphonServer : MonoBehaviour {

	/* Nothing to be done for initialization */
	/* DLL is programmed to auto-initialize with "Weiv" upon first publish */

	/* Cleanup */
	[DllImport ("SyphonUnityPlugin")]
	private static extern void syphonServerDestroyResources();

	/* To help handle resizing */
	private float mScreenWidth;
	private float mScreenHeight;

	void Start ()
	{
		mScreenWidth = Screen.width;
		mScreenHeight = Screen.height;
	}

	void LateUpdate()
	{
		if ( Application.platform == RuntimePlatform.OSXEditor || Application.platform == RuntimePlatform.OSXPlayer )
		{
			if ( mScreenWidth != Screen.width || mScreenHeight != Screen.height )
			{
				mScreenWidth = Screen.width;
				mScreenHeight = Screen.height;
				syphonServerDestroyResources( );
			}
		}
	}

	void OnApplicationQuit( )
	{
		if ( Application.platform == RuntimePlatform.OSXEditor || Application.platform == RuntimePlatform.OSXPlayer )
		{
			/* Cleanup everything and close the server */
			syphonServerDestroyResources( );
		}
	}

}

[WeivCo]

Thank you gentlemen, I can’t tell you how much I appreciate your suggestions and speedy responses. We are trying to solve this problem during work hours so I definitely am grateful. 🙂 Our app is a Unity app with it’s only output via Syphon, so I guess there’s an incentive for us, heh.

I also want to say that Syphon is awesome, and you guys have done a great job sticking with it. We are also thankful and impressed by the unified support it has received within the VJ software space. I’m glad we’ll be built around Syphon from the get-go!

Anyway, we’ll look into this more and keep the thread updated.

Side note, I also found a couple interesting links while searching around for IOSurface stuff regarding web browsers using it:
http://src.chromium.org/svn/trunk/src/content/browser/renderer_host/accelerated_surface_container_mac.cc
https://bugzilla.mozilla.org/show_bug.cgi?id=598425

[WeivCo]

Thanks Brian, I appreciate your feedback.

I guess I forgot to mention that our app is running windowed at 1024×640 and then we’re using RenderTextures to send frames at various resolutions to our live view and then to Syphon, as the code above demonstrates. But our resolutions settings are saved, so when I launch the app it can load at a set res and it will just stay there. But would it be a problem to have the app’s resolution be different from the one sent to Syphon, with Syphon mistakenly thinking the resolution has changed and the context needs to be recreated every frame? It seems like that would slow things down a lot more than it is right now.

And I forgot to mention what is almost the most important thing! We’ve found an odd pseudo-workaround.

The effect pretty much goes away completely when you change the timeScale below 1. Even just 0.5 makes things look much better.

[WeivCo]

Ok, further data that points to some kind of Syphon serving issue. I figured out how to attach the Unity profiler to our standalone build and went through a build with VSync on and with it off with a client connected and then VSync off without a client where the visual result is good. Most of our time is spent in Device.Present, which seems to mean waiting for the GPU. It runs pretty well, except for the occasional spike, which seems to correspond directly with a “chunking” effect that looks similar to 1-2 dropped frames. And sometimes the spikes are more frequent, resulting in a “grinding” effect.

Outputting to client, VSync ON: http://img846.imageshack.us/img846/3762/screenshot20111011at112o.png

Outputting to client, VSync OFF: http://img684.imageshack.us/img684/2909/screenshot20111011at124.png

Not outputting, VSync OFF: http://img17.imageshack.us/img17/2909/screenshot20111011at124.png

Brian, thanks for the response, yeah same behavior happening in both.

[WeivCo]

We’ve done quite a bit of work in this exact field developing our own VJ software built on top of Unity that explicitly supports ProPresenter.

Our first step was to try to support pre-PP 4.2 by developing a “video input device” Unity plug-in, which in developer-terms is referred to as a video digitizer or VDig. The result was pretty bad. It worked ok only in about 640×480 res, and even then wasn’t that great.

It’s important to note that a VDig uses the “old QuickTime”, which is 32-bit and gets translated to 64, and there doesn’t even seem to be any plan in sight to make a VDig use the new QT.

After making a Unity VDig, Syphon released a first beta, and we knew right away it was worth scrapping a couple weeks of work to use it instead. So now, to plug in to ProPresenter, we just use the Syphon Quartz client with PP 4.2+ and it works fine.

[WeivCo]

Bah, I’m stupid…it wasn’t installed at all! I recently upgraded laptops and forgot to install the plug-in on this one.

Thank you for the help.

[WeivCo]

I just found this: http://forums.v002.info/topic.php?id=53

That seems to confirm my issue. I’m using Transparent/Diffuse, which is bad I guess. I switched over to Transparent/VertexLit for now, which does the job. If I need something more it looks like Brian’s shaders in the repos will work.

[Author]

Posts