This could also be an issue simply with :
a) Fill Rate – you may be drawing way too much. GPUs can only fill so many triangles per second. Both Unity and VDMX can easily swamp a GPU with draw calls and fill rate issues on their own. Going full screen 100% effects fill rate, as you are painting a larger portion of the screen.
b) VRam – you may be using more VRam and hitting swap. VRam in OS X is virtual, so you can allocate textures all day long, but once you get over a hardware limit, you will use software (main memory) and things will drastically slow down. Going full screen changes back buffer sizes which, if you are near VRam max, can push you over the edge to swap land.
You can check for both with OpenGL Driver monitor. Look for your video memory and see if it gets too high. For fill rate, I think its “GPU core” usage, but CPU wait for GPU is also a good indicator of rendering latency.
Does changing the unity scene to something simpler make a difference? Does VDMX with a single movie the same size, no effects, one layer, cause the issue? Basically I am curious if you are abusing your system, or if this is a genuine weird issue. Not saying there is not an issue, but this info will help us trouble shoot the issue.