First off, recent versions of VDMX have a Movie Recorder plugin which will record inside VDMX, saving the need for the Syphon stage. Performance is good.
If you’d still like to use Syphon Recorder for whatever reason and it’s not working for you, please post a photo of the Preferences window inside Syphon Recorder showing the settings you are attempting to use.