Nothing else need be installed. If you have the Syphon checkbox marked in Avenue like this, and nothing is happening, try posting to the Resolume forum for help.
The VDMX record function should work. If you’re having problems with that use the built-in bug-reporter in VDMX (Help > Report Bug).
In Modul8, use Output > Syphon (or command-Y).