H264 takes too long to encode to be well-suited to live recording – the frame-rate is generally terrible, and your CPU usage would rocket. GPU support is for H264 decoding only, when it is present at all.
There is untested support in Syphon Recorder for Matrox hardware encoding to H264 and MPEG 2. As it’s untested I can’t report on its performance, but if you can lay your hands on a Matrox encoder, you could let us know.
We’re always open to adding other codecs to the supported list – if there’s an Apple or third-party codec you’d like us to test then name it and we can see how it performs. Anything with a lower data-rate would be a good addition to the list, for sure.
Sounds like a lot of video!