Presumably you are talking about Syphon Recorder..? File size shouldn’t be a limitation.
1. Make sure you’re using the latest version from http://syphon.v002.info/recorder/
2. Post a picture of your Preferences window so I can recreate the settings you’re using
3. Post a picture or the exact text of the error message you get