these are my specs: https://www.asus.com/Notebooks/N551JX/specifications/ (i got the i7 version with ssd 8gb ram)
That CPU is pretty darn impressive for a laptop. I recommend using
Task Manager or
Process Explorer to see how much CPU time you use when playing COH2. If you have around 50% CPU time free, definitely look at using x264 (straight CPU encoding) instead of the nVidia one. Your CPU outranks your video card, I'd say.
Since that CPU also has Intel HD Graphics 4600 built in, you may find it useful to stream with Quick Sync since that will (theoretically) use hardware that is otherwise not being used when you game. It is similar to NVENC in that it prioritises speed over video quality though, so I wouldn't use it for Youtube unless desperate.
In terms of settings (mostly relevant to x264):
Encoding:
- Max Bitrate: 3500 (including however much you allocate to audio). This is mostly a Twitch streaming limitation. People with poor upload speeds (generally Australians) need to have more powerful CPUs to improve the low image quality of lower bitrates.
Go up to 10000 for Youtube recordings. The higher the bitrate, the less effort your CPU has to put in to make each frame look nice and sharp.
- Use CBR. Twitch requirement.
Video:
- Resolution Downscale: Downscale to 1280x720. Definitely consider going lower if your CPU is struggling. A sharper small image is better than a blurry large image.
- Filter: Start with Bilinear. Try increasing it if your performance is ok but the image isn't sharp enough.
- Stick to 30 FPS. Don't experiment with 60 FPS unless you get everything else working fine first.
Advanced:
- x264 CPU Preset: This is the main setting you will be adjusting. Faster settings will ease the load on your CPU, improving performance but at the cost of image quality.
Start with a 'fast' setting, and if Task Manager says your CPU usage isn't peaking above 80-90%, try adjusting it another step higher until you are.
Changing the above settings, such as resolution, *will* change what setting you use here.
Edit: Cleaned up a few sections after some restorative sleep