Rename "quick-play" to "ranked" and put a disclaimer in it, saying that it's not a casual mode (maybe even direct people to custom matches). Then restrict the skill-range in that mode to a reasonable number, to prevent incredibly one-sided matches.
This solves basically every problem in the current match-making system.
That said, I doubt this will actually happen, since it requires UI and MM code changes, which seem to be 'out of scope' these days. Hopefully "CoH3" (if that's ever a thing) will implement a proper competitive system similar to other games.
You are just alienating the playerbase. It's better for matchmaking to have more people searching. If you try to enforce too many strict rules, less people is gonna play.
Ranked is never the most popular mode. I'm playing HoTs (Blizz MOBA) which has probably dozens or hundreds of times the playerbase of CoH. And each time they try to implement this type of "solutions" to limit and be more strict, they end up screwing it more and more.
I'm not saying that no kind of rule could be implemented but nowhere near close to what other more popular games have.
If i were to set rules i would do this:
1- A loss is a loss. You don't like to lose due to quitters, get a partner.
2- Simplify rankings. AT partners who search outside of the number of people in the group counts towards random. Ex: 2 people searching in 3v3 counts towards random 3v3. Group of 3 in 4v4 as 4v4. This is both for ladder and matchmaking.
3- First quit/drop from a game in the first 20mins has no penalties. Further games in a 24hs puts you on a 1 hour cooldown for automatch outside of 1v1 and complete groups. If you have internet issues/bugsplat, you might as well not play till it's fixed on that hour.