Profile of Grumpy

I would skip on the first "free" (250mp/35 totally worth for tech that allows you to build one unit only Kappa) squad any day if I would get weapon racks, nades and snares all at once from it instead

Everybody who plays USF would do this if they could, but I think the balance team said it was off-limits as "free" officers are part of the core of the faction.

We do throw out outlying data as well, but really, even 3-0 are not that unusual and 3-1s should be as common as 3-2s if the outcome of a match is totally random (perfect match of player skills and everything is totally balanced). Sounds strange?

Ok, after one match the result will be 1-0 or 0-1 with a 50:50 split. After two matches the options are 2-0, 1-1, 1-1 and 0-2. After three matches, you have 3-0, 2-1, 1-2, 2-1, 2-1, 1-2, 1-2 and 0-3, so 2 out of 8 or 25% of matches will have ended in either 3-0 or 0-3 (last post I correctly stated that the chance of 3-0 is 12.5%, but forgot that the 0-3 would go in the same bucket...).

So, the remaining 75% of rounds will be at 2-1 or 1-2 after three rounds. Again, if the chance of winning is 50:50 for either player, we will have a 50:50 split between 3-1/1-3 or 2:2 results (which would end up in 3:2/2:3, obviously).

As a result, if match outcomes are completely random, 3-0/0-3 are expected in 25% of rounds, 3-1/1-3 in 37.5% and 3-2/2-3 in the final 37.5%. This was kind of unexpected for me, but the thing is that we do not actually play 5 games. If one player has three wins, the series stops. I guess we tend to sort of extrapolate that after 5 games the 3-0 would certainly end up in a 5-0, but potentially the other player could have won games 4 and 5.

If we look at the actual results, we see that we have 20 3-0/0-3 (which obviously are way too many to say that these where even matches), 7 3-1/1-3 and 5 3-2/2-3 so the 3-1/1-3 are not much of an outlier, really. Actually, if the player skills would be similar and all rounds end in 3:2s, you can be pretty certain that something else was unbalanced...



Sounds reasonable, but if you break it down that much you would end up with a small single digit number of matches per case. Match length seems like a good indicator, but on the other hand you have cases where a player throws the towel early because he lost a crucial unit (we even saw this in the last two rounds). Still, if you are interested, I could give you this data. And yeah, I guess statiticans would probably tell us to "Get more data!".



Well, seems like we overall agree then

Problems with automatch stats are, well: First up, I think there is a considerable skill gap even in the top 50 of players; I didn't check but I wouldn't be surprised if all players that made it to the main round are in the top 100 with at least one faction. And back when I did the post on the automatch stats, I actually looked into what I get when using only games of the top players; you quickly get to the point that you are left with very few matches. I don't recall the exact numbers (might be like 100 matches across all factions, a larger map pool (with uncertain starting positions) for the top 50 or so over three months) but there were so few that I dropped the idea of putting that in the article.

Another issue was pointed out by Momo4Sho in a similar discussion: Top players will often try out new strategies in automatch games, so they might not actually play to win. This is different in tournament games.

So, at the end of the day I guess this is actually close to my work enviroment where decisions have to be made based on multiple different pieces of information (which here could be player feedback, tournament stats, automatch stats...), none of which are conclusive by themselve but together hopefully form a somewhat consistent picture.

Overall we agree
The statement that I changed to bold was one of the points I tried making to Doomlord. If the factions were badly unbalanced, there would be a lot of 3/2 splits.

At this point, I still think it would be more helpful to interview tournament participants, especially the upper players, to find out why they're not picking Ost, rather than trying to do any more statistical analysis. Here is an analogy from my work. I was once asked to look into a problem on a paper machine. When the paper machine has the problem, the paper gets sent to a rework bin. That is really expensive. If it is happening a lot, the workers often slow the paper machine down, which is also expensive. I was asked to check into what is less expensive. I got a really surprising result. When I normalized for speed, slowing down didn't improve the defect rate at all, contrary to what people were saying. When I asked some of the workers about it, the general reply was "Ya, we already know that but it means we have to get up and fix it less often."

People will optimize what is best for them, regardless of what statistics say, and that is especially true for the small group of elite COH2 players. Some of the players switched from OST to OKW. Why? Does the heavy tank meta make OST a worse choice? Something else?

I also agree on your comments about the skill gap between players, but think it can extend a long way down the ladder. This has an effect that a lot of people don't realize when they are complaining about "balance." If someone's ranking is XXX, even if there is a balance patch, the XXX-1 players that are better than them are still going to be better, and the players that were beating them before are still likely to beat them, so their ranking and win rates aren't going to change much.

It would be interesting to look at both the tournament and automatch data sets sometime, if you can share. If you're not supposed to share it then no worries.

So, who wants to setup a "balance" tournament: an obnoxiously long series where top-ranking players need to play a TON of games against each-other as every faction (and combination), solely for the purpose of getting win:loss statistics? I'd suggest a similar format WCS, except each player matchup needs to consist of a "best of 5" of every faction combination (5x USF vs. Ost, 5x USF vs OKW, 5x etc....).


Really though, considering the player base of CoH2, getting a complete statistical breakdown of balance that's as accurate as it needs to be is very unlikely. If the game had daily concurrent player peaks over 100k it would be possible, since top-level play would actually consist of only top players. Unfortunately, even with the free give-away a little while ago, its only reached 22k peak concurrent players, and that average is now down to around 7k (based on steamcharts).

Even so, I still think the WCS is a better source of balance data than Automatch. Players in the WCS were all in the top 150 for 1v1, whereas in automatch it's not uncommon for top 100 players to be matched against top 300-500 players, since that's the closest available.

Sure, but thow out UKF because nobody is going to play it anyway. Then instead of having the #1 seed play the #32 seed in the first round, have the #1 seed play #2, #3 play #4, etc.

The early rounds in the WCS were usually mismatches because they were supposed to be mismatches (and the selection team did a good job). The finals were 3/2 splits because they really were the few elite players. They were the #1, #2, #3, and #5 seeds.

Yeah, pick rates are at least a relatively solid measure of the player's perceived faction performance (distinct from the actual faction performance), so in itself those are already a pretty good measure of faction balance (within the parameters of the tournament).

And I certainly wouldn't say that you can "prove" anything with these numbers. Hence my formulation that the numbers at best could indicate a trend. I figured this would be vague enough, no? Thing is, this is kind of the best shot we get. We certainly also can check out automatch stats (and I did so in the past), but these come with there own set of problems. These can all be pieces in a puzzle.

Ok, thinking about this the other way around: Assuming that a certain faction IS actually OP. This certainly will have an effect on the win percentages, right? Now, as we probably all agree, there are soo many other factors that contribute to these numbers which will skew the results in one way or the other. So, at the end of the day you'd have to make a judgement on how much of the resulting number is due to the OPness of a faction and how much is due to other factors. If you think that all of the other factors have a much bigger impact on the numbers, fair enough. But to some extent the actual faction balance will be buried in there...

Btw, I was wondering about your 3-0 and 3-1 issue: True, the 3:0 tell you little about details of faction balance as those matches more likely were mostly decided by the discrepancy in skill level rather than faction performance.

Sidenote on 3-0 rounds:
Show Spoiler

But the 3-1: Well, actually I would consider these very interesting points. Clearly, one of the players was good enough to realistically beat the other as he showed in the one match he won. We still don't know whether it was faction performance, the map or luck (as we never do with these kind of stats), but I wouldn't know why you would consider this "datum" worth less than a 3:2 round or even an outlier.

Some disclaimer here: I'm a reliability engineer so I analyze industrial processes instead of any type of sport. Normally I have access to lots of historical data, which is good because I'm normally looking for small differences. When you look for small differences, you have to throw out results that are far from the mean (or sometimes median). At work, I would throw out the 3-1's because they're far from the mean and I can always get more data. Here, it isn't that simple and I'm sure if you ran a bunch of tests with similarly skilled players, you would probably get some 3-1 results. On the whole, it still seems like the effect of the player is *probably* more than the effect of the faction in a 3-1 match. To make a more informed decision, I'd need faction, map, pick, and match length at a minimum. Match length would be especially helpful because equal players usually seem to battle longer.

Today's data scientists know more techniques and might be able to get something more out of it than I could, but I think even they couldn't get around the small observation size and multiple independent factors. If the balance was horrible, the small data set wouldn't be such an issue.

Also, I wasn't trying to say that OST definitely isn't UP, just that there isn't enough data to say it with such certainty. Doomlord might be right but this tournament isn't a good proof either way.

stuff

Now its a mistake and not a shout-box worthy lie? Okay.

Redo your bombing run tests with a PIV with the bombing run parallel to the axis of the tank like people would use it in game. Then come back here and post the best and worst results, instead of just saying that it does 960 damage (or whatever number you claimed). The on-axis results that I got were usually in the 160-320 range, which isn't good for 180 munitions. You had to have ran the tests a bunch of times and know when it doesn't work well, so, IMO, you were misleading when you said it's great. It would be like taking a couple of UKF or OST victories from the WCS tournament and then saying there was nothing wrong with the faction, because somebody won with those.

Then run that same test with the strafe from the new airborne doctrine that costs about the same.