People keep asking me for my thoughts on the changes Valve implemented to the rating system, so in lieu of rehashing it every time I’ve opted to go over it here.
I’ll admit that most of this comes from my perspective as a participant in ranked matchmaking, but will touch on why this has produced a headache for your average Dota league admin. Note that it’s still fairly early on, and that makes it tough to judge the system’s merits in the long term; additionally, I’m liable to guess my way through some of the things I mention, and am fairly likely to be wrong on some points. The overall takeaways aren’t massively impacted by this.
Alright, so what happened?
With the New Frontiers update, Dota moved away from using the Elo rating system, and instead operates using Glicko instead. I’ll save you my attempts at paraphrasing the technical side of things to highlight the differences, and focus instead on major mechanics and outcomes that impact our experiences.
If you’ve been playing Dota long enough, you’ll recall a time when MMR gains and losses weren’t static; depending on what the matchmaker would whip up, you could occasionally win (or lose) single digit amounts, or conversely, amounts far above the average of ~25 MMR (back then, that is). This became a static +/-30 (20 in party queue), but with the new patch is again variable — the major difference being that, alongside the MMR of everyone in a given lobby, the gain/loss is also determined partially by the rank confidence of the players, which is a new metric.
With the patch, everyone’s MMR got reset — or, rather, everyone became uncalibrated. In order to regain their MMR, they’d need to play enough games for their rank confidence to surpass 30%. People started at different amounts when the patch dropped, mainly based on how much they played before it — some started off at 0%, and have had to painstakingly string together 20 to 30 ranked games to get that number to 30%, while others started at 30% and simply had to play one match to get their placement.
In this recalibration process, players (particularly those with low confidence) are placed into games with varying averages, and their MMR changes become very high in this period — I’m talking +/-100s, even more sometimes, though this then gradually decreases as more games are played.
Key takeaway #1 pops up at this stage: over- or under-performing in these early games with low confidence rating can massively impact someone’s final MMR.
Now, one major boon of having variable rank change is that, in theory, people who are placed in a bracket they don’t belong in will get where they need to be much faster than before, as they’re not stuck gaining (or losing) a static 30 MMR per game. This isn’t currently panning out quite that way in practice however, which I’ll get to in a bit.
Before that, let’s go back to Valve’s motivation for making this change. One thing they emphasized is how, over time, a cluster of players has established in the Herald bracket. The chief reason for this is fairly straightforward: a player with less than 30 MMR cannot lose more (ie, go below 1, or previously 10, as the fourth digit of MMR was rounded), but the opposing team nevertheless gets their MMR. The graphs pre and post patch show that their mission to get rid of this clump has been successful; the MMR distribution curve is now bell-shaped, and there’s no massive chunk of it in the Herald bracket.
This brings us to key takeaway #2: in lower MMR brackets in particular, players have been shifted up in MMR by a LOT. Any conceptual understanding we may have had as to what a 500 MMR player is, versus a 1k, versus a 2k, etc, is completely meaningless now — and this extends into every bracket, though is most egregious in lower brackets. Without getting too into it, this has ramifications on a number of league logistics. How do standins work now? How do you approach MMR adjustments, if at all (more on this later)? How do you weigh the new MMRs for purposes of coin allocation of captains?
This also leads into one of the most puzzling parts of the change. Why did the people who got MMR gains do so? Why did others not? Hell, why do some lose MMR?
It’s a conundrum that leads to what is, in my opinion, the most problematic bit for the pubs themselves, and it’s key takeaway #3: how do you compare players who are now the same MMR, despite being thousands apart pre patch? You sorta don’t.
Consider the scenario that you have three players. Player X was 4.6k prepatch, player Y 5.6k, player Z 6.6k. After the patch, all three calibrate and land at 5.6k. Who’s representative of what a 5.6k player is? Are we accepting that all three are on equal footing now, despite there being a 2000 MMR gap between players X and Z prepatch?
This brings us to key takeaway #4: any doubts surrounding this kind of issue are resolved by the fact that dynamic rank change will inevitably get whichever player should be higher or lower into their real bracket. Except, that’s not the case.
Once you hit the magical 30% rank confidence barrier, your rank change per game will drop to an average of +-40, very very rarely going above that, and then gradually decreasing with every single pub you play as it increases your rank confidence. As your rank confidence becomes very high, you’ll see sub 20 MMR +- matches, albeit irregularly. Suddenly, this system is worse than the prior one at getting someone to their correct MMR. The harsh drop-off in the change that happens at 30% confidence is staggering, and is in my view a pretty significant misstep.
You’ve aired your grievances — now what?
Imagine now you’re tasked with running a league event.
We can’t accept anyone not calibrating their MMR — we have no clue whatsoever what to estimate someone’s MMR at, because they may well end up thousands of MMR above where they used to be, and the players you might use as points of comparison for skill level against any given uncalibrated player are scattered across three different MMR brackets.
You cross that bridge, and are supposed to adjust people’s MMRs. Doing anything with the new ratings is utterly pointless; do you instead work off of prepatch MMR? Extend that to standins instead. Yes, we can tackle everything case by case, but lord knows a massive chunk of standins become necessary about 3 minutes after a game was supposed to start.
Frankly, the bottom line is essentially that there isn’t one. When season 8 starts, we’ll do our best to provide people with the best amount of info we can, but past that, it’s a bit of a lottery — as is, more or less, every part of the new system.
Fun times lay ahead.
