Announcement

Collapse
No announcement yet.

World War Matchmaking - Design Spotlight 2019

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • World War Matchmaking - Design Spotlight 2019

    Hello DomiNations fans! Joe “Muet” Grubb here with Big Huge Games. This spotlight covers a long awaited change to DomiNations: World War Matchmaking.

    Why Change Matchmaking? What are the Problems We’re Trying to Solve?

    The high level goal is twofold: Improve the competitive integrity of World War and improve the match quality.

    Bear with me. I’m trying something new with this Spotlight; we’re going to go deep. We’re going to talk about how the old system worked and how the new system is attempting to solve problems. If that’s not appealing, there’s a tl;dr “what are the changes?” if you skip to the end. It’ll be clearly marked and I won’t fault you for skipping forward, after all you’ve got a Nation to run and World Wars to fight.

    Didn’t skip forward? You sure? Cool. Let’s take a look at the problems in the current system and break down each of them:
    1. Method for comparing the Offensive and Defensive Ratings of a base
    2. Sandbagging/Imbalance in Alliance compositions
    3. The Leaderboard was not part of the evaluation
    4. Queue functionality
    5. Is this match fair to my Alliance?
    6. Sparse queues
    Method for Comparing the Offensive and Defensive Ratings of a Base

    Did you know that the matchmaking system creates a Rating of your base’s offensive and defensive capabilities? It does. Everything on your base contributes to these Ratings. However, the method that the old system (the current matching system on live) used to compare the final Ratings with other bases wasn’t a great predictor of win rate. There was some directional value in its ability to predict win rate. But, we knew this was an area where we could improve match quality.

    That old system compared the absolute value of your Alliance’s Offensive/Defensive (O/D) Ratings with other potential Alliances. We ran some tests internally and discovered that when we match based on a percentage delta between those same Alliance’s O/D Ratings, we actually get a much stronger predictor for winning. So, we’re updating the system to compare Ratings by the percentage delta rather than absolute values.

    Fun fact: If your Alliance has a ~10% higher Defensive Rating in the new system, your Alliance’s chances of winning are increased to 60% (Yes, defense matters. Start upgrading your neglected Catapults!).
    Additional fun fact: The new system would not consider a 10% delta a “good” match. So don’t worry.

    Sandbagging/Imbalance in Alliance Compositions

    As with other team-based games, there is an ability to “carry” in DomiNations. If we only checked the percentage delta between two Alliance’s O/D Ratings, we may end up making the following match:
    Alliance A: 5 Cold War, 5 Medieval
    Alliance B: 1 Cold War with a mix of Enlightenment+.
    Seem familiar? I’m sure all of us have been in Alliance B’s position before. The O/D Ratings may look fair, but this is not a fair fight; Alliance B is going to lose this War.

    We don’t want to limit Alliances to bringing specific compositions to War. Sometimes you just don’t have enough players at close Ages. But we do want to address the sandbagging issue. So, we’re adding a second comparison to the Ratings formula. The new system now also compares the delta within an Alliance and tries to find other Alliances with similar deltas. The result? The system now knows that Alliance A and B are wildly inappropriate to match. It’ll find Alliance A another Alliance with a large Ratings delta in its players while protecting B from an unfair fight.

    Ready for another fun fact?: If your Alliance is sandbagging in the new system, it may result in increased queue times as the system waits to find another sandbagging Alliance for you.

    The Leaderboard was not Part of the Evaluation

    The old system did not use Glory as part of its calculation. In order to preserve the competitive integrity of our Leaderboards, we felt that it was important to start using Glory. In order to do that, Glory gained/lost needed to be updated.

    We had several discussions about potential directions to take Glory and what systems to use.
    A system like Elo (initially popularized as a chess ranking system) provides a single rating that can be used to predict the winner between opponents. Since Glory is serving as our Leaderboards, using a system that would suggest the higher ranking Alliances are more likely to win made sense. Afterall, if you’re Rank 1 on the Leaderboards, you should have earned it. Elo also does some other great things for Glory like awarding you more Glory if you happen to beat an Alliance that is higher ranked than you. That was a tough War and you deserve it.

    Elo is somewhat of a barebones rating system, however. Even chess has been modifying parts of the formula based on player ratings. So, we also considered more robust systems like Glicko-2, a more complicated evolution on the Elo formula that introduces a confidence interval. This confidence interval is a way for the system to determine how accurate it thinks an Alliance’s rating is. The interval can then modify points gained/lost in the system based on this confidence. For example, if your Alliance is 20-0 in War and the system says you have a rating of 18,000 Glory, what are the odds that your Alliance is probably better than 18,000? Probably high. Glicko-2 would take this into account and likely accelerate your subsequent Glory gained.

    While more complex systems may have offered some extra bells and whistles, we ultimately decided that we could get most of what was important to DomiNations by using a modified version of Elo we’re calling DOM MMR (DomiNations Matchmaking Rating). It’s pretty close to standard Elo with some targeted modifications to K-factor and streaking.

    It’s a bit of an oversimplification, but you can think of K-factor as a constant in the Elo formula that sets a maximum on the number of points that can be won or lost from a given match. There’s a handful of problems with this value being constant. Even chess has been modifying the K-factor of players based on their rating.

    So what modifications are we making? DOM MMR will dynamically change an Alliance’s K-factor based on the following:
    1. New Alliance - New combatants in any ranking system are hard to properly evaluate. It is near impossible to properly evaluate a new Alliance with only 2 Wars compared to an Alliance that has 300 under their belt. Therefore, new Alliances will have a higher K-factor for their first several Wars. The result should help get sharks out of the kiddie pool and help new Alliances find a win faster.
    2. Streaking - Winning or losing several Wars in a row will also cause the system to modify your K-factor. Sometimes you’re just not as good as you used to be, you know? Or maybe your Alliance recruited some top-tier players and it’s time to step into the big leagues.
    3. High Glory Alliances - K-factor modifications sound great but K-factor can break down at the extreme ends of a ranking system. Therefore, Alliances with particularly high Glory will have their K-factor treated differently. The main difference is that streaking won’t apply here.
    Now that we’re modifying how Glory works, we’re going to need to crunch the Leaderboards back toward 12,000 Glory. When the new matchmaking system goes live, all Alliances will be pulled toward 12,000 Glory (the strength of the pull being relative to your Alliance’s current Glory). This shouldn’t affect standings much. But, it will bring everyone closer together in Glory for a fresh start.

    Think of it as a season reset.

    Queue Functionality

    The old queue would search the list of available Alliances at the moment your Alliance entered queue. If you were not matched, you were then put in the bucket for other Alliances to potentially pair with when they entered queue.

    In theory this system sounds fine. And it worked okay for the most part. But, this created a rigid queue. Your potential pool of matches never expanded except when other Alliances entered the queue, regardless of how long you waited.

    This is changing in the new system to better incorporate industry best practices of time vs. match quality. The result is a system that is constantly evaluating the entire pool of Alliances against each other. The longer you wait, the less restrictive the system will be.

    Is this Match Fair to My Alliance?

    Some of you that are Alliance leaders may have noticed that your Alliance would sometimes instantly get matched with what appeared to be a poor match. What gives? Why? To the forums! The matchmaking is awful! Well, not quite. This was because the old system didn’t have a concept of handshaking. It compared Ratings at the time your Alliance entered queue and if they were within an acceptable range of another Alliance, bam. Matched.

    Moving forward, the new system will require the match to be acceptable from the perspective of both Alliances. This isn’t an in-game option or agreement that is made. Instead, you can think of it as the system viewing the potential match from both Alliance’s perspective to see if it thinks it is fair.

    What is fair to my Alliance after being in queue for only 10 seconds may be different from what another Alliance is looking for after 5 minutes.

    Sparse Queues

    I know that this will be one of the least popular changes. I apologize in advance for killing your Alliance’s favorite queue. The reality is that DomiNations can’t—and has never truly been able to—support seven different queues. Think of it this way: Each queue added to DomiNations splits the player base. Each of those splits greatly reduces the number of potential matches your Alliance can make and reduces the quality of those matches. Theoretically, the most optimal system would be to force everyone into a single XvX queue. We can’t really do that and aren’t entertaining the idea. But four is better than seven.

    We’re reducing the queues to the following:
    10v10
    20v20
    30v30
    50v50

    We feel that this division of queues will flow more Alliances into more pools and increase the odds of finding your Alliance a better match. It should be noted that 50v50 is not and has never been a popular queue. But, we felt it was important to preserve 50v50 for community events like the Continental World Cup (https://continentalwar.wordpress.com/). If you find your Alliance jumping into 50v50 expect long wait times and extremely relaxed matching requirements.

    Conclusion (“What are the Changes?” tl;dr is here!)

    In short, this is a complete overhaul to World War matchmaking. Standard PvP battle matching is not being touched. The changes are as follows:
    1. The matching logic has changed how it compares base power between Alliances. The new logic is proving to be a strong predictor of win rate and should allow us to make more fair Wars.
    2. The matching logic now understands imbalances within an Alliance’s composition. It understands “sandbagging” and will try to find Alliances opponents with similar compositions.
    3. How Glory is calculated has been redesigned.
    4. All Alliances are being pulled back toward 12,000 Glory when the update goes live. The pull is relative based on your Alliance’s Glory and it shouldn’t affect the standings much. Think of it as a season reset.
    5. Reduced the number of queues from seven to four:
      1. 10v10, 20v20, 30v30 and 50v50
    6. The behind-the-scenes functionality of the queue itself is changing to constantly evaluate the entire pool of Alliances in queue.
    7. The longer you wait in the queue, the more relaxed your matching requirements become.
    Finally, the entire system has been built with iteration in mind. Every piece of the system I explained has a knob that can be turned. We don’t expect the system to be perfect when it first launches but we’ll be actively studying it and tweaking behind the scenes to improve the quality of your Wars. We built these changes based on a lot of data from our internal systems and tools, but a major factor comes through player comments and discussions from all of our community pages. We'd love to get more feedback on this blog and the upcoming changes, so let us know what you think!

    Thanks for reading!
    - Muet
    Follow Us!
    Facebook: https://www.facebook.com/thebighugegames/
    Twitter: @BigHugeGames

  • #2
    Looking forward to seeing this new matchmaking system in action
    If you are looking for an alliance come check us out at Dominion. We currently have 5 alliances: Dominion Elite, Dominion Títans, Dominion Empire, Dominion Legion, and Dominion Force. Over 200+ members all connected by the required LINE app. Add knattynate and message me to be added to our groups.

    Comment


    • #3
      Bin reading the notes and looks like a good start. Do think the new system needs some time to settle how ever..
      Join us at Outcasts United. All ages welcome.

      Comment


      • #4
        Now if you could just give a preview of what are your plans towards getting rid of hackers with unlimited citizens, that would be great.
        *************************************************

        Lord Stark leader of Dragonstone 1.0

        Comment


        • #5
          BHG_Muet
          We keep our fingers crossed that this will introduce fairer matching. But in all honesty WW is completely broken until you address the cheating.
          Would you not agree having 5 teams out of the top 10 are Cheat Alliances is rather embarrassing ?
          Restoring the integrity of the Glory system can only be achieved by culling these teams that have been endlessly reported by the community. #1 has been using hacks for over a year.
          Pink Unicorn alliances popping up with every player level 332 at every turn. What is the point of a Glory System if you can lose two months gains overnight against these scum ?
          I believe in a few weeks time the Top 10 will be completely filled with Garuda Sakti, Top Killers,Romania Stars,Swift/Fatality,Ardor TW and the other tossers.

          So please if WW matchmaking work is concluded direct your attention to sorting out the cheats once and for all.

          One further question..will the new system allow CS to modify an alliances glory if it has been matched with a cheating team ?

          PS, But first fix the AR, it must be killing your revenue stream currently,as nobody in their right mind spending on the game while it is so hopelessly bodged
          Last edited by Omegaman; 04-05-2019, 07:54 AM.

          Comment


          • #6
            Will the glory payout calculation be fixed/tweaked in the new system? In the current system, you could face a reboot of a top 10 alliance and lose 1000 glory simply because you have more glory than they do. That, along with cheating alliances not being punished, are the main issues with the current system IMO.

            Comment


            • #7
              So I’ll ask the obvious question.... how do you define a sandbag in the new matchmaking system?

              Comment


              • LordStark263AC
                LordStark263AC commented
                Editing a comment
                I'm guessing anyone below Medieval Age?

              • wrathchild_78
                wrathchild_78 commented
                Editing a comment
                So, we’re adding a second comparison to the Ratings formula. The new system now also compares the delta within an Alliance and tries to find other Alliances with similar deltas. The result? The system now knows that Alliance A and B are wildly inappropriate to match. It’ll find Alliance A another Alliance with a large Ratings delta in its players while protecting B from an unfair fight.

              • BHG_Muet
                BHG_Muet commented
                Editing a comment
                It looks at all members the Alliance is bringing to the War and does a calculation to determine the delta in their Offensive and Defensive Ratings to create a Variance Rating. The matching system then uses that Variance Rating as an additional window to compare against other Alliances and tries to find one with a similar rating.

            • #8
              Thanks for the information and thanks for trying to improve the game. Please test this update before releasing that’s all Im asking. Looking forward to see fair wars.

              Comment


              • #9
                Sounds really good and thought out! Let's see how these changes perform. One thing though:

                7. The longer you wait in the queue, the more relaxed your matching requirements become.
                This was in my opinion a problem in the old system. A Leader searching for a good match had to constantly quit and restart the search. This leads to less alliances in the pool, reducing match quality. How will this practice work in the new system?
                Wouldn't it be a good idea to reduce preparation day to 22h and adding 1h minimum search time to accumulate a bigger pool for the system to choose from? Maybe even match only every full hour?




                Comment


                • #10
                  If pourcentage delta is the the same as standard deviation, it will finally be the end of what is called sandbagging. I calculated standard deviation basically from the start of world wars and at 99% accuracy, the highest number always win. At least it matches same level players so that skill rise to the top.

                  Standard deviation is basically the difference between your tops, your bottoms, compare to your overall average. To understand, if you have 10 max SA player and 10 iron age, you would have the highest std dev, and you could have been matched with 20 Global players, which would be the worst std dev (all players same level). Standard deviation is the greatest indicator of win/loss. Matching std dev will result in close exciting matches for all.

                  Elo, or glory points will not be a good indicator on the other hand. An alliance with 10 crazy good Atomic players who never lose, can't be matched by rank with 10 SA players. The leaderboard will be a race to who can recruit more max SA players. It should be about skill compared to your level. If you want to separate leaderboards into a Spage age leaderboard, an Atomic leaderboard, etc... where your top player is of that level, that's fine, but you can't put SA and AA in competition just because they're both high on the leaderboard.

                  Comment


                  • #11
                    I read your long article well and look forward to the change of matching system in the future.

                    I wrote comments in your last article, but I don't know if you remember.
                    Let me introduce myself again. I am a user who is currently using ID SuLBiN, the leader of 3rd place ARES.

                    1. The point about your matching system ELO is very interesting.
                    I have been actively suggesting to the management and development team the introduction of the ELO system.

                    Your example tells of a alliance that is defeated helplessly by too wide a gap.
                    But you know what?
                    It is also very boring and annoying for the upper alliance.
                    We want to have a better World War.
                    But in the existing system, it's not easy to do that quality World War. (I've been waiting for more than eight hours now. LoL)
                    So I like your debate very much.

                    But there's one thing you're overlooking.
                    Did you know that there are two cheater guilds in the top 10 in honor?
                    How many are there in the top 100?
                    Users ask countless report, but they're not being processed.
                    All the top alliance were damaged.
                    But you don't listen to that question at all.
                    The first time I reported them was last October and it didn't work.
                    It's meaningless to change the match system in this situation.

                    For a fair game, such bugs, cheater alliance, need active sanctions against alliances beyond individuals.
                    Can you do that? This is a very important matter.
                    You need to understand the helplessness of normal alliances.
                    Such fairness is essential to the introduction of the ELO systems.


                    2. Do you know why the top-ranked alliance put in the sandbag?
                    In the current system, matching is determined by level, so without a sandbag, matching is not formed.
                    So I have to play a game with a sandbag.
                    But you don't have a definite answer to that in your words.
                    You're just saying it's taking a long time.
                    I want you to be clear.

                    Does the new system allow you to match without sandbags?
                    Or are you still going to need a sandbag for the level?

                    If it's the same as it is now, the new system you're talking about is useless.


                    3. I heard that you will reset your glory score to 12000. So, are you ready to compensate for the glory score you have earned?
                    We simply lose what we've built up by changing your matching system.
                    Are you preparing to make amends for it?

                    I want to introduce a league system if this matching system changes.
                    I hope there's a reward for that when the league is over.
                    The current World War does not have a goal.
                    The reward is too salty.
                    Users need something to focus on.

                    I'd like to say more, but I'd like to make sure I'm asking these three questions.
                    Don't selectively reply to either one, but think and answer clearly as a whole.
                    Last edited by sulbin; 04-05-2019, 10:12 AM.

                    Comment


                    • BHG_Muet
                      BHG_Muet commented
                      Editing a comment
                      Hi sulban. See my post below regarding the topic of cheating. For your other questions:

                      One of the pitfalls of the old system is that it didn’t take time in queue into account. Your Alliance could have been waiting for six hours and it wouldn’t have mattered for how it was attempting to match you. The new system is a dance between time in queue and match quality. As your Alliance waits longer in queue, the range of acceptable opponents widens. While the result may be a lower quality match, one will be made if you wait long enough. So, to answer you directly, yes. The new system will eventually match you without adding sandbags (even if it can’t find another perfectly paired Alliance).

                      The Glory reset isn’t a reset as much as it is a pull toward 12,000. The formula will pull harder the further away from 12,000 an Alliance is, but position to other Alliances shouldn’t really change much, if at all. If an Alliance had less Glory than you before, they’ll have less than you after. I did an exercise when designing the new system and the current top 100 leaderboard should end up somewhere between 15,000 and mid 17,000s.

                    • sulbin
                      sulbin commented
                      Editing a comment
                      Then do we have to wait every time without a sandbag? Is adding sandbag shortening time?

                      So what's different from what we're doing now is that if we wait for a long time, it'll just take us?
                      I don't know how long it'll take, but I hope it won't take too long.

                  • #12
                    So, there's been a few questions crop up regarding cheating. I don't have a lot to share right now (sorry). But know that we are looking into the issue and are working through how we want to address it. As with anything, I don't like band-aid solutions to these types of problems and I want to make sure we're taking the correct steps. Hopefully the deep dive on the matchmaking overhaul is a good demonstration of that. We'll certainly message it loud and clear when we have more to share.

                    Comment


                    • Thebigtfish
                      Thebigtfish commented
                      Editing a comment
                      It's not very sensical to fix match making and expect players to continue to funnel money to pay for a broken game. The imbalance created by the "AR Fix" the fact that players are showing up maxed space age over night, while cheating players out of paid for troops. How after 2 years of none stop reporting by hundreds of players. I alone have hundreds of screen shots to CS and in other forums calling for action against cheaters and nothing has been done. And we get a bracketed sorry, well that's one sorry response.

                    • sulbin
                      sulbin commented
                      Editing a comment
                      This problem is very old. Even cheater alliance is now number one alliance. Money is used at every World War. Even if you make a very good system, it's useless if it doesn't turn out to be a fair game. If you are technically unable to catch that point, you can prevent them from doing so in a way that will ensure that you are punished for it. Punish not only the individual but also the alliance of the punishment. (ex> glory reset ) There are so many cheater alliance now.There are always questions in the leadership community. " enemy is clean? " Isn't this a funny situation? we first think about whether the other person is a cheater or not. I've been reporting on this point, but nothing has changed. I don't want to hear sorry in parentheses. Show me more movement.

                    • {[Fresh]}Kratos
                      {[Fresh]}Kratos commented
                      Editing a comment
                      Looking into the issue and how to address it, one month... two months... three months... hmm...

                      One would think 3 months is enough time, not to actually address the issue... but to at least look into the issue... but not even...

                      These guys operate on a different timeline. I guess that is well presented by the upgrade times in this game!
                      Last edited by {[Fresh]}Kratos; 07-03-2019, 09:31 AM.

                  • #13
                    Before the new matchmaking system is released:

                    - Everyone is excited and cant wait.

                    After it is released:

                    - You broke the game, bring back the old matchmaking system

                    #SameThingsAllOverAgain.
                    Ch0s3nByG0dz - leader of DopeReach.

                    Comment


                    • No Angel
                      No Angel commented
                      Editing a comment
                      Human nature at its best! 😍

                  • #14
                    Originally posted by TinSoldier View Post
                    Method for Comparing the Offensive and Defensive Ratings of a Base

                    Did you know that the matchmaking system creates a Rating of your base’s offensive and defensive capabilities? It does. Everything on your base contributes to these Ratings. However, the method that the old system (the current matching system on live) used to compare the final Ratings with other bases wasn’t a great predictor of win rate. There was some directional value in its ability to predict win rate. But, we knew this was an area where we could improve match quality.

                    That old system compared the absolute value of your Alliance’s Offensive/Defensive (O/D) Ratings with other potential Alliances. We ran some tests internally and discovered that when we match based on a percentage delta between those same Alliance’s O/D Ratings, we actually get a much stronger predictor for winning. So, we’re updating the system to compare Ratings by the percentage delta rather than absolute values.

                    Fun fact: If your Alliance has a ~10% higher Defensive Rating in the new system, your Alliance’s chances of winning are increased to 60% (Yes, defense matters. Start upgrading your neglected Catapults!).
                    Additional fun fact: The new system would not consider a 10% delta a “good” match. So don’t worry.
                    Hi, can you provide a practical example of this. Maybe its just friday and a week of work has my mind melted already. You couldnt be looking only at the delta, right? So, an alliance with an average of Space Age offense, and Atomic age defense would have a 2ish age delta. An alliance with an average of atomic offense and industrial defense would also have a 2ish age delta. I know there are a lot of factors that go into the total weight of offense and defense, and it doesnt scale perfectly with age, but just pretend it does for this example? Those two alliances wouldnt match, would they? It doesnt seem like that would violate the StDev check you make in the next point either? I think a better example might help me understand
                    Proud member of ProdigalThieves

                    Comment


                    • Cannibals
                      Cannibals commented
                      Editing a comment
                      I think the delta described is the one between the aggregate O/D war weights, not the delta between the players in an alliance. The change is to compare the difference between the two alliances as a percent rather than an absolute difference. The effect would be that the range of those you can be matched is wider when you are at the high end and more narrow when you are at the low end. This is in addition to the std-dev or whatever it is comparison to make sure sandbagging alliances (who would have high variance between members) aren’t matched with non-sandbagging alliances (who would have low variance between members). Upon reflection, I think they had to change to use a percent delta so that their fix for sandbagging would scale correctly down to low ages.
                      Last edited by Cannibals; 04-08-2019, 03:21 PM.

                    • S_How
                      S_How commented
                      Editing a comment
                      BHG_Muet Ok, I think I understand a little bit better. But, say for example there is a space age team with an offense rating of 10, and a defense rating of 8. The delta would be 20% or 25% depending on how you calculate it. It is still possible to have an industrial age team with an offense 20 or 25% higher than their defense is my point. What prevents those two teams from matching, this is my question, and what I had trouble understanding.

                      Cannibals agree that its the delta between aggregate O/D war weights. But, if it is only delta between those things being used as the primary match mechanism, what is preventing a space age team with a 25% higher O v D from matching with an Atomic team with 25% higher O v D?

                    • Cannibals
                      Cannibals commented
                      Editing a comment
                      Maybe an example would be helpful. Some of the values will have to be made up of course.

                      Under the old system the match is based on a delta calculation. Two alliances would be matched if they were within some fixed delta of each other’s O/D. If Team 1 had O/D rating 4500 and Team 2 had O/D rating 5000, then their delta would be 500 and if the matchmaking threshold was 500, then the two teams would be matched.

                      Under the new system, the match range is a percent delta calculation. In the above case, the % delta between the two teams would be 500 / 5000, or 10%. Based on the OP, the new system does not consider 10% to be a good match and the two teams would not be matched.

                      So under the old system that had a fixed delta range for matching, it was more likely that teams would be mismatched when they were of lower ages because their O/D would be lower. Under the new system, the matchmaking range scales up or down with the O/D level, making it more likely to get a fair match at whatever O/D level you are.

                      My thought is that they would have calibrated the new % delta range for matching so that it would not impact top levels much, and it’s impact would be most felt at lower O/D levels.

                  • #15
                    A big thanks for your thoughtful efforts, looking forward to the new system

                    Comment

                    Working...
                    X