On the 23rd January, the KeyForge Premier League hosted their second set of qualifiers for spaces in Season 2 of the league. These were sealed qualifiers, where each player had a choice between 4 decks - 1 deck from each set. Each player who had a record of 4 wins and 2 losses or better after 6 rounds of Swiss was put into a single elimination bracket, with any byes being allocated to those with the best records. The top 3 players after this process (with the 3rd place player determined through a play-off) were invited to join Season 2 of the KeyForge Premier League.
As the deck selection process was much more pragmatic than the archon choices, this analysis is going to be very different from the first article.This write-up is to examine the options available, the deck choices players made from those pools, and the success of the top 16 players. I will also be spotlighting several case studies.
In this write-up, I will be using the following terms to refer to different groups of decks:
‘Pool’ - this was every deck which was an option in the qualifier (4 per player)
‘Decks played’ - this is every deck which was chosen from the pool and played.
‘Top 16’ - this is every deck which made the top 16 decks in the event after the challenger round of the single elimination bracket. Some decks were given a bye into the top 16 due to numbers and individual performance.
One of the criticisms of sealed as a format is that it is reliant upon the player's luck in pulling a strong deck. Primarily the goal of this analysis is to answer the question: "Were your chances of winning completely dashed if you got a low-SAS set of decks?"
The decks were drawn from Decks of Keyforge (DoK) by using a date as a filter - the most recent 1000 decks of each set were downloaded, with no other restrictions, so deck quality was completely randomised. Once downloaded, the decks were sorted randomly into pots, which were sent to each player. This ensured an entirely random selection of decks. (Note: random does not mean equally random.)
One of the quirks of doing this is that a bulk listing of Cull The Weak’s Brobnar89’s decks were caught in this - approximately 20, all of which were listed for sale. This was quite funny to discover after the fact, and great free advertising for him! (Perhaps an idea for the future is to only use decks with a For Sale tag to generate the event pool…)
Prior to the event, there were some concerns about drawing random Mass Mutation (MM) decks from DoK. The most concerning of these was the risk of pip manipulation to create broken combos, particularly on cards which recur such as Relentless Creeper, Q-Mechs and Rad Penny. Despite this, the KFPL organisers chose to draw decks randomly, deeming the risk to be very low. I will be examining some of the MM decks which made the top 16 to see if this was an issue in practice.
Note on Analysis
Most of the SAS analysis conducted for this write-up is presented with box-and-whisker plots. Due to me grappling hopelessly with Excel ( >:( ), these don’t have labelled axis, and have an additional item on the key for each type of deck. To read these as intended, please ignore the colours which don’t show up, and read the vertical axis as SAS rating, and horizontal axis as Set. Dots on the charts are outliers.
I would also like to give a disclaimer that I’m not an expert in statistics, although I have checked my methodology with people who are more knowledgeable than me.
Qualifier 3 - 9AM GMT / 4AM EST
80 players registered to participate in this event, resulting in a total pool size of 320. The median for each set ranged between 60 - 67 SAS, with Age of Ascension (AoA) being the weakest set on average, and Mass Mutation (MM) the strongest. MM had the highest SAS peak at 79, and Worlds Collide (WC) the lowest value at 43. The top end of all the sets was fairly consistent (76 - 79), but the low end varied significantly (43 - 57), with MM having the highest value (57) of the lowest scores.
Played Decks Analysis
As the MM decks in the pool were on average much better than the other options, it is unsurprising that it was the most popular choice for this event, making up 45% of the field. WC was second-most popular at 22.5%, and CotA third at 18.75%. Although it was least popular, AoA had a respectable showing at 13.75%.
The SAS ranges of the played decks indicated that those who received the top-rated SAS decks almost always chose those, as the peak values are identical to the general pool. AoA was generally rejected unless it was significantly better than other options. Players who used CotA decks were more forgiving towards their SAS ratings. WC had the highest concentration of high-SAS decks (50% were >=73 SAS).
There was a significant correlation between the SAS of the deck played and its performance in the Swiss rounds [r(78)= .28, p= .011.].
However, this isn't to say that players who chose a lower-SAS deck were doomed to failure - the lowest SAS deck to qualify for the top cut in this event was 62 SAS. One player who chose a 77 SAS deck from their pool went 0-3 and then dropped from the event. You can see the SAS ranges and their success in the Swiss rounds in the box plot below:
The take-home from this, I think, is - you don't have to have a high-SAS deck to do well in a sealed event, but it helps.
Top 16 analysis
The conversion rate of each set to the top 16 was largely proportional to the set share in the decks played.
SAS was a strong predictor of whether or not a deck would reach the top 16, with the range of SAS being 66 - 79. The peaks for AoA and CotA decks were slightly lower than those across the decks played. The top 3 players' decks were 77, 73 and 70 SAS respectively.
Luck or Skill?
So, were the results of this event down to the strength of the decks? To check this, I calculated the expected wins for each deck based on their statistics, and then checked the performance against the average for each of the top 16 players.
The method for this was the following:
On advice from someone with DoK know-how, I used AERC as a metric for measuring CotA and MM decks, and SAS for measuring AoA and WC decks. I was advised that those statistics are better indicators for those sets.
I calculated the average Swiss wins for each AERC and SAS score represented, and then used those to calculate an average number of wins in the event per unit, (This ended up being 0.038 for AERC, and 0.037 for SAS.)
I used this number to calculate the expected wins for each deck. Then, I compared this with the actual wins, then took away 1 from each result to get the performance of that player/deck above the average.
As a formula, this was:
The results, displayed on the right, show that every player in the top 16 performed well above the level of expected wins for their deck in the Swiss rounds. This suggests that there was an enormous element of player skill which contributed towards individual performance. Interestingly, this factor peaks within the rank 3-8 players and then tapers off in the top 2, suggesting that those in the top spots had a slightly easier time due to the power of their decks. That being said, though, the values of the performance above average in the top 16 are universally high - well done!
Any players who would like to know their individual result for this metric for Qualifier 3 or 4 is welcome to DM me on Discord (Muffins#7150) - I won't be publishing them publicly!
Mass Mutation - The Pip Connection
There was a significant correlation between the pip surplus (i.e. having more pips available than the number of enhanced cards in a deck). [r(34)= .35, p= .039].
However, there was not a significant correlation between SAS or AERC and the number of surplus pips. [SAS; r(34)= .17, p= .33]. This implies that decks which had any number of surplus pips had a significant advantage over those that didn't, regardless of other metrics.
Out of 37 MM decks, 11 had a pip surplus (29.7%), of which 4 made the top 16. MM decks with a pip surplus in the top 16 made up 10.8% of MM decks played, and 36.3% of MM decks played with a pip surplus.
Couteeyr the Annoyingly Aegian (1 pip surplus)
Vacuous Pierce of the City (3 pip surplus)
Demoledor el Somnoliento (3 pip surplus)
Spitepeace, Spawn of Triumspinax (3 pip surplus)
The question is whether these decks placed the pips in such a way as to give them a large advantage over their opponent. We checked the event footage to verify the location of the pips and see what impact they had upon the games.
Couteeyr the Annoyingly Aegian - 1 surplus pip
The surplus here was used to assign 2 capture pips to Observ-u-Max. With the double Transporter Platform, this means recurring the capture effect 4 times in one turn across any friendly creatures, if the stars align. Use of this can be seen at 59 mins on the event Twitch vod.
In theory this was highly impactful - however, on the footage I was able to find, the Observ-u-Max was used to upgrade an ANT1-10NY, which cancelled out the additional capture benefit.
Vacuous Pierce of the City - 3 surplus pips
The Keyfrog upgrade is enormously impactful and entirely changes the card - rather than having to time its destruction well, it effectively becomes an action and resolves its destroyed effect immediately as it is played.
Demoledor el Somnoliento - 3 surplus pips
This player used the entirety of their surplus on Bo Nithing, which ended up having two aember pips, a capture pip and a draw pip.
These upgrades, combined with Safe House, made it a very low risk but high impact choice. This card was never purged during the tournament.
Spitepeace, Spawn of Triumspinax - 3 surplus pips
This player appears to have spread out their pips, potentially as a risk mitigation strategy against cards like Infurnace and Mindfire. Placing both capture pips on the Ancient Power would have been extremely effective, but also a massively high-value target.
Given this information, it is clear that players who had a surplus used the opportunity of placing their own pips to create extremely powerful cards which gave them a significant advantage.
In terms of placement, here's how these four examples did:
Demoledor el Somnoliento (4 pip Bo Nithing + Safe House) - This player lost in the top 8 to the 2nd place finisher.
Spitepeace, Spawn of Triumspinax (spread out pips) - This player came 4th after a play-off against Vacuous Pierce of the City, narrowly missing out on qualifying for the league based on their overall performance.
The caveat to this analysis is that I haven't looked at the individual cases for the MM decks with a pip surplus which did not qualify for the top 16 - there may have been more egregious cases of optimal pip placement in that sample. I don't believe the event was a foregone conclusion for players who drew MM decks with pip surpluses, but I believe the data suggests it was a very influential factor.
Eagle-eyed readers will have noticed that the number of decks in the deck pool stats is 79, while I noted there were 80 players. For a reason I can’t really identify, two players in this qualifier were accidentally assigned the same pool of decks. We’re going to have a deep dive into their choices and journey throughout the event.
These were the decks both players had to choose between:
Biggs, Sognatore Curioso: This deck has a huge amount of disruption and 14 aember pips, which should make it consistent enough at high levels.
Quill of Hippocart Lair: This deck has similar aember control (A) and expected aember (E) values to the previous deck, but contains two copies of Miasma and a Too Much To Protect to stall and threaten your opponent, as well as Brend the Fanatic and plenty of ways to destroy him for the bonus aember.
Tessier le fougueux: While this deck has decent speed and disruption, its A is 3.9 and E is 8, which are both incredibly low.
The Virus that Sublimates Dinosaurs: This deck has excellent board control (particularly against non-MM decks with the two Dark Waves) and definite burst potential with the 2 Cleansing Waves. However, there is a significant amount of antisynergy between the board wipes and the capture in the deck.
After reviewing these choices, both players chose Quill of Hippocart Lair for the event.
One of these players went 5-1, and the other 3-3 in the Swiss rounds. This is despite facing approximately the same strength decks, and set breakdown. A possible reason for this is player match-ups. I checked the performance over the average for each opponent that faced either player, which can be viewed in the table below.
After Round 1, player 1 (5-1) faced opponents which consistently performed slightly above average (37.05%). The strength of player 2's opponents oscillated dramatically after Round 1, and despite having a worse record going into later rounds, the opponents they faced performed much better against the average than player 1's. This leads me to believe match-ups are a huge factor in tournament performance, and just another reason why the numbers being big doesn't equal an auto-win.
Next, I'll look at the statistics for Qualifier 4, before gathering my thoughts on both events.
Qualifier 4 - 5PM GMT / 12PM EST
98 players participated in this event, which made Qualifier 4 the largest event the KFPL has ever ran. This made the pool a total of 392 decks. As the pool was larger, it was perhaps inevitable that more high-SAS decks would be caught by the net, and this was confirmed by the existence of several 80+ SAS decks in the pool. Despite this, the median SAS in each set was roughly similar to those in Qualifier 3: the AoA and CotA medians were the same, the MM median was lower, and the WC median was higher. One major difference between the pools are the upper and lower SAS limits of each set - the CotA decks have a much wider range of quality (48 - 83, versus 51 - 76 in Qualifier 3), and the WC decks have a larger upper limit (81 versus 77). Asides from the MM decks, which have consistently higher lower limits in both events, the lower limits of each set is roughly the same. MM is consistently more highly-rated by SAS in this pool, but lower upper limits.
Played Decks Analysis
The proportion of each set played is almost identical to those in Qualifier 3. The main difference here is the reduced number of AoA decks played, probably due to being lower in SAS overall and having a much lower upper SAS limit. The popularity of CotA and WC as choices reflects their comparatively high upper limits.
The data confirms this, showing that the highest extremes of every set in the pool were chosen for use in the event. WC decks were mostly chosen if they exceeded 70 SAS, while players who chose decks from other sets were much more forgiving towards the values of their decks. Interestingly, the range of SAS in the MM played decks is identical to the range of those in the general pool, but the median of MM decks played is lower than the pool, meaning these were worse than the average available.
Like in Qualifier 3, there was a significant correlation of deck SAS with the number of Swiss wins [r(96)= .29, p= .0036].
The SAS ranges at each number of wins has an increasing floor which supports this. The range on the decks which won 4 games in Swiss is much larger than any other category, and also includes the highest SAS peak (83). Once again, while higher SAS numbers are correlated with better performance, this isn't the be-all and end-all.
Top 16 Analysis
Surprisingly, CotA made a disproportionately large showing in the top 16 decks. AoA also punched above its weight, with twice as many decks in the top 16 than in Qualifier 3, with half as many decks in the decks played group. Despite having the highest SAS on average in the decks played,the conversion rate of WC decks into the top 16 was only 11%.
In this group, the highest SAS decks were almost exclusively MM decks (peaking at 79), closely followed by CotA (77). Interestingly, the lowest SAS deck was also MM (66). The highest decks from the general pool (81 - 83 SAS) were unable to progress to this point. The majority of decks in this group were higher than 70 SAS. The top 3 performing decks were 74, 70 and 71 SAS respectively.
So why did the two 80+ decks not make it into the top 16, despite the odds appearing to be stacked in their favour? Let's have a look at one journey and see what happened.
The deck we're going to look at is Shucrystal Komarec-Frampton, Infiltrator, a WC deck which clocks in at 81 SAS. As we found earlier, SAS is a strong predictor of WC performance according to DoK. Certainly this deck looks like it has a lot to offer - 13 printed aember, high efficiency from Logos (Eyegor, Jargogle, Tautau Vapors) and Star Alliance (Captain Val Jericho, Lay of the Land), and strong aember control. There is virtually risk-free set-up of a Tribute-Exile/Sic Semper Tyrannosaurus/The Callipygian Ideal play thanks to the two Praefectus Ludos, and flexibility in loading your creatures with aember with two The Golden Spirals.
Its Round 1 opponent was Mallinson of Metplace Chamber, a 67 SAS/65AERC MM deck which had very little recourse to the threats presented by Shucrystal. Mallinson lacked meaningful creature control for this match-up in the form of bounce effects, and had one major board clear (Krrrzzzaaap!!!). Shucrystal took a confident 3-1 win here.
Round 2 saw Shucrystal face off against Gutbender, the Rogue of Metamorphosis, a 66 SAS/60 AERC MM deck with a +2 pip surplus (but with no obvious targets for pip-related shenanigans). One potential advantage Gutbender may have had is the Armory Officer Nel - Shucrystal has 5 upgrades, and playing these would have given Gutbender a lot of additional speed. Shucrystal would have struggled to remove this high priority target as their spot removal is low damage or conditional (Twin Bolt Emission, Phalanx Strike, Zap), especially if Nel was equipped with both Blast Shieldings for a +4 armor increase. This was a much closer match-up, but ended with a 2-3 loss for Shucrystal.
The third match-up was against Maxidash, the Artisan of The City, which was a 63 SAS/58 AERC MM deck with a +3 pip surplus, with the most obvious target for pip manipulation being the Lyco-Thief paired with Safe House. Maxidash had a lot of disruptive possibilities, with two Impspectors, two Vandalizes, and a Subtle Otto. There is also a lot of significant spot removal in Imp-losion, Harvest Time, and Tempting Offer, which would have been consistent at removing the Ludos, and therefore taking advantage of the aember build-up. This was again a close game, but ended as a 2-3 loss for Shucrystal.
At this point the player chose to drop from the competition, leaving the win score for this deck at 1.
The quality of this player's opponents varied: Mallinson's pilot's relative performance was -100%; Gutbender's was 87.85%; and Maxidash's was 45.74%. There is an argument to be made here that Shucrystal's pilot was matched with very skilled opponents on average, and that it was poor luck on their part. In addition, another factor could have been the draw order for Shucrystal in each game - with much of the aember control being reliant on a few cards (Tribute, Rhetor Gallim, Xenotraining), a poor draw for Shucrystal would have made it vulnerable to rush-style decks.
Based on this journey, I think it's clear that you need to back up strong pulls with a bit of luck.
Looking at the statistics from the event, it is clear that the concerns which the community had were valid: MM decks with a pip surplus were more likely to do better than their counterparts, and the strength of your deck pulls did have a significant correlation with your performance. I hope that it is clear, though, that the fact there is a correlation doesn't mean you automatically do poorly if you get a comparatively low-SAS set of decks. Pilot quality makes a massive difference to overall performance, as shown by the relative performance metrics in Qualifier 3's top 16. The best preparation for a sealed event, therefore, is to get a lot of plays in with a huge variety of decks, learn more about combos and card interactions, and practice like mad! You can't influence the random draw you get from decks, but you can influence your piloting ability, which has a huge effect on how you do on the day.
Regarding the MM pip surplus issue - given that it was a significant predictor of a decks performance, there is an argument to be made for mitigating this in future. However, it's difficult to see a way to manage this without curating the pool, which of course compromises the randomness of the decks drawn. Only using decks with verification pictures would overwhelmingly result in high-level MM decks being selected (who bothers verifying a low quality deck?). Randomising the pips with a program and asking players to use those presents two significant logistical issues - creating a resource for randomising the pips, and then verifying that these are the placements used throughout the event - which may be a poor use of time and energy by event organisers compared to the pay-off. Unfortunately due to the way online play works, it is unlikely that a solution for this will be found any time soon.
Finally, in answer to the question "Were your chances of winning completely dashed if you got a low-SAS set of decks?". My answer to this would be a little bit - but SAS was not the only factor in how well a player did at the event. Due to the nature of the KFPL, most of the players are at a similarly competitive level, so the differences between pilots of different quality decks is far more stark than it would have been for in-person play.
Once players progressed into the top 16, the importance of the deck choice became much more impactful. The top 3 decks in each event were all at or above 70 SAS (70 - 77 SAS), but the average SAS of a top 3 deck was 72.5 (roughly in the top 7% of decks as assessed by DoK). Going into imaginary maths-land here, if there was a completely even split of decks by quality in the Qualifier 4 pool, then 27 of those decks would be at or above 72.5 SAS. These would then be distributed between 7 - 27 different players. If they were evenly distributed, then 28% of players in the event would have access to a deck of this quality. The larger the event, the more the maths lands in your favour (as the more players there are, the more likely it is that higher quality decks will be in the overall pool).
Ultimately, I believe that the players who qualified were the best performing players, who were helped by a little bit of luck in their deck pulls and match-ups.
The next KeyForge Premier League Qualifier event is on Saturday 30th January at 4pm GMT/11AM EST. This is the Last Chance Qualifier, and 9 places in KFPL Season 2 are up for grabs for the highest performing players!