Guilds of Ravnica First Look: Controversy Abound

October 29, 2018 By Dan Troha 7 Comments

By Arseny Khakhalin, Dan Troha, and Bobby Mills

Believe it or not, we’re already four weeks deep into the Guilds of Ravnica draft environment! We’ve had a couple of Grand Prix tournaments, tons of drafting on Arena and Magic Online, and of course there’s still a Pro Tour coming up in a couple weeks.

Much has been made around the internet about ranking the various guilds and colors in GRN. We’ve seen plenty of people say that “green is bad” and have seen others respond with “you just don’t know how to draft green.” So it’s fair to say there’s been a bit of controversy about how to approach the format.

And that’s what we’re going to get into today. Not only will you get some great new visualizations of the overall Guilds draft format, we’re going to dive deep into a new draft statistic we’ve developed: the controversy of a card among drafters.

For today’s article, we took our data from around 118,000 drafts on Draftsim – collected throughout the first week of October. As before, we anonymously saved the cards and the order in which you picked them given the other cards presented to you in the pack.

Synergies

Let’s start with our now-familiar visualization of card synergies. In case you missed our description of how we previously created them, check out this summary. In short, each card on this plot is represented by a point matching its color. The distances between the points reflect how frequently any two cards are drafted together; the more often they end up in the same deck, the closer they are shown on the plot.

The axes here aren’t labeled because the values they represent are not really interpretable. For this type of analysis, only the relative position of different points matters. Colorless cards and nonbasic lands are shown in gray, and multicolored cards are all shown in purple.

First up – drumroll please… the synergy plot for GRN:

Note how each color is closest to its two guild companion colors

Isn’t it lovely? For you to appreciate just how different GRN is from a “normal set,” recall the synergy footprints for all four sets we’ve analyzed so far:

While DOM and M19 were more or less “normal” sets, and RIX/XLN was clearly a tribal block, GRN is built around guilds. What’s the difference between a tribal block and a guild one? What makes the synergy landscape so different?

Two things probably: one is that in GRN an awesome bomb card does not lock you into a guild that firmly. As long as your mana base supports the awesome card, you can still experiment. An Aurelia, Exemplar of Justice you drafted will still mentor cats, if necessary. It doesn’t force you to only draft angels, or mentor cards, as was the case with Ixalan. The second important difference is that two of the tribes in XLN spanned three different colors, while the guilds are all nice and evenly spread, like a flower.

If I told you GRN didn’t look like this, I’d be lion…

With its symmetrical shape and tight card clusters, almost no single card breaks formation. Rare exceptions are the two somewhat lonely red points at the very bottom that are biased towards Izzet. These are, of course, Electrostatic Field and Erratic Cyclops — cards that strongly benefit from being in a spells-based blue/red deck.

And you can see a black card on the very right that is attracted to Dimir: it’s Whispering Snitch, with its surveil synergy.

It really is astounding to look at the way single-colored cards sort themselves out toward their stronger color pairing. In blue, Leapfrog and Maximize Altitude pull towards Izzet while Thoughtbound Phantasm and Enhanced Surveillance push up against the Dimir pile.

You can see a large 2000×1500 px file with all cards labeled here:

A chart showing all the cards in GRN with relative synergies — Click on the image to expand in a new window

And we’ve also got an interactive visualization you can see here:

grn interactive visualization screenshot — Click here to use the interactive visualization

Color Pairs

We also looked at the color pairs that were drafted most often, as we did for M19 in our last post. Remember that for each final pool of cards drafted, we determined the color that was drafted the most (“Color 1”), and at the next most abundant color (“Color 2”). Then we built a heatmap, showing which combinations were drafted with the highest frequency and which were the least favored.

The plot for GRN looks pretty strange, but the sparseness is by design. Nobody is trying to draft Simic this time around, so only some color pairs in the heatmap table are even shaded:

What do we see here? Dimir (and more specifically, Dimir heavily biased towards blue) is by far the most popular. Note how strong the asymmetry is: 20% of decks feature lots of blue spells and some black, but only 7%, almost 3 times fewer, feature lots of black and some blue. Perhaps this is because both blue guilds are considered to be “good,” while one of the black guilds (Golgari) has been widely criticized for being underpowered. So people hedge by starting with being blue-based.

The next most popular guild is Boros (red-skewed), then Izzet (blue-skewed), and Selesnya (green-skewed). Golgari, on the other hand, though it may be strong in constructed, came in last place in our popularity contest.

Interestingly, it took our drafters some time to come to terms with the fact that green is a tough color to draft in this environment. If we look at how the popularity of different colors changed during the first several thousand drafts, we see that green was becoming ever slightly less popular, while black grew on people. The effect is weak, but it is definitely there.

Controversial Cards

We also had a new idea this time: to look for cards that are “controversial” in terms of their relative rank and rating. The intuition is simple: obviously, there are cards which everyone agrees are great and are always drafted first. If you open Dream Eater, you obviously take it, it’s a no-brainer, and it will probably strongly influence you to draft blue for the rest of the draft.

On the flipside, there are cards that are obviously “bad” that no one wants to take. These are the chaff, the very last cards in a booster. There are also cards in the middle that are sort of OK, and which everybody agrees are just OK. Think Devkarin Dissident, for example.

But we thought that there could be other cards that are drafted early by some people and hated by others, thus dividing the drafting community. And wouldn’t it be fun to identify them?

Now, while the idea is simple, the analysis turned out to be unexpectedly hard and weird. But if you like stats and data science, bear with us.

Calculating pick variance

To find controversial cards, we looked at the average pick number within the draft at which each card was taken, the same way we did to compare card rankings for previous sets.

Numbers in a data set can have a small mean (average) value, or a large one. They may also have small variance (be huddled together), or high variance (be spread wide).

But this time, we also calculated the variance of when each card was taken. For example, if a card is always picked first, its mean pick order would be 1, and the variance would be 0. If the card is picked 2nd in 50% of cases, and 3rd in 50% of cases, the average pick order would be 2.5, and the variance, for large sample sizes, would approach 0.25 (var([2 3 2 3…]) ~ (0.5)^2). We calculated these means and variances for every card, and plotted the variances against the mean, which gave us this graph:

As you can see from the legend, the color scheme is new here: black is for commons, gray for uncommons, gold for rares, and red for mythics.

The cards in the bottom left corner are the awesome mythics and rares, like Doom Whisperer, Dream Eater, and Aurelia, Exemplar of Justice. Everybody likes them, everybody takes them the moment they see them, so they have an average pick order of 1, and a variance of 0 (everybody agrees that they are the first cards to be picked).

On the other end of the spectrum there are points on the very right that represent universally unloved commons: Pause for Reflection, Vicious Rumors, and all five lockets. These cards are always picked last — or almost last — and people more or less agree that it is appropriate to pick them near-last.

Between these two extremes, we have a nice gentle curve. Cards in the middle have higher variance — people feel less strongly about drafting them earlier or later. They are more situational and are also dependent on the guild or deck type the drafter is shooting for. As a result, the more “average” the card is, the higher variance of drafting order it has. Which gives you this pretty hill shape, with the highest variance around the middle.

But then you can notice that there is some extra structure to the hill-shaped plot. Notice for example that commons form a band, uncommons sit on top of this band, and rares are pushed to the upper edge of the uncommons. Why is that?

Before we discuss which cards caused variance among humans, we first need to see where variance occurs naturally.

Inherent draft variance

At first we thought that it may be an artifact of human psychology, but then realized that the stratification appears in this plot even if we analyze decks drafted by Draftsim bots (below). That means that it has something to do with a card’s power, color, and its relative frequency, as these are the only three things our bots “know.”

So, where do these bands come from? They seem to be a byproduct of the drafting dynamics themselves. Imagine a card that is reasonably good, but narrow and demanding: something like a multicolored rare, or the very mono-colored card Gigantosaurus from M19. A player who opens this card in a booster would rarely pick it first, as it’s not good enough to commit to a guild or color. Such a card would get passed around the table, until it hits a player who is already drafting this color or guild anyway, at which point they’ll probably take it.

For these narrow, situational cards, that are decent in some decks, but horrible in others, a mediocre average score comes from a mix of early on-guild (or on-color) and late off-guild (or off-color) picks. These are “cards of opportunity”, that either ask a lot out of your deck (like Thousand-Year Storm), or just have a very limited application (like Drowned Secrets). As an example, here’s a histogram of pick frequency for one of the most controversial cards in GRN, Ionize:

Here’s a practical scenario for you that is hopefully easy to relate to. Imagine that 8 people sit at a table to draft. They open 8 boosters, and one of the boosters has Ionize (a high-variance card) as the pack’s best card in Izzet. The card’s two-colored casting cost (and solid-but-not-amazing power level) makes it not good enough to commit to Izzet right away. There are almost certainly more flexible or powerful cards in the booster, so the player that opens it will probably pass it to the next person to the left.

That person will look at the Ionize, and if they are drafting Izzet, they will be happy to take it. But if they are drafting anything else, they will pass it along, because this card has a low chance of fitting their deck. So it will move across the table until it finds a player drafting Izzet, or until it is drafted as an off-color card towards the end of the booster, because no better options are available.

There is almost no competition for these cards, which leads to this flat histogram, and to a cliff-like decrease in probability of being drafted between the 8th pick (almost full cycle) and the 9th pick (beginning of the next cycle). For a card like that to be drafted on the second cycle, someone would have to pass it the first time around, but later firmly settle on Izzet by the time the booster goes around the second time; something that does not happen that often.

For bears, decent commons, and other “so-so” cards, the situation is very different: you never draft them first (they are not good enough for that), and you almost never get to draft them last, as they fit in many decks, and somebody will almost certainly draft them at some point. Which means that if they have an average score, that’s because they are actually average!

These cards are competitively drafted by many players who compare them to other on-color cards in each booster. As the outcome of these comparisons varies from draft to draft, the histogram of pick orders looks vaguely normal-ish (the Central Limit Theorem starts to kick in), centered somewhere in the middle of the rank scale.

For example, here’s a histogram for one of the least controversial cards with about the same average pick order as Ionize (around 7th), Fresh-Faced Recruit:

So Wizards R&D prints more situational and complex cards at rare or uncommon, creating these bands of gray and gold above the curve of black (commons).

This is our best explanation for the fancy Var/Mean curve, and if you have a better one, please let us know in the comments! The effect is real, and has nothing to do with psychology, but it is also tricky, and hard to wrap your mind around.

OK – here’s the list

But enough with all this talk; what are the most controversial cards? Taking into account the baked-in draft dynamics we just described, the list of controversial cards consists of medium power mythics and rares, mostly two-color, that are situationally good. With raw pick variances, the cards score high on this “controversy” scale not because they polarize drafters necessarily, but because of the logic of the drafting “game.” For GRN, the most “high-variance” or “volatile” cards are:

No controvesry here… — No controversy here…

The least controversial cards, on this scale, are:

All the top choices from the bottom left corner of the graph: awesome, flexible bombs. In GRN, it would be cards like Light of the Legion, Doom Whisperer, Niv-Mizzet, Parun, Status/Statue, Aurelia, Exemplar of Justice, etc.
Bottom-feeders and sideboard cards that everyone either universally hates or knows they can get late: Pause for Reflection, all 5 lockets, Vicious Rumors, Crushing Canopy, Never Happened, etc.
Run-of-the-mill commons (see below for a list).

Human-induced variance

Unsatisfied with this scale, we kept thinking: can we still somehow look past the drafting mechanics and quantify the uniquely human contribution to variance?

How do you find results that do not stem from permutations and distributions, but that are driven by psychology and differences in human opinion?

We realized that we can actually estimate that, because we track both human drafts and bot drafts! All effects of stats, distributions, and relative scores should be the same for bots and humans (except that the ratings used by the bots, and average opinions used by human players, would of course be slightly different).

But humans would have some additional personal variability added to their actions. But bots always use the same ranking system, carefully assigned by Dan Troha, so we can estimate human volatility by subtracting bot pick variability from human pick variability! Here’s the result of this analysis for GRN:

On this plot, the horizontal axis shows how much earlier humans drafted each card, compared to bots. If on average humans drafted something 2 picks earlier, it would have an X of 2 here. The vertical axis shows just how less certain humans were about this card, in terms of bringing extra variance to their pick order.

For example, judging from the yellow cloud on the right side of this square, humans tend to rare-draft (humans pick junk rares and mythics 2-3 picks earlier than we recommend our bot do it). They are also more variable in these preferences, probably because some users raredraft and some don’t. In fact, there’s been a long-running debate among Draftsim users about whether the bots should raredraft because many people do in “real life” scenarios.

And so, finally, here are the most controversial cards, the points in the right top quadrant:

As you can see, most cards that are hard to incorporate in a deck because they are situational and narrow (as we described above) were also confusing for human players. In other words, these are the weird rares.

Which, in hindsight, makes lots of sense. When we are faced with tough decisions, we each solve them differently. That’s what is nice about humans: some like to take risks, some are careful; some stick to strategies that worked in the past, some like to experiment.

So when cards were hard to fit in a deck, this “volatility” was further amplified by human psychology, making them “controversial” not just in a mechanical sense, but also in the psychological sense of this word.

We then looked at the most controversial commons:

Now, this is interesting! In raw, absolute numbers, the five lockets were relatively less controversial than most other commons (remember, they all sit in the right bottom corner of the inverted parabola). Yet if you look past the systemic variance into human-specific variance, you actually see thathuman players had different strategies here. Some were drafting lockets, and some were not!

People love to experiment when they use a draft simulator, and drafting a “sweet” four or five-color locket deck is a nice thing to try out. People were also fairly uncertain about where the lockets fell as far as power level at the beginning of the format.

For the sake of completeness, here are the most controversial uncommons:

A nerdy footnote on mathematical paradoxes

Now, here is an even more mathy aside, for those of you who like to read Frank Karsten and Felix Weidemann for fun. Did you notice that the plot of human-controversial cards actually looks a bit weird, as humans seem to draft virtually all cards earlier than bots do? Look at it again: most of the points have positive X values, meaning that they are drafted by humans a few picks earlier (anything between a fraction of a pick, to 5 picks for Divine Visitation).

When we saw this plot for the first time, we thought that it may be a bug in the code, because how could it possibly be that humans would draft all cards earlier? Surely if they draft some cards earlier, they would draft some other cards later, right? The average values should be the same for humans and bots, as they are all in the same boat, subject to the same rules, wouldn’t you agree?

It turns out that it is actually not a bug, but rather a real full-blown paradox, directly related to Simpson’s Paradox (check out the UC Berkeley admissions controversy, which is closely related to our problem). It turns out that it is possible for one player to draft all or almost all cards earlier, on average, than another player. It is actually a sign of inefficiency. Humans are inefficient when compared to bots — but we knew that!

If you find this counterintuitive and hard to believe, here’s a small trinket example. Say two players are drafting from boosters of 4 cards, and there are only two types of cards, there, A and B. A is more powerful than B. They draft 2 times, and get the following final draft pools:

Draft 1: Player 1: AB, Player 2: AB
Draft 2: Player 1 BB, Player 2: AA

Let’s calculate average pick order for both cards and both players. For Player 1 here, the pick order of A is 1 (it was always drafted first). The pick order of B for the first player is (2+1+2)/3 = 1.7. For the second player, the average pick order of A is (1+1+2)/3 = 1.3, and the pick order of B is 2/1 = 2.

If you compare these average picks, the second player, on average, drafts both cards, card A and card B, later than player 1 does (because 1.3 > 1, and 2 > 1.7). And also, incidentally, they win on average, as AB against AB is a draw, but AA wins over BB.

So, surprisingly, the “Unique Human Preference” plot above not only illustrates some features of human psychology, such as raredrafting, but also testifies to the average inefficiency of human drafts compared to bots. While humans are hunting for fancy cards, bots calmly draft more efficient cards, resulting in more consistent draft pools, and most probably, more powerful decks overall (although we didn’t try to directly estimate that).

Concluding Thoughts

Given that one of the big reasons that you like to use Draftsim is to get better and learn how to improve at drafting, what can you take away strategically from all this?

If you want a quick idea of the format, I highly suggest looking closely at the mono-colored cards on the synergy plot with all the cards labeled to see what guilds they are most closely associated with. This will give you probably the quickest overview of “how to draft the set” because you’ll see which cards work best in conjunction with which other ones.

Secondly, talk to your friends and people you play with about the list of controversial cards. Have they put them to good use? What kinds of decks want them? When can you expect to get them in the pack and how early do you need to draft them? That way when you’re at your next FNM, you’ll know whether to take that weird rare or whether you should just pass it on to the next person.

**(Unless you are a raredrafting human, that is!)

If you enjoyed this article, be sure to follow Draftsim on Facebook and Twitter and please help out the site via Patreon if you can.

Draft Data Analysis 2: How Much Does a Format Change in One Month? [M19]

September 17, 2018 By Dan Troha 1 Comment

By Dan Troha, Arseny Khakhalin, and Bobby Mills

As we finally say goodbye to M19 and welcome in Guilds of Ravnica (appearing on Draftsim.com this Wednesday), our team at Draftsim wanted to take a final look at Magic Core 2019 to see what we can learn from this boatload of data we collected.

This time around we’re going to look at how M19 has evolved since July. Have you ever been able to see an actual list of cards that you over/underrated in draft? We’ll be going through that and a lot more. I have to admit, I was amazed how much the data resonated with my own understanding of the format as time passed.

Plus, there will of course be many pretty new visualizations.

M19 – One Last Time

matrix graphic — It’s time to re-enter The Matrix (matrix math, that is)

Just to recap from our previous article, we visually represented the past three formats with a massive statistical analysis. We collected the picks from over 100,000 M19 drafts in July, then another 50,000+ in August. Then we slice and diced the results in a bunch of different ways, but we mainly looked for pairwise synergies of different cards to find out how they clustered together when people repeatedly picked them for the same decks.

And that analysis produced some sweet plots that looked like these:

2D plot with dots representing cards — M19 August 2018

Note: The units on the axes do not mean anything. They are only there to show the relative relationships between points. For a further explanation, please check out our last article.

We had quite a few people ask us for the data — which was really exciting, but also was kind of overwhelming. It’s clear that the Magic community is awesome and is just as nerdy as we are! We’re planning on releasing this data eventually, but we need to do a lot more work and analysis with it before we can turn it loose on the internet. So stay tuned on that front.

We also received several requests and suggestions for interactive visualizations, and I’m happy to say they are here! Here’s a D3.js version of the 2D visualization above (click on the image):

m19 interactive visualization screenshot — Click here to play with the interactive visualization

But enough of this snapshot stuff. This time around, we’re going to look at … ch-ch-changes.

With a data set this large, we are able to look at relative changes between July and August in card-by-card pick orders and even in overall color pair preferences. For example, we might be able to make guesses at questions like “Which colors became more popular in August?” or “Which archetypes fell out of favor?”

And how could you adjust your drafting strategy if you knew this?

Format Migration – “Flocking”

Let’s first start with the graphic that we teased in the last article:

m19 shown with before and after card movements — “Flocking” of relative changes in card preferences. Click here for a labeled version

The plot above shows every card in the M19 draft format and where it stood before August and after. The large squares (the “heavy heads”) represent the new position, while smaller “tails” represent where each card started in July.

To build this plot we projected July and August drafts onto a 2D picture, and then found a rotation of one picture that matched the positions of cards in the second picture as well as possible (on average). Then we looked at the relative movement of each card within the full cloud.

The most striking thing is that cards mostly seem to move in big flocks! While there are some dramatic individual card changes, it really looks like most of the movement comes from overall changes in the color, not just specific cards here and there. Why would that be?

Overall, it could be that archetypes are becoming more clearly defined and the colors are becoming slightly more polarized. As a format develops during the initial weeks, everyone goes through a learning curve in an attempt to discover what each color combination is trying to do, which color combinations support good archetypes, and which colors are underwhelming when paired with other colors. The flocking shows the format becoming less homogeneous and more clearly defined.

For example, take a holistic look at the color green. The majority of cards appear to be moving in the opposite direction away from the black/white cluster. If you think about what archetypes are represented here — black/green nothing and white/green auras, it certainly could give us some clues.

It is reasonable that players would be less inclined to pair green with black because the deck has somewhat of a bad rap for having no synergy — and therefore is harder to “make work” in a draft. As for white, GW’s primary strength and game plan relies upon an auras theme that takes very specific cards to build. Although green is often trying to ramp in M19, a GW deck is much less interested in Colossal Dreadmaw, Elvish Rejuvenator, or even Druid of the Cowl.

Red, on the other hand, appears to be biasing away from green and slightly toward the white/black cluster. This also would indicate GR losing some popularity (one fun exception is that Onakke Ogre took a hard turn up toward GR). As we’ll see in our archetype and individual card analysis, RW certainly got more popular, and you can see Trumpet Blast, Dark-Dweller Oracle, and Goblin Instigator moving in this direction.

Note: Let me take a moment to say that I also updated the ratings in the system a couple times during this period, which affects which cards are presented to people and the order they are drafted. Not only would the bots take the cards/colors that I bumped up higher, players using the “Suggestions” function of Draftsim would also incorporate the same biases.

screenshot of draftsim's suggestion engine — Draftsim’s AI provides pick suggestions to users — which probably influences some picks

Color Preferences Changed… A Little

Okay, so now that we’ve taken a look at the format as a whole, let’s drill down a little bit and look at each color individually compared to its counterparts.

The tiles below show the share of decks drafted in each pair of colors. The x-axis, “Color 1,” represents the primary deck color — the color with the most cards drafted at the end of the draft. The y-axis represents the color with the second-highest number of cards in the drafter’s final pool.

july 2018 color distribution chart — July 2018 M19 color distribution

In the first pass at the format, we see a strong preference for UW, BW, UB, and GR. Remember, the second color here is not necessarily the color that was drafted second, but rather the color that was second in size in the final draft pile.

UW artifacts and BW lifegain in particular seemed to be pretty obvious synergies right out of the gate. The gold cards told you what to do, there were strong-looking uncommons that supported those themes, and plenty of commons available that acted as enablers. So it’s not too surprising people would start by drafting these decks.

In July, the most popular draft combination contained lots of blue cards and a handful of white cards. But overall, people were drafting most of the color pairs quite diversely. Only WG got no love.

Now let’s compare that to August…

august color data chart — August 2018 M19 color distribution

It seems that after a month of practice, fine-tuning, and mulling over the metagame, drafting strategies started to pull further toward a handful of popular color pairs. BW stayed dominant, RW joined it, while UW, UB, and GR shifted around and became less focused on their primary colors.

Essentially, in August drafters converged on a stronger preference:

1. Draft black, red, or white
2. If the first color isn’t white, draft white for support. A whopping 27% of decks had white as the secondary color. Look at how dark the bottom row on the visualization is!

A special note about white in M19

White was heavily represented as a secondary color. If I had to bet, I would guess this was because of increased discussion in the community about how great white’s commons were, which made people a little more willing to look at white cards for their decks.

Not only does white have a premium removal spell at common in Luminous Bonds, it also has three very strong creatures headlining its slate of top commons. Each of these cards effectively puts the screws on an opponent, and together they are almost unbeatable in their uninteractivity.

Credit to Lords of Limited for dubbing this band the ~~“Three Amigos.”~~ (edit: Three Musketeers, thanks guys).

Hot Decks, Cold Decks

Now let’s give our flocking chart another look, comparing it to the heatmap of changes in color pair popularity, shown on the right.

Here red cells show pairs that became more popular, and blue cells show those that were now drafted less often.

If you let your eye drift up the column for each primary color, you can see that as time passed, Draftsim users really tried to force white, black, and red as primary colors, while they practiced drafting blue and green a bit less frequently. And also, they became increasingly interested in drafting both WR and RW combinations.

Do these observations match the flocking picture? Sort of. The green flock moved away from the other colors, as primarily green decks became less popular overall. Both WR and RW became more popular, so white flock moved a bit towards the red flock.

But overall, the flocking chart does not provide crystal-clear results. Therefore, it might be a little less useful than we hoped. Large-scale movements of colors obscure the individual changes in card-to-card synergies (if these changes exist), and even large-scale movements are hard to interpret. For example, what’s the deal with some of the seemingly-conflicting rearrangements of the blue swarm? Tezzeret only knows.

Still, we tried to look into the movement of individual cards (click here for a full 2000×2000 px version). It’s pretty obvious from the picture that multicolor cards (magenta) were the ones to move the most, which is fun, but reasoning for the movement isn’t obvious.

For example, a whole bunch of cards in the upper left quadrant moved towards green, including Draconic Disciple, Vaevictis Asmadi, the Dire, and Poison-Tip Archer, perhaps implying that you need to be more green than the other color to make use of these cards. This is a phenomenon that you see sometimes in draft — because of the mana or what a particular strategy is trying to do, a gold card can be better in a deck that has a stronger base in a certain color.

Brawl-Bash Ogre and Heroic Reinforcements each move in the same direction as their black and white flocks respectively, while staying equidistant from red, which makes sense.

Ratings Changes

Now that we’ve discussed final color configurations and changes in card synergies, let’s move on to individual card ranks.

Were there any changes in pick order in August compared to July?

Of course there were some changes, but they weren’t overwhelming. Exciting cards were still picked early in the draft (lower left corner), while weak and weird cards were picked late (top right corner). Still there was some curious reshuffling of individual picks, and with our sample sizes (from 1031 for the rarest card to 24733 for most common) we hoped to suss out some real changes, even if they were subtle.

We looked at the top 20% of cards that either lost or gained popularity between July and August. Here’s the data for the most volatile cards, with more popular cards (those that were picked early) shown closer to the top, and less popular cards shown at the bottom. Generally, these popularity rankings correspond to the power level of the card (but not always).

Lines show how much the rank of each card changed. If a card became less popular, the line is blue. If a card’s popularity increased, then the line is red. The intensity of each line represents the absolute change in popularity.

You can read the changes from left to right, where the card name represents the ranking in August after the changes.

So many cards, so many stories to tell. I’m going to dive in by categorizing these in three groups: “subtly powerful cards,” “weird cards,” and “synergy cards.”

Subtle power and format context

By now everyone knows how good Dryad Greenseeker is. But that wasn’t completely obvious at the beginning of the format. It was slept on a little until people got to see it for the card-advantage-machine-with-good-stats that it really is — and now it’s considered a “mythic uncommon.”

The same goes for Departed Deckhand. In a format without many good two-drops, this was one of the absolute best. You could play it in almost any deck, it was good on turn 2, and great on turn 8. The fact that you could get such an impactful two-mana card made it very important to prioritize because there just aren’t that many cards in the format that can provide that amount of impact for the price.

And if there aren’t many good twos, you shouldn’t be putting the bad twos in your deck. But then how do you not die to your opponent’s bad two drops? Essence Scatter quickly rose in stock in M19 because it was a great early play, but unlike a 2/1 for 2, it’s also great on turn 5-6 when you can nab a Thornhide Wolves or Colossal Dreadmaw.

Reclamation Sage experienced the biggest jump, and this is one I’m not completely sure about. Certainly between the existence of a WU artifacts deck, some powerful enchantment-based removal, and gamebreaking bombs (Patient Rebuilding, Sarkhan’s Unsealing) it is quite maindeckable. But there isn’t a huge reason for it to be so relatively underrated initially.

Finally I have to give a shout out to one of my favorites, Vampire Neonate, which had to be one of the most annoying cards to play against in the format. It blanked many of the aggressive two drops and allowed control decks and slower midrange decks to climb back into the game after falling behind. And it was a good mana sync. Not to mention having “the combo” with Ajani’s Pridemate. All for an unassuming one drop 0/3.

Weird cards

Transmogrifying Wand kills some stuff and makes 2/4 oxen. That’s not amazing, right? But wow was this thing powerful. It turns out that if you’re a control deck, or playing against someone who has green creatures in their deck, the card would dominate the game and 3-for-1 your opponent by neutralizing their best threats or even let you upgrade your own creatures.

Inferno Hellion initially looks like a pretty bad card, but there are just enough weird things you can do with it like Thudding or Sarkhan’s Unsealing that it turned out to be quite a bit more playable.

Synergy cards

The presence of a card like Elvish Clancaller (or Goblin Trashmaster) makes you assume there are goblin and elf decks in the format. Unfortunately, M19 did not have these, so they turned out to be much worse in context. Same for Desecrated Tomb — it looks like the card should have ways to trigger it, but there just were not many options in M19 limited, and so it proved to be a very narrow — basically unplayable — card.

On the upswing, you can see that Angel of the Dawn and Heroic Reinforcements jumped to near the top. Word got out (beginning probably with Ben Stark) about how good RW was, and these cards that are a tremendous payoff for a go-wide strategy as it rose in stock.

Concluding Thoughts

You hear a lot about the collective Magic hive mind working to “solve” a standard environment, but the same scale of testing and information sharing goes on with limited. You can see that people combine their own experiences with those of influencers in the community and change their behavior accordingly.

Speaking of, during the first few weeks of a format, I love to hang out on Twitter and look through 3-0 draft decklists to see what’s working for people. Feel free to tweet at me if you discover something cool (or super uncool) about a card or archetype.

Like all of you, I can’t wait for Guilds of Ravnica to drop soon. This is going to mean a whole shiny new data set and some pretty new visualizations of a completely different and guild-based format. I can’t wait to see what a guild synergy clustering visualization looks like…

Oh yeah, and I guess there’ll be a sweet new Magic set to actually play with too!

Be sure to follow Draftsim on Twitter and Facebook to be notified when GRN goes live and when we post more in-depth articles like this.

Draft Data Analysis: What We Learned from Analyzing Over 200,000 Draftsim Drafts

September 3, 2018 By Dan Troha 18 Comments

Hi! I’m Dan, and I run Draftsim, a website that helps people practice by drafting in a simulated environment against seven “bots” programmed with artificial intelligence. Along with some key collaborators, I’ve been working on a big project this year that I’m excited to finally share with you.

Have you ever wondered what a draft format looks like in one picture? Or what makes a format really “synergy dependent”? You’re in luck!

This past year we’ve been collecting anonymous data from the thousands of drafts that occur each day on Draftsim. It’s finally time to take a look at what we can learn from all that sweet, sweet data.

I owe all the credit for this project to Arseny Khakhalin (thank you Reddit for helping us connect!) and Bobby Mills from Bard College in New York. They helped me collect all the data on the site, do the analysis, and create these awesome visualizations.

Methodology

How much data are we talking about? Well, a lot.

Here’s how many human drafts we analyzed for each format:

M19 Part I – 108,000 drafts (July 2018)
M19 Part II – 48,000 drafts (August 2018)
DOM – 51,000 drafts (April 2018)
RIX (including XLN) – 10,000 drafts (February 2018)

In all, that’s actually over 200,000 drafts!

We simply captured the format being drafted, the cards presented to the user in the pack, and the card the user picked given his or her current card pool.

Animated gif of someone drafting M19 — We recorded thousands upon thousands of anonymous user picks

For each draft, we looked at the final pile of cards a user ended up with, and dutifully marked which cards were drafted together in a table that reflected all possible “card pairs.” After going through thousands of drafts, we compared these co-occurrences of cards in final draft pools to numbers that you could expect to get by chance, as if all drafts were completely random. Obviously, some pairs of cards were drafted together way more often than you’d expect by random chance, while some cards were drafted together less often.

At this point we used a standard statistical technique called multidimensional scaling to visualize the allegiances of different cards in a nice plot. To do so, we imagined that each card is represented by a point, and distances between these points depend on how often they are drafted together. Cards with a synergy were assigned small distances between them, while cards that are only drafted together by chance were assigned large distances.

We then utilized a function in the statistical computing language R called cmdscale (the whole analysis was performed in R), and asked it to find an arrangement of points in a 2D or 3D space that would achieve these predefined pairwise distances as well as it possibly could. The function finds an arrangement of points, and then we visualize it either with ggplot (for 2D plots), or with rgl package (for 3D), to get the pictures you’ll see below.

Results and Analysis

So enough with the talk about how we got there – let’s see what we found out! We’ll start with the recent M19 data, mostly from August.

Visualizing M19

On this plot, each point represents a card. Red cards are red, green ones are green, and so on, except that the white cards are gold colored. Colorless and artifact cards are gray. All multicolored cards, regardless of guild, are shown in purple for simplicity’s sake.

If two cards are often drafted together, they are shown close to each other on this plot. If the cards are almost never drafted together, they will be very far from each other on the plot.

Note also that the X and Y axes are not really interpretable; they are just a way to plot the points, but there is no deep meaning to them.

If you’re wondering which card each dot represents, a similar picture that has each point labeled can be found here.

As you can see, it is not easy to represent a complex drafting landscape in a 2D image, so a card may be in the center both if it is drafted with any color (like most artifact cards), and also if it is drafted in decks of two opposing colors (like Gruul or Izzet on this plot). But at least you can spot things that are suspicious and investigate them.

Plotting in 3D might be more informative, but the trouble with 3D plots is that you still need to show them on 2D screens, which really limits their utility. Here’s a 3D projection of M19 drafts: pretty, but hard to use.

a rotating cube with x y z axes depicted relative M19 card positions — M19 live and in 3D

Observations

The first thing we can see here is that cards of the same color are drafted together, which seems obvious. But even then, exceptions prove the rule, and we have quite a few interesting exceptions.

For one, the black and white cards (represented by golden dots) don’t form distinct clusters, which hints that in M19 drafts, players tend to pair W with B more often than with any other color, and vice versa.

The reality is a bit more complicated: white and black cards actually do form distinct clusters, but these clusters are closer to each other than any other two color clusters. So when the multidimensional scaling tried to show all cards on a 2D picture, and looked for some complexity to sacrifice, it sort of folded them together. But if you look at the 3D graph earlier, you’ll see that the white and black clusters are actually distinct. This is an artifact of the 2D visualization method, but it is an artifact that appeared for a reason.

We can also hunt for individual points that don’t belong to the cloud. See that red dot on the very left, far from the main red cluster? That’s Sarkhan’s Unsealing, a card that benefits from large, beefy creatures, and even though it is strictly red, players treated it as a Gruul (RG) card. Or look at that green card that is closer to artifact cards than to other green ones: that’s Scapeshift — which is completely unplayable, however some Draftsim users like to raredraft, so it was drafted as if it were color agnostic.

And while we’re on the subject of unplayables, you’ll also find Omniscience and Mistcaller sitting in the middle since they wouldn’t reasonably be included in any M19 draft deck either.

What about that lonely white card on the far right? It is Aethershield Artificer, which pumps artifact creatures, making it an obvious synergy payoff for the blue/white Artifacts Matter deck.

Enough about relationships, what about power?

We can also mine draft results for other, more straightforward types of information. For example, which cards are the absolute winners that are always picked first? Here’s the top five list:

This is interesting. While these are mostly great pack-one-pick-ones, they’re certainly not what I have rated as the top five cards in the set. Sarkhan especially is strange, but people love planeswalkers and the card is deceptively worse than it looks.

And what cards don’t see any love from drafters? Here are the bottom five, most frequently picked last (so shown backwards, from worst to slightly less worse):

Apparently, “tower” cards are bad.

RIX/XLN visualization

Do all draft formats have similar shapes? Not at all, and of course the Ixalan and Rivals of Ixalan formats are special here, because their synergies are based largely not only on color pairs, but also on tribes that are in those colors. So the plot ends up looking completely different:

Click here for version with cards labeled

Here we have Vampires on the left (with Anointed Deacon and multicolored Legion Lieutenant leading the charge), Merfolk on the right, and the red guys on the top — Pirates and Dinosaurs alike. What is the half-ring of multicolored cards surrounding the red cloud? Pirates are on the left (leaning towards the black Pirates), and Dinosaurs on the right (towards green, and to a lesser extent, white creatures).

Look how closely WB Vampires and UG Merfolk are clustered. These are the two tribes in the set that were represented by only two colors, so they end up grouped very closely together.

Contrast that with Grixis Pirates (B/R/U) and to Naya Dinosaurs (G/W/R). The relationships are much more spread apart and less tightly correlated than the two-color tribes.

Red is in a special place because the color “belongs” to neither Vampires nor Merfolk. When you pick a red card, assuming an equal archetype distribution, it has an equal chance of ending up in a tribal deck of any of the four colors — either a Pirate deck (RB or UR) or a Dinosaur deck (GR or RW). In contrast, it’s not so with black, where you can end up in Pirates (RB or UB) or Vampires (WB) only, if you stick to the prescribed tribal archetypes and groupings of synergy.

In XLN/RIX, GB and UW were “tribeless,” making their synergies — if they existed at all — much less pronounced. It makes sense that we don’t see them bumping up against each other here.

DOM visualization

Click here for version with cards labeled

Dominaria seems to have the most separation between colors of all three. Perhaps this has to do with the high individual power level of many of the cards in the set. Moreover, in DOM, artifacts do not form an amorphous cloud in the center, but make a cluster of their own, very close to U/W/B, which happen to be the colors that care much more about historic triggers.

Not too surprisingly, the blue cards that pull closer to red are Wizards synergy cards (eg Naban, Dean of Iteration, Vodalian Arcanist, and Wizard’s Retort), and the ones that are closer to white are either artifact/historic payoffs or enablers (The Antiquities War, Artificer’s Assistant, Zahid, Djinn of the Lamp, Karn’s Temporal Sundering, Sentinel of the Pearl Trident).

I’m not sure how to explain why white is so closely clustered together versus red, however. This possibly indicates a replacability/lack of synergy among white’s cards or perhaps a tendency of the white cards to only be drafted with each other.

Bonus: Just for fun, here are the other two 3D visualizations that we generated for RIX and DOM:

a rotating cube with x y z axes depicted relative RIX/XLN card positions — RIX 3D visualization

a rotating cube with x y z axes depicted relative DOM card positions — DOM 3D visualization

Where To Go From Here? Some Exciting Future Possibilities…

So this is all really cool, but what can we do with it?

Training Draftsim’s AI to use this data and capture synergy

Synergy in Magic is a very difficult thing to measure because it shifts from format to format. In one block, R&D may introduce a new mechanic that explicitly drives synergy (such as historic, kicker, or cycling), but another might primarily use tribes, such as in Ixalan. And let’s not even get started with cards that are “better for aggro decks” or “control” decks. After all, most decks in limited are just some flavor of midrange, right?

Rather than individually scoring or programming all these factors for each set, it would make a lot of sense to enlist Draftsim’s wonderful user base to help train the algorithm to recognize synergistic pairings of cards.

We could then in turn train the AI to make more realistic decisions based on the clusters of cards that are frequently drafted together.

Encapsulate the shifting “draft metagame”

It would be fascinating to track how cards and color pairs become progressively more or less overvalued during the course of the format. You might see certain color pairs move around as archetypes go in and out of vogue or one color gets recognized as “bad.” And, like in constructed, a draft metagame seems like it would eventually settle somewhere.

And wouldn’t it be really fun to be able to quantitatively measure what cards people “slept on” at the beginning of a format? Or to assign archetypes and see how our understanding of what cards belong in them changes? You could then use this information to improve your card and archetype evaluation methods.

In fact, this leads into our next article where we’ll do a quantitative analysis of how much a format changes over the course of a month. Here’s a little taste of what we’ll be looking at:

That’s right – we have before and after data that we can use to measure changes. Double the data means double the fun!

double rainbow guy saying double data — Stay tuned to find out…

Clustering decks instead of cards

This whole article we’ve been looking at defining a format by the relationships between individual cards. However, we can also zoom out and examine the the end product of your draft as well: your final collection of drafted cards.

By looking at the features of these collections, we can use unsupervised learning algorithms to group decks into archetypes. This will give us a clear picture of the kinds of decks that you will likely face in a draft metagame, the key cards in those decks, and frequency of those decks being drafted.

Not only would this help all of us learn a ton about drafting, we could start exploring draft bot “personalities.” Maybe one bot at your table is leaning towards drafting something aggressive, while another bot just can’t wait to Cancel every bomb you draft. These personalities could very well mimic the same drafting preferences you see from people at your local game store.

Final Thoughts

One thing that really surprised me was the shapes of the formats — even in the more meaningful 3D visualizations. RIX looks incredibly different than the more “normal” M19 and Dominaria formats. Perhaps by looking at these shapes and patterns, we can quickly figure out some important or valuable points about approaching a format. As as we get more data from different sets, more patterns should emerge.

This is such a treasure trove of data that I’m sure there are many ways we could use this methodology to learn more about Magic and about its different draft formats. Let us know what other awesome things you think we could do or what else you’d like to see measured and analyzed!

If you enjoyed the read, be sure to follow Draftsim on Twitter or Facebook for more. And why not give the draft simulator a try?