Draftsim - Draft and Sealed Analysis and Strategy

Draft Data Analysis 2: How Much Does a Format Change in One Month? [M19]

September 17, 2018 By Dan Troha 1 Comment

By Dan Troha, Arseny Khakhalin, and Bobby Mills

As we finally say goodbye to M19 and welcome in Guilds of Ravnica (appearing on Draftsim.com this Wednesday), our team at Draftsim wanted to take a final look at Magic Core 2019 to see what we can learn from this boatload of data we collected.

This time around we’re going to look at how M19 has evolved since July. Have you ever been able to see an actual list of cards that you over/underrated in draft? We’ll be going through that and a lot more. I have to admit, I was amazed how much the data resonated with my own understanding of the format as time passed.

Plus, there will of course be many pretty new visualizations.

M19 – One Last Time

matrix graphic — It’s time to re-enter The Matrix (matrix math, that is)

Just to recap from our previous article, we visually represented the past three formats with a massive statistical analysis. We collected the picks from over 100,000 M19 drafts in July, then another 50,000+ in August. Then we slice and diced the results in a bunch of different ways, but we mainly looked for pairwise synergies of different cards to find out how they clustered together when people repeatedly picked them for the same decks.

And that analysis produced some sweet plots that looked like these:

2D plot with dots representing cards — M19 August 2018

Note: The units on the axes do not mean anything. They are only there to show the relative relationships between points. For a further explanation, please check out our last article.

We had quite a few people ask us for the data — which was really exciting, but also was kind of overwhelming. It’s clear that the Magic community is awesome and is just as nerdy as we are! We’re planning on releasing this data eventually, but we need to do a lot more work and analysis with it before we can turn it loose on the internet. So stay tuned on that front.

We also received several requests and suggestions for interactive visualizations, and I’m happy to say they are here! Here’s a D3.js version of the 2D visualization above (click on the image):

m19 interactive visualization screenshot — Click here to play with the interactive visualization

But enough of this snapshot stuff. This time around, we’re going to look at … ch-ch-changes.

With a data set this large, we are able to look at relative changes between July and August in card-by-card pick orders and even in overall color pair preferences. For example, we might be able to make guesses at questions like “Which colors became more popular in August?” or “Which archetypes fell out of favor?”

And how could you adjust your drafting strategy if you knew this?

Format Migration – “Flocking”

Let’s first start with the graphic that we teased in the last article:

m19 shown with before and after card movements — “Flocking” of relative changes in card preferences. Click here for a labeled version

The plot above shows every card in the M19 draft format and where it stood before August and after. The large squares (the “heavy heads”) represent the new position, while smaller “tails” represent where each card started in July.

To build this plot we projected July and August drafts onto a 2D picture, and then found a rotation of one picture that matched the positions of cards in the second picture as well as possible (on average). Then we looked at the relative movement of each card within the full cloud.

The most striking thing is that cards mostly seem to move in big flocks! While there are some dramatic individual card changes, it really looks like most of the movement comes from overall changes in the color, not just specific cards here and there. Why would that be?

Overall, it could be that archetypes are becoming more clearly defined and the colors are becoming slightly more polarized. As a format develops during the initial weeks, everyone goes through a learning curve in an attempt to discover what each color combination is trying to do, which color combinations support good archetypes, and which colors are underwhelming when paired with other colors. The flocking shows the format becoming less homogeneous and more clearly defined.

For example, take a holistic look at the color green. The majority of cards appear to be moving in the opposite direction away from the black/white cluster. If you think about what archetypes are represented here — black/green nothing and white/green auras, it certainly could give us some clues.

It is reasonable that players would be less inclined to pair green with black because the deck has somewhat of a bad rap for having no synergy — and therefore is harder to “make work” in a draft. As for white, GW’s primary strength and game plan relies upon an auras theme that takes very specific cards to build. Although green is often trying to ramp in M19, a GW deck is much less interested in Colossal Dreadmaw, Elvish Rejuvenator, or even Druid of the Cowl.

Red, on the other hand, appears to be biasing away from green and slightly toward the white/black cluster. This also would indicate GR losing some popularity (one fun exception is that Onakke Ogre took a hard turn up toward GR). As we’ll see in our archetype and individual card analysis, RW certainly got more popular, and you can see Trumpet Blast, Dark-Dweller Oracle, and Goblin Instigator moving in this direction.

Note: Let me take a moment to say that I also updated the ratings in the system a couple times during this period, which affects which cards are presented to people and the order they are drafted. Not only would the bots take the cards/colors that I bumped up higher, players using the “Suggestions” function of Draftsim would also incorporate the same biases.

screenshot of draftsim's suggestion engine — Draftsim’s AI provides pick suggestions to users — which probably influences some picks

Color Preferences Changed… A Little

Okay, so now that we’ve taken a look at the format as a whole, let’s drill down a little bit and look at each color individually compared to its counterparts.

The tiles below show the share of decks drafted in each pair of colors. The x-axis, “Color 1,” represents the primary deck color — the color with the most cards drafted at the end of the draft. The y-axis represents the color with the second-highest number of cards in the drafter’s final pool.

july 2018 color distribution chart — July 2018 M19 color distribution

In the first pass at the format, we see a strong preference for UW, BW, UB, and GR. Remember, the second color here is not necessarily the color that was drafted second, but rather the color that was second in size in the final draft pile.

UW artifacts and BW lifegain in particular seemed to be pretty obvious synergies right out of the gate. The gold cards told you what to do, there were strong-looking uncommons that supported those themes, and plenty of commons available that acted as enablers. So it’s not too surprising people would start by drafting these decks.

In July, the most popular draft combination contained lots of blue cards and a handful of white cards. But overall, people were drafting most of the color pairs quite diversely. Only WG got no love.

Now let’s compare that to August…

august color data chart — August 2018 M19 color distribution

It seems that after a month of practice, fine-tuning, and mulling over the metagame, drafting strategies started to pull further toward a handful of popular color pairs. BW stayed dominant, RW joined it, while UW, UB, and GR shifted around and became less focused on their primary colors.

Essentially, in August drafters converged on a stronger preference:

1. Draft black, red, or white
2. If the first color isn’t white, draft white for support. A whopping 27% of decks had white as the secondary color. Look at how dark the bottom row on the visualization is!

A special note about white in M19

White was heavily represented as a secondary color. If I had to bet, I would guess this was because of increased discussion in the community about how great white’s commons were, which made people a little more willing to look at white cards for their decks.

Not only does white have a premium removal spell at common in Luminous Bonds, it also has three very strong creatures headlining its slate of top commons. Each of these cards effectively puts the screws on an opponent, and together they are almost unbeatable in their uninteractivity.

Credit to Lords of Limited for dubbing this band the ~~“Three Amigos.”~~ (edit: Three Musketeers, thanks guys).

Hot Decks, Cold Decks

Now let’s give our flocking chart another look, comparing it to the heatmap of changes in color pair popularity, shown on the right.

Here red cells show pairs that became more popular, and blue cells show those that were now drafted less often.

If you let your eye drift up the column for each primary color, you can see that as time passed, Draftsim users really tried to force white, black, and red as primary colors, while they practiced drafting blue and green a bit less frequently. And also, they became increasingly interested in drafting both WR and RW combinations.

Do these observations match the flocking picture? Sort of. The green flock moved away from the other colors, as primarily green decks became less popular overall. Both WR and RW became more popular, so white flock moved a bit towards the red flock.

But overall, the flocking chart does not provide crystal-clear results. Therefore, it might be a little less useful than we hoped. Large-scale movements of colors obscure the individual changes in card-to-card synergies (if these changes exist), and even large-scale movements are hard to interpret. For example, what’s the deal with some of the seemingly-conflicting rearrangements of the blue swarm? Tezzeret only knows.

Still, we tried to look into the movement of individual cards (click here for a full 2000×2000 px version). It’s pretty obvious from the picture that multicolor cards (magenta) were the ones to move the most, which is fun, but reasoning for the movement isn’t obvious.

For example, a whole bunch of cards in the upper left quadrant moved towards green, including Draconic Disciple, Vaevictis Asmadi, the Dire, and Poison-Tip Archer, perhaps implying that you need to be more green than the other color to make use of these cards. This is a phenomenon that you see sometimes in draft — because of the mana or what a particular strategy is trying to do, a gold card can be better in a deck that has a stronger base in a certain color.

Brawl-Bash Ogre and Heroic Reinforcements each move in the same direction as their black and white flocks respectively, while staying equidistant from red, which makes sense.

Ratings Changes

Now that we’ve discussed final color configurations and changes in card synergies, let’s move on to individual card ranks.

Were there any changes in pick order in August compared to July?

Of course there were some changes, but they weren’t overwhelming. Exciting cards were still picked early in the draft (lower left corner), while weak and weird cards were picked late (top right corner). Still there was some curious reshuffling of individual picks, and with our sample sizes (from 1031 for the rarest card to 24733 for most common) we hoped to suss out some real changes, even if they were subtle.

We looked at the top 20% of cards that either lost or gained popularity between July and August. Here’s the data for the most volatile cards, with more popular cards (those that were picked early) shown closer to the top, and less popular cards shown at the bottom. Generally, these popularity rankings correspond to the power level of the card (but not always).

Lines show how much the rank of each card changed. If a card became less popular, the line is blue. If a card’s popularity increased, then the line is red. The intensity of each line represents the absolute change in popularity.

You can read the changes from left to right, where the card name represents the ranking in August after the changes.

So many cards, so many stories to tell. I’m going to dive in by categorizing these in three groups: “subtly powerful cards,” “weird cards,” and “synergy cards.”

Subtle power and format context

By now everyone knows how good Dryad Greenseeker is. But that wasn’t completely obvious at the beginning of the format. It was slept on a little until people got to see it for the card-advantage-machine-with-good-stats that it really is — and now it’s considered a “mythic uncommon.”

The same goes for Departed Deckhand. In a format without many good two-drops, this was one of the absolute best. You could play it in almost any deck, it was good on turn 2, and great on turn 8. The fact that you could get such an impactful two-mana card made it very important to prioritize because there just aren’t that many cards in the format that can provide that amount of impact for the price.

And if there aren’t many good twos, you shouldn’t be putting the bad twos in your deck. But then how do you not die to your opponent’s bad two drops? Essence Scatter quickly rose in stock in M19 because it was a great early play, but unlike a 2/1 for 2, it’s also great on turn 5-6 when you can nab a Thornhide Wolves or Colossal Dreadmaw.

Reclamation Sage experienced the biggest jump, and this is one I’m not completely sure about. Certainly between the existence of a WU artifacts deck, some powerful enchantment-based removal, and gamebreaking bombs (Patient Rebuilding, Sarkhan’s Unsealing) it is quite maindeckable. But there isn’t a huge reason for it to be so relatively underrated initially.

Finally I have to give a shout out to one of my favorites, Vampire Neonate, which had to be one of the most annoying cards to play against in the format. It blanked many of the aggressive two drops and allowed control decks and slower midrange decks to climb back into the game after falling behind. And it was a good mana sync. Not to mention having “the combo” with Ajani’s Pridemate. All for an unassuming one drop 0/3.

Weird cards

Transmogrifying Wand kills some stuff and makes 2/4 oxen. That’s not amazing, right? But wow was this thing powerful. It turns out that if you’re a control deck, or playing against someone who has green creatures in their deck, the card would dominate the game and 3-for-1 your opponent by neutralizing their best threats or even let you upgrade your own creatures.

Inferno Hellion initially looks like a pretty bad card, but there are just enough weird things you can do with it like Thudding or Sarkhan’s Unsealing that it turned out to be quite a bit more playable.

Synergy cards

The presence of a card like Elvish Clancaller (or Goblin Trashmaster) makes you assume there are goblin and elf decks in the format. Unfortunately, M19 did not have these, so they turned out to be much worse in context. Same for Desecrated Tomb — it looks like the card should have ways to trigger it, but there just were not many options in M19 limited, and so it proved to be a very narrow — basically unplayable — card.

On the upswing, you can see that Angel of the Dawn and Heroic Reinforcements jumped to near the top. Word got out (beginning probably with Ben Stark) about how good RW was, and these cards that are a tremendous payoff for a go-wide strategy as it rose in stock.

Concluding Thoughts

You hear a lot about the collective Magic hive mind working to “solve” a standard environment, but the same scale of testing and information sharing goes on with limited. You can see that people combine their own experiences with those of influencers in the community and change their behavior accordingly.

Speaking of, during the first few weeks of a format, I love to hang out on Twitter and look through 3-0 draft decklists to see what’s working for people. Feel free to tweet at me if you discover something cool (or super uncool) about a card or archetype.

Like all of you, I can’t wait for Guilds of Ravnica to drop soon. This is going to mean a whole shiny new data set and some pretty new visualizations of a completely different and guild-based format. I can’t wait to see what a guild synergy clustering visualization looks like…

Oh yeah, and I guess there’ll be a sweet new Magic set to actually play with too!

Be sure to follow Draftsim on Twitter and Facebook to be notified when GRN goes live and when we post more in-depth articles like this.

Draft Data Analysis: What We Learned from Analyzing Over 200,000 Draftsim Drafts

September 3, 2018 By Dan Troha 18 Comments

Hi! I’m Dan, and I run Draftsim, a website that helps people practice by drafting in a simulated environment against seven “bots” programmed with artificial intelligence. Along with some key collaborators, I’ve been working on a big project this year that I’m excited to finally share with you.

Have you ever wondered what a draft format looks like in one picture? Or what makes a format really “synergy dependent”? You’re in luck!

This past year we’ve been collecting anonymous data from the thousands of drafts that occur each day on Draftsim. It’s finally time to take a look at what we can learn from all that sweet, sweet data.

I owe all the credit for this project to Arseny Khakhalin (thank you Reddit for helping us connect!) and Bobby Mills from Bard College in New York. They helped me collect all the data on the site, do the analysis, and create these awesome visualizations.

Methodology

How much data are we talking about? Well, a lot.

Here’s how many human drafts we analyzed for each format:

M19 Part I – 108,000 drafts (July 2018)
M19 Part II – 48,000 drafts (August 2018)
DOM – 51,000 drafts (April 2018)
RIX (including XLN) – 10,000 drafts (February 2018)

In all, that’s actually over 200,000 drafts!

We simply captured the format being drafted, the cards presented to the user in the pack, and the card the user picked given his or her current card pool.

Animated gif of someone drafting M19 — We recorded thousands upon thousands of anonymous user picks

For each draft, we looked at the final pile of cards a user ended up with, and dutifully marked which cards were drafted together in a table that reflected all possible “card pairs.” After going through thousands of drafts, we compared these co-occurrences of cards in final draft pools to numbers that you could expect to get by chance, as if all drafts were completely random. Obviously, some pairs of cards were drafted together way more often than you’d expect by random chance, while some cards were drafted together less often.

At this point we used a standard statistical technique called multidimensional scaling to visualize the allegiances of different cards in a nice plot. To do so, we imagined that each card is represented by a point, and distances between these points depend on how often they are drafted together. Cards with a synergy were assigned small distances between them, while cards that are only drafted together by chance were assigned large distances.

We then utilized a function in the statistical computing language R called cmdscale (the whole analysis was performed in R), and asked it to find an arrangement of points in a 2D or 3D space that would achieve these predefined pairwise distances as well as it possibly could. The function finds an arrangement of points, and then we visualize it either with ggplot (for 2D plots), or with rgl package (for 3D), to get the pictures you’ll see below.

Results and Analysis

So enough with the talk about how we got there – let’s see what we found out! We’ll start with the recent M19 data, mostly from August.

Visualizing M19

On this plot, each point represents a card. Red cards are red, green ones are green, and so on, except that the white cards are gold colored. Colorless and artifact cards are gray. All multicolored cards, regardless of guild, are shown in purple for simplicity’s sake.

If two cards are often drafted together, they are shown close to each other on this plot. If the cards are almost never drafted together, they will be very far from each other on the plot.

Note also that the X and Y axes are not really interpretable; they are just a way to plot the points, but there is no deep meaning to them.

If you’re wondering which card each dot represents, a similar picture that has each point labeled can be found here.

As you can see, it is not easy to represent a complex drafting landscape in a 2D image, so a card may be in the center both if it is drafted with any color (like most artifact cards), and also if it is drafted in decks of two opposing colors (like Gruul or Izzet on this plot). But at least you can spot things that are suspicious and investigate them.

Plotting in 3D might be more informative, but the trouble with 3D plots is that you still need to show them on 2D screens, which really limits their utility. Here’s a 3D projection of M19 drafts: pretty, but hard to use.

a rotating cube with x y z axes depicted relative M19 card positions — M19 live and in 3D

Observations

The first thing we can see here is that cards of the same color are drafted together, which seems obvious. But even then, exceptions prove the rule, and we have quite a few interesting exceptions.

For one, the black and white cards (represented by golden dots) don’t form distinct clusters, which hints that in M19 drafts, players tend to pair W with B more often than with any other color, and vice versa.

The reality is a bit more complicated: white and black cards actually do form distinct clusters, but these clusters are closer to each other than any other two color clusters. So when the multidimensional scaling tried to show all cards on a 2D picture, and looked for some complexity to sacrifice, it sort of folded them together. But if you look at the 3D graph earlier, you’ll see that the white and black clusters are actually distinct. This is an artifact of the 2D visualization method, but it is an artifact that appeared for a reason.

We can also hunt for individual points that don’t belong to the cloud. See that red dot on the very left, far from the main red cluster? That’s Sarkhan’s Unsealing, a card that benefits from large, beefy creatures, and even though it is strictly red, players treated it as a Gruul (RG) card. Or look at that green card that is closer to artifact cards than to other green ones: that’s Scapeshift — which is completely unplayable, however some Draftsim users like to raredraft, so it was drafted as if it were color agnostic.

And while we’re on the subject of unplayables, you’ll also find Omniscience and Mistcaller sitting in the middle since they wouldn’t reasonably be included in any M19 draft deck either.

What about that lonely white card on the far right? It is Aethershield Artificer, which pumps artifact creatures, making it an obvious synergy payoff for the blue/white Artifacts Matter deck.

Enough about relationships, what about power?

We can also mine draft results for other, more straightforward types of information. For example, which cards are the absolute winners that are always picked first? Here’s the top five list:

This is interesting. While these are mostly great pack-one-pick-ones, they’re certainly not what I have rated as the top five cards in the set. Sarkhan especially is strange, but people love planeswalkers and the card is deceptively worse than it looks.

And what cards don’t see any love from drafters? Here are the bottom five, most frequently picked last (so shown backwards, from worst to slightly less worse):

Apparently, “tower” cards are bad.

RIX/XLN visualization

Do all draft formats have similar shapes? Not at all, and of course the Ixalan and Rivals of Ixalan formats are special here, because their synergies are based largely not only on color pairs, but also on tribes that are in those colors. So the plot ends up looking completely different:

Click here for version with cards labeled

Here we have Vampires on the left (with Anointed Deacon and multicolored Legion Lieutenant leading the charge), Merfolk on the right, and the red guys on the top — Pirates and Dinosaurs alike. What is the half-ring of multicolored cards surrounding the red cloud? Pirates are on the left (leaning towards the black Pirates), and Dinosaurs on the right (towards green, and to a lesser extent, white creatures).

Look how closely WB Vampires and UG Merfolk are clustered. These are the two tribes in the set that were represented by only two colors, so they end up grouped very closely together.

Contrast that with Grixis Pirates (B/R/U) and to Naya Dinosaurs (G/W/R). The relationships are much more spread apart and less tightly correlated than the two-color tribes.

Red is in a special place because the color “belongs” to neither Vampires nor Merfolk. When you pick a red card, assuming an equal archetype distribution, it has an equal chance of ending up in a tribal deck of any of the four colors — either a Pirate deck (RB or UR) or a Dinosaur deck (GR or RW). In contrast, it’s not so with black, where you can end up in Pirates (RB or UB) or Vampires (WB) only, if you stick to the prescribed tribal archetypes and groupings of synergy.

In XLN/RIX, GB and UW were “tribeless,” making their synergies — if they existed at all — much less pronounced. It makes sense that we don’t see them bumping up against each other here.

DOM visualization

Click here for version with cards labeled

Dominaria seems to have the most separation between colors of all three. Perhaps this has to do with the high individual power level of many of the cards in the set. Moreover, in DOM, artifacts do not form an amorphous cloud in the center, but make a cluster of their own, very close to U/W/B, which happen to be the colors that care much more about historic triggers.

Not too surprisingly, the blue cards that pull closer to red are Wizards synergy cards (eg Naban, Dean of Iteration, Vodalian Arcanist, and Wizard’s Retort), and the ones that are closer to white are either artifact/historic payoffs or enablers (The Antiquities War, Artificer’s Assistant, Zahid, Djinn of the Lamp, Karn’s Temporal Sundering, Sentinel of the Pearl Trident).

I’m not sure how to explain why white is so closely clustered together versus red, however. This possibly indicates a replacability/lack of synergy among white’s cards or perhaps a tendency of the white cards to only be drafted with each other.

Bonus: Just for fun, here are the other two 3D visualizations that we generated for RIX and DOM:

a rotating cube with x y z axes depicted relative RIX/XLN card positions — RIX 3D visualization

a rotating cube with x y z axes depicted relative DOM card positions — DOM 3D visualization

Where To Go From Here? Some Exciting Future Possibilities…

So this is all really cool, but what can we do with it?

Training Draftsim’s AI to use this data and capture synergy

Synergy in Magic is a very difficult thing to measure because it shifts from format to format. In one block, R&D may introduce a new mechanic that explicitly drives synergy (such as historic, kicker, or cycling), but another might primarily use tribes, such as in Ixalan. And let’s not even get started with cards that are “better for aggro decks” or “control” decks. After all, most decks in limited are just some flavor of midrange, right?

Rather than individually scoring or programming all these factors for each set, it would make a lot of sense to enlist Draftsim’s wonderful user base to help train the algorithm to recognize synergistic pairings of cards.

We could then in turn train the AI to make more realistic decisions based on the clusters of cards that are frequently drafted together.

Encapsulate the shifting “draft metagame”

It would be fascinating to track how cards and color pairs become progressively more or less overvalued during the course of the format. You might see certain color pairs move around as archetypes go in and out of vogue or one color gets recognized as “bad.” And, like in constructed, a draft metagame seems like it would eventually settle somewhere.

And wouldn’t it be really fun to be able to quantitatively measure what cards people “slept on” at the beginning of a format? Or to assign archetypes and see how our understanding of what cards belong in them changes? You could then use this information to improve your card and archetype evaluation methods.

In fact, this leads into our next article where we’ll do a quantitative analysis of how much a format changes over the course of a month. Here’s a little taste of what we’ll be looking at:

That’s right – we have before and after data that we can use to measure changes. Double the data means double the fun!

double rainbow guy saying double data — Stay tuned to find out…

Clustering decks instead of cards

This whole article we’ve been looking at defining a format by the relationships between individual cards. However, we can also zoom out and examine the the end product of your draft as well: your final collection of drafted cards.

By looking at the features of these collections, we can use unsupervised learning algorithms to group decks into archetypes. This will give us a clear picture of the kinds of decks that you will likely face in a draft metagame, the key cards in those decks, and frequency of those decks being drafted.

Not only would this help all of us learn a ton about drafting, we could start exploring draft bot “personalities.” Maybe one bot at your table is leaning towards drafting something aggressive, while another bot just can’t wait to Cancel every bomb you draft. These personalities could very well mimic the same drafting preferences you see from people at your local game store.

Final Thoughts

One thing that really surprised me was the shapes of the formats — even in the more meaningful 3D visualizations. RIX looks incredibly different than the more “normal” M19 and Dominaria formats. Perhaps by looking at these shapes and patterns, we can quickly figure out some important or valuable points about approaching a format. As as we get more data from different sets, more patterns should emerge.

This is such a treasure trove of data that I’m sure there are many ways we could use this methodology to learn more about Magic and about its different draft formats. Let us know what other awesome things you think we could do or what else you’d like to see measured and analyzed!

If you enjoyed the read, be sure to follow Draftsim on Twitter or Facebook for more. And why not give the draft simulator a try?