ralphmelton: (apple)
[personal profile] ralphmelton
In February, I posted about my preliminary results with Virulent Strain. After 100 games with four players and five Epidemics with Virulent Strain, my win rate was nearly the same as without Virulent Strain. Was this a lucky streak, or a sign of something else?

I've now played 500 games of 4P5EV. My win rate for 500 games has been 84.0% +/- 3.2%; my win rate for 586 games of 4P5E without Virulent Strain was 84.1% +/- 3.0%.
Some times I look at stats and think that with some more games to narrow the confidence intervals, it might develop into a statistically significant difference. This is not one of those times. To my eye, these stats give no basis for any belief that I win less often with Virulent Strain than without.

But it's obvious that Virulent Strain is harder. (Obvious does not necessarily mean true.) The Virulent Epidemics add ways to fail and don't add ways to succeed.

So maybe Virulent Strain turns easy games into hard games? Not really.
Percentage of easy games: 37.5% +/- 3.9% without Virulent Strain vs. 39.2% +/- 4.3% with Virulent Strain.

Maybe Virulent Strain makes it harder to get Eradications?
For Eradications per successful game: 0.52 +/- 0.06 for 4P5E vs. 0.83 +/- 0.08 for 4P5EV. That's a statistically significant difference of more Eradications with Virulent Strain.

Down at the nigh-anecdotal end of the scale, I got 4 four-Eradication games in 500 games of 4P5EV, compared to none in 586 games of 4P5E.

In my last post about Virulent Strain, I described three possible explanations:
1. I've been having a lucky streak, and am likely to get slapped down soon. This is the most likely possibility.
2. Over the thousand-plus games I've played since my 4P5E games, I've honed my Pandemic skills a bit.
3. The actual increase in difficulty from Virulent Strain is much smaller than it seems.

The 'lucky streak' hypothesis is less plausible now after 500 games.

The 'I've improved my skills' hypothesis has some appeal. I'd like to think that I could improve my skills incrementally over a thousand games, and I can think of ways that I've improved my skills.

And there's a point in favor that Virulent Strain does cause me to lose some games, so I must be making it up somehow. Here's some additional data about how much difficulty Virulent Strain adds:
After the first 100 games, I started recording games in which I felt afterward that I had lost because of Virulent Strain. This is necessarily subjective, and some of the games that I lost because of Virulent Strain would have turned out to be losses even if I had not been playing with Virulent Strain. But with those qualifications, I recorded 21 games out of 400 lost due to Virulent Strain, or about 5.25% of those games. In 3 of those games, I recorded that I thought I would have lost anyway. But still, that suggests that Virulent Strain was the difference between failure and victory in at least 2 or 3% of my games.
But the final victory percentage was almost identical, so I must be making up those games - which seems to suggest an improved skill.

But here's the counterargument: if we hypothesize that I improved my Pandemic skills while playing 500 games of 3P5E and 700 games of 5P5E. It seems obvious (again, not the same as true) that if I improved while playing 5P5E, I would particularly improve at 5P5E. Although I have not done a rigorous analysis, I did not notice any improvement at my win rate with 5P5E over the course of those games - my overall win rate drifted slightly down as I played.

The data shows pretty clearly that with four players and five Epidemics, I win as often with Virulent Strain as without. It's very perplexing, and I don't have a good explanation.

Date: 2016-07-03 09:58 pm (UTC)
cellio: (gaming)
From: [personal profile] cellio
You've been continuing to play non-VS games alongside the VS games? That's what it sounds like, as opposed to your non-VS data being older than your VS data. (In that case you'd want to collect some new non-VS data to test the "improved player" hypothesis.)

Date: 2016-07-11 02:10 pm (UTC)
From: [identity profile] ralphmelton.livejournal.com
I was playing purely VS games. My 4-player 5-Epidemic non-VS games mostly came in two big batches:
Games 124-544, spanning Dec-13-2013 to Feb-18-2014
Games 950-1344, spanning Jun-8-2014 to Sep-22-2014
(There are both games in those ranges that weren't 4P5E and 4P5E games that weren't in those ranges.)

My Virulent Strain games started Jan-3-2016 (game 2002) and continued to Jun-6-2016 (game 2576).

I agree that collecting new non-VS data is an obvious way to test the "improved player" hypothesis. But it seems hard to prove that:
Since I recorded that in 18 of 400 games I lost due to Virulent Strain and wasn't obviously doomed otherwise, let's assume for discussion that I'm now 4.5% more likely to win 4P5E (non-VS) than I was when I played those games. Could I prove that?
If I played another 500 games, it would take me six months and I'd get a confidence interval of about 3% - too wide a confidence interval to prove that I've gotten better. Confidence intervals are proportional to 1/sqrt(sample size), so to narrow the confidence interval enough to prove a 4.5% improvement, I'd have to play about 2000 games, taking me two years.

I'm not absolutely opposed to doing that, but I'm not feeling very eager. So first I will engage in speculation about possible causes and enlist the help of statistician friends to see whether there's more information I can tease out of the data I have so far.


ralphmelton: (Default)

September 2016

45678 910

Most Popular Tags

Page Summary

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 20th, 2017 01:57 am
Powered by Dreamwidth Studios