Notes on new team difficulty settings
By
Abigail Frost,
Posted on: 2009-08-26 15:02 in The Market and Inventions
Quote:
Which teams size and difficulty settings were you using?|
Originally Posted by EarthWyrm
Bosses, no AVs:
|
Am I right in thinking that you used team size 8 for your first two tests? (The one's with 2484 and 617 defeats)
Quote:
Sorry about that. My most recent tests have all been +1/8, since I get the most data fastest that way. The 2484 was a combination of +1/8 and +1/1, and I believe (I'd have to go back through the thread and check) that the 617 was the +1/1 part of that.
|
Originally Posted by Archie Gremlin
Which teams size and difficulty settings were you using?
Am I right in thinking that you used team size 8 for your first two tests? (The one's with 2484 and 617 defeats) |
My postings to this forum are not to be used as data in any research study without my express written consent.
Summary so far:-
434 recipes dropped by 24543 minion equivalents giving a drop rate of 1.77% with a standard error of 0.08 and a 95% confidence interval of between 1.61% and 1.94%.
1513 salvage dropped by 20021 minion equivalents giving a drop rate of 7.56% with a 95% confidence interval of between 7.19% and 7.93%.
3 more runs: 1134 enemies (918/216)
Expected recipes drop: 36 (28/7/1)
Effective recipes drop: 19 (14/2/3) = 53% of expected
Expected salvages drop: 97 (76/18/3)
Effective salvages drop: 83 (66/14/3) = 86% of expected
Total for me;
3780 ennemies (3060min, 720 lieut)
Expected recipes drop: 120
Effective recipes drop: 88 = 73.3% of expected
Expected salvages drop: 288
Effective salvages drop: 284 = 98.6% of expected
I think I may have found a pattern in the data! 
One of the things that's been bothering me is the wide variety of drop rates being reported. If we were looking at a single drop rate then plotting the rates for each run should give a bell shaped curve. It doesn't. What we actually get is a flat line between 0.5% and about 3%. This strongly suggests that we're getting different drop rates under different circumstances. (As we all suspected.)
I grouped our runs by team size. As most of our runs used sizes 0,1, 6 or 8 I created three groups. 0 and 1, 5 and 6 and 8. I excluded the runs I did on the wall in Cimerora as I don't know what team size applies in that case.
Code:
Notice that the confidence intervals for the three groups don't overlap. I'm not sure how to test this for significance but it certainly looks significant from the confidence intervals.Size Defeats Recipes Drop Rate 95% Interval =================================== 8 4583 35 0.76% 0.53% - 1.06% 5,6 10915 201 1.84% 1.60% - 2.11% 0,1 5805 159 2.74% 2.33% - 3.19%
I've also noticed that these drop rates are quite close to 1/3, 2/3 and 100% of the old drop rate. (0.89%, 1.78% and 2.67% respectively.)
One note of caution here is that that team size correlates with other factors such as difficulty setting and map choice.
(Edited to get the team sizes in the right order. Doh!)
I definately appreciate the work being put in by the folks in this thread. Great job!
Question for the statistically inclined: At what point do we have enough data?
Without crunching the numbers and just reading through the thread I am convinced that recipe drops have dropped from approximately 20% of live and that it appears most of that is attributable to Lts.
Well, I take that back . . . a bit at least. No one, to my knowledge, has compiled boss drop information. So bosses may or may not also be a problem.
50s: Inv/SS PB Emp/Dark Grav/FF DM/Regen TA/A Sonic/Elec MA/Regen Fire/Kin Sonic/Rad Ice/Kin Crab Fire/Cold NW Merc/Dark Emp/Sonic Rad/Psy Emp/Ice WP/DB FA/SM
Overlord of Dream Team and Nightmare Squad
Quote:
What team size and difficulty settings were you using Beber?
|
Originally Posted by Beber
3 more runs: 1134 enemies (918/216)
|
Team size = 6 (always, with no bosses)
I change difficulty settings each time:
First run = +0 ----->11 recipes
Second run = +1 -> 5 recipes
Third run = +2 -----> 3 recipes
Quote:
If you look at the spread in the confidence intervals in my last post, it's still pretty wide in terms of percentage of the presumed live drop rates. The minion value ranges from about 50% to about 120% of the presumed live probability, for example. This makes sense, because the presumed "real" values are quite small - 2.333% chance for a minion - so small uncertainties represent a big chunk of a small "real" value.|
Originally Posted by Bionic_Flea
Question for the statistically inclined: At what point do we have enough data?
|
That's for about 1100 minions. I'd like to shrink the confidence interval by a factor of 2 or so, which (assuming I correctly understand how to estimate this) is going to take about 4x as many samples.
Quote:
| Well, I take that back . . . a bit at least. No one, to my knowledge, has compiled boss drop information. So bosses may or may not also be a problem. |
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA
Quote:
Archie's breakdown above is, to me, a sufficient summary of the testing we've all been doing to present a compelling picture of what's wrong. Given that his analyses can be replicated by the Dev team by datamining everything that's happened across the various stages of I16 beta, I am comfortable concluding that some unintended change has occurred in drop rates that seems to be related to the virtual teammate slider. |
Originally Posted by Bionic_Flea
Question for the statistically inclined: At what point do we have enough data?
|
Put more succinctly, over 20k observations is way more than enough for me to be confident that the patterns are incredibly unlikely to be the result of random chance.
I'm hopeful that whatever the issue is can be identified and dealt with prior to I16 going live. Given the somewhat questionable decision on the part of the competition to engage in large-scale nerfing on their launch day, I don't think the CoX team should feel any compulsion to push I16 live until it's as close to perfect as possible. Another couple of weeks, hopefully, should get this last major bug squashed.
Edit: I just mentally kicked myself. The one character I used who got higher drops than my brute was my dom, who I ran at -1/8. I should drop more characters down to 0 or -1, instead of the +1 I've used consistently on the brute.
My postings to this forum are not to be used as data in any research study without my express written consent.
You know I was thinking about this the other day but didn't want to mention it at the time but the I16 overview describes this as:
Quote:
Just because the map is going to be "populated for 8" doesn't mean they're gonna do squa-doosh (aka anything at all) to a solo scrapper's - "populated for 8" drop-rate. "...control over the level they are detected to be when it comes to the dynamic spawn system." Doesn't necessarily mean the drop rate system is gonna change, but the Devs should know darn well - that's exactly what people are expecting/wanting from this new system!|
Enhanced Difficulty Options - Players can now fine-tune their difficulty levels by telling the game exactly how they want to be treated. They can choose to be regarded as a specific size team (even if they are soloing), and even have control over the level they are detected to be when it comes to the dynamic spawn system. So if you ever wanted to see if your Scrapper could take on a map that is populated for an 8 person team, four levels higher than your own, now you can! The difficulty settings of the mission owner dictate how the mission is handled, just as before, and you can change your difficulty settings at any Hero Corp.'s Representative in Paragon City or Fateweaver in the Rogue Isles. |
Let's assume for a second that's this is the case - I'm thinking that when people go to complain about it, they would get a response along the lines of:
"Oh... We thought you just wanted the "challenge" of a "map spawned for 8" - and we gave it to you! Surely, you didn't REALLY THINK - we were gonna give you the drop rate of a "padded for 8 map" to go along with it, Hah, yeah right!
Oh. and good luck asking for "fillers" to "pad" your maps now, because they'll all think "you don't need fillers anymore", HAHAHA!
"That's just the Devil's Advocate in me talking there - above.
I don't honestly think they would try to pull any shenanigans like that right now.
As for the perceived drop rate discrepancy, it might have something to do with the way they're trying to handle the - "what happens to a team where everyone is running at "set for 8" (and possibly on different difficulty settings).
I read something about this - that in this situation it would set the spawns based on the actual number of people on the map... or something to that effect - I'm not sure which set of "Build Notes" or Dev Digest post I read that in - but something like that was/is in there somewhere.
IIRC, it was in answer to one of the most asked questions, "What happens if 8 people "set for 8" - and what if they're at different difficulty slider settings?", Etc.
I applaud your efforts so far, given the Pre-download that happened yesterday - I hope Synapse (or whomever) can figure out what's up with the drop rates ASAP. The other thought I had about this was - when Synapse recently replied to this and (essentially) said, "the drop rate code hasn't changed in I16".
I was thinking - maybe it slipped in sometime before? - maybe during I15 or one of the patches or something??
Heck IDK, it's just some food for thought - so to speak. Something to tide you over - while you're busy banging your heads on the walls of the Cargo ship map, *shrug*
Quote:
|
Originally Posted by Arcanaville
City of Heroes didn't fail, City of Heroes was killed. If a 747 dropped on your house, you'd say you were killed, not you failed to find a safer dwelling.
|
Quote:
In regards to this, the drop rate on live has also been checked, and is in line with the older data. Any difference that's showing up is a difference between the current live build and the build on test.
|
Originally Posted by Cardiff_Giant
The other thought I had about this was - when Synapse recently replied to this and (essentially) said, "the drop rate code hasn't changed in I16".
I was thinking - maybe it slipped in sometime before? - maybe during I15 or one of the patches or something?? |
Arc#314490: Zombie Ninja Pirates!
Defiant @Grouchybeast
Death is part of my attack chain.
Can anyone who build a log parser find a place to host it? I'd love to use it to continue to gather more testing date, but don't know enough to build one myself.
119088 - Outcasts Overcharged. Heroic.
Quote:
You can find mine here, but it's kind of a hack job. You need a Python interpreter that can run Python 2.5.x. (You can find one for Windows here.) If you know what you're doing, it might work under Python 2.6, but I haven't tried, and it will probably complain of deprecated language usages.|
Originally Posted by Lakanna
Can anyone who build a log parser find a place to host it? I'd love to use it to continue to gather more testing date, but don't know enough to build one myself.
|
I've done all my drop testing with Nemesis, and the parser is mildly hard-coded to recognize only Nemesis names. (It looks to see if the defeated critter was a boss or LT, and if it wasn't, it assumes it was a minion.) Since Python is fairly easy for someone with a bit of programming experience to read, it's not too daunting to expand the parser's horizons. The critter and salvage name lists are pretty obvious and wouldn't be hard to modify with a text editor. The rest of it should work pretty generically, as far as I know. The code isn't going to win any beauty contests, but hopefully it's not real buggy.
It's meant to be run from a command prompt, like so:
python dropParser.py file1 file2 ... fileN
It will read the files in the order provided and treat them as one long stream of data.
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA
Quote:
One reason I'm still bothering with testing this is that measured solo drop rates still did not match live values. They also did not match live values if one created an actual team and then soloed it. Both tests indicated that this was a universal change to drops, even if it wasn't consistent by team size.|
Originally Posted by Cardiff_Giant
Heck IDK, it's just some food for thought - so to speak. Something to tide you over - while you're busy banging your heads on the walls of the Cargo ship map, *shrug*
|
Another reason I'm still bothering with testing this is that we were explicitly told by pohsyb in closed beta that neither "virtual" team size nor foe level settings are supposed to affect drop rates. Above and beyond that, we were explicitly told by Synapse that there was no intended change to drops at all. That means that we need data to show the devs if the drop rates actually have changed.
Even if the devs did intentionally reduced drop rates for virtual teams, it should be in the patch notes. This is not an exploit fix. People like are posting in this thread would be basically wasting dozens of hours testing something that's working as intended. In fact, I posted several times on the closed beta forums how the devs really should let us know if there was any intended change here just on the basis of how much time would be needed to get good data. It took them a while, but they did get us that response.
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA
Quote:
I've tarted up my log parser and put it on the web. You can download it from http://www.glasspaw.com/dropstats/. There are full instructions on installing it and running it.|
Originally Posted by Lakanna
Can anyone who build a log parser find a place to host it? I'd love to use it to continue to gather more testing date, but don't know enough to build one myself.
|
It gives a pretty good summary of your drops including defeats, salvage and recipes. It splits defeats into bosses, lieutenants and minions which is handy if you're reporting to this thread.
It works pretty well if you're a lvl 50 and you're soloing. It has trouble counting defeats otherwise. (But it does get the drops right.)
It recognises the most important mobs but I haven't included all the mob groups yet so let me know if it has trouble with your favourites.
Any feedback would be very welcome. Enjoy.
I've stopped posting my results, but not because anything has changed. 4 more runs of the same mission, three with one character on a live instance and one with another character on a flashback instance, set for +0/8 or -1/8 (the single flashback instance), resulted in 0, 1, 2, and 2 recipes per run. Each map had the same proportion of lieutenants to minions as virtually every other instance of the map (total sample size per map = 344 = 268+76). Salvage continues to vacillate around expected values.
Thing to note: The spawn rules seem incredibly non-standard on this map. When I enabled bosses, I only got 8 total (breakdown 268/68/8, I think - it's further up the thread). 4 of them are hard-coded, and then each boss spawn added one more standard boss. None of the regular groups, spawned for 8, offered anything but minions and lieutenants, even when I enabled bosses.
My postings to this forum are not to be used as data in any research study without my express written consent.
Archie: if you have time, could you incorporate Carnies into the parser? I've done a few missions with them in it, and they're unknowns.
If you update the .pl file, all old data can be run through it again, right? you don't have to scrap old runs when you update the parser?
119088 - Outcasts Overcharged. Heroic.
Quote:
Ok. Carnies coming up. I'll add a few more high level groups while I'm at it.|
Originally Posted by Lakanna
Archie: if you have time, could you incorporate Carnies into the parser? I've done a few missions with them in it, and they're unknowns.
If you update the .pl file, all old data can be run through it again, right? you don't have to scrap old runs when you update the parser? |
Yes. You can run old log files through the new version. DropStats doesn't modify your log files in any way. (Or affect what goes into them in the first place.)
Quote:
I've put up a new version of DropStats. I believe this includes all mobs that you can encounter at lvl 50. It also tracks merits, Vanguard merits and tickets now.
|
Originally Posted by Lakanna
Archie: if you have time, could you incorporate Carnies into the parser?
|
I ran a Council paper mission a few times over the weekend on difficulty -1, team size 8 no bosses. I did a short CoT run first with bosses enabled which generated a huge proportion of bosses. That's why there's a few bosses in the results.
The results were:-
1,515 defeats including 11 bosses and 304 Lts.
36 Pool A recipes, 140 salvage and 16 enhancements.
This gives drop rates of 1.96% for recipes and 8.66% for salvage.
This suggests that my theory that team size 8 gives extra low results is wrong. Hey ho.
What was the min/lt/boss breakdown on that last run? Not too much data on bosses so far.
50s: Inv/SS PB Emp/Dark Grav/FF DM/Regen TA/A Sonic/Elec MA/Regen Fire/Kin Sonic/Rad Ice/Kin Crab Fire/Cold NW Merc/Dark Emp/Sonic Rad/Psy Emp/Ice WP/DB FA/SM
Overlord of Dream Team and Nightmare Squad
I've just re-run the same paper mission at difficulty +1/8. The results are much the same.
Total defeats 1,317 including 0 bosses and 265 lieutenants.
Recipes 27, salvage 123.
Drop rates recipes 1.71%, salvage 8.75%.
Both of these runs have been pretty close to the average drop rate reported here.
P.S. My last post includes the number of bosses and lts. The minions are the difference.
Quote:
Your results stand out as being much worse than virtually all the others. I'm beginning to wonder if this map has a particularly bad drop rate.|
Originally Posted by EarthWyrm
Thing to note: The spawn rules seem incredibly non-standard on this map.
|
Would you care to try a different map on difficulty 1 team size 8? I suspect that you'll get a much better drop rate.
I set about figuring out the confidence interval math for these trials, and the LT values definitely do look suspicious.
Edit: I switched from normal approximations to the math from the calculation page noted earlier.
Using the same data set I last pasted above, I now get this.
Defeats: Minion Defeats: 1099 LT Defeats: 354 Recipes: Minion Drops: 23, Rate: 2.093% (95% in range 1.331%-3.124%) (49.919%-117.142% of expected) LT Drops: 10, Rate: 2.825% (95% in range 1.363%-5.134%) (25.552%-96.254% of expected) Salvage: Minion Drops: 76, Rate: 6.915% (95% in range 5.487%-8.579%) (68.586%-107.243% of expected) LT Drops: 29, Rate: 8.192% (95% in range 5.555%-11.553%) (52.208%-108.584% of expected)Drop probability info on ParagonWiki falls within the 95% confidence interval for every calculation except the LT recipe rate, though it's close. More samples should help close up the ranges, though it unfortunately shrinks roughly with the square root of the sample size.
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA