Beta Testing: The weight of subjective feedback
By
7thCynic,
Posted on: 2009-07-01 10:35 in The Global Training Room Test Server
Well, all i have to say is I13 and PvP. They really don't care about it, if they sink enough time into it, even if its good or bad, they will ignore every1 and just say that it was a success.....
I suppose thats a good example.
My initial reaction is to say, well maybe others (the non-pvp mass majority) told the devs they liked it. But in thinking abt it, i doubt they were testing it much, especially compared to the hardcore PvPers.
So yeah, you're right.
The problem with subjective feedback boils down to, there is rarely a consensus in the reports on these topics.
In addition there is a self selection going on that people are more liely to post if the have a subjective issue, than if they dont. For every 1 subjective 'I dont like X because' there will be Y folsk who tried and simply dont give feedback because they like it the way it is.
@Catwhoorg "Rule of Three - Finale" Arc# 1984
@Mr Falkland Islands"A Nation Goes Rogue" Arc# 2369 "Toasters and Pop Tarts" Arc#116617
Excellent point. So then, how are the devs to figure it out? Based on how many are screaming? If so, then i13 PvP should have been a no-brainer, for as you suggest, those that may have liked it, probably didnt bother to say anything.
On the other end of the spectrum I am sure there were also plenty of players that did not like it and didn't give feedback either.
In all honesty a small portion of Cox players even frequent the forums. I hope the dev's read forum feedback but also have other tools that help them make decisions regarding the games design.The last four issues are turning this player into a skeptic though 
<QR>
I imagine that sometimes the subjective feedback of the players informs them of a situation they weren't fully aware of and they make changes accordingly. (Witness the massive amount of changes that went into the new SF text that may have only happened because of player feedback)
Other times, like the PvP changes, the subjective feedback might not have as much weight, because they were already expecting some people to not be happy with it. If you were already a "hardcore" PvP'er prior to I-13, well, that is because you liked PvP the way it was basically.
I think it was BaB's who had the basketball court analogy, if you show up with a group of people at a basketball court and then when you get there you find that it is full of potholes and that one of the baskets is missing a backboard, most of the people just up and leave. Some will stay and make a game out of what is there, it might not be basketball in the purest sense, but they might end up liking what they play quite a bit.
But that is no reason to not fix the court. And when you do it is inevitiable that the people who liked the broken court will complain. So, I would guess that going into the PvP changes, negative subjective feedback was to be expected and probably given less weight.
Otherwise, I've particpated in most of the Closed Betas since I-9 and in every singe one of them, without exception, changes have been made that were based completely on player feedback. It is simply untrue to say that the Devs don't listen to feedback, I've seen it with my own eyes too many times to believe that.
But isnt the new TF/SF a good example of possibly not listening enough? With it being the bulk of i15, I would think it would be quite imperitive that they got it right. Yeah maybe they didnt have enuff time or resources, but in the end, if you want people to love it, youll have to go back and make drastic changes regardless. Which is somthing Im not sure theyre willing/able to do. So were stuck with it like were stuck w/positron. Unless that is, despite the feedback and similarly to pvP, they dont think it needs the drastic change.
Everyone was running ITFs when it came out and still are. I dont really hear any calls for BSFs, and without many changes, I dont think I will.
[ QUOTE ]
The problem with subjective feedback boils down to, there is rarely a consensus in the reports on these topics.
[/ QUOTE ]
I can't think of one person who thought the new TF's storyline was good.
We ALL had consensus on this. It was ignored. We offered SO many suggestions, quick fixes and the like... ignored. I will be the first person to tell the Devs they did an excellent job (Invasion was such a winner), but I won't sugar-coat.
[ QUOTE ]
But isnt the new TF/SF a good example of possibly not listening enough? With it being the bulk of i15, I would think it would be quite imperitive that they got it right. Yeah maybe they didnt have enuff time or resources, but in the end, if you want people to love it, youll have to go back and make drastic changes regardless. Which is somthing Im not sure theyre willing/able to do. So were stuck with it like were stuck w/positron. Unless that is, despite the feedback and similarly to pvP, they dont think it needs the drastic change.
Everyone was running ITFs when it came out and still are. I dont really hear any calls for BSFs, and without many changes, I dont think I will.
[/ QUOTE ]
I'm not sure if you've seen Horatio's (new redname) post about the SF, but there are fixes coming down the line. If they had the option of waiting, I would have urged them to wait until those fixes got in, but there really is no way of knowing what other factors were in play that made them decide to release it as it was. And the fact is that most every TF/SF that has ever gone Live had bugs at release, that's just part of the deal.
And here I'll say that there is something about the SF that it seemed most players agreed should be changed that wasn't. That was the Temp Powers being tied to AT's. I made my feelings known about this aspect as did many others, and there is no denying that consensus opinion was that it was a bad idea.
But, I won't presume to think that the Devs didn't have many knockdown drag-out meetings about that very subject. I don't think tying powers to AT's in a SF is something that would be done lightly, so I wasn't completely surprised that they decided to go with that concept despite the player feedback. This is their full-time job, we're all part-timers at best, and our paycheck doesn't depend on the outcome of these decisions. So at the end of the day, I think it is fair to assume that most of the issues that players bring up have been brought up internally.
[ QUOTE ]
I can't think of one person who thought the new TF's storyline was good.
We ALL had consensus on this. It was ignored. We offered SO many suggestions, quick fixes and the like... ignored.
[/ QUOTE ]
This certianly isn't directed at you, RadDidit or any of the others out there that were testing the TF, but one of the problems in this Beta was that there just weren't that many people testing in general.
For whatever reasons, a lot of the serious testing of the TF didn't get underway until it was very late in the process. At one point the SF feedback thread had about 30 pages of feedback while the TF thread languished at under 10 pages. People stepped up and started planning TF runs and such, but that was only a couple of weeks ago and may have been just a little too late to get changes in before they felt things had to go Live. (There is so much we don't know about what is going on behind the scenes that we're all kind of guessing with most of this stuff.)
But, as this post from our newest redname Horatio (and the person responsible for the SF and TF) indicates, changes are going to come for the story still:
[ QUOTE ]
While I did make some adjustments to the story due to the feedback from the training room, I didnt get all the details nailed down as best I could. Due to a tight translation deadline, I was unable to get final text passes into the build that was going to the Live servers, but rest assured Ill be doing my best to address them over the next week or so to bring the story of Reichsmans return in line with the existing history of the 5th Column.
[/ QUOTE ]
I think you'll hear more calls for BSF runs once the bug for the MM power is fixed and the merit count increased.
Subjective feedback really IS hard to deal with, though. A lot of subjective feedback with the dominator changes was just bad. It wasn't bad because they disagreed with the changes, the problem was that the feedback was poorly thought out or consisted more of people shouting down the idea of there being a change period, rather than offering good suggestions. A lot of the feedback that WAS well thought out was actually implemented into the dominator changes, e.g. higher target cap for the low damage psi powers, reducing end cost on the tier 1 attacks etc.
For the TF story, I'd imagine that it was just not enough people providing feedback early enough in the development cycle. They have builds that things get put into, and if the feedback is offered too late, it may get INTO a build, but that build may not come until after the issue goes live.
As for costume pieces...Well, feedback with that is more difficult to integrate. By the time a costume piece goes onto test, it's basically done. Unless looking at it makes a small man jump out of your monitor and murder your family, the piece is probably going to go live as is. Any changes would be pretty small, since the content is basically done. To a lesser extent you can apply this to things that go into beta.
One possibility would be opening up the development process to player input even earlier, but seriously. Given how few people wanted to test the I15 changes, do you really think there will be a substantial number of people ready to offer subjective input on changes THAT early in the development cycle? The changes would end up being at the mercy of a very small group of players who have time to test that early.
Wow, I typed a lot.
Active (Freedom): Setna (Ice/Psi Dom), Arram (WP/KM Tank), Tesmiel (Elec/SS Tank), Astredax (Robot/Dark Mastermind), Operative Vidali (melee fortunata)
Retired (Virtue): Gaav (Inv/EM Tank), Baqra (Fire/SS Tank)
Altho Horatio - I mentioned him in the 1st post - says theres changes coming, I dount its anything drastic, as he's already stated the 5th wont be showing up as Axis America, as many have requested. If anything, it'll prolly be more dialogue and such. And Im not convinced that will enuff to get people to think the TF is fun (or makes sense), MM temp power or not.
And you know, even tho Horatio is takin the responsibility/blame for the TF, it really shouldnt stop @ him. Someone had to approve it, no? Kinda sad. You know with MA now avail to us, the devs really have to raise the bar in regards to content. A basic starting point may be to take the subjective feedback a lil more seriously.
If the TF was an MA arc, how many stars would it have? Really.
Just chiming in again, subjective feedback is hard to interpret, its true, but when pvp in general takes a decline, thats beyond feedback its hard core data.
Triumph, one of the servers i regularly play consisted of maybe 5-6 hard core pvpers, several casual, now we are down to maybe 3 hardcore and very few casuals(Speaking of RV). I have even noticed subtle signs of decline on freedom. Arena is artificially inflated due to arena badges and pvp ios, but if you pay attention, the arena population has declined as well, and those who stayed play without the new TS and HD as well as wanting to get rid of DR but not wanting to include the huge squishy res.
It's not a case of subjective feedback when the data backs it up. Only thing keeping the devs from admitting failure is the amount of manpower they put into truely genius changes, but refuse to admit they implemented them the wrong way.
Just a thought.
This is a staple of mmo culture stuff gets released people come up with many many many good bad and neutral ideas to improve and the devs have to evaluate what is "broken" and what would just be nice.
Over all i dont think the devs ever intend to change things like costumes or alter major features of a beta unless they end up causeing crashes or errors. likewize most zones will never change after launch because after its done people forget the code details and what its hooked to this makes a job much harder to go back to later. for example the guy who origionally coded positron's task force may not even be working for them any more.
Now occationally there is a suggestion that is so good it stays in the back of the devs heads for a few issues and gets added later because they were probibly working on it bit by bit as a low priority thing. but if they were to try to release it with issue 15 theyd have to not release i15 possibly for several months and that would make people sad.
In conclution you never know some update 6 months from now vinesplus may show up and it will be all 3d like and stuff. also in various updates im sure youlll be seeing tiny changes to the new task force as they debug it small details may be added or removed. so its really not fair to say noone listens because nothing changed between beta and release.
The PvP topic is tricky. Im not saying I disagree w/you in any of your points, only that they could have convinced themselves that in the 'long run' the changes would help the game. But I have no doubt that the majority of the feedback on the changes were negative, but I also think that the majority of the feedback on PvP BEFORE the changes were ALSO negative.
I think in that case they actually listened to the feedback in initiating a PvP change (al beit late), but ignored the feedback they got during testing.
Selective hearing maybe?
[ QUOTE ]
...likewize most zones will never change after launch because after its done people forget the code details and what its hooked to this makes a job much harder to go back to later.
[/ QUOTE ]
Ok so lets take zones. Say they create a new zone. Most feedback in beta is negative, and not very constructive ("this zone sux"). At this point they are on a tight sched for release, and have most resources allocated to future content, so they cant make a change, so it is released as is. Sure enuff, the community doesnt like it and it ends up a ghost town (like boomtown or DA). So, we end up with another dud zone, possibly permanently.
This could work with TF/SFs as well.
Perhaps if the sunjective feedback would have been taken a lil more to heart, there would have been a sense of urgency to make the necessary changes.
But if they make bizarre changes due to X or Y player who is in the super secret early alpha agents club for testing stuff, you might end up with the same situation. So subjective feed back is really really tough to use. I'd imagine most of the subjective feedback on this set of costumes and TFs will end up being incorporated into I16 and Going Rogue, but may not change anything that has actually been released.
The PvP fixes are a whole other monster, though. I was in a pretty hardcore PvP VG at the time, and a lot of the members were actually in the small group that was doing the PvP testing, and it sounded more like an issue of too many cooks driving the devs nuts.
The real problem with CoH pvp, as I see it, is that the powers were never developed for fighting one or two superfast moving players. The whole game and player powersets would have to be redesigned for PvP to be not completely ridiculous, and is something they should have held off on until around now, when they have a lot more money and devs.
Active (Freedom): Setna (Ice/Psi Dom), Arram (WP/KM Tank), Tesmiel (Elec/SS Tank), Astredax (Robot/Dark Mastermind), Operative Vidali (melee fortunata)
Retired (Virtue): Gaav (Inv/EM Tank), Baqra (Fire/SS Tank)
There were a lot of major rewrites to the SF due to subjective opinions.
A game is not supposed to be some kind of... place where people enjoy themselves!
[ QUOTE ]
There were a lot of major rewrites to the SF due to subjective opinions.
[/ QUOTE ]
Yeah, wish there had been some for the TF. >.>
The devs do not care about player opinion. There is no point in trying to understand why this is the case, because unless you are willing to complain to ncsoft about it enough to get someone replaced, then it isnt going to change. Sad but true.
@Viking Queen.
[ QUOTE ]
The devs do not care about player opinion. There is no point in trying to understand why this is the case, because unless you are willing to complain to ncsoft about it enough to get someone replaced, then it isnt going to change. Sad but true.
[/ QUOTE ]
Text of Barracuda Strike Force changed, based on Player Feedback.
Text of Kahn Task Force changes in the pipeline, based on Player Feedback.
Science Jacket 2 potenitally being changed to accept a second color on the center panel, based on Player Feedback.
Tier 1 Dominator Assault powers changed to faster recharge / lower endurance / lower damage for the Issue 15 Dominator changes, based on Player Feedback.
Power Customization (or at least, control over the hue of our Powers) coming in Issue 16, oft requested by players since the game's release.
ALL emotes that are available to all characters (not via the Boosters) being added to the quickchat menu, based on Player Feedback.
Combat Auras from the Cyborg Booster available from level 1, based on Player Feedback.
Should I go on? Or are you covering your eyes while reading this post so you can continue to feel wronged by Positron because he slapped your wrist over the way you and your friends were abusing the Mission Architect?
New story arcs coming soon (ARC IDs will be aded when I finish the arc):
So, you want to join the Hellions? (level 1-14 Villainous arc)
Sparks & Steel (level 5-20 Heroic arc)
and
So you want to join the Skulls? (level 1-14 Villainous arc)
Funny, I have never "abused" anything here in the game except newbs whose insight and/or strategic vision were lacking. Do you REALLY think that adding a few fun emotes is a balance to the gutting of the pvp system, and the resultant loss of the majority of our experienced pvp player base ? Tell this to the folks that say the Doms and Controllers fare better in pvp than tanks, blasters and scrappers. You are talking about minor cosmetic changes that have zero effect towards basic playability, and I am talking about changes that alienate or drive away a large percentage of the paying customer base. It is NO MYSTERY to me why the experienced pvp players left the game, maybe you will figure it out one day as well.
If you are happy that you can now add another color to your "science jacket" then great, you got what you are paying for. Me, I would love to see base raids come back. But Sunstorm says that they are off line because pvp isnt as balanced as it could be for all archtypes....funny, I thought Positron said the new pvp was great, but apparently not good enough to be used for base raiding, oh well. You may continue to believe the developers "doublespeak" if you wish, but it is hard to reconcile the 2 statements from 2 different devs there.....either pvp is good enough to be used as a combat system, or it isnt. Indoor map, outdoor, base map, where ever... it either works or it doesnt.
@Viking Queen.
Oh, and another thing....."feel wronged because Positron slapped my wrist..." is really super funny to me. I have never been banned, had my account locked, kicked from forums, had M.A. missions nerfed, characters deleted, nothing. Perhaps my opinion is based on 4 years of playing the game, but I understand that it may not be EVERYONES opinion. In any case, it certainly is not because I am mad for being disiplined.
@Viking Queen.
The problem is allot of what is on a Devs plate was planned months if not years ago, they HAVE to complete those tasks or they are sent packing. Then when there is a break in a schedual they can go back and look at ideas until the next project is forced on them.
ArchRex Dojhrom x ?
* Sidus Loricatus: B-NRG2, S-BS/Reg, T-Fire/Ice, MM-Bots/FF, St-NRG2, Dom-Psi/NRG, Cor-Son/Traps, Cor-Ice/Kin, Ctrl-Fire/Kin, PB-LB/LA
* Arachnos Loricatus: Soldier, Widow
* Praetoria Loricatus: B-DP/Dev, Cor-Elec/Elec
Oh, and one final point to make.....you list 7 items of note as examples of how responsive to player feedback developers have been, but if you read these items closely, you will notice that 3 of the 7 are called "potential" changes that are "in the pipeline".....we were not talking about how responsive they could have been, or say they will be in the future, so your example shows they are responding to player feedback (on minor issues) a little over half the time. I suppose that is better than not at all.
@Viking Queen.
So how much SHOULD the devs listen to subjective feedback?
I was wondering this as I read a thread that complained that the vines pattern was released w/i15 without any changes. (Apparently the OP thought there was a concensus that it needed a revamp.) Obviously, other posters mention that altho the devs do appreciate that kind of feedback, they're primarily looking for bugs. Sounds right, but should that be the case?
Look @ the Positron TF. Not very popular due to it being long & boring. But when we ask for a revamp, they say theyd rather use their resources into creating new, than 'fixing' old. (This has been their recent stance w/zones as well.) So looks like we're stuck with it the way it is, right?
Now take the new TF/SFs, as far as I know, most people were not happy with them. And altho they have been since tweaked, they were released w/many of the complaints still there. (Yes, I know Horacio posted abt possible furth adjustments.)
So, if you release something that the majority dont like, even tho they tell you so in beta, while knowing that you dont have the resources to modify it later, isnt that just bad business? (Understand I mean story and content, not bugs.)
Should subjective feedback be taken more seriously?
What do you guys think?