Discussion: Issue 11 Closed Beta
By
Angry_Citizen,
Posted on: 2007-10-09 10:27 in News, Events and Announcement Discussion
[ QUOTE ]
you make it sound like players don't matter at all when it comes to closed beta testing.
[/ QUOTE ]
In terms of how it benefits them, beyond having a better game overall? It doesn't.
[ QUOTE ]
but if we as players don't matter at all when it comes to beta testing, then the lack of fairness there isn't an issue at all. we could go on for each new beta and you and i will never be no-kidding-really-invited and that should not matter.
[/ QUOTE ]
There is no "lack of fairness", not in the sense of "being unfair" that you seem to imply.
The very concept of "fair" never even enters into it; efficiency is the name of the game here. Because they hold a closed beta, and have a proven stable of testers to rely upon as a group, we get issues in amazingly short spans of time.
Instead of being on test for 2-3 months, we're likely to see a 5-6 wqeek testing cycle.
If that means someone else gets into the closed beta, and I don't? Hey - whatever it takes. I still remain hopeful that, eventually, I'll get into a closed beta and be able to contribute.
...
And now, I know how to increase those odds: as a new issue becomes imminent, START SPENDING TIME ON THE TEST SERVER. Then ... just cross my fingers.
[ QUOTE ]
I simply believe that a more optimal way of selection would be to change the criteria from the above to a random selection across all servers. A random selection of participants across all servers would allow everyone equal chance and could include great testers. There are many great testers who play on a specific server, know how to get on Test, but rarely log on to it.
[/ QUOTE ]
Oh, I can agree that there could be improvements. But my agreement stops cold at the point of "exclude prior beta testers, to give more people a chance".
I do think, instead of "logged on to Test recently", those slots could have been better used in a PlayNC-wide random selection. Or a CoH-related contest, even.
But to those who say "noone should be in two beta tests in a row", or whatever?
Grow up. Seriously. The more you make it about your chance, the more I become convinced that you would be the worst sort of "tester" any Beta could be saddled with.
[ QUOTE ]
[ QUOTE ]
Say... Just a thought here, and not meant to be a negative comment... But can we get back to discussing I11 Closed Beta as opposed to discussing what people are grumping about CONCERNING I11 Closed Beta? Nothing constructive is happening here, and nothing is being gained. However, we ARE losing validity.
Just my two pfennigs.
And this is directed at both sides. not ANY individual.
[/ QUOTE ]
Given NDAs and such (can't ask who is in the Closed Beta, those in it can't say anything), what else is there to talk about?
[/ QUOTE ]
I'll go you one better on that front, and directly quote the "no telling" part of my invitation email - and before anyone squeaks about me being a tester, I subsequently got an email form Lighthouse rescinding that invitation, explaining it was sent in error, and apologising for the mistake.
"CONFIDENTIALITY
For this phase of testing, the Training Room server is only available to select accounts. Involvement with this testing is a privilege and we ask that you:
- Do not discuss details of this test to anyone not in the test
- Do not talk about who else is in the beta
- Don't tell others that you are in the beta or
- [Do not] discuss any other Issue 11 information that you learn about in this test with anyone outside of the test.
Thank you for keeping the details of this test confidential!"
So. Can't give any details of the test; can't talk about who else is in the beta; can't admit if you are yourself in the test; can't discuss any other i11 information with anyone who isn't part of the test.
...
All that leaves for us to discuss, I must sadly agree, is the selection process. And I agree if could maybe be improved. But no act of tossing out the current, established pool of testers is an "improvement" IMO, so any idea that starts from there, I would have to oppose.
[ QUOTE ]
All that leaves for us to discuss, I must sadly agree, is the selection process. And I agree if could maybe be improved. But no act of tossing out the current, established pool of testers is an "improvement" IMO, so any idea that starts from there, I would have to oppose.
[/ QUOTE ]
Why?
What is so bloody special about these people that they need to be included in this and any future testing?
We have no knowledge of what the requirements of being selected entails. For all we know, it is 'Be born under a blue moon, be able to turn your eyelids inside out and know how to do the Chicken Dance'.
Unless and until there is 100% transparency of the selection process, this will continue. No 'mystery' criteria whatsoever.
And don't say 'The releases have been smooth'. All that is proof of is that the releases have been smooth, nothing more. It is in no way, an indication of the skills/talent/dumb luck of the Chosen Ones.
That, and the Closed Beta forums really need to be opened up once the Closed Beta is over. That way, everyone can partake of the same information these select few are privy to.
Edit - And if the ranks aren't purged, then eventually the number of closed beta testers (assuming any future ones are ever added) will get so big as to there being no point to the closed beta.
[ QUOTE ]
[ QUOTE ]
All that leaves for us to discuss, I must sadly agree, is the selection process. And I agree if could maybe be improved. But no act of tossing out the current, established pool of testers is an "improvement" IMO, so any idea that starts from there, I would have to oppose.
[/ QUOTE ]
Why?
What is so bloody special about these people that they need to be included in this and any future testing?
We have no knowledge of what the requirements of being selected entails. For all we know, it is 'Be born under a blue moon, be able to turn your eyelids inside out and know how to do the Chicken Dance'.
Unless and until there is 100% transparency of the selection process, this will continue. No 'mystery' criteria whatsoever.
And don't say 'The releases have been smooth'. All that is proof of is that the releases have been smooth, nothing more. It is in no way, an indication of the skills/talent/dumb luck of the Chosen Ones.
That, and the Closed Beta forums really need to be opened up once the Closed Beta is over. That way, everyone can partake of the same information these select few are privy to.
Edit - And if the ranks aren't purged, then eventually the number of closed beta testers (assuming any future ones are ever added) will get so big as to there being no point to the closed beta.
[/ QUOTE ]
One thing that is always looked for in choosing beta testers is prior beta testing experience. Why? Simple really, because they have a much better chance at spotting the less obvious things, and they also know what information to include when they report their findings. That doesn't mean that just because you're a prior tester you're guaranteed a spot. The quickest way to get your name removed from a beta contact list is to beta test something and not give any feedback at all while testing.
Also, opening the tesing boards would do nothing more than open a floodgate of useless information that is no longer relevant, as it has been changed and changed again. Thus you would have people attemting to build their charaters with incorrect information, then complaining about said incorrect information and why it was even made available in the first place. The final build of a beta never happens during the closed beta phase, as anyone who has ever been part of a beta test can tell you. Any information released from that phase would be nothing but detrimental to the entire process.
Rikti Inquistition: Unexpected. Chief Weapon: Fear and Surprise. Chief WeaponS: Fear and Surprise and Uniforms: Snappy Red. Exclamation: Profanity! Declaration: Restarting.
While I agree with your point, might I say that telling people to "Grow up" isn't the best way to discuss. Those "selfish" arguments do have a point in that the selection process is flawed.
Along that same line, purging the testers is a valid idea. Personally, I would not go as far as to say one selection means no more selections. I would, however, say that to constantly add testers to a base number will overwhelm the intent of a "closed beta".
With that said, I'm going to add what I would suggest they do as a selection process.
- Include those who, in previous betas, gave feedback which in some what helped the development team. This can be bug reports, it can be good feature ideas, it can be encouraging testing. This way, you have a wide array of existing users that can reliably assist you. Those who give no helpful feedback should be removed from selection -- since this part is already selective, what does more selectiveness hurt?
- Select a random group across all servers. If you have 100 existing testers, choose an exponential amount from all servers. That way the existing base, while existing, is not the core of the beta. Fresh eyes, fresh ideas, possibly a better beta.
- Should Group B perform well enough, their inclusion in Group A is possible. Say Joe Bob reports a lot of aesthetic fixes and Bob Joe spots a lot of bugs. Both should be given that chance to be pre-selected.
This way, should a beta tester not perform well, they no longer take up a space that another tester could occupy. Test Server instructions, if they are not, should be included in every e-mail invite to the beta to ensure everyone has the ability to log in.
Since you are regularly adding and subtracting from A, the groups remain both fluid and manageable. Thus no exponential growth that defeats the purpose of closed betas.
Again, tl;dr: Fixed group A, random group B. If someone from B does well, they can become A. If someone from A or B doesn't help, they become group SOL.
[ QUOTE ]
[ QUOTE ]
but if we as players don't matter at all when it comes to beta testing, then the lack of fairness there isn't an issue at all. we could go on for each new beta and you and i will never be no-kidding-really-invited and that should not matter.
[/ QUOTE ]
There is no "lack of fairness", not in the sense of "being unfair" that you seem to imply.
The very concept of "fair" never even enters into it; efficiency is the name of the game here. Because they hold a closed beta, and have a proven stable of testers to rely upon as a group, we get issues in amazingly short spans of time.
[/ QUOTE ]
i think when it comes to equity/fairness we're just going to disagree. there is a difference between people continually being allowed into a closed door environment to help test new features in the software and people who are not. i see the difference as enough to create an unfairness. i am not saying the devs are intentionally trying to be unfair or that they do not care that their actions are creating an inequity, but i do believe their approach to selecting who gets to help them is creating an inequity among us, and given their approach i believe it gets larger with each new closed beta they conduct. and when it comes to this inequity, their intent doesn't cause the inequity to not exist, no matter how pure. it is still there. now i think a better question is--is that unfairness worth it to get the advantage their getting? and my personal answer is no--not given the current approach that they're taking. i think they can continue to improve their selection process, and it may even cause more people outside the activity to believe in what they are doing despite the impact to them. sure, no one's always going to be pleased, but i think things can be improved.
i don't think the right answer for us is to spend more time on the test server either. i don't think that even fits the intent of what the devs were looking for this time around. i think they were picking people who recently logged into the test server (which would include people testing new builds, testing io load-outs, doing pvp, doing cross server events, etc) to get at those of us who have time and a desire to continually test new features without needing an incentive to be invited to a closed beta.
[ QUOTE ]
[ QUOTE ]
All that leaves for us to discuss, I must sadly agree, is the selection process. And I agree if could maybe be improved. But no act of tossing out the current, established pool of testers is an "improvement" IMO, so any idea that starts from there, I would have to oppose.
[/ QUOTE ]
Why?
What is so bloody special about these people that they need to be included in this and any future testing?
[/ QUOTE ]
Proven track record. I9-->I10-->i11 has been a faster process than ever before, and partof that is due to the greater intensity and focus of testing in the Closed Beta stage.
YAY FOR I11!!!!!!
It's not paid time. I get my 42 month vet badge at the end of this month and was first round CoV beta, yet I'm still looking for that magic email in my inbox. 
[ QUOTE ]
With that said, I'm going to add what I would suggest they do as a selection process.
- Include those who, in previous betas, gave feedback which in some what helped the development team. This can be bug reports, it can be good feature ideas, it can be encouraging testing. This way, you have a wide array of existing users that can reliably assist you. Those who give no helpful feedback should be removed from selection -- since this part is already selective, what does more selectiveness hurt?
- Select a random group across all servers. If you have 100 existing testers, choose an exponential amount from all servers. That way the existing base, while existing, is not the core of the beta. Fresh eyes, fresh ideas, possibly a better beta.
- Should Group B perform well enough, their inclusion in Group A is possible. Say Joe Bob reports a lot of aesthetic fixes and Bob Joe spots a lot of bugs. Both should be given that chance to be pre-selected.
This way, should a beta tester not perform well, they no longer take up a space that another tester could occupy. Test Server instructions, if they are not, should be included in every e-mail invite to the beta to ensure everyone has the ability to log in.
[/ QUOTE ]
This, I can get behind. It's not a purge (as in, "total emptying of the list"), it's a WINNOWING (as in, "culling the herd of the least-useful, leaving only the best behind").
I'm not going to say "the process is perfect and cannot be improved", I just won't participate in any effort to start "improving" things by chucking the baby with teh bathwater, just so that "more people get a turn".
The goal of the selection process should be "produce the best beta tester pool possible", not "give as many people a turn as possible". The two goals are not compatible, IMO.
[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
but if we as players don't matter at all when it comes to beta testing, then the lack of fairness there isn't an issue at all. we could go on for each new beta and you and i will never be no-kidding-really-invited and that should not matter.
[/ QUOTE ]
There is no "lack of fairness", not in the sense of "being unfair" that you seem to imply.
The very concept of "fair" never even enters into it; efficiency is the name of the game here. Because they hold a closed beta, and have a proven stable of testers to rely upon as a group, we get issues in amazingly short spans of time.
[/ QUOTE ]
i think when it comes to equity/fairness we're just going to disagree. there is a difference between people continually being allowed into a closed door environment to help test new features in the software and people who are not. i see the difference as enough to create an unfairness.
[/ QUOTE ]
And because you obviously have the players' interests closest to yourheart ... you're not an optimal candidate for beta testing. This or, really, anything. No offense mind you; not everyone is cut out to be a beta tester. Not even me, necessarily ... although, when I'm in one, I do at least try to actively find bugs I can test, and report.
[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
All that leaves for us to discuss, I must sadly agree, is the selection process. And I agree if could maybe be improved. But no act of tossing out the current, established pool of testers is an "improvement" IMO, so any idea that starts from there, I would have to oppose.
[/ QUOTE ]
Why?
What is so bloody special about these people that they need to be included in this and any future testing?
[/ QUOTE ]
Proven track record. I9-->I10-->i11 has been a faster process than ever before, and partof that is due to the greater intensity and focus of testing in the Closed Beta stage.
[/ QUOTE ]
You cannot say, with any degree of accuracy that the testers themselves have proven anything other than they can follow instructions.
The quickness could be attributed to: 1) better coders on the Devs part gaining a better knowledge of the system; 2) the Devs being able to focus on one specific system at a time in the Closed system; 3) the feedback not getting lost in the noise of umpteen thousand people clamoring to be heard; 4) rather than taking large chunks of systems the updates have been smaller and more manageable; and other reasons I'm not thinking of this early in the morning.
But saying that these few people out of over 140k active subscribers and over 108k registered board members, are the only ones that can do so is a logical leap not supported by the evidence at hand. Saying that they 'deserve' to be there or are the best possible testers is not evident.
[ QUOTE ]
And because you obviously have the players' interests closest to yourheart ... you're not an optimal candidate for beta testing. This or, really, anything. No offense mind you; not everyone is cut out to be a beta tester. Not even me, necessarily ... although, when I'm in one, I do at least try to actively find bugs I can test, and report.
[/ QUOTE ]
i don't have the players' interest closest to my heart, i just value fairness. i'm not saying that you don't, but when i see a situation where there's enough of a lack of fairness and i see reasonable ways to lessen it or avoid it entirely then i am going to speak up and suggest them. there are improvements to this selection process that can be made. not only can those improvements improve the testing crop that the devs work with every issue, they can improve fairness for us as players.
as far as me not being an optimal beta tester--or me not being an optimal tester of any software--you state that as if it were a fact. i think if you spent a day with me on the job you would think differently. and given what you've said so far in this thread, i think you would go home with a headache.
Selecting people who recently loged onto test is stupid.
How many people only go there to PVP?
I think quite a few probably. I have a friend who got into closed beta because she and other PvPers do their thing in test quite a bit.
The only problem I see with whom they put in test is the ones who have logged into test recently. The problem I have with that is as the person above has pointed out they are the PVPers mainly. These are not necessarily main stream COXers and will have an aggenda - PVP not PVE. Many would get rid of the PVE entirely - and make everything PVP - yes just look here in forums and ive overheard the same thing in game.
To me that means that there will be mostly suggestions on the new powers as how they apply to PVP not what a majority play. The new story lines will also get less attention as that is PVE content.
I think there should be a fairer way of doing it (meaning a better represetitive sample of the whole COX population) and yes those that report bugs should get in first because they are doing it normally not just skipping and going on.
P.S. No I am not in Beta and wish i were but I can wait.
yes i report bugs - but its been a while since ive found one not already mentioned.
SFC America Assault/Dev Blaster LVL 50
Cpt Patriot Inv/SS Tanker LVL 40+
Entropy MA/Inv Scrapper LVL 30+
Warbot En/En Blaster LVL 40 +
[ QUOTE ]
[ QUOTE ]
And because you obviously have the players' interests closest to yourheart ... you're not an optimal candidate for beta testing. This or, really, anything. No offense mind you; not everyone is cut out to be a beta tester. Not even me, necessarily ... although, when I'm in one, I do at least try to actively find bugs I can test, and report.
[/ QUOTE ]
i don't have the players' interest closest to my heart, i just value fairness.
[/ QUOTE ]
And you define "Fairness" in terms of "this player gets X, that player doesn't" - well, that's the players' interest. For a beta test, it doesn't matter who gets what, EXCEPT: the game gets tested well.
[ QUOTE ]
Selecting people who recently loged onto test is stupid.
How many people only go there to PVP?
[/ QUOTE ]
This, I can agree, is a flaw in their approach. A better approach would have been, as others suggested, a random drawing from the general playerbase-at-large, for those extra spaces ... simply because a wider range of opinions, viewpoints, and desires would be represented in that mix.
Alternately, some sort of contest, via the website and/or forums, might be a better approach.
[ QUOTE ]
I think quite a few probably. I have a friend who got into closed beta because she and other PvPers do their thing in test quite a bit.
[/ QUOTE ]
Exactly, just because they are on the Test Server doesnt mean they are actually 'testing' anything.
[ QUOTE ]
Dark One, it doesnt matter. Not to these people. They dont care that the same people get in time and time again or that you cant get the Bug Hunter badge.
All they care about is calling people like you and me who express our opinion about the testing selection whiners.
[/ QUOTE ]
Truthfully, I don't care about your opinion. I know the truth, and that is you are whining, and nothing will come of it. Absolutely nothing. It is out of your control, for the next closed beta they could decide to pick every 5th person that logged into Infinity on the 3rd Tuesday of April, and you still couldn't do [censored] about it. My suggestion? Let this thread die, please.
[ QUOTE ]
My suggestion? Let this thread die, please.
[/ QUOTE ]
Will do, boss.
[ QUOTE ]
The very concept of "fair" never even enters into it; efficiency is the name of the game here. Because they hold a closed beta, and have a proven stable of testers to rely upon as a group, we get issues in amazingly short spans of time.
[/ QUOTE ]
No sir. You are wrong. Community is the name of the game here. That's what this short of thing should be about - getting people involved. However, it is not so.
Yes, I understand your oft-repeated point about it being a good idea to have the proven, veteran testers participate each time. However, if those people are as necessary as you make them out to be then put them on the payroll (as someone else suggested).
[ QUOTE ]
Instead of being on test for 2-3 months, we're likely to see a 5-6 wqeek testing cycle.
[/ QUOTE ]
Again (as has been pointed out, I believe), you have no proof that this is the case. And no, your use of the term "likely" does not preclude my stating that you have no proof. Were you around for the release of Issue 2? Now there was an issue that could have benefited from a lot more testing. Every issue release since then has been a breeze by comparison.
One of the most difficult things about the closed beta phase, whether one is part of it or not, is dealing with the insufferable smugness of the others. You end up with the word "whine" being bandied about with considerable regularity (the "better be 14 years or younger" comment was a real gem). You end up with those who cry "favoritism" being met with "you're paranoid and unstable". Just because one person sees favoritism doesn't mean that they are paranoid and unstable.
No. Closed beta has ABSOLUTELY NOTHING to do with community, except that community is where they can get willing and hopefully educated and/or capable testers for free.
[ QUOTE ]
Say... Just a thought here, and not meant to be a negative comment... But can we get back to discussing I11 Closed Beta as opposed to discussing what people are grumping about CONCERNING I11 Closed Beta? Nothing constructive is happening here, and nothing is being gained. However, we ARE losing validity.
Just my two pfennigs.
And this is directed at both sides. not ANY individual.
[/ QUOTE ]
Given NDAs and such (can't ask who is in the Closed Beta, those in it can't say anything), what else is there to talk about?
Triumph: White Succubus: 50 Ill/Emp/PF Snow Globe: 50 Ice/FF/Ice Strobe: 50 PB Shi Otomi: 50 Ninja/Ninjistu/GW Stalker My other characters