Any QA Left?

Well played sir

Technical Issues & Bugs is a few boards up. But if I were you OP, I would try to be more specific

This is a section for questions to other players, from other players. Surely, you can't expect any of us to answer your question. Perhaps posting your question here would be better?

You have given us, the other players, no idea what you're even talking about. As such, we really can't help you. That is what I was trying to point out. Does Paragon Studios have a QA department? Sure, they probably do. That doesn't mean that bugs won't make it to live, though.

Yes, they do. I have a friend that works there. He works hard and long hours. I assume others in his unit do as well. But they are only human and are limited by time and resources.

CONSUMER COMPLAINT FORM



We appreciate any communications from our customers, please place your questions/comments in the square below.



☐


Thank you.

Angel Red
"We're heroes. Fighting injustice is the thing we do." ~Rebel Scum

Then, I, as a player, will tell you what I know: Mako.

It's a legitimate game-related answer by a player.

(For a legitimate question, it sure looks a lot like rhetorical sarcasm.)

Yes it does.

You're so clearly wrong, Zombie. The answer is obviously: [Tranquilizer Darts]. Sheesh!

No no, the answer now is [walk]. Oh yes.

Perhaps [Don't Panic] coupled with a [Towel] may be better suited.

While I feel certain that this is a sarcastic post with no expectation of a real answer:

Welcome to MMO Game Design Basics 101

The code base for a MMO, or any game for that matter, is exceedingly complex. What many players may view as a simple button press or click may involve hundreds of underlying sub-routines. When changes to one part of a code cause problems in another part of code that were not intended, you have Software Regressions.

Yes, NCSoft and Paragon Studios do employ a certain number of Quality Assurance Testers. Yes, these testers do attempt to find bugs and fix those bugs in-house.

However, the actual content in an MMO is, as the name implies, massive. It is quite literally physically impossible for a small Quality Assurance team to police all of the content possible in a game such as City of Heroes.

This is why MMO's have CLOSED AND OPEN BETA tests. The company depends on external users to play through the content and locate bugs, then report those bugs. While a MMO company can simulate the effect of 5,000 players on a server at once, they won't actually know how the system performs and works until they actually have 5,000 players... at once. For more background information on the point and purpose of betas in an MMO, read this article: http://saist.fateback.com/what-part-of-beta.html

In the case of recent patches to City of Heroes, the developers have been making small changes to the game, such as enabling the Halloween Event content, and then speeding up the Halloween event timers.

However, these superficial changes have created additional problems that the QA teams simply either didn't have time to look at, or simply couldn't reproduce on their limited internal systems.

It is very likely that some issues simply didn't occur to the QA team since the affected code shouldn't have been touched or modified, indicating deeper problems.

What-ever the case, patches causing problems that they weren't supposed to is a part of life for this type of game.

Is that statement similar to what's commonly referred to as a bald-faced lie?



Hey, look, my question is an actual (though possibly rhetorical, which i think is only fair in this case) question that other players are eminently qualified to answer since it's not about aspects of the game's development that most players would have no direct ability to know. (je saist's response to the OP is far superior, but i'm used to that sort of thing. )

Similar, yes, though I think technically it's closer to "utter bollocks".

Oh, well then the answer is "yes."

Any more questions?

Then perhaps you could ask your bud how in the bluest of blue hells that last patch made it live.

Courtesy of je_saist

Maybe they HAVE been QAing and hadn't been able to pin it down until now.

I haven't heard from him in awhile, but I'll just guess that it was a small patch to fix one thing, which they tested, and it broke something(s) else, which was not tested.

It is practically impossible to test every mission, powerset, and every other little thing each patch. Unless you want to wait a year or more between patches.

I'm rather certain the testers don't attempt to fix the bugs.

All? Certainly not. Very few complex software systems can claim to have policing of how changes affect all their components. However, given that you mentioned software regression, it seems reasonable to consider the discipline of software regression testing. Regression testing can often be automated. If automated, it can also be interwoven with stress or load testing.

Of course, you might change your system in a way for which you actually have no test in your regression test scripts. When you discover such a situation, you have found an opportunity to improve your test scripts by adding a new test case.

There is some truth to this, but only in the sense among that those 5,000 players are going to be some people who are doing something which was not tested for. There's nothing magical about having 5000 users logged in as part of a test suite - this is done with real, mission critical software systems all the time.

(Some problems don't show up just under load - a system I work on recently had a sequence of failures because the vendor added a cache flushing mechanism that kicked in every 12 hours after the software was started. None of the load or regression testing lasted 12 hours, so the issue was never seen until the system went live. After this was found, the vendor provided a formal patch which removed this behavior entirely, agreeing it was a poor design.)

There's no technical barrier preventing one from building testing client interfaces that can either simulate or drive actual client interfaces performing certain scripted player actions. There may be cost barriers, of course. Whether the testing is automated or manual, the goal is to build a list of tests which concentrate on those things which have been found problematic in the past. Mob spawn levels has broken several times in the recent past - if I was in charge of testing, there would be a test case for that in the AE, on TFs, and in Ouroboros.

User beta tests are not where any bugs that could have been caught by existing unit, integration or regression tests should be caught. Betas with the user community should be testing for user acceptance (feedback on new interfaces, for example), for errors for which there are no test cases (which you then correct), or for outright design flaws (trying to expose things like exploits).

Unfortunately, enough old bugs get resurrected in CoH, perhaps through branch merging, it raises the question of whether there are enough (or even any!) internal regression test cases performed before stuff goes to player testing.

One of the other glaring weaknesses of how CoH betas are conducted is a frequent lack of information given to testers about how things should actually be working. Beta testing is not the time to be coy about how things work to try and keep it interesting. If the players do not know how things should work, they cannot conclude whether changes are bugs or intended behavior. They can guess, but making testers guess is a huge waste of time and potential.

A fine example was the recent change in how influence is earned at level 50. Pohsyb recently commented on how he couldn't believe that no one had caught this, which was a bug that had existed at least since exemplaring was introduced in the game, if not since the game's release. The truth is that no one caught it because no one knew how it was working was wrong.

This is where regression testing is invaluable. You make the change, run through test cases, and see what, if anything failed. Then you investigate the failures and make a judgment about whether what's broken is more important than what your patch fixes. If not, you proceed with the patch, if it is, you hold off and fix the new problem too.

Edit: Regression testing even for small changes is important, but often stress or load tests are not performed for small changes. These are often even more time and resource intensive than regression testing. If your regression test is large, you may choose not to run it for small changes, but doing so is begging for problems you actually have test cases for to slip through.

That doesn't mean it's acceptable. Continuous improvement is an important part of any successful IT system's management. Improvements in your test cycle are a key aspect of that. Failing to improve testing methodologies leads to repetitive failures, which contribute to discontented users. Where I work, enough discontented users gets you replaced with another product.

Ultimately, test cases, automated testing suites, and test planning come down to investing in tools that allow a team to work smarter, not harder. These tools should be part of any serious software house's equipment. Whether they're home grown or purchased, there are a lot of things testers and developers can do to improve the quality of their releases.

The idea that Paragon Studios might be lacking good testing discipline is worrisome in the face of the massive set of new features we can expect soon. Obviously, I don't know what they have, but I can make guesses based on recent quality that whatever they have could use some work.