Do the Devs even test their code anymore?
By
all_hell,
Posted on: 2011-12-07 13:27 in Player Questions
Quote:
And the part YOU are glossing over is the fact (and it IS a fact, the devs have confirmed as much in the past) that quite frequently, the cause of the bug has no apparent relation to what the bug is affecting. |
Originally Posted by FourSpeed
The part you are glossing over completely is that these types of bugs should
not even be IN the build - many of these should have been caught at the programmer's desk |
It is entirely possible, and in fact likely, that these bugs aren't being caught at the programmer's desk because they are being caused by something that programmer had nothing to do with. The programmer's code could be working perfectly, and another code could be working perfectly, but when those two pieces of code start interacting it could have an unexpected effect.
It could be something like: Your map will break if you have ever teamed with someone in a different phase than you. In that case, the map code could be working correctly, and the phasing code could be working correctly, but the 2 codes break things when combined. Unless one person is working on phasing technology and the UI simultaneously, it is unlikely that connection will be made immediately, because the code appears to be correct upon examination.
Your example of the missing driver's side mirror is a poor one. In that case it is painfully obvious what the problem is, and the solution is just as obvious.
In the case of bug fixes in this game, it's more like the car is missing exactly one bolt and you have to figure out which bolt is missing before you can replace it.
And the bolt might not be missing, it might just be broken and still in place. Which means you'd have to go through the entire car and test each bolt individually until you find the broken one.
I don't think the game's programmers are incompetent in the slightest. I just think they have a mountain of unpredictable code they have to deal with that can do weird things when new code is added to it.
Quote:
|
Originally Posted by Dechs Kaison See, it's gems like these that make me check Claws' post history every once in a while to make sure I haven't missed anything good lately. |
Quote:
The first? Haha.
|
Originally Posted by VoodooGirl
 Congratulations on being the first receipient! |
Quote:
I think one of the issues with video games is that traditionally, the game is released in a more-or-less fixed state, and while there may be a patch or two to clear up particularly horrible bugs, generally the developers will then move on to the next project. No one needs to think about what could be happening ten years down the line, when the whole team has changed, and so have the requirements. The keys are quick development, and code that works, but isn't necessarily easy to change or maintain.|
Originally Posted by Wing_Leader
I have noticed that this is true to wildly different degrees depending on the industry we're talking about. I don't think it is accurate to make the blanket assertion that all (competent) programmers in all application domains are going to be equally rigorous or effective at debugging and testing. I can't speak to what is "typical" in the video game arena, but in the two industries I've worked in as a software engineer, the attitudes towards testing and debugging have been almost diametrically opposed.
|
Unfortunately, MMOs are a lot more like enterprise software in that the same code needs to be continuously maintained and developed for years, while it's in use by the customer. I would hope that things have changed now, and new processes have been brought in, but I can understand why an old MMO like CoX might now have a lot of problems.
Arc#314490: Zombie Ninja Pirates!
Defiant @Grouchybeast
Death is part of my attack chain.
Some general thoughts:
We are not paid to beta test for the Devs!
We are not paid to beach and moan on the forums either but ...
I've been a programmer for ...
There's a large numbers of code monkeys in this forum (I am one myself) but how many of us develop systems with thousands of end users, each with unique system configurations? Not many. Those that do probably also work in the games industry.
I work on financial and costing systems in the Higher Education sector, these are delivered internally onto "merely" hundreds of corporate desktops through the use of HTML. Yet even within this predictable environment we still experience unforeseen problems due to differences in desktop and browser configuration, configurations that are supposed to be essentially the same! Plus the systems I (and my team) develop have far fewer possible permutation of actions than a large free form game like City of Heroes, yet performing a full end-to-end test of all those permutations would be prohibitively expensive, if it were even possible. So we test effected areas and do "light" general testing as befits the perceived risk. Occasionally we get it wrong.
So given all that it will be a long time before I am overly critical of Paragon Studios. And if I am critical at all, it will be with a sympathetic "I've been there" undercurrent 
This is a song about a super hero named Tony. Its called Tony's theme.
Jagged Reged: 23/01/04
Quote:
What's the saying... "A picture is worth a thousand words" O_o
|
Originally Posted by Frostbiter
I don't have anything non-inflammatory to say so please enjoy this picture of puppies.
 |
It's better to save the Mystery, than surrender to the secret...
Quote:
Agreed.|
Originally Posted by Jagged
I think I must be in the wrong game forums because apparently the game most of you guys are playing is Buggycrap. Can I recommend you try City of Heroes instead? This last week I've played it more than I normally do, doing my first Underground Trial, quite a few Keyes and one or two story base arcs. I also created new characters using totally new abilities and newly purchased costume pieces. All without any problem. I have recommended it to most of my friends.
|
I must be one of the lucky ones. I'm not running into that many issues to affect my game play experience. Overall, I think the Devs are doing a good job.
Say it with me, folks:
BETA TESTING IS MORE THAN JUST LOOKING FOR BUGS.
The Devs are looking for both subjective and objective feedback, and they DO change things based on that feedback before the Issues go live. Go read any beta forum feedback threads if you don't believe me. Ask the regular testers, and they'll tell you the Devs change things...maybe not in time for launch, but they do fix things. Look at how many times the Keyes Trial has been tweaked, for example.
You can keep saying Beta is useless, and you'll still be wrong.
Loose --> not tight.
Lose --> Did not win, misplace, cannot find, subtract.
One extra 'o' makes a big difference.
Quote:
You're skipping the "go live" part. Writing bugs into the code with an addition is expected. Taking time to find them is good. Knowing that they exist for weeks and shoving them out the door anyway is problematic.
|
Originally Posted by ClawsandEffect
Would you rather wait 6 months for something new because they haven't tracked down exactly which piece of 7 years worth of code is causing it to act weird?
|
Quote:
Absolutely.
|
Originally Posted by ClawsandEffect
Would you rather wait 6 months for something new because they haven't tracked down exactly which piece of 7 years worth of code is causing it to act weird?
|
Quote:
Isn't it obvious? They are prioritizing releasing new material. There seems to be a consensus building that the priorities need to be adjusted slightly. The level of bugginess SEEMS to be increasing. So does customer dissatisfaction.|
Originally Posted by Wing_Leader
Out of curiosity, what is your explanation for why Paragon has not implemented this painfully simple and obvious strategy? If it is so simple, surely they would have considered it themselves. If they rejected it, what could the possible reasons have been?
|
Quote:
|
Originally Posted by VoodooGirl
Congratulations on being the first receipient! |
Paragon City Search And Rescue
The Mentor Project
What's the reason that mis-spellings and typos don't get corrected?
Sure, they don't interfere with the functionality of the game, but they certainly leave an impression on a player.
And, really, how much time can it take to edit some text?
Quote:
You raise some good points. I stand by the position that it is a reasonable comparison|
Originally Posted by MajorPrankster
As a professional in the software development of financial software myself, I can see where the OP is coming from, but there is more to it than code. I happen work in a building that also has a big-name game making company in it. I get to talk to both kinds of developers and to the people that run the companies.
First, the financial code we create is FAR more 'focused', making it far easier to debug and maintain, IMO. The fact that many of the same technologies used in highly mission critical software, like programming languages and databases, are used to create CoH as well, does not mean they are used in the same way. In fact, many of the underlying technologies were developed to create financial software and have been shoe-horned into creating games. Often, just as it does with financial software, the bugs lie not in the code written by the product developer, but somewhere in the underlying technology. Second, and more importantly, highly mission critical code has an enormous amount of money and resources thrown into the QA phases as compared to games, where the development resources are placed elsewhere, like art and storytelling, etc. The amount of money spent on QA in the highly critical software, IME, often outweighs by far the amount spent on the actual development. Games are art, where highly critical software is engineering. There are two completely different mindsets at work between the two types of software, regardless of the fact they happen to use the same tools, like a welder that builds cars vs one that builds replicas of Transformers. I am NOT saying this game developers should not unit test like everyone else. I am not saying that QA should not be done. But QA is not where the game industry spends the majority of it's money. The margins they have are small enough as it is, games would not get made if held to the same standards as the PCI complaint software of the modern financial world, or they would cost a whole lot more. Comparing game code development to highly critical code development is not a reasonable comparison, for a number of reasons; to entertain is art, to track how much money I have in the bank is something else entirely. IMO, it's like telling an artist his work has to be held to the same scrutiny as a bridge builder. |
as we are discussing code and end-user application systems - we're not debating
apples and oranges.
In fact, the comparison is highlighting some key (and valid imho) differences
between various disciplines, largely in terms of funding, focus and prioritization.
Your point about a higher priority for the artwork aspects is a good one, although
I'll point out that there is plenty of room for quality improvement there also -
given clipping issues, pieces that don't fit, pre-tinting (a "feature" argument),
and various changed/missing items.
I can accept, and have essentially said, I don't expect QA to be as high here as
I would in other industry segments I've mentioned, but I definitely think it needs
to be raised higher than where it appears to be currently.
Good Points though - I appreciate you raising them.
Quote:
|
Originally Posted by "ClawsandEffect
And the part YOU are glossing over is the fact (and it IS a fact, the devs have confirmed as much in the past) that quite frequently, the cause of the bug has no apparent relation to what the bug is affecting.
It is entirely possible, and in fact likely, that these bugs aren't being caught at the programmer's desk because they are being caused by something that programmer had nothing to do with. The programmer's code could be working perfectly, and another code could be working perfectly, but when those two pieces of code start interacting it could have an unexpected effect. It could be something like: Your map will break if you have ever teamed with someone in a different phase than you. In that case, the map code could be working correctly, and the phasing code could be working correctly, but the 2 codes break things when combined. Unless one person is working on phasing technology and the UI simultaneously, it is unlikely that connection will be made immediately, because the code appears to be correct upon examination. Your example of the missing driver's side mirror is a poor one. In that case it is painfully obvious what the problem is, and the solution is just as obvious. In the case of bug fixes in this game, it's more like the car is missing exactly one bolt and you have to figure out which bolt is missing before you can replace it. And the bolt might not be missing, it might just be broken and still in place. Which means you'd have to go through the entire car and test each bolt individually until you find the broken one. I don't think the game's programmers are incompetent in the slightest. I just think they have a mountain of unpredictable code they have to deal with that can do weird things when new code is added to it. |
esoteric voodoo that is unfathomable to the average mortal.
It isn't.
It can be complex, it does require meticulous attention to detail, and in some
cases, interactions between various subsystems can cause erratic, and
difficult to isolate/fix results.
But by and large, it follows the 80/20 rule like a lot of things - 80% of the issues
are usually simple "Duh" mistakes that are often pretty easy to find and fix,
and 20% (or less) are the weird things where programmers really earn their pay.
I'm not complaining in this thread about that 20%.
I AM saying that lately, it appears to me that much more of that 80% is ending
up in the live, production game, and I think that some steps should be taken
to address that (ie. slow down a bit, test code, act on beta feedback sooner,
and prioritize quality a bit more than they seem to).
Now, if you wish to think that this stuff is all miraculous and incomprehensible
magic, fine by me. I don't know what your programming qualifications are, but I
do know what mine are - and based on those, I don't think we're seeing a magical,
issue here. I think it is much more an issue of attention to detail and elevating the
quality standards a bit.
Regards,
4
I've been rich, and I've been poor. Rich is definitely better.
Light is faster than sound - that's why some people look smart until they speak.
For every seller who leaves the market dirty stinkin' rich,
there's a buyer who leaves the market dirty stinkin' IOed. - Obitus.
Part of the reason for the Test Server is supposed to be so that the player base can do things with the power sets/costume creator/missions that the devs have not thought of, or have not had the time/resources to test appropriately. I am not a programmer, but I would expect that there is a vast difference from running an updated CoH on their internal servers with a limited number of players and running that same update with 10x (or 1000x) the number of people using it.
It would be impossible for the devs to try out every configuration of computer systems, every combination of powers accross all ATs, every possible combination of missions/maps/contacts and whatever. That seems to be why these bugs pop up.
I would be truly curious to see a list of the bugs that the devs find and fix on the new updates before we ever see them. It might make a bit of a difference to the players upset about these current bugs if they knew that for every bug that makes it to the live servers they have found and fixed 100 bugs that we never knew about. Of course I have absolutely NO idea what the actual ratio would be, the numbers I used were purely arbitrary and simply a shot in the dark. But I would think that the number of bugs that we never see is quite a bit higher than the number we do see.
Quote:
This sentence brought to you by the Department of Redundancy Department.
|
Originally Posted by Ironblade
When you don't have a good response, make a stupid and fatuous post. (shrug)
|
Quote:
|
Originally Posted by PleaseRecycle
it has gone from unconscionable to downright appalling that we have no way of measuring our characters' wetness.
|
Quote:
|
Originally Posted by Brillig
It's hard to beat the entertainment value of Whackjob Wednesdays.
|
Quote:
This is where people lose me in this debate. Why would anyone want to play untested, unreleased content? Heck, I don't even want to play the new stuff when it hits live. I still want to go back to pre-ed when things were simple. Now, there're all these extra tools and tricks to "purple-out" a toon, and soft cap, etc. |
Originally Posted by ClawsandEffect
While I won't say it is anyone's job, you DO get compensated for it.
You are allowed to play new content before it is technically released. They are in no way obligated to allow us to do that. In return for that privilege, it is understood that you will be telling the developers about any problems you encounter while doing so. Playing unreleased content is not a right like some people want to claim. If you want to play unreleased content, hold up your end of the deal and report a bug when you find it. It takes, literally, less than a minute to file a bug report, but you don't want to be bothered because it's "not your job". Even though the devs are letting you play stuff that hasn't been released yet. |
I never, ever, ever asked for new content. I never expected it. I don't see the point in having new content.
New animations, new powers, new quality of life stuff, sure. But Croatoa? I like it, but I never asked for that. I still don't understand what was wrong with continuing to run missions through Talos, IP, Bricks and Founders.
I DO understand that people get tired of stuff. I get that. (I'm not totally crazy) But, I don't. I like the same things, the same way. I like to compete with myself. Take a tank, run a mission, see how long it took. Take a scrapper, do the same mission, same setting, compare. Repeat with brute, scrapper, different kind of tank, etc. That's just me, and I know it. I can't expect everyone (or anyone) to be like me.
However, you're suggesting that Beta is a privilege - that's your perspective, and you've a right to feel that way. But me, I only go there when the servers are down. I have no interest in looking for bugs or even playing as normal - because if I do get a purple drop - it's wasted. I can't bring it to live where I'd use it. I can't sell it, or email it to myself - it's counter-productive. To me, beta is a time sink. The only purpose it serves to me is to see if Afterburner is fast enough, or if Spring attack is any different from Shield Charge.
But, like I say, that's just me.
I know on my server, so many people are trying Titan Weapons, and were eagerly awaiting it. I wasn't. I really could not care less about it.
"Most people that have no idea what they are doing have no idea that they don't know what they are doing." - John Cleese
@Ukase
Quote:
There are a lot of people who seem to think there was a golden age of commercial programming. There wasn't. The industry has never valued quality. There are few programmers and software engineers that are a) more than minimally competent, b) willing to draw a line in the sand to hold the line on quality, and c) still around.|
Originally Posted by FourSpeed
On another sidenote, I'll add and agree that there are a LOT fewer programmers
who actually understand wth goes on inside code these days (Thank You .NET et al, where a lot of the inner workings are increasingly inaccessible blackbox mysteries they don't want a programmer to see). Sadly, the field is becoming more of a lego block process, but that's also a different discussion altogether. |
We expect - demand, even - that certain professions have a code of ethics that includes minimum levels of quality We expect doctors, pilots, engineers, to say "either you let me do this correctly, or you fire me and find someone else because I won't do it incorrectly." When they don't, we feel justified in basically crucifying them. Even when lives are not at stake, we still believe accountants are supposed to do the books right or not at all. We have *never* as a profession even *pretended* that we aspired to that level of basic competency.
And it shows.
[Guide to Defense] [Scrapper Secondaries Comparison] [Archetype Popularity Analysis]
In one little corner of the universe, there's nothing more irritating than a misfile...
(Please support the best webcomic about a cosmic universal realignment by impaired angelic interference resulting in identity crisis angst. Or I release the pigmy water thieves.)
Quote:
Why do you suppose they are prioritizing new material over fixing (certain) old bugs?|
Originally Posted by Ironblade
Isn't it obvious? They are prioritizing releasing new material. There seems to be a consensus building that the priorities need to be adjusted slightly.
|
Personally I think the consensus you refer to is but a squeak in a hurricane of indifference, but that's just one opinion battling another I suppose.
Quote:
|
Originally Posted by Ukaserex
This is where people lose me in this debate. Why would anyone want to play untested, unreleased content?
|
Quote:
|
Originally Posted by Ukaserex
I don't see the point in having new content.
|
Quote:
|
Originally Posted by Arcanaville
We have *never* as a profession even *pretended* that we aspired to that level of basic competency.
And it shows. |
Reminds me of an old observation I read once that "software engineering" barely deserves the moniker, because if we built bridges the way we build software, nobody would drive over them.
NOR-RAD - 50 Rad/Rad/Elec Defender - Nikki Stryker - 50 DM/SR/Weap Scrapper - Iron Marauder - 50 Eng/Eng/Pow Blaster
Lion of Might - 50 SS/Inv/Eng Tanker - Darling Nikkee - 50 (+3) StJ/WP/Eng Brute - Ice Giant Kurg - 36 Ice/Storm Controller
The map issue is a database issue. Didn't we get the same thing like, issue 7? It was related to map IDs shifting or something like that. I honestly can't remember, but I don't see it as a problem, especially with Reveal and VidiotMaps.
Quote:
Because they're trying to make a philosophical statement about English spelling and grammar?
|
Originally Posted by Wing_Leader
Why do you suppose they are prioritizing new material over fixing (certain) old bugs?
|
Quote:
Oh Dear! |
Originally Posted by Ukaserex
This is where people lose me in this debate. Why would anyone want to play untested, unreleased content? Heck, I don't even want to play the new stuff when it hits live. I still want to go back to pre-ed when things were simple. Now, there're all these extra tools and tricks to "purple-out" a toon, and soft cap, etc.
I never, ever, ever asked for new content. I never expected it. I don't see the point in having new content. New animations, new powers, new quality of life stuff, sure. But Croatoa? I like it, but I never asked for that. I still don't understand what was wrong with continuing to run missions through Talos, IP, Bricks and Founders. I DO understand that people get tired of stuff. I get that. (I'm not totally crazy) But, I don't. I like the same things, the same way. I like to compete with myself. Take a tank, run a mission, see how long it took. Take a scrapper, do the same mission, same setting, compare. Repeat with brute, scrapper, different kind of tank, etc. That's just me, and I know it. I can't expect everyone (or anyone) to be like me. However, you're suggesting that Beta is a privilege - that's your perspective, and you've a right to feel that way. But me, I only go there when the servers are down. I have no interest in looking for bugs or even playing as normal - because if I do get a purple drop - it's wasted. I can't bring it to live where I'd use it. I can't sell it, or email it to myself - it's counter-productive. To me, beta is a time sink. The only purpose it serves to me is to see if Afterburner is fast enough, or if Spring attack is any different from Shield Charge. . |
Don't you realize that without beta testing there is no game. But for the hours beta testers put in testing stuff for you and yours there wouldnt even been a game to play. Beta testing is how the game's developers find and fix problems - it's the only way they can do it. So they don't get it right 100% of the time, do you perform at 100% at your job/school/life all the time? No! of course you don't.
Sure, the game has developed and evolved from what it was seven and a half years ago, sometimes for the better sometimes for the worse. If you liked it then but don't like it now, you have the choice of either not doing the new content or leaving the game. Nobody is forcing you to either play the game or beta test it. It's your money so it's your call.
There is always a lot to be thankful for, if you take the time to look. For example, I'm sitting here thinking how nice it is that wrinkles don't hurt. ~Author Unknown
Quote:
Yes there was!!! You never saw bugs like this in Pong!|
Originally Posted by Arcanaville
There are a lot of people who seem to think there was a golden age of commercial programming. There wasn't.
|
Quote:
Just noticed this after rereading the thread. |
Originally Posted by all_hell
For various reason I don't bother with beta.
However, there's a new wrinkle involved since we have gone to micro transactions. It's possible that there's a financial benefit to allowing us to try out new stuff before we buy. So things aren't quite as simple as they used to be. |
No, MT's have not changed anything. The devs still don't have to let you check out microtransactions before they go live. You can wait until they're released like everyone else, and check them out on the test servers after they are in the store but before you make a purchase on the live servers.
Quote:
Sure they dont and sure they could. But that's not the point.|
Originally Posted by Forbin_Project
Just noticed this after rereading the thread.
No, MT's have not changed anything. The devs still don't have to let you check out microtransactions before they go live. You can wait until they're released like everyone else, and check them out on the test servers after they are in the store but before you make a purchase on the live servers. |
The point is that there's potentially a benefit to them for letting us check them out on beta. This is different than when there wasn't.
Quote:
My sarcasm detector is waffling on this one...|
Originally Posted by Ukaserex
This is where people lose me in this debate. Why would anyone want to play untested, unreleased content? .... [snipped]
I never, ever, ever asked for new content. I never expected it. I don't see the point in having new content. ... [snipped] |
If City of Heroes was a standalone game, one that you played by yourself and you only paid a fee -once- (i.e. the initial $29.99 or whatever it was) - then I would totally understand your bafflement over people wanting new content.
However, we have in the past, and some of us continue to, paid a monthly fee to play this game - and if new content wasn't added, I probably wouldn't continue to pay after the second month; I'd have grown bored.
Since I continue to pay for this game, then I continue to expect new things to keep me amused and keep me willing to pay that monthly fee.
And it is because I enjoy this game so much (I have only ever once before played an MMO past the 3 month mark and that was Ultima Online) that I am willing to give up my play time on "Live" to give my input on this "untested, unreleased content," because I want it to be the best it can be for my fellow players and myself.
/end rantamble
Now I have to:
create a toon
request a level bump
re-pick powers and slots up to 50
buy and craft IO's
I don't think so. I'm not going to spend an hour on menial tasks for the honor of doing free testing for the devs. Previously, I had a stable of toons available for testing things, as well as a supergroup and base. Then they wiped the test server and the character copy tool was either broken previously or at the same time. I've been trying the copy tool every 1-2 days to see if it's fixed yet.
Congratulations on being the first receipient!