Quote:
Oh, sorry I thought this was excessively obvious. What we are talking about is a simulation, where the developer gets to specify everything about the simulation. The inputs are all generated by their software and the outputs don't have to conform to much of anything except what they say they should. Matter of fact the developer can change things have systems perform as they want them to work, but still blame the user for not using it in a manner they failed to inform the user about.|
Originally Posted by Ironblade
Then I guess you're deliberately acting the fool. We're not talking about events in the game. We're talking about software development. Sounds like 'real life' to me.
|
That is about as non real world as you get.
Quote:
An excuse is generally something that could have been reasonably avoided, but wasn't. A plausible reason usually doesn't mean that it couldn't have been avoided, but that it required unreasonable effort or resources, or it was understandable why it couldn't be done. Not the same thing. Seriously, are you even reading the body of these posts? Many people have given you plenty of reason as to why this kind of thing can't be avoided. It seems that you're just cherry picking the parts of our posts that you can visibly disagree with and ignoring the rest. Do you have a reasonable way that they can do a proper load test on every single possible hardware/software/OS combination that can run this game? Of course not, you're not "financially invested" in this. I'm betting you don't even have a realistic idea of what's involved. Cherry pick all you want, it doesn't make you any more right.|
Originally Posted by Another_Fan
No there is not. There are reasons for everything in the world. When you take a reason and say this should be overlooked because of reason X, the reason becomes an excuse. Whether you choose to accept the excuse or not is entirely a personal issue.
What kills me here is that the OP made a perfectly valid observation, and all of a sudden the amateur red names are trying to jump down his throat and making excuses for NCSoft. |
Things happen. While we aren't necessarily excusing it, we are understanding it, and realizing that our Devs are doing everything they can to make it right. They wouldn't have rolled back that patch otherwise.
Quote:
|
Originally Posted by Darkfaith
An excuse is generally something that could have been reasonably avoided, but wasn't. A plausible reason usually doesn't mean that it couldn't have been avoided, but that it required unreasonable effort or resources, or it was understandable why it couldn't be done. Not the same thing. Seriously, are you even reading the body of these posts? Many people have given you plenty of reason as to why this kind of thing can't be avoided. It seems that you're just cherry picking the parts of our posts that you can visibly disagree with and ignoring the rest. Do you have a reasonable way that they can do a proper load test on every single possible hardware/software/OS combination that can run this game? Of course not, you're not "financially invested" in this. I'm betting you don't even have a realistic idea of what's involved. Cherry pick all you want, it doesn't make you any more right.
Things happen. While we aren't necessarily excusing it, we are understanding it, and realizing that our Devs are doing everything they can to make it right. They wouldn't have rolled back that patch otherwise. |
Actually I do read them, its what slows up my replies. Usually where the post is going to go is contained in the first few lines.
Anyway back to the "Reasons" what people have presented here as excuses are in no way shape or form reasons, they are little more than excuses.
Look let me take an example that is mathematically well described and well known.
I have a salesman I need to send him to cities I want to do this as efficiently as possible. I can't figure out how to do this myself so I hire a consultant. The consultant tells me there is no easy way to do this as there no easy way to deal with the complexity that adding extra cities entails. I say OK, I still need to schedule my salesmen so I go to another consultant and ask him can he do anything with this. He comes back and has a bunch of shortcuts techniques that don't produce a perfect solution but keep it manageable. Upshot I pay the second consultants bill and I see if I can take first to court to recover my funds.
I have heard everything that has been said by the people above about how difficult software engineering is. Mostly I have heard it along time ago from programmers I was managing. I used to tell them the above story. I would also point out to them I had written a debugger for expressly parallel code as my thesis, so elucidating the details of software complexity to me was vaguely insulting. Edit: It is up there with the person who tried to use mil spec development procedures as data why you can't develop software efficiently.
Anyhow the upshot was our customers didn't care about how hard software was, they wanted their salesmen scheduled.
Quote:
Then you have no right to complain when problems crop up that stem from the load. People go around thinking "Oh, someone else will do it" or "it's not my job". The test server is RIGHT THERE, easily accessible and available for testing. They WANT us to go and test it, unlike other games that have no test server. You want to complain how bad the bugs are? Take a shot at finding them yourself on the test server.|
Originally Posted by Another_Fan
LOL its not my product. I have no financial interest in it. I will derive no career advancement from it. If it doesn't work right long enough and often enough and the outages are severe enough, I will simply suspend my sub.
|
Honestly, how can you expect them to find a bug that only crops up with 2500 people logged in when only 300 people play on the test server? Especially when there is only ONE test server and ELEVEN live servers? They can't duplicate that in house, when there are people all over the world with varying connection speeds, doing thousands of different things in the live game.
Quote:
Honestly why aren't you in Detroit how can you expect them to find problems that only crop up when the traffic is stop and go.| Honestly, how can you expect them to find a bug that only crops up with 2500 people logged in when only 300 people play on the test server? |
Oh as to the how
http://www-01.ibm.com/software/rational/
Seems someone has managed it. I wonder if any other companies have products like this or if somehow they are the only one..
This does illustrate the cultural divide though. Some people are invested in the game others just play it.
Quote:
Step right up ladies and gentlemen and get your silver bullets. If you just sign up and pay mere thousands of dollars per developer per year for these silver bullets you will have the ability to make all your development problems disappear.|
Originally Posted by Another_Fan
Honestly why aren't you in Detroit how can you expect them to find problems that only crop up when the traffic is stop and go.
Oh as to the how http://www-01.ibm.com/software/rational/ Seems someone has managed it. I wonder if any other companies have products like this or if somehow they are the only one.. This does illustrate the cultural divide though. Some people are invested in the game others just play it. |
Each product is carefully crafted to use only the latest in buzzwords that you have seen talked about in your favorite software developer management magazines or raved about at your last software developer management conference (that also featured an open bar and was adjacent to a world renown golf course).
----------
All kidding aside, yes these tools and techniques exist. Not all of them are flash in the pan, some are actually good, some are good for only certain situations but few if any can be dropped onto a project that already existed for years and magically make everything better. Most require the developers to be aware of and use them from day one of coding and when that is done, it can significantly improve quality.
But I'm not kidding on the costs. Companies that sell these packages are just this side of drug dealers. The first year or two are at a "reasonable" price but then once you are hooked, BOOM, huge price hikes. Now you are stuck between paying to still use the software and laying off 10-15% of your staff.
Father Xmas - Level 50 Ice/Ice Tanker - Victory
$725 and $1350 parts lists --- My guide to computer components
Tempus unum hominem manet
I'm trying hard to remember patch rollbacks in the game. I think we have had 4 in total.
The fact that two of them have been recently could be just bad luck or could be a sign of a decrease in the quality of the QA process.. I'm thinking its the former.*
*future events may change this thinking.
@Catwhoorg "Rule of Three - Finale" Arc# 1984
@Mr Falkland Islands"A Nation Goes Rogue" Arc# 2369 "Toasters and Pop Tarts" Arc#116617
Quote:
|
Originally Posted by Father Xmas
Step right up ladies and gentlemen and get your silver bullets. If you just sign up and pay mere thousands of dollars per developer per year for these silver bullets you will have the ability to make all your development problems disappear.
|
That was for automatic load testing not software design. The post I was replying kept on insisting that if I wasn't playing on test I was somehow causing the lack of load testing.
Just out of curiosity and perversity if you are building something that is massively multiplayer wouldn't you at the start want to have some means of seeing how it works under massive load, aside from the obvious "Have lots of people use it"
Quote:
Artificial load testing is (somewhat) like testing drugs on animals that are similar to humans. It can give us an indication of things will go, but if it passes animal testing it doesn't go straight to the masses, it undergoes human trials- which don't always catch all the issues a drug may have. |
Originally Posted by Another_Fan
That was for automatic load testing not software design. The post I was replying kept on insisting that if I wasn't playing on test I was somehow causing the lack of load testing.
Just out of curiosity and perversity if you are building something that is massively multiplayer wouldn't you at the start want to have some means of seeing how it works under massive load, aside from the obvious "Have lots of people use it" |
And someone earlier brought up a comparison to cars. Code iis a funny beast that can leave the strangest "ghosts in the machine". Unlike cars where you don't have to worry if some tweak to the engine will affect how the car door opens.
Orc&Pie No.53230 There is an orc, and somehow, he got a pie. And you are hungry.
www.repeat-offenders.net
Negaduck: I see you found the crumb. I knew you'd never notice the huge flag.
Quote:
|
Originally Posted by Rajani Isa
Artificial load testing is (somewhat) like testing drugs on animals that are similar to humans. It can give us an indication of things will go, but if it passes animal testing it doesn't go straight to the masses, it undergoes human trials- which don't always catch all the issues a drug may have.
And someone earlier brought up a comparison to cars. Code iis a funny beast that can leave the strangest "ghosts in the machine". Unlike cars where you don't have to worry if some tweak to the engine will affect how the car door opens. |
http://www.popularhotrodding.com/fea...nes/index.html
If you go with their first entry there, Cadillac V-8-6-4 and the unmentioned problem that it had with occasionally shutting down while on the highway, it kind of undercuts the idea that designing an automobile is less complex than writing a program. I am sure there are more. In GM's case they let a product go into production that wasn't fully tested and probably failed to listen to anyone that was saying so. Do I feel that automotive design is complex, yes. Do I feel that automobiles are complex systems with many interacting systems ? Yes. Was I driving a Cadilac then ? Yes that's how I know about this. Will I again ? No, These days I drive an Acura, I hear GM has its quality up to par again, ehh so what.
Just another note, I see the new Fords incorporate a system from Microsoft these days. The jokes almost write themselves. If for whatever reason these cause significant difficulties do you think the software is difficult reason will hold water ?
Why are you guys even arguing anymore? Another_Fan has way too much invested in his opinion to even think about changing it, and ya'll are not going to change your opinions (because they're generally correct).
Just stop. It got dumb at least ten posts ago.
Paragon Wiki: http://www.paragonwiki.com
City Info Terminal: http://cit.cohtitan.com
Mids Hero Designer: http://www.cohplanner.com
Quote:
|
Originally Posted by Dispari
I don't know why Dink thinks she's not as sexy as Jay was. In 5 posts she's already upstaged his entire career.
|
Quote:
No, but only becuase customers expect more than what is possible, and people who delegate but don't do the work often misunderstand the potential problems.
|
Originally Posted by Another_Fan
If for whatever reason these cause significant difficulties do you think the software is difficult reason will hold water ?
|
There is an old programmer saying, one that I've heard many versions of.
"Software Engineering is the only discipline that adding a wing to a building is considered to be maintenance."
I used that quote in a meeting about 15 years ago. The head of marketing for my project replied "Why would anyone want to add a wing to a building?", of course she was thinking airplane wing.
Imagine the entire group from the engineering division at the meeting face palming at the same time.
Father Xmas - Level 50 Ice/Ice Tanker - Victory
$725 and $1350 parts lists --- My guide to computer components
Tempus unum hominem manet
Quote:
LOL.|
Originally Posted by Aggelakis
Why are you guys even arguing anymore? Another_Fan has way too much invested in his opinion to even think about changing it, and ya'll are not going to change your opinions (because they're generally correct).
Just stop. It got dumb at least ten posts ago. |
Opinion: Software development is complex: Agreement
Opinion: It's too complex to test except by inflicting it on the userbase: Disagreement
As to investment since I have been playing this game and reading the forums, I have seen many suggestions that their proponents felt were reasonable and well within the realm of possibility. What is more at least on the overall level there is very little to make them impossible. The same people over and over come out puff up their chests and proceed to beat down the proposer with rants about how COX is enormously complex, and they know little about the inner workings. This is also true.
The crux is being right about the complexity of COX has little to do with anything.
Recently the game has had the following added to it.
The ability to resize characters between costumes and change genders.
The ability to customize powers.
Its amazing what becomes possible with motivation.
Quote:
|
Originally Posted by Thirty_Seven
No, but only becuase customers expect more than what is possible, and people who delegate but don't do the work often misunderstand the potential problems.
|
There is a bit of truth about being in a position where delegate producing a certain lack of comprehension. The strange thing is, it gets worse the more knowledgeable a person is about a topic. I know that when I moved up into management I became almost completely deaf to people saying the exact same things I had been just a short while ago.
Quote:
|
Originally Posted by Father Xmas
There is an old programmer saying, one that I've heard many versions of.
"Software Engineering is the only discipline that adding a wing to a building is considered to be maintenance." I used that quote in a meeting about 15 years ago. The head of marketing for my project replied "Why would anyone want to add a wing to a building?", of course she was thinking airplane wing. Imagine the entire group from the engineering division at the meeting face palming at the same time. |
So very true.
Of course if architecture operated on the same standards as software engineering, spontaneous building collapses would be excused by the complexity of the discipline.
QR
Personally I understand the time restraints when it comes to the quality assurance portion of this game and in all products in general but I do have to say that it does suck from time to time when you run into situations where the bugs or issues seem too abundant to look like anything is being done.
In regards to this game the only reall issues i've had with this game in the recent past was that damned infernal chat bug and the broken drop rate which were both fixed (the later sooner than the former.)
You really can't blame them for all of the bugs that slip by them or have to ignore in order to take care of the more important fixes.
Quote:
This example is too easy. If the problem is actually triggered by 2500 people logging in, as opposed, to, say, 2500 people being logged in and 10 of them simultaneously combining HOs, then that problem should be caught by a simple load test that logs in 2500 automated clients.|
Originally Posted by SunGryphon
Honestly, how can you expect them to find a bug that only crops up with 2500 people logged in when only 300 people play on the test server?
|
If you log in 2500 automated clients and the actual problem doesn't happen until you hit 3000, well, that's the breaks. You learn about that and next time maybe you test 3000 clients.
You can't expand testing in such ways forever - eventually test complexity or capacity will become more expensive than dealing with the problems you want to catch. Figuring out what tests to run, and how far/hard is part of an ongoing process. That doesn't mean you shouldn't build test facilities, test cases or automation. Having them will almost improve your the quality and efficiency of your QA process. If you set some of the tests aside later to reduce complexity or save time, you can always go back to them if you need to explore an old problem. "Hey, look, something like this happened before, and here's the test case we built for it...."
We aren't talking about silver bullets. We're talking about the difference in using power tools instead of rocks and flints. If you aren't a good builder, you're going to build a poorly constructed house with the best power tools money can buy. But if you are actually competent, you'll build a nice house a lot faster with power tools instead of stone-age ones, and you'll probably drive more of the nails straight on the first try.
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA
Quote:
But this is my point exactly. We have NO WAY of knowing what they are or aren't doing to test this stuff. We can only speculate. How do we know this isn't exactly what happened in this case? People are going to whine and complain that they didn't do enough testing no matter WHERE the line was.
|
Originally Posted by UberGuy
If you log in 2500 automated clients and the actual problem doesn't happen until you hit 3000, well, that's the breaks. You learn about that and next time maybe you test 3000 clients.
|
Quote:
Personally, I think this is a horrible example. This is something that very clearly is begging for automated regression testing. The animation system has clearly defined inputs and outputs. Someone should be able to write something that iterates over the input combinations and validates that it gives the correct output. The only thing about this example that makes it hard to test is the combinatorial complexity, which computers excel at slogging through for us.|
Originally Posted by Dispari
A really good, and recent example of this is the animation bugs. BABs and crew have been fixing animation bugs. When they do, it causes another domino to fall over. After they fix that, something else breaks.
|
Based on comments BaB has been making over the last couple of years about the effects scripting, it seems likely that the devs have long been saddled with two problems. (1) A lot of the game is controlled by manually managed files. Excel spreadsheets, manually modified FX scripts, etc. (2) No one had the skill to manipulate or possibly even improve generation of most of these inputs using automated tools. One of the main things that BaB told us enabled Power Customization was them getting someone on board who could write something to make bulk updates to the FX scripts. They still required a manual inspection/tweaking pass, but the man-hours required was massively reduced because someone wrote a batch parser/updater.
That's the kind of tooling that improves your process. You have to work smarter, not harder. QA is no different. You don't QA just by throwing bodies at it; that's just not efficient. You identify things you can verify in an automated way and then you verify those on the cheap every time you make a change unless you just cannot afford the time to test for some extraordinary reason.
You can leave an automated test running all night, and you don't have to pay it overtime.
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA
Quote:
Actually, we do know they don't have this sort of thing. They talked about it a little bit at the latest HeroCon.
|
Originally Posted by SunGryphon
But this is my point exactly. We have NO WAY of knowing what the are or aren't doing to test this stuff. We can only speculate. How do we know this isn't exactly what happened in this case? People are going to whine and complain that they didn't do enough testing no matter WHERE the line was.
|
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA
Quote:
"This sort of thing" meaning what? They don't have QA at all? Or they don't have a load-testing program.
|
Originally Posted by UberGuy
Actually, we do know they don't have this sort of thing. They talked about it a little bit at the latest HeroCon.
|
Quote:
Interesting that certainly isn't my perception. The original post was certainly short and on the irate side exactly what is expected from a customer when things aren't working. Then there seem to have been a raft of posts seeming to say that its unrealistic to expect software to work. properly.
|
Originally Posted by SunGryphon
But this is my point exactly. We have NO WAY of knowing what they are or aren't doing to test this stuff. We can only speculate. How do we know this isn't exactly what happened in this case? People are going to whine and complain that they didn't do enough testing no matter WHERE the line was.
|
Quote:
Broadly, their QA is primarily achieved via low-pay, human testers, and increased QA is achieved by longer hours or more bodies. There is apparently neither automatic nor load testing. They do, in fact, rely heavily on Test Server beta testing to find bugs of all complexity, because a staff of human QA testers can only cover so much testing. The Test Server is their way of throwing more bodies at it by tapping a (potentially large) body of volunteer testers.
|
Originally Posted by SunGryphon
"This sort of thing" meaning what? They don't have QA at all? Or they don't have a load-testing program.
|
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA
Quote:
That's what I figured. Thanks for confirming.
|
Originally Posted by UberGuy
Broadly, their QA is primarily achieved via low-pay, human testers, and increased QA is achieved by longer hours or more bodies. There is apparently neither automatic nor load testing. They do, in fact, rely heavily on Test Server beta testing to find bugs of all complexity, because a staff of human QA testers can only cover so much testing. The Test Server is their way of throwing more bodies at it by tapping a (potentially large) body of volunteer testers.
|
Really ?
http://www.healthdangers.com/medical...iley/index.htm
There are more. L-Tryptophan dietary supplements come to mind as another. But let me cut to the heart of the matter here. It goes something like this.
1. Most disciplines are held to certain standards of performance accountability
2. Software is inherently harder than other disciplines
3. Therefor Software should not be held to the same level of accountability.
Some counters go as follow.
1. Nobody who isn't in the software development community gives a rats rear about 2.
2. No 2 really isn't true there are many things that are more challenging than software design that regularly perform at a higher standard. To take your example, in medicine: Reboot the patient and restore from the backup just doesn't cut it.
3. Even taking 2 for granted, there are way to deal with difficulties that isolate the end user from them.