More Server downtime?!?


Airhammer

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
In response to Tex:

Quote the letter of the law as much as you like. You know as much as I do what happened yesterday, while not a breach of contract, shouldn't have happened and that many in your player base were inconvenienced. As I'm reading the terms of service, nowhere does it state that you absolutely cannot make reparations, only that you are not legally obligated to. Do the right thing and consider making things right for the folks that pay your bills.
I agree that making and maintaining an MMO is not rocket science. I should know. I've worked on life-critical software systems for the Space Station and Space Shuttle. Even with all of the reviews and testing that goes on in changes to the Mission Control Center software, one of the software changes introduced by one of my colleagues forced the delay of a shuttle launch.

Things break. Bad things happen. We try to fix them as best we can. Life goes on.


 

Posted

Quote:
Originally Posted by Magus_Prime View Post
In response to Tex:

Quote the letter of the law as much as you like. You know as much as I do what happened yesterday, while not a breach of contract, shouldn't have happened and that many in your player base were inconvenienced. As I'm reading the terms of service, nowhere does it state that you absolutely cannot make reparations, only that you are not legally obligated to. Do the right thing and consider making things right for the folks that pay your bills.

Jeeze. Entitlement issues? MUCH?

This is software development at it's base. Tens, thousands, possibly even millions of lines of code. And no, it's not something any reasonable person would be able to read and immediately understand without annotations.

Small changes in various places can have drastic effects that may or may not be evident in small-scale testing. And because all these systems interact, it is nearly impossible to tell what may or may not break when updating various things.

Occasionally things break. And sometimes in non-obvious ways. And while, on one level YES it's the fault of the devs. They're human. Not omniscient. It's not like it's intuitively OBVIOUS that a piece of bit-wrangling over HERE is going to cause instability elsewhere.

If YOU can do any better, please apply for a job there. It'd be great if the service could achieve 5 9's.

Until then, accept that the service has downtime. Most planned. But, occasionally, some is unplanned.

And essentially begging "GIMME SOMETHIN' FER MUH TROUBLES!" puts you on par with the window washers who come up and slather your car in water and rub a dirty squeegee over your windshield and then demand money for their "labors".

You have your access to the game. You aren't "owed" anything else.



Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
In response to Tex:

Quote the letter of the law as much as you like. You know as much as I do what happened yesterday, while not a breach of contract, shouldn't have happened and that many in your player base were inconvenienced. As I'm reading the terms of service, nowhere does it state that you absolutely cannot make reparations, only that you are not legally obligated to. Do the right thing and consider making things right for the folks that pay your bills.
I'd like to make a suggestion:

When talking with people about a point over which there is some dispute, try to avoid telling them that they "know" whatever point it is that you're advancing. Just try this for a minute:

Case 1:

You know as well as I do that software bugs cannot be anticipated in all cases.

Case 2:

I have usually found that software bugs can be hard to anticipate.

If you're human, you probably found "Case 1" vaguely offensive; it probably got your pulse up a bit, made you angry. It's condescending. It's rude. It's insulting. It's not a good way to persuade people.

I don't personally think that any particular compensation is in order; I accept the risk of a day or two of downtime here and there as part of my life as an MMO customer. That's my understanding of the spirit of the agreement. A downtime in excess of 12 hours, I might well think it'd be nice to get a bonus day of sub time for, maybe. But really, I don't expect that, because I don't think there's anything unusual about an occasional unplanned outage. These happen, and honestly, they don't happen too often in CoH.


 

Posted

Quote:
Originally Posted by SinisterDirge View Post
You took time off work yesterday so you could switch alignments on your buddy's toon? To each thier own I guess.
I've taken time off work for recreation before. That's why they give me "vacation" days.


 

Posted

Quote:
Originally Posted by seebs View Post
I've taken time off work for recreation before. That's why they give me "vacation" days.
Ah yes, but the way in which the OP put it across though was that (well at least in my eyes) they took unpaid time off.

Then again, when i was working, i was well known for being able to go 11 months without using up any holiday up, so ended up taking the *entire* last month off due to the "use it or lose it" part of our contract....

And during those 11 months, i was still able to take long weekends off to go to the occasional player meet, visit family for new year/christmas, have friends visiting me, play expansions on day of release etc etc.

Depending on your job, it is actually quite amazing what you can end up getting away with... especially if you were like me and willing to work 14 straight days covering other peoples holidays.


 

Posted

Entitlement issues? Am I really being that unreasonable? Something went wrong not just once or twice but throughout the day on several levels. You're all focusing on writing code and I'm not so implacable that I don't realize one mistake can be very pivotal. This detail alone isn't inexcusable but the lack of notice upon the second shutdown and their inability to tell us what went wrong and how long it would take to fix on top of the fact that I had to wait 9 hours to fix? I'll say it again, a good business would know the right thing to do and how to do it just to ease the concerns of the clientele regardless if they signed a waiver. Really, I think it's nice that everyone rallies to the defense of the programming developers but if it weren't for watchdogs to call them on their flaws who's to say they wouldn't take the liberty to run roughshod all over you? Checks and balances is really all this is and I have no compunctions making my thoughts known on the threads.


Magus Prime- lev 50 kin/ elec defender
Meta-Human- lev 50 fire/ ss tank
Cabal Bravo- lev 50 merc/ ff master mind
Schwarzchild- lev 50 grav/ ff controller
Shanghai Storm- lev 50 ma/invinc scrapper
Nicodemus- lev 50 db/ regen scrapper
Dragonhyde- lev 50 wp/ sm tank
On The Pinnochle server!

 

Posted

Quote:
Originally Posted by Tex View Post
Things break. Bad things happen. We try to fix them as best we can. Life goes on.
I hope you don't work customer service.


Magus Prime- lev 50 kin/ elec defender
Meta-Human- lev 50 fire/ ss tank
Cabal Bravo- lev 50 merc/ ff master mind
Schwarzchild- lev 50 grav/ ff controller
Shanghai Storm- lev 50 ma/invinc scrapper
Nicodemus- lev 50 db/ regen scrapper
Dragonhyde- lev 50 wp/ sm tank
On The Pinnochle server!

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
Entitlement issues? Am I really being that unreasonable?
It seems pretty unreasonable to me.

Quote:
Something went wrong not just once or twice but throughout the day on several levels. You're all focusing on writing code and I'm not so implacable that I don't realize one mistake can be very pivotal. This detail alone isn't inexcusable but the lack of notice upon the second shutdown and their inability to tell us what went wrong and how long it would take to fix on top of the fact that I had to wait 9 hours to fix?
Usually, the very nature of looking for a bug is that you don't know what's wrong, and therefore, you don't know how long it will take to fix.

Imagine that you have lost your keys. Do you think it's reasonable for someone to demand that you immediately notify them as to where you lost your keys, and how long it will take for you to get them, before you go off and start looking? Because that would be a comparable accomplishment.

Quote:
Really, I think it's nice that everyone rallies to the defense of the programming developers but if it weren't for watchdogs to call them on their flaws who's to say they wouldn't take the liberty to run roughshod all over you?
Common sense, really. If they don't provide service often enough, they don't keep subs and they don't keep their jobs. Furthermore...

Read the patch notes, play the game a bit. The devs love this game. They are not gonna do a shoddy job on it, because they care. Very few people work in game development, as programmers, without caring a lot. It is, apart from the love of the work, generally not regarded as a particularly great line of work. Long hours, lots of hassles, not the best pay.

The people who are working on I21 are not people who would give less than their best effort to coding and running the game, maintaining servers, and so on. If you wanna point fingers, feel free to point them at ncsoft, who are pretty clearly unaware of the relationship between customers, money, and quality of service.


 

Posted

So you've NEVER had something go wrong in your line of work? You've never been in a situation where a critical system went down while in use? You've never had a power outtage at the office during business hours?

Dude, problems happen. Something malfunctioned, they fixed it. You ARE being unreasonable, and we are every bit as entitled to tell you so as you are to complain.


Main Hero: Chad Gulzow-Man (Victory) 50, 1396 Badges
Main Villain: Evil Gulzow-Man (Victory) 50, 1193 Badges
Mission Architect arcs: Doctor Brainstorm's An Experiment Gone Awry, Arc ID 2093

-----
Quote:
Originally Posted by Nethergoat View Post
it's NEVER too late to pad your /ignore list!

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
Entitlement issues? Am I really being that unreasonable?
Rather than just waiting until everything crashed or corrupted, they brought the servers down to fix it.

Or would you have preferred to have lost XP, Inf, and stuff you bought on the market from a possible server crash and accompanying corruption of your character's config table?

Quote:
Something went wrong not just once or twice but throughout the day on several levels.
Which is why the eventually brought it down. One person crashing and burning is an anomaly. Dozens or hundreds of people experiencing aberrant behavior in the game is NOT an anomaly. A server going *POOF* on them is an indicator that something major is wrong.

They had a couple of the servers go away on them and brought the rest down in a controlled manner so that they would lose as little as possible.

Quote:
You're all focusing on writing code and I'm not so implacable that I don't realize one mistake can be very pivotal. This detail alone isn't inexcusable but the lack of notice upon the second shutdown and their inability to tell us what went wrong and how long it would take to fix on top of the fact that I had to wait 9 hours to fix?
So you want a phone call every time Paragon needs to bring down THEIR servers? You need to be included on the action memo? And they can't proceed until you've acknowledged them?

Howsabout a big helping of NO there.

As to what was wrong. You don't need to know the details. You simply need to know something was wrong and they're working to fix it as rapidly as possible. Downtime is EXPENSIVE. You have a lot of people getting paid overtime to get your stuff back up and running. You're losing revenue every second the systems are down. And your devs can't continue with their new work because they're busy trying to get the live stuff back up.

As to knowing how long it would be to fix. HOW THE *BLEEP!* are they supposed to know that? When's the next time you're going to have to take a ****? How long until your next car accident? How many beers are you going to drink after 3PM on October 17th 2014?

When there's a problem, they bring the servers down and they have to spend time assessing the problem. Then trying to come up with a solution that ALSO needs testing or they're simply dumping bad patch on top of bad patch. All of this takes an indeterminate amount of time. Throwing SWAGs at players doesn't help. Setting artificial deadlines WHEN THEY DON'T KNOW does nothing.

Quote:
I'll say it again, a good business would know the right thing to do and how to do it just to ease the concerns of the clientele regardless if they signed a waiver.
You can say it a thousand times. It just makes you look like a greedy *****. They ARE a good business. They DID the right thing. They brought the service down until they could properly patch it, rather than just leave it up until it bombed out COMPLETELY.

Quote:
Really, I think it's nice that everyone rallies to the defense of the programming developers but if it weren't for watchdogs to call them on their flaws who's to say they wouldn't take the liberty to run roughshod all over you? Checks and balances is really all this is and I have no compunctions making my thoughts known on the threads.
I'm not NCSoft staff. Therefore, I have no corporate mandate to "make nice" to an unreasonable person chucking out the "disgruntled customer" schtick and trying to get something they're not entitled to on the "squeaky wheel" theory.

As such, I have no compunctions about telling you that you don't know what the hell you're talking about. I've actually done programming work. And I do work for customers now as well. In some cases, where EVERYTHING is routine, I can give my clients a timeframe. But when something screwy is happening, I refuse to give a timeframe. Because, until I've studied the problem, I don't KNOW.

The only check needed here is upon your greed.



Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
I hope you don't work customer service.
C'mon. Your bridge is getting lonely. Don't make us get the torches again..



Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

 

Posted

Quote:
Originally Posted by seebs View Post
Read the patch notes, play the game a bit. The devs love this game. They are not gonna do a shoddy job on it, because they care. Very few people work in game development, as programmers, without caring a lot. It is, apart from the love of the work, generally not regarded as a particularly great line of work. Long hours, lots of hassles, not the best pay.
And it's self-important douchebags making unwarranted demands on them, while simultaneously slinging insults that usually cause them to burn out so damn hard. It's really hard to keep caring about something when somebody keeps jabbing you with a poker every time you try to address it.



Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
Entitlement issues?
Since you asked: yes.


 

Posted

Quote:
Originally Posted by Chad Gulzow-Man View Post
So you've NEVER had something go wrong in your line of work? You've never been in a situation where a critical system went down while in use? You've never had a power outtage at the office during business hours?

Dude, problems happen. Something malfunctioned, they fixed it. You ARE being unreasonable, and we are every bit as entitled to tell you so as you are to complain.
I'm not saying I'm infallible. I've made more than my fair share of mistakes in my line of work and when I screw up on the one thing I'm paid to do I forgo getting paid by not turning in my billable hours. This wasn't a mild slip up. The whole game went down. For nine hours. My phone company has reimbursed me for less. I'm asking in the context of being a business, why should I expect anything less from NCSoft?


Magus Prime- lev 50 kin/ elec defender
Meta-Human- lev 50 fire/ ss tank
Cabal Bravo- lev 50 merc/ ff master mind
Schwarzchild- lev 50 grav/ ff controller
Shanghai Storm- lev 50 ma/invinc scrapper
Nicodemus- lev 50 db/ regen scrapper
Dragonhyde- lev 50 wp/ sm tank
On The Pinnochle server!

 

Posted

Quote:
Originally Posted by Hyperstrike View Post
And it's self-important douchebags making unwarranted demands on them, while simultaneously slinging insults that usually cause them to burn out so damn hard. It's really hard to keep caring about something when somebody keeps jabbing you with a poker every time you try to address it.
Yeah. And the thing is... CoH is a real labor of love. I think it always has been, but the team's really gotten into an excellent groove with it these days. Everything I see makes me think that morale is high. Going Rogue worked. It was a really good expansion, and the following issues have been good, and I think they have a real shot at being very successful with Freedom. And they're doing their best... and they did a major hardware upgrade, and whaddya know, something went wrong, and they fixed it.

I don't think forum rules permit me to compare their handling of stuff to that of other MMOs. I will say I think I've seen better, but not by much, and worse, by a long shot.

BTW, a side note: There was a thread in another MMO's official forums about best community rep. I named Zwillinger, linking to a recent post, and one of their community reps affirmed that, yes, Zwillinger is awesome.

When your competitors acknowledge that you are doing a good job, you are doing a very, very, good job.


 

Posted

Quote:
Originally Posted by Hyperstrike View Post
And it's self-important douchebags making unwarranted demands on them, while simultaneously slinging insults that usually cause them to burn out so damn hard. It's really hard to keep caring about something when somebody keeps jabbing you with a poker every time you try to address it.
Way to go with the name-calling. I've made concessions, acknowledging my faults and trying to convey that I'm not so obstinate that I can't empathize but the company messed up on several approaches. I'm not going to let it go so unleash the hounds. As for me slinging insults, by all means go over all my previous posts. I think the most caustic thing I've said is that the developers screwed up and I've made sure that I remain respectful while stating my views. I believe when you resort to being rude your case holds less water and what you have to say becomes less relevant.


Magus Prime- lev 50 kin/ elec defender
Meta-Human- lev 50 fire/ ss tank
Cabal Bravo- lev 50 merc/ ff master mind
Schwarzchild- lev 50 grav/ ff controller
Shanghai Storm- lev 50 ma/invinc scrapper
Nicodemus- lev 50 db/ regen scrapper
Dragonhyde- lev 50 wp/ sm tank
On The Pinnochle server!

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
I hope you don't work customer service.
I thought I made that clear in my sig.


 

Posted

Quote:
Originally Posted by Magus_Prime View Post
Entitlement issues? Am I really being that unreasonable?
Opinion: Yes.
Furthermore . . . how to say this without breaking any rules . . . 'some' posts in this thread come across as really . . . jerky. Many of them are from you.


Paragon City Search And Rescue
The Mentor Project

 

Posted

Quote:
Originally Posted by Ironblade View Post
Opinion: Yes.
Furthermore . . . how to say this without breaking any rules . . . 'some' posts in this thread come across as really . . . jerky. Many of them are from you.
Nice. Real nice.


Magus Prime- lev 50 kin/ elec defender
Meta-Human- lev 50 fire/ ss tank
Cabal Bravo- lev 50 merc/ ff master mind
Schwarzchild- lev 50 grav/ ff controller
Shanghai Storm- lev 50 ma/invinc scrapper
Nicodemus- lev 50 db/ regen scrapper
Dragonhyde- lev 50 wp/ sm tank
On The Pinnochle server!

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
I hope you don't work customer service.
In turn, I hope nothing actually bad ever happens to you, if this is how you react to a little unscheduled online-service downtime. You seem to be pretty short of cope.


 

Posted

Quote:
Originally Posted by Magus_Prime View Post
I'm not saying I'm infallible. I've made more than my fair share of mistakes in my line of work and when I screw up on the one thing I'm paid to do I forgo getting paid by not turning in my billable hours.
And are you working on thousands of lines of code with 100% PERFECT understanding of the interrelations of EVERY system and how it interfaces with every other?

I kinda think not. If you had, we wouldn't be having this discussion right now.

Quote:
This wasn't a mild slip up.
Actually yeah. Yeah it was. To my knowledge, no data was lost or had to be backed out and undone. There've been several bugfixes in the last couple years that have caused people to lose stuff. Like...WHOLE CHARACTERS.

Quote:
The whole game went down.
Because when HALF the game is down, people are screaming for the devs to bring it all down and FIX IT. Apparently you don't understand that to do the maintenance properly the game HAS to go down.

Quote:
For nine hours.
So you lost AN evening (out of 31) of gameplay. Gimme a sec here. I'll summon up a Boo Frickin Hoo for ya...

Okay. No I won't.

Quote:
My phone company has reimbursed me for less.
Your phone company has a different contract with you. They're also a utility and subject to consumer laws and government jurisdictional precedents a game provider like Paragon isn't bound by.

Quote:
I'm asking in the context of being a business, why should I expect anything less from NCSoft?
Because it isn't the same thing at all. Being without a game for an evening doesn't do anything beyond force you to seek other venues of entertainment.

Being without phone service can endanger your life.



Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
Way to go with the name-calling.
Wasn't talking about you, but if the foo *****...

Quote:
I've made concessions, acknowledging my faults and trying to convey that I'm not so obstinate that I can't empathize but
But you still want them to give you something for nothing...

Quote:
the company messed up on several approaches.
Again, unless you have a knowledge of programming on large multi-user products, you really are in no place to pass judgement.

Again, the code in all areas could have been PERFECT. But certain modules interacting with one another (even done right) may have been the issue. Software interactions like this happen on a regular basis. Why do you think Microsoft patches every Tuesday?

Quote:
I believe when you resort to being rude your case holds less water and what you have to say becomes less relevant.
Believe whatever you want. Your beliefs are irrelevant here, as has been stated before.



Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

 

Posted

Quote:
Originally Posted by Magus_Prime View Post
Nice. Real nice.
You asked.

If someone asks if you want the NICE answer or the HONEST one, don't ask for the honest one and then get ticked when it doesn't turn to be the nice one.



Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

 

Posted

Quote:
Originally Posted by Captain_Photon View Post
In turn, I hope nothing actually bad ever happens to you, if this is how you react to a little unscheduled online-service downtime. You seem to be pretty short of cope.
I hear they have prescriptions for that.

And, failing in that, I've heard good things about this "lobotomy" procedure...



Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

 

Posted

Except in this case it's probably not a code problem per se but a server/raid/network configuration and integration problem that cropped up during the server hardware upgrade.

While they could pre-install the server side software on the new equipment, mirror a copy of the user accounts onto the new storage arrays, it isn't until you start final assembly of the whole system, shard servers, instance servers, e-mail server, global chat server, market server and get them to properly sync with each other that configuration problems crop up.

Once you think you've worked all of that out you open up the cluster to all us clients, you then can find other problems syncing between the servers as well as network traffic bottlenecks.

But once it appeared that player data was getting corrupted you have to hit the big red button and take the whole thing down, hoping you have captured enough data the track down the cause.


Father Xmas - Level 50 Ice/Ice Tanker - Victory
$725 and $1350 parts lists --- My guide to computer components

Tempus unum hominem manet