Discussion: Live Server Maintenance - 9/17/11
By
15bribri15,
Posted on: 2011-09-16 18:53 in News, Events and Announcement Discussion
Thanks for coming in on the weekend, guys! 
Quote:
Actually, quite a few of those folks are, and have been working Saturdays. |
Originally Posted by Arcanaville
Thank you subset of devs?
|
I, however am posting this while having breakfast with the family from The Original Pancake House.
Om nom nom.
Andy Belford
Community Manager
Paragon Studios
Quote:
Oh-ho HIGHROLLER. No shady second class pancake houses for you huh!?
|
Originally Posted by Zwillinger
Actually, quite a few of those folks are, and have been working Saturdays.
I, however am posting this while having breakfast with the family from The Original Pancake House. Om nom nom. |
Apparently I'm to stupid right now to make an awesome link with a picture and stuff but neverthelss sign the petition! [u]http://www.change.org/petitions/ncso...city-of-heroes[u]
Quote:
Shame on you. You should have taken this opportunity to enjoy the fabulous Bacon Waffles at Holder's Country Inn.|
Originally Posted by Zwillinger
Actually, quite a few of those folks are, and have been working Saturdays.
I, however am posting this while having breakfast with the family from The Original Pancake House. Om nom nom. |
Tsk tsk.
If the game spit out 20 dollar bills people would complain that they weren't sequentially numbered. If they were sequentially numbered people would complain that they weren't random enough.
Black Pebble is my new hero.
Y'know, I could leap in here and get all huffy being an EU player but I can't be bothered right now. All I want to be able to do is log in which is something I'm hoping this downtime fixes. If not, my mood is likely to go further southwards.
All I'll say is that when it comes to making decisions that affect the whole playerbase, it's inevitable that changes that will affect the least number of players are preferable. As the EU is smaller than the NA playerbase, we get to feel the brunt of this. No official events, no competitions, no presence at conventions, downtime smack bang in our peak times and of course, the accursed global rename travesty where inactive and banned accounts were put as a higher priority than paying, loyal customers.
I know all this and economically, it makes sense. Is it good customer service though? No. Because you do it to a particular section of the playerbase long enough and they get to feel pretty looked down on and even persecuted to some degree. As an EU player, I'm resigned to this. I don't even expect EU players' perspectives to be considered when it comes to change making decisions. Ho hum.
What I will say is that on the whole non-EU players have been helpful, supportive and sympathetic towards EU players who are often out on a limb. And for that I'm very grateful.
Now I'm going to go and cross my fingers and send think positive thoughts for the men and women attempting to fix the servers and have been kind enough to come in on the weekend to do it. Good luck to all of you I say. 
@Dante EU - Union Roleplayer and Altisis Victim
The Militia: Union RP Supergroup - www.themilitia.org.uk
Quote:
I sort of disagree with this. For the last 21 years my job has pretty much dictated my life to some degree. I've worked on holidays MANY, MANY, MANY holidays when others were home. I've worked for days on end. Just recently I worked 12 to 15 hour days, 6 days a week for over 30 days. My one day off which was never the same day of the week I was inundated with phone calls. |
Originally Posted by Hyperstrike
And asking the people who would be patching it, who've already been at work 12+ hours, to sit on their thumbs for another 4-6 hours then come back in the middle of the night is unconscionable. |
I've worked in all sorts of environment rain, sleet, snow, hurricanes, floods. I did not have the luxury of saying I can't come in when my car is blocked in with 2 feet of snow all around it and all mass transit is NOT working. They expect me to get up earlier start clearing that snow and make my way in.
End result, you want a game that people pay a lot, you want a game that people pay to play. You want to have a pay check and support you family. You do what is necessary.
I also want to add I could have retire and collect a good pension and moved on to a different career in the computer industry and I'm still there.
And I guess I will add that being in my current position I deal with many software companies. Everyone wants to say our organization uses their software. When we test something out and it doesn't work we give developers enough time to fix what we find faulty or lacking. If they don't we drop them. They hate when we do that. When we buy software, one its not one copy and two the software is so industry specific its thousands of dollars. So they make money and get "street cred" when dealing with us.
I'm NOT a programmer but I will say this, having to tag market items as "1 of" and "2 of" shows me something about the programming or the limitations in the code being used. And I think anyone who is a programmer can clearly understand what I am talking about. Nutshell I am pretty sure that is something they could have done and tracked on the back end and not hard code into market items.
I find that people give too much leeway to game developers because it is entertainment. Imagine if you will this was the program that ran you business and put food on your table just to find out every update put you out of work. Or even better imagine if this was the OS that ran your computer and every time you had some big service patch it would crash your machine.
Regardless of what this might be business or pleasure it should be working and I shouldn't be dreading every time I see an new update because I know what comes along with it. Add on top of this I play on a MAC so its even worse.
If anything I can say welcome to the world that MAC users have to deal with everyday in this game.
And I know some will say or are at least thinking if it's so bad, then why not leave. All I can honestly say is I am truly considering it. I know it doesn't matter and I'm not gonna break any hearts leaving. Matter of fact I am pretty sure the Devs and Mods would be happy to see me leave as I have never paint them in a good light. I think they are a biased lot and enforce rules as they seem fit and is convenient to them and I have said that to them anytime I had to interact with them, which was very limited. I don't considerably think this company handles it customers very well over all.
Some are complaining about being down for 4 hours on a Saturday. MAC users on i20 were down for over 10 days with ZERO compensation.
In i21 BETA we couldn't even play. By the time they got stuff fixed BETA was closing out. While many were on the BETA servers getting free giveaways, I couldn't even get in if I wanted. And only through the very knowledgeable players that were kind enough to help out on the MAC forums ( CuppaMaga to name one of a few ) were players able to even get in when i21 went live.
Am I going to die because I didn't get a free costume code ? Please of course not. But being down for 10 days you think you could just credit my Credit card a few dollars or even the month ? Are they that tight for money ? If so then they have bigger problems.
Again at this point of the game giving away even 50 merits to an account is not going imbalance the game and would just give customers a good feeling. Its about perceptions, I can screw you over but if my overall perception is I am helping you out then its all good. Again anyone who knows someone in Law Enforcement or is in LE will understand this concept immensely. That is how confessions are obtained many times.
I just don't know really. I think this game is a gold mine but they are just so short sighted sometimes.
1. Why Soft Cap is Important : http://dechskaison.blogspot.com/2011...important.html
2. Limits: http://paragonwiki.com/wiki/Limits
3. Attack Mechanics: http://wiki.cohtitan.com/wiki/Attack_Mechanics
4. Rule of Five: http://wiki.cohtitan.com/wiki/Rule_o...e_Law_of_Fives
Quote:
That's just how I roll. |
Originally Posted by MrDead
Oh-ho HIGHROLLER. No shady second class pancake houses for you huh!?
|
Pancake.
Andy Belford
Community Manager
Paragon Studios
Quote:
Careful, some bite back.|
Originally Posted by Zwillinger
That's just how I roll.
Pancake. |
Triumph: White Succubus: 50 Ill/Emp/PF Snow Globe: 50 Ice/FF/Ice Strobe: 50 PB Shi Otomi: 50 Ninja/Ninjistu/GW Stalker My other characters
Quote:
I didn't know they made patch cables that reached that far.
|
Originally Posted by Zwillinger
Actually, quite a few of those folks are, and have been working Saturdays.
I, however am posting this while having breakfast with the family from The Original Pancake House. Om nom nom. |
[Guide to Defense] [Scrapper Secondaries Comparison] [Archetype Popularity Analysis]
In one little corner of the universe, there's nothing more irritating than a misfile...
(Please support the best webcomic about a cosmic universal realignment by impaired angelic interference resulting in identity crisis angst. Or I release the pigmy water thieves.)
Quote:
I never saw that show, but I'm glad you got to meet some celebrities!
|
Originally Posted by Zwillinger
I, however am posting this while having breakfast with the family from The Original Pancake House.
|
Quote:
No fair i wnat BACON |
Originally Posted by Zwillinger
Actually, quite a few of those folks are, and have been working Saturdays.
I, however am posting this while having breakfast with the family from The Original Pancake House. Om nom nom. |
Quote:
I may never eat pancakes again.....|
Originally Posted by Snow Globe
Careful, some bite back.
|
<shudders in horror, goes back to licking bacon>
Guardian server is Life!
Founder and Leader and all around greatest member of the Solo-SG, The Charybdis Clan.
Quote:
Yeah, the "hot swap" term was what this sounded like to me too. If DAoC was using the term "hotfix" like that, as far as I know that was basically unique to them. Hotfix has meant to me what you described since the 90s.|
Originally Posted by EricHough
Just because one group of folks (the DAoC dev's) used it one way doesn't mean that was the original usage. I think both you and they are confusing it with other tech uses of the term hot - like hotswap drives or hardware.
|
Edit:
Quote:
|
Originally Posted by Arcanaville
Actually, the *original* term "hotfix" referred to fixes that were applied or executed on running systems. Keep in mind that back in the old people days computers were rarely turned off or rebooted in the first place.
However, that usage persists to today mostly in niche areas, particularly self-correcting hardware systems like drive arrays. The modern usage - modern meaning anything past disco - refers to a patch specifically designed to target a specific problem sent to select or all customers generally outside the normal development cycle of the product or system. |
Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA
Quote:
|
Originally Posted by LSK
Well we all know the real reason for this do we not? IT'S A RANGLE PLOT
|
Yeah. But seriously. Two teams are standing by with "eliminate with prejudice orders".
TEMPT NOT HAPPY FUN GOD!
Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.
Quote:
I'm sure they'd hate if if we started typing in all caps, and pantomiming in-game to get our point across.|
Originally Posted by LSK
Hey I wanted to extend it to the French, and German one but like you said the language barrier. We would like to extended it to there. We just need some help from ppl that play there to help with translation. I can speak little German, and no French
 . Even though I have tanks there. |
THIS GAME /em airguitar ROOOOOCKSSS!
Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.
Quote:
I wonder what obscenity Zwill really typed in there...
|
Originally Posted by Zwillinger
...
I, however am posting this while having breakfast with the family from The Original Pancake House. ... |
@Zethustra
"Now at midnight all the agents and the superhuman crew come out
and round up everyone that knows more than they do"-Dylan
and round up everyone that knows more than they do"-Dylan
Quote:
Actually yeah. I have. That's the problem with assumptions. There's a big difference between HA server farms in electronic trading houses and a game. Primarily the amount of money backing the venture. Downtime for the latter costs someone a couple thousand bucks an hour. In the former, it's millions or billions.|
Originally Posted by Siolfir
As for the rest of it... I assume that you've never been any sort of work environment where the words "mission critical" mean "I don't care who you are or how long it takes"?
|
Secondarily, the proper staffing levels to support an HA environment. I don't claim to know how many people Paragon and NCWest have on staff to support their game(s). But I'd find it damnably difficult to believe their development staff can run on a five-nines basis with appropriate staffing levels 24x7x52.
Now if you want to go ahead and pony up a half billion or so REAL dollars, and keep doing it every year, I'm sure Paragon and NCSoft would LOVE to go to an HA format for the game. Then they can actually put the staff on to support that kind of thing.
Until then, reality shows its ugly face.
Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.
Quote:
Translation: rabblerabblerabble|
Originally Posted by Hyperstrike
Actually yeah. I have. That's the problem with assumptions. There's a big difference between HA server farms in electronic trading houses and a game. Primarily the amount of money backing the venture. Downtime for the latter costs someone a couple thousand bucks an hour. In the former, it's millions or billions.
Secondarily, the proper staffing levels to support an HA environment. I don't claim to know how many people Paragon and NCWest have on staff to support their game(s). But I'd find it damnably difficult to believe their development staff can run on a five-nines basis with appropriate staffing levels 24x7x52. Now if you want to go ahead and pony up a half billion or so REAL dollars, and keep doing it every year, I'm sure Paragon and NCSoft would LOVE to go to an HA format for the game. Then they can actually put the staff on to support that kind of thing. Until then, reality shows its ugly face. |
Go. Hunt. Kil. Skulls.
I don't know what you "fixed" server side, but the game is practically unplayable for me currently. I cannot zone into any mission maps without the client freezing at the Warworks/Statesman load screen, forcing me to close the process through task manager and relaunch the client. After doing this and relogging, it lets me zone in.
This is happening on every mission map, on all of my characters. Any time I go into a mission, I have to force close the client and relaunch to be able to enter. I've rebooted, reset my network connection, checked to ensure my drivers are fully up to date, and otherwise checked everything it could be on my end.
This is getting ridiculous now. Since the servers have come back up, I've had to force close the client at least a dozen times so far just to be able to get into missions. As you can imagine, this is not conducive to group play, or playing in general.
Partial Character List:
LENINA 198-DELTA - 50+3 DP/Kin Corruptor (Main) / Captiosus - 50+3 Energy/Energy Blaster
Highlands.Hellraiser - 50+1 DS/Fire Tanker / SpazRat - 50 Claw/SR Stalker
Col. Gregor Aktaybr - 45 Robots/Time MM
Quote:
It's rare for people to appreciate it. Too often people assume that if they spend £5k for a server that somehow that server will provide them with 24/7 service and 100% uptime and the balk at any downtime requests for patching and hardware replacements.|
Originally Posted by Hyperstrike
Actually yeah. I have. That's the problem with assumptions. There's a big difference between HA server farms in electronic trading houses and a game. Primarily the amount of money backing the venture. Downtime for the latter costs someone a couple thousand bucks an hour. In the former, it's millions or billions.
Secondarily, the proper staffing levels to support an HA environment. I don't claim to know how many people Paragon and NCWest have on staff to support their game(s). But I'd find it damnably difficult to believe their development staff can run on a five-nines basis with appropriate staffing levels 24x7x52. Now if you want to go ahead and pony up a half billion or so REAL dollars, and keep doing it every year, I'm sure Paragon and NCSoft would LOVE to go to an HA format for the game. Then they can actually put the staff on to support that kind of thing. Until then, reality shows its ugly face. |
You have to understand that there is no such thing as 100% uptime. "Five Nines" or better is possible but very expensive. You not only need redundant hardware (which is generally easily enough for most things except CPU, RAM and the system board) but also some mechanism which can protect against software failure (which gets exponentially more complicated as the complexity of the software increases).
Even with a multiple host, VM-based solution with failover clustering on a well designed SAN with full network redundancy across multiple physically-separated sites is only good as long as the application itself keeps working correctly and the users don't do anything stupid like delete a load of important records or accidentally lose $2bn in dodgy stock market trading.
Omnes relinquite spes, o vos intrantes
My Characters
CoX Chatlog Parser
Last.fm Feed
Quote:
|
Originally Posted by Captiosus
I don't know what you "fixed" server side, but the game is practically unplayable for me currently. I cannot zone into any mission maps without the client freezing at the Warworks/Statesman load screen, forcing me to close the process through task manager and relaunch the client. After doing this and relogging, it lets me zone in.
This is happening on every mission map, on all of my characters. Any time I go into a mission, I have to force close the client and relaunch to be able to enter. I've rebooted, reset my network connection, checked to ensure my drivers are fully up to date, and otherwise checked everything it could be on my end. This is getting ridiculous now. Since the servers have come back up, I've had to force close the client at least a dozen times so far just to be able to get into missions. As you can imagine, this is not conducive to group play, or playing in general. |
Start a thread in the technical forum. Use the "Help Me!" guide in my sig to provide your CoH Helper and Hijack This logs. If you're hanging and crashing that isn't the server. That's you.
Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.
Quote:
|
Originally Posted by The_Spad_EU
It's rare for people to appreciate it. Too often people assume that if they spend £5k for a server that somehow that server will provide them with 24/7 service and 100% uptime and the balk at any downtime requests for patching and hardware replacements.
You have to understand that there is no such thing as 100% uptime. "Five Nines" or better is possible but very expensive. You not only need redundant hardware (which is generally easily enough for most things except CPU, RAM and the system board) but also some mechanism which can protect against software failure (which gets exponentially more complicated as the complexity of the software increases). Even with a multiple host, VM-based solution with failover clustering on a well designed SAN with full network redundancy across multiple physically-separated sites is only good as long as the application itself keeps working correctly and the users don't do anything stupid like delete a load of important records or accidentally lose $2bn in dodgy stock market trading. |
Heh. I still remember having an entire IT department go pale when I suggested rebooting their server. Essentially they were all helpdesk guys who left the server alone and never messed with it. Because it took on the order of 40 minutes for the machine (an AD box running a MONSTROUSLY HUGE Exchange) and they were all but crippled when it was down.
Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.
Quote:
heheh nice.|
Originally Posted by Hyperstrike
Heh. I still remember having an entire IT department go pale when I suggested rebooting their server. Essentially they were all helpdesk guys who left the server alone and never messed with it. Because it took on the order of 40 minutes for the machine (an AD box running a MONSTROUSLY HUGE Exchange) and they were all but crippled when it was down.
|
The company I work for; I get to reboot some servers on Mon. and Fri. nights and one of those servers is the Exchange server (on a Fri. night too of course).
Some times that servers takes 5-10 mins just to turn off (reboot) and another 5 mins to turn on for me to log in to it.
This past Friday, it took 12 or so mins. to reboot it
Grr...not fun when trying to leave work early (or at least on time)!
Leader of The LEGION/Fallen LEGION on the Liberty server!
SSBB FC: 2062-8881-3944
MKW FC: 4167-4891-5991
$DAYJOB had a server failure recently. Multiple-disk failure of THE central source code repository for my group. All the automated builds started failing.
It was back a day later. I think it may have cost me as much as five minutes of productivity. Obviously, it soaked at least a solid day from someone, probably two or three days total, but we use distributed stuff and our source system is git, so basically we were fine and lost nothing.
Yes, all of this downtime can be annoying when you want to jump into the game, however, this isn't the first time they have done emergency maintenance on a weekend. Honestly, I expected some of this for the first week or two of open access, so it's really no big deal. I'll take some downtime now before they open up the servers to everyone else if it makes the final release better for the new players.
"I never said thank you." - Lt. Gordon
"And you'll never have to." - the Dark Knight