The Three Common Types of Downtime

Hey All,

I wanted to take a few moments to chat with you about downtimes since I know they can be a point of contention for you sometimes. Downtime can definitely be a bummer, for they prevent people from logging into the game and using the service. We try and minimize them as much as possible because although they are a necessity we really don’t enjoy inconveniencing anyone. Of course, we want the game to be up and available to as many people as possible all of the time.

Like I said, however, downtimes are necessary for a variety of reasons. The first reason is maintenance. Maintenance in any online service is a fact of life. That being said, we're always working hard to reduce the length and amounts of downtimes that need to be performed. That’s why we've switched to the new trial schedule of only one maintenance period a week. We need to perform maintenance to keep the game functioning, but we need to reduce it to an acceptable level.

The second common cause of downtime is content patches. Patches introduce new features into the game, or change how features in the game currently work. These patches occur less frequently than maintenances. The upside to these patches is that you get some new content or system (or bug fix) along with them, that hopefully adds to your enjoyment of the game. When we make these kinds of patches, we try to include as many things as possible in them so that we can execute them fairly infrequently.

The final cause of downtimes is the most infrequent and is one used to address a problem in the game – for example – something that is not working as designed that is negatively impacting your game playing experience. Sometimes we are not at liberty to discuss the details of these issues, but please know that we only take the game down to address them only in the most serious of circumstances.

Anytime we have downtime one of the issues that OCR raises is “How will this affect players, particularly those players who live in the Oceanic region and whose prime play time is most impacted by these decisions?” This issue is always a consideration for us, but depending on the situation… regardless of who may be most impacted, a downtime may still be required. We've stepped up our efforts internally to properly notify the players of downtimes and will continue to try and give players as much notice as possible. We just do ask that you realize that true “emergency situations” do sometimes occur and this might affect the amount of notice that is given.

Sometimes, downtime caused by necessary fixes to the game follows a content patch, and this can make a downtime last longer than we’d like. We recognize this is an issue, and are very focused on providing stable content patches that reduce the need for fixes that cause additional downtime. When possible, we also fold in content upgrades with our regular maintenance schedule so as to not increase the frequency of downtime more than usual.

We wanted to go ahead and share this information so that you all really understood what was going on during downtimes and didn’t think that we did them simply to annoy you. There are always very good reasons for each and every one we initiate, although that might not always be visible on your end. So with that, we’d just like to thank you for taking the time to read this and educate yourself on the subject. We also want to thank you for your continued patience during any and every downtime that we might need to execute!

Thank you for taking the time to post.

And I'm really happy about only one scheduled maint per week.

I realise that the maintenance isnt personally directed at us in Oceanic time zones. And I glad there is an understanding that they effect us badly.

The schedule makes it hard to organise activities and bring new people in to play. Its a self fulfilling prophecy of sorts, the downtime is picked to minimise disruption as there are lower numbers of people on at that time, but the more down times are at those times will lead to less people in our timezone being on at all.

Thanks for the post Ocho. I'm glad to hear that we are considered in any discussions about additional maintenance.

But mostly, I appreciate that you have changed the start time for tonight's extra maintenance ( ) to two hours later, back at the old time.

Yes, somewhere in NC a manager is getting heat for slipping some "low risk" fixes into the halloween frequency patch. Turns out they weren't so low risk and the servers are unstable. Could someone give them a poke for me too, I know this behaviour far too well.

I do appreciate that people are paying attention to our plight. Thanks to TheOcho for coming over here and taking the time to address our concerns. That was very nice. Thank you for moving the maintenance time in an effort to be less obnoxious.

Seriously, if you guys need a hand we can organize a working bee. Lili, Min Min and Frozen Motion all all professional programmers of the highest skills and quality. It would be a great opportunity for us to return the love.

Then there's the problem of someone pulling out the server power cord so they can plug in the coffee machine...

oh drek, more maint tonight.

Please consider that rotating schedule. So once in a blue moon it doesnt effect our prime time. And you dont get much more prime time than 9:30 on a friday.

Thanks Ocho, but like a number of the rest of us, I was already aware of the problems with managing apps on servers and are quite aware that patching and fixing has to happen and developers aren't godlike beings that always get things right first time. So the issue is not that it has to happen, but that it has to happen in my prime gaming time every time. Regular maintenance yes, we Aussies can plan other things for Thursday nights (though the TV schedule is lousy:-))but patches and fixes surely could sometimes be done at another time.

Thanks for posting Ocho, I think that a little feedback and forethought in this area goes a long way.

Quote:

Originally Posted by Liliaceae Yes, somewhere in NC a manager is getting heat for slipping some "low risk" fixes into the halloween frequency patch. Turns out they weren't so low risk and the servers are unstable. Could someone give them a poke for me too, I know this behaviour far too well.

Ironically enough, I'm absolutely that guy in my day job. Difference is that I've never made a mistake in my life.

Thanks for the clarification Ocho, but to be honest it just sounds like "Nothing's going to be done about this, it's just the way it goes and you people happen to be the ones who suffer every time. <shrug>"

Is there *nobody* actually doing change management at NCSoft?

Looks like if we weren't all aware of what scheduled maintenance involves and why it needs to be done, we can't claim ignorance now! I'd like to thank TheOcho for commenting on the issue.

For the other posters, here's something to remember, this is a trial. I'd quite like the trial to become permanent. Once a week has got to be better than twice a week right?

The rolling schedule idea has merit, but I don't think it would be economically viable to pay people to get out of bed in the middle of their night to do it.

Let's give the trial some more time, see how it flies when there aren't events like Halloween and icky server stability issues to sully our view of it.

One issue with this trial during this period of the year is that it comes with a lot of holidays that in turn increases the load on the servers.

However, I'm not expecting an end-of-year patch (it would be in beta by now if they planned to have an I17 out before Xmas). So perhaps the maintenance will be more 'normal' in that regard.

I'd be very happy if they just stuck with the Thursday one, making an effort to keep it as short as possible, and making patches as infrequent as they can and on the scheduled day.

They've indicated that this is what they intend and I'll be thrilled if they can carry it through.