Maintenance is a bit excessive...
By
Aett_Thorn,
Posted on: 2011-09-30 06:04 in Suggestions and Ideas
Quote:
The current realities of this game are always going to be complicated by its relative age. Not only are they reworking the overall game for the F2P changes but they are doing it to a system that probably has some elements to it that are approaching being 10+ years old (depending on just how far back before April 2004 they were laying the groundwork for the game). Having to work in changes to the account databases and the new Paragon Market transaction servers into the existing rat's nest of code must be a semi-nightmare for them no matter how disciplined they've been.|
Originally Posted by quickfire
And you make a fair point on both counts. Sorry if I inferred you made that contention about taking it off line...that was meant for another, kind of a "take care of in one shot" sort of thing.
Yes, I would have to say that even the best stress test can't beat live. People will simply not load the server like live does. Even with scaling. A load test can only provide information on the OSI protocol stack, and certainly isn't dynamic enough to cover regression. But I still think Sharding is viable. Naturally, again, it depends on setup. I can say they are using LIMITED sharding right now, as I have seen evidence of this, but I think they need to expand it. My problem is that I'm unaware of what they are using to shard...my guess is Codefutures...maybe dbShards? If its active record (ORM) than it's probably likely to be Octopus, part of Ruby if I remember right. Actually I'm not sure if Ruby is outdated. You would know better than I. The point is Sharding could reduce load while the upgrades are made. It's hard to believe that a consistent load of say...10000 players...and that's probably high end...could be served by limited sharding during an active hardware upgrade. Now, if it's a database upgrade that's obviously a different story. They have made SOME database changes obviously. But I'm not sure if that translates to a code change. |
I honestly couldn't tell you much more about their hardware setup beyond what they've told us. We do know that as recently as a week or two ago they were "physically moving server hardware to a new location" so it's likely that quite a bit more is going on than how they are handling their high level server instance applications.
Once again I can totally sympathize that any downtime is bad for a game like this. But given everything they are trying to accomplish I almost think it's amazing they've managed to do what they've done with so -little- overall downtime.
Loth 50 Fire/Rad Controller [1392 Badges] [300 non-AE Souvenirs]
Ryver 50 Ele� Blaster [1392 Badges]
Silandra 50 Peacebringer [1138 Badges] [No Redside Badges]
--{=====> Virtue ♀
Quote:
Again, fair. You have no insight on what they might be using for database software? I didn't recognize any markers myself...Oracle always has a tell. SQL has a few markers as well.|
Originally Posted by Lothic
The current realities of this game are always going to be complicated by its relative age. Not only are they reworking the overall game for the F2P changes but they are doing it to a system that probably has some elements to it that are approaching being 10+ years old (depending on just how far back before April 2004 they were laying the groundwork for the game). Having to work in changes to the account databases and the new Paragon Market transaction servers into the existing rat's nest of code must be a semi-nightmare for them no matter how disciplined they've been.
I honestly couldn't tell you much more about their hardware setup beyond what they've told us. We do know that as recently as a week or two ago they were "physically moving server hardware to a new location" so it's likely that quite a bit more is going on than how they are handling their high level server instance applications. Once again I can totally sympathize that any downtime is bad for a game like this. But given everything they are trying to accomplish I almost think it's amazing they've managed to do what they've done with so -little- overall downtime. |
The non-stacks (inventory) are curious to me. I wonder if that is a marker. But I can't find any indicators anywhere.
At any rate, I'm guessing you feel increasing the sharding will not help.
And look here...we are now debating (or debunking if you want to call it that) my idea. This is a better way to discuss it methinks.
any idea how long this is going to last today
Prof Radburn controller,Celtic Ice Maiden,blaster,Miss Knockout scrapper,Mistress Davina controller,Stone Hart,tank Split Personality PB.Queen Lostris controller,Fridgid Mary blaster,Shocking Fire blaster Future Elfling defender, Little Weed controller,Capo Angelo MM, Commander Buzzsaw MM, Justice Tank tank all 50,s
Quote:
Eh, I never really said "increasing sharding" wouldn't help. Based on the scant few words the Rednames have said about what they are doing with trying to better "load manage" their players may mean something along those lines anyway. It'll be as transparent to us as possible in any event - ongoing changes to something like that might not even make it to the patch notes.|
Originally Posted by quickfire
Again, fair. You have no insight on what they might be using for database software? I didn't recognize any markers myself...Oracle always has a tell. SQL has a few markers as well.
The non-stacks (inventory) are curious to me. I wonder if that is a marker. But I can't find any indicators anywhere. At any rate, I'm guessing you feel increasing the sharding will not help. And look here...we are now debating (or debunking if you want to call it that) my idea. This is a better way to discuss it methinks. |
If you're interested is some of the "under the hood" stuff you could always sniff into the messaging that goes on with some of the third party apps like HeroStats and Titan Sentinel. Those things would give you some hints without having to do anything remotely reverse-engineery against the EULA. Beyond that I've never tried to "look behind the curtain" on this game too much myself because I'm mostly content to play it without wondering about the details. I try to keep work and play as separate as possible if you know what I mean.
Loth 50 Fire/Rad Controller [1392 Badges] [300 non-AE Souvenirs]
Ryver 50 Ele� Blaster [1392 Badges]
Silandra 50 Peacebringer [1138 Badges] [No Redside Badges]
--{=====> Virtue ♀
Quote:
During the I21 Beta, they used a bot program for a stress test. They had over 50 Atlas Parks full, and had all the bots spam the chat server at once. IOW, a massive stress test. More than the servers are ever likely to take at once in Live. But still not able to replicate a Monday Mothership raid on Virtue where we regularly make the server lag. (And my screen turns into a slide show even on minimum settings.) Running bot programs gives them valuable information, but it is still not enough compared to what can happen on live, with live people using their powers in different ways, in different combinations, with different colors even.|
Originally Posted by quickfire
It's a fair point, but there are ways to create a stress test. It's not a perfect solution, but it can reveal the hardware problems they may be having.
|
Quote:
|
Originally Posted by quickfire
Again, fair. You have no insight on what they might be using for database software? I didn't recognize any markers myself...Oracle always has a tell. SQL has a few markers as well.
|
Justice Blues, Tech/Tank, Inv/SS
----------------------
Fighting The Future Trilogy
----------------------
Looks like we've got a practical application of what we've been saying. Graphical problems with a lot of powers being reported after the servers came back up, a problem that was not reported on Test before being brought to Live servers. Doesn't seem like it's happening to everyone, or on affected characters in all places.
So, bug appeared that didn't appear on Test. How should they have planned for that?
Let me never fall into the vulgar mistake of dreaming that I am persecuted whenever I am contradicted.
~Ralph Waldo Emerson
"I was just the one with the most unsolicited sombrero." - Traegus
Quote:
I was having graphical problems on top of sound problems ever since Freedom went live. I bugged it and now they are working again. I didn't see anyone else complaining about the same problems I was having on the forums so I know this wasn't happening to everyone else.|
Originally Posted by Aett_Thorn
Looks like we've got a practical application of what we've been saying. Graphical problems with a lot of powers being reported after the servers came back up, a problem that was not reported on Test before being brought to Live servers. Doesn't seem like it's happening to everyone, or on affected characters in all places.
So, bug appeared that didn't appear on Test. How should they have planned for that? |
Edit: By graphic problems I mean power fx, auras, capes, and clickable objects were invisible. And by sound problems again no power sounds, or that sound you hear when your near a glowey. Now I can see and hear stuff again.
Quote:
There is no such thing as a bad opinion. Only an opinion.|
Originally Posted by quickfire
Ok. My guess is that most players of the game won't have 20 years of Software Engineer experience. That's the only response I could give to that.
Funny how people feel experience in a certain field of work justifies a bad opinion. Also funny how they feel they exist in a vacuum. |
Also; Test=!Live servers. Stuff can and WILL crop up on Live that simply never did on Test and Beta, such as the Server List bug and such.
Quote:
|
Originally Posted by Zwillinger
GG, I would tell you that "I am killing you with my mind", but I couldn't find an emoticon to properly express my sentiment.
|
Quote:
|
Originally Posted by Captain_Photon
NOTE: The Incarnate System is basically farming for IOs on a larger scale, and with more obtrusive lore.
|
"It occurs to me, Captain Solo, that this codebase may not be entirely stable."
"Not entirely stable! Well, I'm glad you're here to tell us these things."
Quote:
Good idea.|
Originally Posted by quickfire
I would recommend that a solution be found as soon as possible...
|
I hope the Devs get this message and get started first thing Monday morning.
I hear ya. We all hear ya. The Devs probably feel it the worst.
Hopefully things will get smoother as quick as can be. Unfortunately, you can't ask for anything more than that. As we all know... poodoo happens.
Anyway, the shame of it is that this game has had such a wonderful history regarding lack of downtime. I mean, we have not had anything like the past month for years.
I cannot argue against: They should have taken care of this more before launching it as completely free.
I don't know what problems may or may not have only been caused once they did launch... but... regardless, they are where they are now and they're just going to have to make the best of it as the experts see fit.
Hopefully it won't have a bad impact on the momentum of Freedom's launch.
To paraphrase Ivan Drago... "If it does... It does."
@Zethustra
"Now at midnight all the agents and the superhuman crew come out
and round up everyone that knows more than they do"-Dylan
and round up everyone that knows more than they do"-Dylan
I hope this guy is kidding. What constant down time? Tuesday mornings?
People don't like playing with bugs, even simple ones. I know people who would happily give up a 8 hour downtime to fix that black bar issue there was in i20 (Where teammates showed up on team bar as black.) It cant' be helped and they are working on it. I can't even imagine how stupid someone would have to be to say "Oh, I started playing the game for free, but if I can't play Tuesday mornings I am bailing."
"NO....No clowns" - Positron
50s: Smasha (SS/SD Brute), House Rules (Mind/Thorn Dom), Wind of Mind (Illusion/Storm Controller), Coraxa (Kat/Inv Scrapper), Summer's Dream (Fortunata)
Quote:
Funny how I also have 20 years experience (and more) in the same field, and I'm inclined to completely agree with Lothic.|
Originally Posted by quickfire
Ok. My guess is that most players of the game won't have 20 years of Software Engineer experience. That's the only response I could give to that.
Funny how people feel experience in a certain field of work justifies a bad opinion. Also funny how they feel they exist in a vacuum. |
What you're suggesting is unrealistic for this particular game and client base. It would be a marketing disaster. Incremental fixes and patches are the optimal solution in this case, and, given a few more days or weeks, things will settle down.
The Devs do their best to encourage people to go try things out during the closed and open betas. They even generate "live stress tests" as best they can - basically every time they do things like the Praetorian Invasion that's a stress test or sorts.
But as SolarSentai said there's always things that crop up in a live environment that'll never show up during a beta no matter how much testing you do beforehand. Are there things they might be able to do to further reduce the downtimes we're seeing now? Sure there's always room for improvement. But my guess, for what it's worth, is that they are doing the best they can given the limitations of time, staff, budgets, etc. No situation is going to be perfect - it's how they balance juggle all the factors at once that matters most.
And for the record I never ONCE thought you ever said the current build needed to be completely unloaded from the live servers. Why you thought I jumped to that assumption is anyone's guess. *shrugs*
Telling the Devs to do this is like telling them they need to breath to stay alive. You're not saying ANYTHING they aren't already doing. My guess is that they are always "aggressively" trying to fix the problems ASAP.
Yes, I would have to say that even the best stress test can't beat live. People will simply not load the server like live does. Even with scaling. A load test can only provide information on the OSI protocol stack, and certainly isn't dynamic enough to cover regression.
But I still think Sharding is viable. Naturally, again, it depends on setup. I can say they are using LIMITED sharding right now, as I have seen evidence of this, but I think they need to expand it. My problem is that I'm unaware of what they are using to shard...my guess is Codefutures...maybe dbShards? If its active record (ORM) than it's probably likely to be Octopus, part of Ruby if I remember right.
Actually I'm not sure if Ruby is outdated. You would know better than I.
The point is Sharding could reduce load while the upgrades are made. It's hard to believe that a consistent load of say...10000 players...and that's probably high end...could be served by limited sharding during an active hardware upgrade.
Now, if it's a database upgrade that's obviously a different story. They have made SOME database changes obviously. But I'm not sure if that translates to a code change.