The Great Consignment House Heist of May 9th
By
AmazingMOO,
Posted on: 2007-05-09 19:21 in News, Events and Announcement Discussion
<woot>
QR!!!
Thanx Lighthouse- I have to do some research to find out what I lost.....
"I'm not scared of anyone or anything Angie. Isn't that the way life should be?"
Jack Hawksmoor, The Authority.
i lost about 4 or 5 enhancements, SO's and maybe 1 DO, cant remember, im just glad there is going to be compensation, rather than a OH well y'all SOL. better luck next time 
thanks for the update LH.
i lost some rare salvage and recipes...that i got from the stf
ill be quite upset to lose these...lets hope this works
[ QUOTE ]
One thing to keep in mind with this reimbursement plan is that characters have a hard limit of 10 enhancements. If you have a character that engaged in a number of enhancement transactions, you would be well advised to keep that characters enhancement inventory clear. We will be giving as much advance notice as possible as to when the reimbursement process will take place.
[/ QUOTE ]
So, items will be inserted into the player inventory, rather than the transaction slots in WWs? I understand that there is a hard limit either way, but I would think there would be fewer problems with placing them into transaction slots versus storage slots.
I'm going to go out on a limb here and assume you'll be reimbursing the same items? I'd like my rare costume piece back above everything else, and don't really want anything in place of that. I had a lot of things up for sale, but the costume piece was most important as it is rare. You've been perfectly clear about how inspiration and enhancement items will be returned, but haven't really been clear on recipes/salvage and or if they'll be returned the same way 
Thanks for the heads up though. If we could just squeeze a tiny bit more info out of you, I'd be happy
We'll see....
And here we have an exceptionally good reason to not have hard limits on inventory slots. The system should keep track of items that won't fit in the inventory and place them as slots become available during the reimbursement period.
Come to think of it, that would be a great thing to implement as a general feature. That way, you'd never miss out on rewards, but they wouldn't be usable by the player until the player made open slots available.
<QR>
Here's the odd bit. I put a handful of items into WW last night and today, all of the ones I recall putting in there were there or sold, minus the one selling for 30 million influence. Even though I put them all up at the same time. Maybe it was something reflecting on when people bought them instead of me placing them in WW.
I'm not angry at this point, but a little annoyed. The issue for things like this is since the economy is in a constant state of evolution, what was worth 30 - 40 million influence today to the right people, might only be worth 1 - 2 million influence tomorrow; and it keeps chugging along while we wait for our items in limbo. Again, not complaining, per say, but pointing out the dirty facts of the matter. And yes, I'm quite aware the swing could also go the other way.
Regardless, thanks for keeping us posted as opposed to in the dark.
Is this related to the consignment house doing this:
- letting you put a bid on item A
- letting you win the bid based on (it seems) the correct logic
- giving you the correct item
- charging you unbelievably different amounts of inf for the item
?
I noticed this happening last night... after I bought maybe a dozen items for 1-2k each, and was suddenly short 12 million influence! If that 12 million inf comes back I will be SOOOO happy. Although, I was hoping to debug it today and try to earn Bug Hunter to make up for the lost inf. Still, better it be fixed, then I can just play instead of work!
As a SQL Server DBA, I've always kind of wondered on a professional basis what database backend you use.
I have some idea of what kind of effort this kind of restore takes, although of course I don't know the specifics of what you have to do...
Might I suggest a spot of Database Mirroring?
It's probably some kind of trade secret though so I won't press any further.
Currently: 50s (5), 40s (3), 30s (5)
Red and blue side, mostly Infinity, Virtue, and Freedom.
I'm very glad to hear that not all is lost; I had a substantial amount of influence and recipes in the house at the time. I'm glad I stayed calm and had faith you guys would do your best to fix it.
Issue 16 made me feel like this.
Warning: This poster likes to play Devil's Advocate.
[ QUOTE ]
As a SQL Server DBA, I've always kind of wondered on a professional basis what database backend you use.
I have some idea of what kind of effort this kind of restore takes, although of course I don't know the specifics of what you have to do...
Might I suggest a spot of Database Mirroring?
It's probably some kind of trade secret though so I won't press any further.
[/ QUOTE ]
Back during CoH beta, Gaffer mentioned that they ran a SQL Server database. I am not certain if this has changed since then.
[ QUOTE ]
...As a result of this roll back to an earlier save, Heroes who had used the Consignment Market after that time may have a variety of issues from duped items to lost items...
[/ QUOTE ]
Someone's dupin' rares!! 
Unique Dragon was right!!!1!eleventy!! 
Statesmonkey Sez: Lighten up! It's a game, for Lincoln's sake!
Also: Six years of casual play begins to look an awful lot like one year of hardcore play.
Lighthouse,
You are talking about keeping the personal character trays that hold Enh, Insp, Salvage, and Recipes free and clear, right? Not the one used in Consignment House auction menu, Vault or another storage units?
SQL Server Represent!
... it was funny in game one time when I came across a character named "Clustered Index". That's a SQL Server (and, okay, yech, Sybase) specific concept. Well, funny is relative. Recognizing it and that someone called their character that was funny. The name and the concept behind it aren't especially.
Currently: 50s (5), 40s (3), 30s (5)
Red and blue side, mostly Infinity, Virtue, and Freedom.
[ QUOTE ]
For those of you who are just tuning in, we want to recap an important moment in the history of Paragon City the Great Consignment House Heist!
You see, today Wentworths Fine Consignments, the excellent new buying and selling service that recently went into service across Paragon City, came to a grinding halt and apparently some customers items went missing! Police officials are very suspicious of Villainous activities, given that underground sources indicate there were no similar problems with the shady Black Markets on the Rogue Isles. Those same officials are currently investigating and cataloging all missing influence, inspirations, enhancements, recipes and salvage and planning on making a full reimbursement, courtesy of Wentworths insurance agency.
Ok, all role play aside, if you havent seen the announcement thread, you can see some of the blow by blow there. I thought I would recap it for any new folks who were tuning in.
The issue only affected the Hero side Consignment Markets no Villains were affected.
A problem with the back end Auction server that supports the in game Consignment Markets became apparent shortly after our morning update on May 9th. The problem was not related to the update except that it showed up after the servers spun up.
We brought down the Auction Server and investigated. Roughly 5 hours later we brought the Auction Server back up. In order to correct the problem, we had to revert back to a save from 10 PM Central on May 8th. Again, this only affected the Hero side Consignments. Now, we do have a number of safeguards and more frequent saves. Unfortunately, as things typically work out in complicated systems, several things came together to give us trouble. That led to the relatively long time frame of the roll back. We are always very concerned with data integrity and these issues will be corrected to give us better back ups in the future.
As a result of this roll back to an earlier save, Heroes who had used the Consignment Market after that time may have a variety of issues from duped items to lost items. To correct this, we are planning on reinserting the lost items from the transaction logs into players inventories. This includes any lost influence, inspirations, enhancements, salvage and recipes. At the time of this writing we are still testing this, but are confident enough that we can pull it off that Im giving you advance notice of it.
One thing to keep in mind with this reimbursement plan is that characters have a hard limit of 10 enhancements. If you have a character that engaged in a number of enhancement transactions, you would be well advised to keep that characters enhancement inventory clear. We will be giving as much advance notice as possible as to when the reimbursement process will take place.
So, thats the Great Consignment Heist of May 8th as it stands at this moment. Our sincere thanks to everyone for their understanding and patience with this incident. Please know that we are putting in additional safeguards to ensure that such a problem does not arise again. Please keep an eye on the Announcement Forum for further details as they unfold!
[/ QUOTE ]
Maybe I need glasses, but I did not see the word "sorry" appearing even once on this post, or any of the half dozen announcements pertaining to the screw up. To all the people who have lost stuff and are inconvenienced, isn't an apology the professional thing to include right next to your promise to fix the errors?
The only thing worse than devs making bad decisions is the hoard of fanboys and bootlickers that keep cheering them on.
<QR>
Apologize for what?
"I'm not scared of anyone or anything Angie. Isn't that the way life should be?"
Jack Hawksmoor, The Authority.
Oh I don't know, having an error which required a rollback which left lots of people missing their influence and items that they are trying to restore?
The only thing worse than devs making bad decisions is the hoard of fanboys and bootlickers that keep cheering them on.
I'm not trying to start anything here but they probably won't apologize until they know what went wrong and if they can entirely fix it.
Just saying sorry now does nothing but raise the ire of people who just want their stuff back.
But, that's how I see it............
"I'm not scared of anyone or anything Angie. Isn't that the way life should be?"
Jack Hawksmoor, The Authority.
A real apology is an admission of liability, which is why soulless lawyers dislike them so much. An expression of regret is marginally safer.
[ QUOTE ]
I'm not trying to start anything here but they probably won't apologize until they know what went wrong and if they can entirely fix it.
Just saying sorry now does nothing but raise the ire of people who just want their stuff back.
But, that's how I see it............
[/ QUOTE ]
Hey give em a break. Data base problems aren't easy to deal with especially when the amount of data is huge. Oh, and not to burst your immersion bubble but "your stuff" is just electrons stored on "their" silicon wafers.... 
-Insanity is doing the same thing over and over again and expecting different results. - Albert Einstein.
-I do not feel obliged to believe that the same God who has endowed us with sense, reason, and intellect has intended us to forgo their use. - Galileo Galilei
-When injustice becomes law, resistance becomes duty. - Thomas Jefferson
<QR>
Positron apologized for the holiday temp power take away- if they are at fault here, I think they will apologize in time.
"I'm not scared of anyone or anything Angie. Isn't that the way life should be?"
Jack Hawksmoor, The Authority.
Lighthouse,
I'm (for once) on the side that came out massively ahead by this apparent heist. I guess the robbers dropped their stash on my safebox on their way out.
Anyway, I have duped about 12m influence, a number of lowbie and mid-range IOs, and a couple of rare recipes -- all stuff I parted with late last night, but that had been up most of the day to be caught in your save,
Should I observe any special cautions while I'm holding all this ill-gotten loot? Will the feds be dropping by my house?
I have no trouble surrendering the extras to GMs. I'm just not sure to go round a bout how to do that.
I certainly hope they can restore what was lost. I'm not concerned about the items I put up for sale so much as the 36 mil influence bid I had out placed on a fairy wings recipe... which by the look of things is still not enough for a winning bid the last couple of days, but that was most of the influence I had as of last night, and at the moment it's vanished into thin air. 
I knew I was being an idiot to bid that much influence but I sort of did expect to either get my wings or eventually reclaim my money...
And I suppose to complicate things, probably about 2 million of that was made on sales that I made just before placing my new bid, so those would have also been rolled back... ah well.
my lil RWZ Challenge vid
[ QUOTE ]
[ QUOTE ]
I'm not trying to start anything here but they probably won't apologize until they know what went wrong and if they can entirely fix it.
Just saying sorry now does nothing but raise the ire of people who just want their stuff back.
But, that's how I see it............
[/ QUOTE ]
Hey give em a break. Data base problems aren't easy to deal with especially when the amount of data is huge. Oh, and not to burst your immersion bubble but "your stuff" is just electrons stored on "their" silicon wafers....
[/ QUOTE ]
Psssssssssssssssssssst-
I know, and I am your side......
"I'm not scared of anyone or anything Angie. Isn't that the way life should be?"
Jack Hawksmoor, The Authority.
For those of you who are just tuning in, we want to recap an important moment in the history of Paragon City the Great Consignment House Heist!
You see, today Wentworths Fine Consignments, the excellent new buying and selling service that recently went into service across Paragon City, came to a grinding halt and apparently some customers items went missing! Police officials are very suspicious of Villainous activities, given that underground sources indicate there were no similar problems with the shady Black Markets on the Rogue Isles. Those same officials are currently investigating and cataloging all missing influence, inspirations, enhancements, recipes and salvage and planning on making a full reimbursement, courtesy of Wentworths insurance agency.
Ok, all role play aside, if you havent seen the announcement thread, you can see some of the blow by blow there. I thought I would recap it for any new folks who were tuning in.
The issue only affected the Hero side Consignment Markets no Villains were affected.
A problem with the back end Auction server that supports the in game Consignment Markets became apparent shortly after our morning update on May 9th. The problem was not related to the update except that it showed up after the servers spun up.
We brought down the Auction Server and investigated. Roughly 5 hours later we brought the Auction Server back up. In order to correct the problem, we had to revert back to a save from 10 PM Central on May 8th. Again, this only affected the Hero side Consignments. Now, we do have a number of safeguards and more frequent saves. Unfortunately, as things typically work out in complicated systems, several things came together to give us trouble. That led to the relatively long time frame of the roll back. Our apologies for this difficulty and any problems or losses it may have caused our customers. We are always very concerned with data integrity and these issues will be corrected to give us better back ups in the future.
As a result of this roll back to an earlier save, Heroes who had used the Consignment Market after that time may have a variety of issues from duped items to lost items. To correct this, we are planning on reinserting the lost items from the transaction logs into players inventories. This includes any lost influence, inspirations, enhancements, salvage and recipes. At the time of this writing we are still testing this, but are confident enough that we can pull it off that Im giving you advance notice of it.
One thing to keep in mind with this reimbursement plan is that characters have a hard limit of 10 enhancements. If you have a character that engaged in a number of enhancement transactions, you would be well advised to keep that characters enhancement inventory clear. We will be giving as much advance notice as possible as to when the reimbursement process will take place.
So, thats the Great Consignment Heist of May 8th as it stands at this moment. Our sincere thanks to everyone for their understanding and patience with this incident. Please know that we are putting in additional safeguards to ensure that such a problem does not arise again. Please keep an eye on the Announcement Forum for further details as they unfold!
Lighthouse
Community Relations Manager
If you have a specific in game, account, tech or billing problem please contact our Customer Support team via The Knowledge Base "Ask A Question" page.