Titan Network: Need some assistance

Also, because it has to be said, go Falcons!


UPDATE: (17:05 UTC Nov 06)

Woot! We have site uppage! I've just removed the restrictions on the following sites:
- The main Titan Network site.
- City Info Tracker
- repo.cohtitan.com (distribution site for Sentinel and Mids' Hero Designer)
- avatars.cohtitan.com (Provides custom avatars)

A few very important caveats to go with that, though:

First, we've made some pretty significant changes on the back end, including locking down the security of the sites a LOT tighter and rolling out a significant upgrade to the CIT back-end framework. We've tested as much as we can, but there's always that bizarre one-off or little-used forgotten feature that we still haven't poked around with. As a result, it's not just possible that there will still be some glitches, but probable. Let us know if you get any error messages or can't do something you used to be able to, and we'll tackle them as we can.

Second, there seems to be a minor issue going on with the webservice that I'm working on ironing out that's preventing Sentinel from logging in to it. I'll post an update as I find out anything.

Third, some folks have reported issues with their passwords. If you have any trouble, please try resetting your password first by filling out the forgotten password form. It will e-mail you a new password. If you don't get the e-mail within a few minutes, check your spam folder! If you find something that's still broken, please report it here, or if it's something that's preventing access to our forums, drop me a PM or e-mail me at tonyv@cohtitan.com and let me know.

I'm going to take a break for a while since I've been working on this stuff for a solid week straight (!) and watch the Falcons game. Yes, I know they're playing the Colts and every bone in my body says I don't really have to bother, but you know what they say about any given Sunday, so I have to see what happens. A little later, though, I'll work on getting Faces upgraded and restored.

Thanks again for all of your kinds words and encouragement. If it's any consolation, we've also taken this opportunity to do some much-needed upgrading of a lot of software including Linux and the core apps (Apache, Php, MySQL), MediaWiki, CodeIgniter, and SMF. This is a huge deal, and even under normal circumstances, entails a lot of work with significant risk of stuff breaking that we have working. I can't say enough how much I appreciate everyone on the back end administration side of the Titan Network pulling together and helping out to make this happen!


UPDATE: (07:35 UTC Nov 06)

Okay, not much of an update, but here goes. Upgrading the sites to the latest version of our framework is proving harder than we anticipated. Fortunately, I was able to get the help of a guru tonight who was able to lend us a lot of awesome help, enough to get CIT not just back up and running, but on the latest version of the back-end software. There are still a few little kinks, but after a few more validation tests in the morning, I anticipate CIT--including the Sentinel and Mids repositories and the custom avatars--being back up and online by noon or so. If the kinks are still present by then, I'll probably still bring them online and just let everyone know what they are as Known Issues until we can get them ironed out. So far, I haven't run across anything significant that would hold up the works, and that's a very good sign.

I'll tackle the webservice site required for Sentinel communication wtih CIT after that. Faces might be late tomorrow afternoon or night. Sorry folks, I know that a lot of you use and like Faces, but being the attack vector used to compromise our system, I'm giving it a LOT of extra attention before I turn it back on. The Ouroboros Portal and RedTomax/City of Data will be back as soon as I can get to them, I promise.


UPDATE: (06:30 UTC Nov 05)

A bit of good news / bad news. The good news is that we've found out more details about how this happened. In fact, I think we've nailed down the exact exploit that was used. Turns out, it wasn't our old install of SMF that was the culprit; at this point, we're almost certain that it is the version of CodeIgniter that the Faces site was running, which allowed an attacker to upload a malicious file.

The bad news is that until we do some code changes, we're still considering the Faces site (and CIT, even though the exploit has been fixed in the version of code it's running) as vulnerable to a repeat attack. It's kind of hard because what very little information is out about the exploit merely acknowledges its existence, so the lead devs and I have been chatting about it for a couple of hours now trying to reverse engineer what happened to make sure we don't still have accessible malware on the affected sites.

Just to give an idea of why we're being so paranoid, apparently our domain names got harvested into the botnet as a compromised host, and even on the new server I've seen several attempts to access the attack vector (which I must emphatically stress HAS BEEN CLOSED) from zombie machines in various parts of the world to try to reinfect it.

Until we can nail it down for sure, we're going to keep the sites that are currently down deactivated publicly. The sites that are up (our forums, the Paragon Wiki, and HeroStat sites) aren't affected, and were never at risk due to the attack vector we're researching. I'm hoping we can figure this out soon rather than Soon™ so that we can restore service.


UPDATE: (00:05 UTC Nov 05)

Okay, it's been a long haul today and I know I've been pretty quiet in this thread, but I think we're aaaaalmost ready to turn the rest of the sites back on. All of them except Faces are physically up and running on the new server, and they're just locked out to the public while we finish some internal testing and validation. We've run into a few snags with the synchronization of passwords between the Titan Key and our forums due to SMF changing their authentication mechanism. We might go ahead and turn the rest of the sites back on and just warn people that they might not be able to post to our forums until we figure it out.


UPDATE: (15:40 UTC Nov 04)

Great news, the HeroStats site is back up and online. This one was pretty easy to restore and had no dependencies on the rest of the Titan Network, so I went ahead and scanned it and popped it back out there. I should have said this earlier, but the HeroStats site was never affected by any of this; it's been clean all along. If you had only visited the HeroStats site and not the Titan Network, you're not impacted by any of this. Apologies to Ineffable_Bob, the builder and maintainer of HeroStats, for us having his site down the past few days.


UPDATE: (07:20 UTC Nov 04)

Doggonit, I was really hoping to have the main site page up tonight. I think it's ready, but I'm not going to turn it on until we complete some internal testing and make sure I haven't hideously broken something.

Tomorrow I have the day off, and my intention is to try to get as many of the remaining sites up as I can. The only hard parts will be the parts where I have to actually modify code, which from here on out, I'm hoping is very seldom. I really am hoping to have made a pretty big dent in getting most if not all of our sites back up before this time tomorrow night.

The order in which I'm concentrating on getting things back up is:
- The main page, required to create and maintain your Titan Network account.
- The download site, so that people can once again get Mids, Waterworks, and Sentinel.
- City Info Tracker
- Webservice (required for Sentinel to interact with CIT)
- Faces
- Avatars
- Ouroboros Portal
- RedTomax/City of Data

In addition, we're hosting the HeroStats site, which is currently offline as well. I'm going to make that a priority as well, up towards the top of the list, though I don't know exactly where yet. It should be really straight-forward and not take long, so I might do it first thing when I get up in the morning. Also in the mix is the Infinity Taxibots site, which is hosted on our server, which is down towards the bottom of the list since I'll probably have to upgrade the back-end software for it.

I'll post more updates tomorrow as I have them. Unlike the past few days, during which I've had to go work my day job (around 10 hours a pop I can't work on the sites ), I'll be dedicated to working throughout the day on the sites, so you'll probably see several updates before all is said and done.


UPDATE: (06:05 UTC Nov 03)

I know it looks like nothing has been done, but we have been really busy tonight. After several of the Titan devs and I nailed down a new security scheme last night, we actually rolled it out today. It's a little more complicated than the old scheme and I expect we'll have glitches now and then, especially as we make tweaks and upgrades to the sites, but since the web server account no longer has write access to any files or directories it doesn't absolutely have to, it should effectively prevent the specific kind of attack we got hit with.

That having been said, we've upgraded the Titan Network forums to the latest and greatest (and most secure) version of SMF. That's the good news. The bad news is that in the process, we lost some of the customization such as being able to navigate to people's CIT and Faces pages from their profiles and different icons for the different forum sections. At some point, we'll look into restoring that, but of course, only after we get the other sites up and running. Also, some of our testers have reported some password sync issues. If you can't log in yet, please be patient. You might have to reset your Titan Key password once we get the home site (cohtitan.com) back up and until then, you might not be able to post there. We're working on it as fast as we can, I promise. The home page is actually up and running, but I decided to delay making it accessible to shore up a few auxiliary scripts that need tweaking. It's really, really close though.

We've transferred over most of the databases, changing all of their passwords along the way. This means that some of our back-end code is going to have to be tweaked with the new passwords, which we're doing as we move filesystems over and set permissions. Also, there were some very minor changes to some tables that we've had to account for. There are a few more big honkin' filesystems that have to be moved over such as the CIT avatars and Faces photos, which we'll probably do tomorrow night after we get the main site back up and fully functional.

I have managed to secure Friday off from work (woot, vacation!) so that I can hopefully get most everything up and running by the weekend.


UPDATE: (06:05 UTC Nov 02)

Hey all, didn't want you to think we've been slacking off tonight.

The main thing we've done is discussed some really boring fundamentals about things like filesystem permissions and user accounts. I've been in a chat session with the other Titan Network devs and admins regarding what will work, what is required, what issues we might face, etc. I've also been testing the heck out of various combinations of permissions and user accounts to make sure everything still works right. I'll be implementing it on the wiki site first just to make sure we have the groundwork laid for going forward with restoring all of the other sites.


UPDATE: (15:17 UTC Nov 01)

Okay, MediaWiki upgrade complete, the site should be fully available now. Off to work, I'll post more updates as I continue work when I get home tonight. If you see any problems with the wiki (other than not being able to make any changes, since I still have it in read-only mode), PM me or drop me an e-mail at tonyv@cohtitan.com.


UPDATE: (14:35 UTC Nov 01)

I can't work for long this morning, I really do have to go to work today. However, I am in the process of creating a backup of the wiki in preparation of an upgrade to the latest and greatest version to make sure we don't have any security holes in MediaWiki. Once I get home tonight, I'll put the wiki back into read/write mode and start validation that it works. If you don't have an account on the wiki yet you won't be able to log in since we don't have the main Titan site (which is used to create Titan Keys) running yet, but we're getting there.

In the meantime, if you notice the wiki slow or down, don't panic, that's just me working on it.


UPDATE: (07:06 UTC Nov 01)

Great news! We have gotten the Paragon Wiki (at paragonwiki.com or wiki.cohtitan.com, whichever you prefer) up and running! It's in read-only mode, so no updates until we get more work done, but the source code has been sanitized and it is running on a shiny new installation of Ubuntu Server 11.10. If you still can't get to it, you might have to wait a while for DNS propagation to complete (technically up to 24 hours, though it hardly ever takes that long).

So there you go, progress! Now I really have to go to bed. I'll pick up the updates tomorrow, probably after I get home from work. Thanks again a TON for all of your support, and we really do apologize for the hassle. For what it's worth, we really are working hard not just to restore service, but to perform all upgrades and lockdown steps to ensure that this doesn't happen again.

Mini-update: For some weird reason, paragonwiki.com is taking longer to propagate than wiki.cohtitan.com is for a lot of people. If one of the links above doesn't work, try the other. If neither of them work, give it a little while longer, it is getting out there, I promise. Within 12 to 18 hours at most, both should be working fine. If you absolutely have to have access to the wiki right now and you're technically proficient enough to know what this means, add the following to your hosts file. (But be sure to remove it in a day or so!):
50.116.49.221 paragonwiki.com wiki.cohtitan.com


UPDATE: (05:00 UTC Nov 01)

Still at it. We've decided to pick arguably the most used part of the site to focus on first: the Paragon Wiki. I'm in the process of moving the files and database over. We'll most likely only make it available in read-only mode initially while other details regarding user accounts are sorted out (and to minimize risk of any more hackage), but at least it will be there for reference.

If I can at least get that far, I'm going to consider that a win and call it a night. It's getting really late here on the east coast (1:00am), and I'm pretty bushed at the moment.


UPDATE: (02:55 UTC Nov 01)

It's safe to say that the sites aren't going to be back up tonight. It's not that we're dealing with anything particularly dangerous at this point, it's just that it takes a loooong time to configure a new server from scratch, especially one that we've been running for years. We're also taking the opportunity to lock a few more things down that aren't related to this incident, but that we want to anyway. In particular, I'm deleting some user accounts and cruft that has built up over the years, small little cracks that we've had to enable for one-off purposes (or that I didn't know about) but that are no longer valid.

I'm in the process of re-establishing the databases now, which involves configuring a bunch of back-end user accounts and privileges. Cleaning the files is probably actually going to be a little easier than I expected, although we're going to take our time on that too and make sure we get the permissions right.

In short, bear with us, we're making headway, but it will still be a while yet. Unfortunately, I can only get away with so much "sick" time before my boss at my day job starts questioning my leeway, so it will probably be at least sometime tomorrow night before we have anything significant up and running. (Though I'm trying to at least get some basic functionality enabled before then.) I'll keep posting updates as I have them.


UPDATE: (00:30 UTC Nov 01)

I've received a note from a user saying that their machine was infected. Also, the malware site that was loading via a hidden iframe does contain a malicious payload. I would highly suggest that anyone who has visited the Titan Network sites in the past 48 hours or so to run a virus scan on your machine. If you have recent copy of Windows (Vista or Windows 7, if I recall correctly), you can use Microsoft Security Essentials. ClamWin, avast! and AVG are other options, though with the latter two, please be sure you disable their adware crap when you install it. Obviously, several commercial antivirus programs exist. Personally, I don't like the 800 pound gorilla in the market (Norton/Symantec's products). Kaspersky is a viable alternative.

Again, I want to emphasize that we do not believe that the user database on the server was compromised, which contains e-mail addresses, usernames, and hashed passwords. If we find out that it has, I'll raise red flags as high as I can, because that's something you really need to know. Fortunately, though, indications are that this was a simple bot attack, something a so-called "script kiddie" put together, not a hack specifically targeted at our sites. Whoever did it was pretty stupid in that they sure weren't very subtle about not being caught. The objective appears to be to compromise as many pages as they could for as many redirects as possible before someone shut the sites down, which we did this morning.

So don't panic, but do take some common-sense precautions.


UPDATE: (23:50 UTC Oct 31)

Unfortunately, my day job has been kicking me pretty hard today. Even though I called in sick, I still got roped into two multiple-hours-long conference calls.

So anyway, right now, I'm working on getting a very basic maintenance page up so that at least something is responding to web requests. You'll probably be seeing it instead of the "Connection Refused" messages very soon as DNS propagation takes place. I should be done with that shortly. After that, I'll work on sanitizing and moving our sites over. Indications are still that the only thing that was compromised was the content pages, not the user database. GuyPerfect is currently digging into the distribution server site to try to find out more info on that front.

Again, thanks a ton for everyone's patience and understanding, it really does mean a lot, and I promise, we're working as hard as we can to restore service.


UPDATE: (18:30 UTC Oct 31)

Not much to report yet. Just a quick note that while we have the sites down, you will receive a "Connection Refused" error (or if using IE, probably a vague "Page cannot be displayed" error). This is normal, and just means that there is no service on the server to respond to HTTP requests. I'm working on getting another server up and running to at least show a maintenance page.

Incidentally, I'm almost certain now that this is what was causing the slowness yesterday. I think it was doing a full filesystem scan, looking for those index.php files, and our filesystem is quite large.


UPDATE: (16:50 UTC Oct 31)

I've called in "sick" today from my day job to try to dig into this more deeply, but we don't have coverage from 18:00 UTC until 22:30 UTC (that's 2:00pm until 6:30pm for you east coasters like me). I promise, I'm working as fast as I can here, but in an hour or so I'm going to have to juggle this with some other tasks. Right now, I can't guarantee that someone doesn't have root access to the server, so here's what I'm going to do.

I'm currently setting up a second server to replace the first one. Instead of trying to clean up all the mess, I'm seriously considering just doing a full restore from Saturday (the last known good configuration) and migrating all of the data and files over to the new server, this one running a newer version of the Linux OS and locked down more tightly. All of this is going to take some time, but it's the best way I know of to ensure that 1) nothing corrupt is copied, and 2) we are locked down more tightly than we were.

If your are a Titan Network developer: Because our forums are down, I'll try my best to get at least a temporary forum up where we can talk about options going forward. In the meantime, you know my Skype name (tonyv.paragonwiki in case you don't), feel free to hit me up there. Please note that at least initially, I'm only granting access to people I know are current, active developers. If you don't have access to something you did, don't get all hurt or offended.


UPDATE: (15:50 UTC Oct 31)

In case you're just tuning in, I've taken all of the Titan Network sites (Paragon Wiki, Planner, Faces, etc.) down temporarily due to a server compromise. In the interest of disclosure and hopefully to assure you that we are on this, here's what I found.

Apparently, someone gained enough access to the server that they have injected code at the top of all index.php files. The code checks the user agent to see if it's as webcrawler. If not, it sends a request for a javascript malware package from a distribution server, which is apparently performing some kind of check on the back end, because it's not sending its package to everyone. I'm still looking into it, especially to find out the nature and source of the compromise, but at this point, I don't want to set the expectation that the server will be back up in the immediate furture. If we're lucky, it may be tonight. (Emphasis on may.)

I still believe that the user database has not been compromised, that this is only a malicious redirection attack, although due to the nature of the changes I'm seeing, I can't 100% rule that out. Obviously, I'll keep everyone up-to-date as I find out more information.

Again, I don't want folks to panic; like I said, we're being paranoid. I want to be absolutely, 100% crystal clear about this: Your security is more important to us than some fan sites. We'll have the sites up against as soon as possible, but not until we're absolutely certain the compromise has been handled.

Apologies again for the inconvenience.


UPDATE: (15:18 UTC Oct 31)

I've definitely found a compromise on the site.

I'm continuing to investigate. In the meantime, in order to prevent malware infection, I've taken down the Titan Network server. I apologize for the inconvenience, but security is our top priority, plain and simple, and until I'm convinced that people's machines won't get infected from visiting our sites, I'm keeping it down. This includes all Titan Network sites, such as Paragon Wiki, the Planner sites, Faces, Ouroboros Portal, RedTomax/City of Data, and our own forums.

I'm pretty sure not all sites are affected, but right now, I'm being paranoid. Again, current indications are that user data has not been compromised; all I've found so far are malicious redirects. I'll continue posting new information as I get it.

I'm available on Skype at tonyv.paragonwiki if anyone has any questions, though if I start getting bombarded, I might go into busy or offline mode.


Original Post: (14:56 UTC Oct 31)

Hey all,

Beginning at 2:00am Eastern this morning (06:00 UTC), I started receiving messages from people saying that they are receiving warnings from malware scanners/detectors that some parts of the Titan Network are giving warnings. We take the security of our sites very seriously, and I'm investigating now.

In the meantime, I could use some help. I can't replicate the message that anyone else is getting and I'm not seeing any indications of compromise, so if you get a message, please do the following:

• Right-click somewhere on the page other than on a link or image.
• From the popup menu, select "View Source"
• Copy the entire source code of the page on which the warning was generated.
• Paste it in an e-mail to admins@cohtitan.com
• Please also include the full URL of the page it was shown on, how you got to the page (did you click a link to get there? please include it!), what browser and, if you know it, browser version you're using, and what program detected it. Also, a screenshot of the page and warning would be really helpful.

Web site attacks really tick me off, so if someone can provide information that helps me determine that there is an attack on our sites and what it is, I'll try to put together some kind of little reward for the help.

As a side note, our sites were responding very slowly yesterday starting around 21:00 UTC. I'm not sure if it's related or not, that's one of the things I'm investigating. Also, although I treat any compromise as very serious, so far based on the reports I've gotten, the site data itself (i.e. your contact info, passwords, etc.) hasn't been compromised. The reports I've gotten so far indicate that some users are getting "black hole exploit" warnings and malicious redirects. I want everyone to be protected as much as possible, so I'll be posting as much info as I can as soon as I learn anything.

See my Guides and Tutorials

in a case like this i think it could be overlooked

and its appreciated that your taking all necessary steps to protect user data

@TURGENEV - Freedom Server / IRON / B.A.N.E / HORDE

Turg Fiction: Ghost in the Machine Acts III & IV coming 2012!
Turg Fiction: IX is now LIVE on Architect Entertainment!

With that profile, I wouldn't think so. The full scan certainly won't hurt though, and I would recommend that anyone who has visited the sites in the past 18 hours or so (since around 17:00 ET/21:00 UTC) to run a scan just in case.

Good plan.

You should get Posi or Synapse to pop in so it shows up in the Dev Digest as well.

Unfortunately, I can't be of much help aside from moral support.

Originally Posted by Dechs Kaison See, it's gems like these that make me check Claws' post history every once in a while to make sure I haven't missed anything good lately.

The nuts and bolts of it is that something has rewritten a whole bunch of index.php files, including most that run the core functionality of all of our sites, to include the following line at the top:
echo (base64_decode('ZXJy[bunches more gibberish]0KfQ=='));

When you decode that, you get a Php function that:

- Turns off error reporting,
- Fetches the IP address of who's accessing the page,
- Fetches the user agent (UA) of who's accessing the page and compares it against a list of known security sites and webcrawlers,
- If it's not in the list of UAs, it does a cURL fetch of a javascript payload from a remote distribution site. The URL is defined as:
'http://[scum domain omitted]/index.php?go=1&ip='.$ip

So it's sending the user who is accessing the page's IP address. Depending on that IP address, they payload may or may not be delivered. When I put my own address in, it's not. When I poked around a little bit, I was able to find an IP that did deliver the payload, which I copied for further analysis as soon as I get a chance.

There are a couple of interesting things that I'm trying to figure out. For one thing, the modification datestamps of the index.php files are unchanged. That means that whatever script made the modifications either 1) is running at a very low-level filesystem level, or 2) took some pains to save the modification date and change it back to what it was originally.

Right now, my efforts are being split in three directions. First, I'm on shift for my day job and, as my luck is going today, got sucked into a teleconference as soon as I logged on two hours ago, which means that my other efforts are currently being severely hampered. Second, I'm setting up a second server that will likely replace our existing Titan Network server that has been completely staged from scratch and onto which sanitized copies of our sites will be moved. Third, I'm trying to get to the bottom of how this happened so that I can prevent it from happening again and, if possible, report the incident to law enforcement.