Tool to save your messages

Hey all, in light of the imminent deletion of some of our private messages, and because I have so durn many of them (501, to be exact) and don't feel like spending hours saving them individually and don't want to delete them, I wrote a little widget to pull them from the City of Heroes forum site. It struck me as something that you all could use too, so I've user-friendlied it up a little and posted it, along with the source code, for you to use if you want.

If you just plain trust me implicitly and don't want to bother with all the whys and wherefores, here are the links:

PM Capture Setup.msi, a Windows installer package
PM Capture.zip, the executable and SQLite component, if you don't want to use the installer
PM Capture Source.zip, the complete source code to compile yourself if you wish or want to make sure I'm not trying to hack your account.

If you use the installer, remember where you installed it, because it doesn't add any start menu items or desktop icons. You'll have to launch it from Windows Explorer or using Start / Run.

Now, the whys and wherefores. Well, the whys are obvious, so let's talk about wherefores...

Please keep in mind that I threw this together over the course of yesterday, so if it doesn't meet your criteria of being glorious, it's because I was trying to focus on getting it out there, not making it terribly pretty or loading it up with uberfeatuers. If I put my mind to it for a few more days, I could probably add some more stuff, but hey, it serves its base purpose (i.e. archiving your PMs) pretty well for now, so it'll do.

It does ask for your forum login information. Yes, I know that's kind of scary, coming from a third party. If it makes you uncomfortable, don't use it. I'm paranoid about things like that myself, but the practical matter of the situation is that it cannot access your PMs without logging in as you to the forums. However, I have also distributed the source code, so if you want to use it, feel free to download the code and, if you have Visual Studio, compile it yourself after checking it out to make sure it's not doing anything stupid. Feel free to run a network trace while it's running, and see for yourself that the only server it's talking to is the forums server and that it is not transmitting anything anywhere else. It does not store your account information anywhere, so every time you use it (do you really need to do a capture more than once?), you'll have to re-enter it.

I'm releasing this as a totally public domain utility, so feel free to copy, modify, upload to other places, tweak to use with other UBB systems, improve, sell, or whatever you want to do with it. I dont' even care if you credit me with it or not. My only caveat is that if you take it and do something evil with it, or if god forbid the code is broken in some way I haven't foreseen, that's your problem, not mine. It's offered stictly as-is, your mileage may vary, no warranty is offered at to suitability of purpose, yadda yadda yadda.

When you save your PMs to a file, it saves them as an SQLite database in a table named "messages." If you're a development wonk, SQLite is built into Perl, most distributions of PHP, has a standalone command-line utility (downloadable at the SQLite site), and I'm using Robert Simpson's SQLite.NET component in C#, also public domain software, with "zero licensing restrictions for private or commercial use." Once you save the file, if you want write your own utility to export it into some other format or do whatever with it, it should be easily accessible. Why didn't I use [insert other format or storage mechanism here]? Because I'm familiar with SQLite, and it seemed like a good tool to use in the 16 seconds I considered the question of how to store the messages. Maybe also worth noting is that the tool stores the original HTML taken from the page in that database, so if my parser is broken or you want to reparse it and render the messages differently and you're capable of writing a better interface, by all means, the opportunity is there for you.

Speaking of which, you might notice some rendering artifacts, like random <blockquote> or <font> tags inserted. My parser makes some assumptions that aren't always true, because UBB has this nasty habit of not actually checking messages for validity, such as making sure that all [quote] tags are closed and such. Oh well, such is life. Like I said, if I get time, maybe I'll update the parser to account for that. Like I said, the original HTML is stored in the file, which is the most important part to keep from losing your messages. Oh, and currently, the parser only tries to reverse-engineer the BBcode from the tags. Maybe someday I'll work on actually rendering bold text in bold and such, but again, this was thrown together quickly.

If you're a Eurpoean forums user, it should, in theory, work with the European boards, too. I say in theory because I haven't tested it because I don't have a European boards account, but my observation is the relative paths are all the same, only differing in server name to access the site, which I've accounted for. Don't be fooled by the French and German skins on the European forum site, they all hit against the same database, so whatever European country you call home, it should still work if you pick EU.

Please don't be stupid with it. The utility will hit against the forums server once for every PM you have, plus three more times. (Once to log in, and twice to retrieve your sent and received message lists.) While it shouldn't bring down the forums server or anything, if you're like me have have a crapton of PMs, it doesn't take long to get your messages, but it does make a lot of web requests in a short period of time. Get your messages, save them, and open them from the file if you want to see them. Don't repeatedly download them all over and over. If you get in trouble for denial of service or something because you're not doing it right, it's your own durn fault, not mine.

Once the forums change over to the new forums, there's a very high probability that the capture function will simply stop working. You'll still be able to open your messages from a file and read them, though. No, I don't have a vBulletin version of it. Will I make one? I don't know, maybe, depending on whether I think it's worth the time and effort to do so. Likewise, will I ever improve this tool to add features or fix any bugs that might crop up? I don't know, maybe, depending again on whether I think it's worthwhile.

Last, but not least, this tool is meant just to help. If the mods ask me to take it down, I'll take it down. There are rules about accessing the game with bots, but to my knowledge, there are none about accessing the forums in an automated manner. Given that it's a constructive use of an automated tool to avoid negatively impacting customers, I'm hoping that it will pass muster if judged. If you write to me and gripe about it, it will probably fall on deaf ears because first and foremost, I wrote it for myself. If you like it and want to shower me with gifts, don't sweat it. Donate to the Paragon Wiki if you feel like you just have to do something.

Edit: I re-read the post about PM deletion, only those older than two years will be wiped, not all of them. Still, I have well over a hundred PMs since before July 2007, so it's still a handy tool. And besides, I feel better knowing that even through the forum changeover, if something goes wrong and the PMs are wiped, I still have copies of them.

---

We've been saving Paragon City for eight and a half years. It's time to do it one more time.
(If you love this game as much as I do, please read that post.)

[ QUOTE ]
Please don't be stupid with it. The utility will hit against the forums server once for every PM you have, plus three more times. (Once to log in, and twice to retrieve your sent and received message lists.) While it shouldn't bring down the forums server or anything, if you're like me have have a crapton of PMs, it doesn't take long to get your messages, but it does make a lot of web requests in a short period of time. Get your messages, save them, and open them from the file if you want to see them. Don't repeatedly download them all over and over. If you get in trouble for denial of service or something because you're not doing it right, it's your own durn fault, not mine.

[/ QUOTE ]



TonyV is the The Great Beast of the Forumpacolypse!

THE END TIMES ARE NIGH! REPENT REPENT!

DOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOMMMMMMMMM!!!!!!!

---

Well, I was just worried that they might see people accessing the server like 500 times in five seconds and think it's some sort of attack. I've been testing the tool yesterday and this morning and haven't been locked out, but if they start seeing thousands of accesses like that, they might decide to take action.

The message is kind of a heads-up to the mods also that if they do see that kind of traffic, it's not a botnet, it's probably just people snagging their PMs, and it will probably go away once the new forums roll out.

Still, there's no point in needlessly taxing the server. Like I said, the tool's purpose is for archival, not as an offline reader that synchronizes with the server. In fact, there is exactly zero synchronization code built into it. I ask that people get their messages, save them, and not fetch them over and over.

---

We've been saving Paragon City for eight and a half years. It's time to do it one more time.
(If you love this game as much as I do, please read that post.)

But... you provided the tool needed to bring about... THE END OF IT ALL! You have signaled the coming forumpacolypse!

THERE WILL BE WAILING AND GNASHING OF TEETH!

AND THE EATING OF THE FRUIT BATS!

REPENT!!!!!!!!!!!!!!!

---

If I had lost all of my PMs, then there would have been wailing and ganshing of teeth. Hopefully this will help others avoid that predicament.

---

We've been saving Paragon City for eight and a half years. It's time to do it one more time.
(If you love this game as much as I do, please read that post.)

As someone with 453 messages in my inbox (after I started cleaning it... and filling it up again), thanks a ton. I was not looking forward to making another huge push to clean it up before Tuesday at all.

---

[ QUOTE ]
If you're a Eurpoean forums user, it should, in theory, work with the European boards, too. I say in theory because I haven't tested it because I don't have a European boards account, but my observation is the relative paths are all the same, only differing in server name to access the site, which I've accounted for.

[/ QUOTE ]
Sorry Europe, it doesn't work on the boards across the pond.

The paths are all okay, but the clues I use to tell me where messages start and such are all different because of the different forum skin. I've asked a European user to e-mail me a few pages, but it's probably going to be late tomorrow at the earliest, if it happens at all, before I can fix it to work with the European boards.

---

We've been saving Paragon City for eight and a half years. It's time to do it one more time.
(If you love this game as much as I do, please read that post.)

Got this big ol' error message after running the program from the msi installer:

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.BadImageFormatException: Could not load file or assembly 'System.Data.SQLite, Version=1.0.64.0, Culture=neutral, PublicKeyToken=db937bc2d44ff139' or one of its dependencies. An attempt was made to load a program with an incorrect format.
File name: 'System.Data.SQLite, Version=1.0.64.0, Culture=neutral, PublicKeyToken=db937bc2d44ff139'
at PM_Capture.IOHelper.SaveMessages(Message[] Messages, String Filename, Boolean Overwrite, SaveProgress Progress)
at PM_Capture.frmMain.mnuFileSaveas_Click(Object sender, EventArgs e)
at System.Windows.Forms.ToolStripItem.RaiseEvent(Obje ct key, EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleMouseUp(M ouseEventArgs e)
at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEven tArgs mea)
at System.Windows.Forms.ToolStripDropDown.OnMouseUp(M ouseEventArgs mea)
at System.Windows.Forms.Control.WmMouseUp(Message&amp ; m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ToolStrip.WndProc(Message&amp ; m)
at System.Windows.Forms.ToolStripDropDown.WndProc(Mes sage& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].



************** Loaded Assemblies **************
mscorlib
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.4016 (NetFxQFE.050727-4000)
CodeBase: file:///C:/Windows/Microsoft.NET/Fra...7/mscorlib.dll
----------------------------------------
PM Capture
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/temp/PM%20Capture.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.4016 (NetFxQFE.050727-4000)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...dows.Forms.dll
----------------------------------------
System
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.4016 (NetFxQFE.050727-4000)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...089/System.dll
----------------------------------------
System.Drawing
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.4016 (NetFxQFE.050727-4000)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...em.Drawing.dll
----------------------------------------
System.Configuration
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.4016 (NetFxQFE.050727-4000)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...figuration.dll
----------------------------------------
System.Xml
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.4016 (NetFxQFE.050727-4000)
CodeBase: file:///C:/Windows/assembly/GAC_MSIL...System.Xml.dll
----------------------------------------
System.Web
Assembly Version: 2.0.0.0
Win32 Version: 2.0.50727.4016 (NetFxQFE.050727-4000)
CodeBase: file:///C:/Windows/assembly/GAC_64/S...System.Web.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

---

That's really weird, it's the 32-bit version of the library, so it should work fine on 32-bit and 64-bit machines. Are you sure you have the System.Data.SQLite.dll file in the same directory as the PM Capture.exe file? They have to go together. (Or you could put System.Data.SQLite.dll in C:\Windows\system32 or whatever path Vista uses, but it's easier if they're just kept together.)

On a related note, I've gotten an HTML capture of the European boards from a user over there, and I think I've figured out why it wasn't working with the European boards. I've rebuilt and re-uploaded it, but he's probably in the arms of Morpheus (the god, not the Matrix guy), and I'm about to go to bed anyway, so I probably won't be able to actually get any results until tomorrow.

In the meantime, now I'm a little baffled. Out of curiosity, has anyone else tried it, and if so, did it work for you? Has anyone else had any problems? For what it's worth, the error indicates a problem with System.Data.SQLite.dll, not my executable. It's little consolation, but if you have verified that the file is actually there, you might want to try snagging a copy off of the developer's home page. Or try downloading the ZIP file and extracting it instead of using the installer. Let me know if it still doesn't work.

---

We've been saving Paragon City for eight and a half years. It's time to do it one more time.
(If you love this game as much as I do, please read that post.)

Got my stuff downloaded, which is great. Not quite up on the whole database thing, though - so if anyone DOES come up with a way to export it into something a bit more friendly, please let me know!

Michelle
aka
Samuraiko/Dark_Respite

---

[ QUOTE ]
That's really weird, it's the 32-bit version of the library, so it should work fine on 32-bit and 64-bit machines. Are you sure you have the System.Data.SQLite.dll file in the same directory as the PM Capture.exe file? They have to go together. (Or you could put System.Data.SQLite.dll in C:\Windows\system32 or whatever path Vista uses, but it's easier if they're just kept together.)

[/ QUOTE ]
The dll and PM Capture.exe are in the same folder. I also tried copying the dll from the zip file and using it instead. When I checked the properties of the dll (running Vista 64-bit here) it had been "blocked" by the system because it was from an "unknown computer". I used the option to unblock it and ran the capture again but got the same error message.

Since it runs a full scan every time, I'll hold off on using it again unless you have any other suggestions.

---

Well, to be honest, I avoid Vista for precisely these kinds of problems I was constantly having. I don't even have it installed on any of my machines to test it.

I'm not sure what to suggest, I don't know what magical machinations you have to go through to tell Vista, "No really, I want to run this!" Anyone else who's more in tune with Microsoft have any suggestions?

If you want, Michelle, I can swing back around and write some kind of export to text or something, but it will probably be after the new forums are rolled out. That's kind of an important point I want to make sure folks understand: Get your messages now, and we'll worry about the details of formatting at our convenience. I'm glad it's working, though.

---

We've been saving Paragon City for eight and a half years. It's time to do it one more time.
(If you love this game as much as I do, please read that post.)

[ QUOTE ]
On a related note, I've gotten an HTML capture of the European boards from a user over there, and I think I've figured out why it wasn't working with the European boards. I've rebuilt and re-uploaded it, but he's probably in the arms of Morpheus (the god, not the Matrix guy), and I'm about to go to bed anyway, so I probably won't be able to actually get any results until tomorrow.

[/ QUOTE ]
Woot! We've both awoken, and I have verification that the modifications worked. European users, you're in business! Version 1.1 build 2 FTW! I've uploaded it in-place, so the above links all download the newest version.

Creole_Ned, I still don't have an answer for you, but I did manage to get access to a Vista machine. I'll try it out before I leave for work, but I can't promise an answer this morning.

Edit: Okay Creole_Ned and the rest of you Vistians out there, I think I figured out the problem. I made a couple of changes to the project build options, and it's working on the Vista machine I borrowed now. Again, I've uploaded the files in-place, so just use the above links, and see if that works for you. Version 1.1 build 3 FTW!

---

We've been saving Paragon City for eight and a half years. It's time to do it one more time.
(If you love this game as much as I do, please read that post.)

The new version worked perfectly. Nicely done!

---

Just a quick reminder. The newest version (downloadable via the links at the top of this thread) is now confirmed working for US and European users on XP and Vista, and that the forum shtuff is going to hit at around 7:00am Eastern tomorrow morning.

If you want to archive your PMs, do it now, or forever hold your peace!

---

We've been saving Paragon City for eight and a half years. It's time to do it one more time.
(If you love this game as much as I do, please read that post.)

Thanks for putting this together. I was messing with Adobe Acrobat web capture to do the same thing, but this is way neater and easier.

---

Champion
Pillars of Might
Darc Ranger [Blas] / Darc Nebula [Cont]
The Bikini Patrol
Darc Lighter:51-[Blas] / Darc Lady:50-[Tank]
The Panty Raiders
Aegis Magnus [MM] / Atomic Spector [Cor] / Dominar Sefus [Dom]
MEGAFORCE
-Darc Ranger [Def]

All my PMs have been archived and deleted from the forums.

There, did my duty to make the transition go faster.

Also, my posts older than 2 years will be saved.


Also, this utility comes with a post-reader so you can read your PMs, you don't have to export it anywhere to read them.

---

Speeding Through New DA Repeatables || Spreadsheet o' Enhancements || Zombie Skins: better skins for these forums || Guide to Guides