.STORYARCs show diff filesizes on different PCs..?

The issue: Two separate machines report an Architect .storyarc file with different filesizes while editing. RESULT: On one machine, Architect will show the file as 98.07%, leaving it open for publishing and testing. On another machine, Architect shows the file at 102.07%, locking it against publishing and testing.

Background: I tend to work on my arcs at both work and home. As such, I've taken to archiving my arcs on a USB flash drive, and copy/pasting the updates to each system before I open them. I copy the following folders:

costumes
Custom_Critter
CustomVillainGroup
Missions

The byte size for these files/folders on each location (thumb drive, work laptop, home PC) is identical.

When I edit the files on my work laptop, it reports it as 98.07% in size. I can edit the file, make changes, and it ups or downs the cost as expected.

I edit the same batch of files on my home PC, and Architect reports it as 102.07% in size. It allows me to edit, but I cannot publish or test from this workstation.

The net difference is around 4%. While that may not seem like a lot, there are two issues at stake here:

1. I talk a lot. I need every character I can make work. I'm ALREADY constrained by the built-in limits, and wish I had about 25-50% more than what's allotted to us. 4% is a considerable difference: It's roughly the size of a quality-written end-of-arc souvenir.

2. Assuming all else is equal, why should COH show a different bytesize between two otherwise identical files and folders?

I'm curious: Can anyone verify this behavior? That taking your storyarc files (and associated folders, for comparison) to a different machine shows different filesizes in use?

---

Hrmm.

Try changing the MA's 'working folder' on your desktop from the hard drive to the thumb drive. Otherwise we run into issues with how each drive is formatted, sector sizes, all that jazz. You'll have to have the thumb drive slotted before you engage the MA interface, of course.

---

"City of Heroes. April 27, 2004 - August 31, 2012. Obliterated not with a weapon of mass destruction, not by an all-powerful supervillain... but by a cold-hearted and cowardly corporate suck-up."

[ QUOTE ]
Otherwise we run into issues with how each drive is formatted, sector sizes, all that jazz.

[/ QUOTE ]

See, that's my theory: That it's reading "size on disk" rather than the file's actual byte size. I'm just too tired right now with resume building to think about the math. XD

---

Are you running XP 64 or Vista 64 on your home machine by any chance? That affects the size of pointers, which is significant for in-memory structured data. (File size is unaffected, but I'm not convinced the MA sizing is strictly related to file size.)

---

And for a while things were cold,
They were scared down in their holes
The forest that once was green
Was colored black by those killing machines

KD: On both machines, running XP Pro 32-bit. Home HD is on a 400GB partition; work HD is a 100GB partition.

---

Filesize turns out to be irrelevant, more or less. You can easily get the filesize down to a third its normal size, and it won't reduce the in-editor % by 2/3rds.

You should check the reported filesizes of the various files on both computers to see where the extra is - it seems highly unlikely that the difference is actual disk usage.

---

It's the only correlation I'm finding, without knowing more, VP. I have a file that's a known, rigid quantity: It's exactly X bytes in length. Take that file into two COH installations on two separate computers, both running XP Pro. One says it's 98%, the other says it's 102%...

Besides "size on disk" being different between systems, I'm stymied. =/ What could cause a 4% increase between two systems when reading a file with exactly the same characteristics between the two locations?

---

You've got me stumped on that one, turg, if the filesize reported by the OS on both computers reads the exact same number (thats what I was saying to check to be CERTAIN of) then I agree, the actual disk usage would be my first guess... but we determined in the IGOR thread that filesize of any sort doesn't seem to directly correlate to the mission % used.

You're welcome to run it through IGOR, or I can tell you what stuff you can easily edit out of the file without causing editing difficulties if you want to see if reducing the filesize will help. Given the small % you're over, it should do the trick, but I'll admit I'd like to know the ultimate reason, not just throw a scuccessful solution at it without ever finding out.

---

o.O

IGOR?

(searches) Oh, dang. Where have you been all my life, baby doll...

VP: Same here - I think taking a file that's borderline around 98%, then passing it around to players and seeing what they report, would be an interesting way to get a real sample of what's going on...

I can't wait to play with IGOR...

---

Well, I need to wait on one user's storyarc file - he says IGOR made an arc that could public before IGOR, into an arc that could not publish! That baffles me, so I need to do some testing when I get that file.

IGOR makes the arc uneditable (it doesn't alter the original, it makes a copy) because it removes things that aren't necessary to get it published properly, but which the in-game editor cries over. If that's a problem I can just tell you which lines in your storyarc to remove without triggering any of those "errors", but the size starts inching up. Either way you should get the extra 2% needed.

---

[ QUOTE ]
Filesize turns out to be irrelevant, more or less.

[/ QUOTE ]

Filesize on the local copy and publishability serverside are not really related. I usually end up with the issue that I show 3-4% free in the arc but it won't publish because it's too big. The server-side copy is what matters and that's a couple percent larger than it shows on my PC.

Changing the file size locally won't fix anything. What matters is it's size server-side

---

[ QUOTE ]
Changing the file size locally won't fix anything. What matters is it's size server-side

[/ QUOTE ]

Feel free to read the rest of the thread too, to understand why we're disucssing this, Smurch.

---

[ QUOTE ]
[ QUOTE ]
Changing the file size locally won't fix anything. What matters is it's size server-side

[/ QUOTE ]Feel free to read the rest of the thread too, to understand why we're disucssing this, Smurch.

[/ QUOTE ] Well you were quite zealous to put him in his place.
I wasn't surprised at his answer, and as an attempt to help answer, it was not as bad as you have tried to make it. But you did indeed succeed in ding-ing him.



I too, at first thought that his answer would be true...
that the filesize the MA would check would be the filesize on the MA server.
But it feels like it is indeed testing the filesize reported from the user's machine.

filesizes are often, well, usually, different, on different devices.

The short answer?
File size measures the size of a computer file. Typically it is measured in bytes with a prefix. The actual amount of disk space consumed by the file depends on the file system.
i.e. depending on how the drive is formatted, and designed, will make a file be a different length on a different drive.

-----------------------------------------------------------------------


FileSize is affected by all the things about the way the storage is formatted.
Things like Block size, sectors, Clusters, and other factors.

Think of the area your file is stored in, as a string of little boxes.
The things mentioned above have determined the size of little boxes on your drive.

• If each box could only contain 2 text characters, then the filesize would be very efficient, because the most empty space (slackspace) that could be in the last box, would be one character long. (this would also make drive-reading very INefficient)
• But if the little boxes could hold 1000 characters, then the last box of any file could have from 1 to 999 empty slots, on average 500 empty. (larger slackspace) .
• the larger the 'little boxsize' is, the wider variation there can be from one drive to the next.
• In the just previous example, if the file were made up of 1000 boxes, 1000characterspace each, and the file had a 995 slackspace, and you copied the file to a drive that had room for one more character in each box, then you would eliminate the last box entirely, and the slackspace in the new 'last box' would only be a few characters long, since the file fit in the new set of boxes more efficiently. The length of the file on the two systems would have a significant difference, in a case like an MAarc file that is at the full mark.
• (The above numbers are example only. They are not likely numbers for the 'box sizes', and there ARE other factors than the ones I mentioned here.)

_____________________________________


So yes... IF you are working on an MA arc that is pushing the limit . . .
AND the MA system is accepting the file size your local machine is reporting, rather than the size the MA file will be when saved on the server, . . .
then the difference from machine to machine can be quite significant.

.

---

I understand the concept just fine, but from testing we've seen that the % on the bar inside the in-game editor is not directly related to the filesize in any way. Since it was stated that the maximum size of arcs was 100k, it was a pretty natural assumption that it would be 100k in some form WE could measure it. I still suspect they were using it as an arbitrary measurement since... telling us the maximum size in a format we can't ever measure is pointless. Arcanaveal, in typical "useful" fashion says it was foolish to assume it would be in some way we could measure, that she knows how its done, but that she won't say. Useful as always, that one.

Given that file size is not what determines the % size in-editor, from testing, and that the file itself has not changed at all during its move between computers, the reason that it reads as two different in-editor %s is a bit of a mystery. Thats why I said he was welcome to try using IGOR to shrink down the filesize (it does remove some things that affect % as well, but its effect on % is nowhere near as profound as its effect on filesize) but that I'd love to know what is causing the discrepancies.

---

[ QUOTE ]
Arcanaveal, in typical "useful" fashion says it was foolish to assume it would be in some way we could measure, that she knows how its done, but that she won't say. Useful as always, that one.

[/ QUOTE ]

Actually, I was very specific, and even wrote several articles about it in closed and open beta. I specifically told you that the size bar was a function of the internal size of the mission, and that the local file size did not correlate to that. You promptly went sailing off into schizo-land based on that statement which is when I decided you are simply insane.

I leave it to the court of opinion on that one:

Me: (this is the first post I made in the thread VP is referencing):
[ QUOTE ]
It might be misleading, but its not intended to be nor does it claim to be the local file size. Local file size is independent from internal mission arc size. Many things affect mission size and local file size in a completely different way. My mythbusting thread unfortunately got eaten, but one of the things I discovered is that the recommendation to set unused costume parts to black is worthless: it doesn't decrease mission size (at least, it did not at the time I tested). It does dramatically reduce local file size, but that doesn't help the author.

Also, the 100k limit is actually 100,000 bytes, not 100k.

I'll take a look at IGOR to see what it does; some of the things it does (like stripping or reducing file references) might help, and then again it might not.

[/ QUOTE ]

And the response:
[ QUOTE ]
That's a nonsensical statement, Arcanaville - what is the point of TELLING us that the maximum size is 100k if that doesn't correspond to anything we can measure ourselves? Given that approach they might as well say the maximum size is "100%" on the little meter, which it theoretically is but practically isn't... no more than the maximum filesize is 100,000 bytes. I have a non-reduced mission that is 92% in the editor, and 105k on disk.

The only rational explanation is that 100k was an estimate they gave based on some level of trial and error with the system, since it corresponds directly to nothing we can measure.

[/ QUOTE ]

I always appreciate it when someone first says I'm nonsensical, then states that their position is the only rational explanation, and then says something that is actually false. Bonus points when it contradicts dev statements and prior experimentation.


For the record, the meter is *not* a trial and error meter. All of the differences in size between the local file and the internal memory counter can be explained by changes in encoding. For example, color values that are stored 0,0,0 do not change the MA size meter when they are changed to 255,255,255 - that's because you're looking at a human-readable encoding, not the actual internal coding of those values.

Different mission objectives have different size requirements for their details. But one thing that is fairly constant is that raw text takes (barring tag insertion) 0.001% per character, entirely consistent with the meter measuring exactly 100,000 bytes. The meter doesn't display that level of resolution, but in beta I tested this by adding one character at a time to long text fields, and recording when the meter moved. There is a roundoff error that (at least at the time of testing) caused the meter to roll in an alternating fashion at 9 characters, then 11, then 9, then 11, in a repeating pattern. But overall, 10 characters = 0.01% of the meter.

Some things you can remove completely from the local file, and doing so will reduce the mission size because that data is no longer in the mission. But some things when removed don't change the mission size, because they are computed values and not intrinsic data, which means the mission contains that data implicitly whether the local file does or doesn't. Some costume part entries are like that in some cases.

However, the meter is definitely measuring something specific: its not guessing or anything. Its behavior is predictable and repeatable under experimentation.

The issue of whether the meter is measuring the local file size or not was a matter discussed and disposed of during closed beta with fairly careful experimentation, not just by myself but others.

Summary:

1. The meter measures actual mission file size, and the limit is exactly 100,000 bytes.

2. Adding mission details adds to that, but without careful experimentation there's no way to know how many bytes each mission objective requires for its own data structures. But text always takes up one byte per character, including spaces and tags.

3. The MA passes missions through a validator during publishing to the mapservers (this includes testing a mission, and not just "publishing") and during that process some data can be converted or altered. This can change the size of a mission from when its created to when its executed. Curiously, some coversions happen *after* the size is validated, which means its on rare occasions possible to test a mission once, and then have it be untestable a second time due to size constraints.


If you construct missions in a particular way, its possible to determine the size requirements of anything you want to add by differential analysis. I have results for a wide range of objectives and objects, but its not in a form that is easy to post. I was working on a way to do that when VoodooPokey elected himself arbiter of reasonable analysis for the MA, whereupon I set it aside and began working on my MA arc statistical analysis instead. I'll return to the subject in a few months once I've given VoodooPokey ample opportunity to preempt my work. That's not specifically out of spite, I generally *always* work on something else if someone is working on the same thing I'm working on. There are forum posters out there that can attest to that fact: why duplicate work if someone else can do it just as well? I'll even give VP full credit for it if he does whenever the data is referenced.

Keeping my data to myself and letting VoodooPokey do all the heavy lifting himself? That *is* out of spite. I ordinarily dump all of my data onto whoever else is working on the subject, and I don't generally ask for credit. In this case, I don't feel like it. Sue me.

---

[Guide to Defense] [Scrapper Secondaries Comparison] [Archetype Popularity Analysis]

In one little corner of the universe, there's nothing more irritating than a misfile...
(Please support the best webcomic about a cosmic universal realignment by impaired angelic interference resulting in identity crisis angst. Or I release the pigmy water thieves.)

Back to the original question...

Turg, are you absolutely, positively sure that everything is the same on both systems? I ask because when you edit story arc files, the system will "absorb" any changes made to custom critters or groups. So if you copied the story arc file to another system, but forgot to copy one of the critter group files, you could end up with two different sizes after the story arc "absorbs" the difference.

One thing I'm curious about is the idea of re-saving the story arc file on the machine that shows the increase, and then using a diff tool to see if there are any differences between that and the original. That would detect not only the case I mention above, but any weirdness involving the ever-popular   spam and the like.

---

And for a while things were cold,
They were scared down in their holes
The forest that once was green
Was colored black by those killing machines

[ QUOTE ]
Back to the original question...

Turg, are you absolutely, positively sure that everything is the same on both systems? I ask because when you edit story arc files, the system will "absorb" any changes made to custom critters or groups. So if you copied the story arc file to another system, but forgot to copy one of the critter group files, you could end up with two different sizes after the story arc "absorbs" the difference.

One thing I'm curious about is the idea of re-saving the story arc file on the machine that shows the increase, and then using a diff tool to see if there are any differences between that and the original. That would detect not only the case I mention above, but any weirdness involving the ever-popular   spam and the like.

[/ QUOTE ]

Absolutely, positively sure. What I did was wipe out all the MA folders from both locations, excepting the USB flash; then recopy from the flash source to the two locations. Verified in both laptop and desktop that the actual file size was X for the storyarcs and the various folders. I also ran a file compare on Leandro's suggestion to see what'd it show, and FC showed zero discrepancies on either side. However, with both editors up and editing the file, one would show 98%, the other 102%. The only real difference I saw was in "file size on disk", which showed a 4% difference. The two sets of facts seem pretty direct in this case.

---

[ QUOTE ]
The short answer?
File size measures the size of a computer file. Typically it is measured in bytes with a prefix. The actual amount of disk space consumed by the file depends on the file system. [/b]
i.e. depending on how the drive is formatted, and designed, will make a file be a different length on a different drive.

[/ QUOTE ]

There's a big difference in the logical size of a file and the "physical" size it requires on storage media, filesystem architecture, etc.. Data applications should never care about the physical size, and indeed are typically completely blind to it. If I load a plain text file containing nothing but 10,000 ASCII letter "X"s into a programming text editor, that file will be exactly 10,000 bytes long no matter what storage media I store it on or even what OS I store it on. Nothing that is not an OS or disk management tool should ever know that the file might actually be taking up 11,264 bytes of storage on my disk drive due to NTFS's block size.

In short, whatever is up, that is most assuredly not it, or some CoH dev doesn't have enough to do if he/she is determining how much filesystem overhead the mission files incur on our individual systems.

---

Blue
American Steele: 50 BS/Inv
Nightfall: 50 DDD
Sable Slayer: 50 DM/Rgn
Fortune's Shadow: 50 Dark/Psi
WinterStrike: 47 Ice/Dev
Quantum Well: 43 Inv/EM
Twilit Destiny: 43 MA/DA
Red
Shadowslip: 50 DDC
Final Rest: 50 MA/Rgn
Abyssal Frost: 50 Ice/Dark
Golden Ember: 50 SM/FA

[ QUOTE ]
Absolutely, positively sure. What I did was wipe out all the MA folders from both locations, excepting the USB flash; then recopy from the flash source to the two locations. Verified in both laptop and desktop that the actual file size was X for the storyarcs and the various folders. I also ran a file compare on Leandro's suggestion to see what'd it show, and FC showed zero discrepancies on either side. However, with both editors up and editing the file, one would show 98%, the other 102%. The only real difference I saw was in "file size on disk", which showed a 4% difference. The two sets of facts seem pretty direct in this case.

[/ QUOTE ]

Well, I'm baffled then. Arcanaville's testing in beta indicated that the progress bar was measuring the sizes of objects in memory rather than how much space they take on disk (the size of all-black costume parts seems to show this pretty clearly). But the only explanation for what you're seeing seems to be that the size is the size on disk, or at least directly related to it.

Does every story arc exhibit this same behavior on these two systems, or is it something particular to this story arc? If the latter, can you try removing missions and/or details to see if there's a specific construct that shows a difference?

Of course you might be sick of looking at the issue, in which case no worries, I'm just really curious now.

---

And for a while things were cold,
They were scared down in their holes
The forest that once was green
Was colored black by those killing machines

Turg, is there any possibility that the two CoH installations have a difference in Client Versions?

---

I believe that a Kheldian Gold Standard should be based on SO's, and for anything above that... there's Platinum!

Save Ms. Liberty (#5349) � Augmenting Peacebringers � The Umbra Illuminati

[ QUOTE ]
[ QUOTE ]
Back to the original question...

Turg, are you absolutely, positively sure that everything is the same on both systems? I ask because when you edit story arc files, the system will "absorb" any changes made to custom critters or groups. So if you copied the story arc file to another system, but forgot to copy one of the critter group files, you could end up with two different sizes after the story arc "absorbs" the difference.

One thing I'm curious about is the idea of re-saving the story arc file on the machine that shows the increase, and then using a diff tool to see if there are any differences between that and the original. That would detect not only the case I mention above, but any weirdness involving the ever-popular   spam and the like.

[/ QUOTE ]

Absolutely, positively sure. What I did was wipe out all the MA folders from both locations, excepting the USB flash; then recopy from the flash source to the two locations. Verified in both laptop and desktop that the actual file size was X for the storyarcs and the various folders. I also ran a file compare on Leandro's suggestion to see what'd it show, and FC showed zero discrepancies on either side. However, with both editors up and editing the file, one would show 98%, the other 102%. The only real difference I saw was in "file size on disk", which showed a 4% difference. The two sets of facts seem pretty direct in this case.

[/ QUOTE ]

So far, I've been unable to reproduce this behavior, and I edit on two different systems myself. I have two suggestions:

1. Backup your MA directories (which you seem to have already done) and then delete them from both computers. Then copy *only* the mission storyarc file to both computers, and see if they show the same discrepancy. The question is whether the problem is actually in the mission arc file or in something related to how the MA resolves references.

2. If you don't mind someone looking at your source code, I can try to analyze the storyarc file to see if there is something in it that might be behaving oddly by testing it on a variety of systems. Contact me directly if you want me to take a shot at that.

---

[Guide to Defense] [Scrapper Secondaries Comparison] [Archetype Popularity Analysis]

In one little corner of the universe, there's nothing more irritating than a misfile...
(Please support the best webcomic about a cosmic universal realignment by impaired angelic interference resulting in identity crisis angst. Or I release the pigmy water thieves.)

[ QUOTE ]
Turg, is there any possibility that the two CoH installations have a difference in Client Versions?

[/ QUOTE ]

If the client reports a version older than the game servers, it shouldn't connect. There is a way to bypass the updater when starting the game client, but doing so will generate this failure whenever the game servers are patched. It would be a very weird bug if he could get an older client to successfully connect to the game servers.

In theory, there are ways for two game clients to get out of sync in a way that the updater doesn't detect, but I don't know of any that would affect the MA. Its possible to alter the behavior of the client by virtual file system override, for example (that's how things like map packs work, and its amazing what actually is overridable in that fashion) so in theory its possible someone could have tampered with their game client in a way that disrupts the MA, but as a practical matter I don't think that is likely.


I did a spot-check last night to see if things like mission objectives are still taking the same amount of static memory now as they were back in beta, and so far as I can see they all seem to be doing so, plus or minus a byte or two (text fields, of course, have changed significantly since then but only in terms of the parser, not the memory usage). So I don't think footprint measurements have changed significantly since then. So at the moment I don't have a good way to account for two identical files generating different mission sizes on two different computers. I would need more information to form a hypothesis there.

---

[Guide to Defense] [Scrapper Secondaries Comparison] [Archetype Popularity Analysis]

In one little corner of the universe, there's nothing more irritating than a misfile...
(Please support the best webcomic about a cosmic universal realignment by impaired angelic interference resulting in identity crisis angst. Or I release the pigmy water thieves.)

[ QUOTE ]
[ QUOTE ]
Turg, is there any possibility that the two CoH installations have a difference in Client Versions?

[/ QUOTE ]

If the client reports a version older than the game servers, it shouldn't connect.

[/ QUOTE ]

The only way I can think of that this might happen is if Turg was accidentally running the Test client on one of the systems... but that seems so unlikely I almost didn't mention it.

---

And for a while things were cold,
They were scared down in their holes
The forest that once was green
Was colored black by those killing machines

[ QUOTE ]
...at the moment I don't have a good way to account for two identical files generating different mission sizes on two different computers. I would need more information to form a hypothesis there.

[/ QUOTE ]
Yup, this needs to be investigated for sure. Off the top of my head, I could think only of the   transformation issue, and since I do not know for sure if that transformation happens on the Client, or on the Server, hence I asked if there may be an actual difference between the client versions (i.e. binaries/configurations) themselves.

It may be quite possible that this is completely a client-configuration issue, but we won't know that without intensive exploration.

---

I believe that a Kheldian Gold Standard should be based on SO's, and for anything above that... there's Platinum!

Save Ms. Liberty (#5349) � Augmenting Peacebringers � The Umbra Illuminati

[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
Turg, is there any possibility that the two CoH installations have a difference in Client Versions?

[/ QUOTE ]

If the client reports a version older than the game servers, it shouldn't connect.

[/ QUOTE ]

The only way I can think of that this might happen is if Turg was accidentally running the Test client on one of the systems... but that seems so unlikely I almost didn't mention it.

[/ QUOTE ]
My past (and professional) experience tells me this is probably a client-configuration issue and when you debug this sort of trouble, you go for the obvious first, i.e. Client Versions... to rule out any such "silly" options.

Like Arcanaville suggested, configuration differences should be ruled out next (i.e. removing all MA/AE related folders and forcing the Client to extract all the files out of the .storyarc original file).

I would not be surprised if the problem is caused by some yet-unknown internal configuration variable that is kept in some .ini file and has different values between the two installations

---

I believe that a Kheldian Gold Standard should be based on SO's, and for anything above that... there's Platinum!

Save Ms. Liberty (#5349) � Augmenting Peacebringers � The Umbra Illuminati