Hall of Fame requirements need revisiting


airhead

 

Posted

Quote:
Originally Posted by Wrong_Number View Post
This is very true. Many people ignore math and go with all sorts of criteria for giving ratings. Simply put:

1 Star = At least 80% of the arcs in the system are better than this arc
2 Stars = At least 60% of the arcs in the system are better than this arc
3 Stars = At least 40% of the arcs in the system are better than this arc
4 Stars = At least 20% of the arcs in the system are better than this arc
5 Stars = This arc is in the top 20% of arcs in the system


WN
But that's not true either. I pretty much reserve 1 star ratings for farms, but more than 80% of the arcs in the system are farms. Here's my chart:

1 Star = This arc is a farm and not worth my time. Worse than 10% of the arcs in the system.
2 Stars = Well at least there's some kind of story or something. Worse than 1% of the arcs in the system.
3 Stars = About the same as classic official content. Worse than 0.1% of the arcs in the system.
4 Stars = About the same as current official content. Worse than 0.01% of the arcs in the system.
5 Stars = Better than current official content. Worse than 0.001% of the arcs in the system.

Interestingly that 0.001% number looks similar to the number of 5-star average arcs with more than 100 plays... Maybe off by about a magnitude but still pretty close.


Winner of Players' Choice Best Villainous Arc 2010: Fear and Loathing on Striga; ID #350522

 

Posted

Quote:
Originally Posted by FredrikSvanberg View Post
But that's not true either. I pretty much reserve 1 star ratings for farms, but more than 80% of the arcs in the system are farms. Here's my chart:

1 Star = This arc is a farm and not worth my time. Worse than 10% of the arcs in the system.
2 Stars = Well at least there's some kind of story or something. Worse than 1% of the arcs in the system.
3 Stars = About the same as classic official content. Worse than 0.1% of the arcs in the system.
4 Stars = About the same as current official content. Worse than 0.01% of the arcs in the system.
5 Stars = Better than current official content. Worse than 0.001% of the arcs in the system.

Interestingly that 0.001% number looks similar to the number of 5-star average arcs with more than 100 plays... Maybe off by about a magnitude but still pretty close.
I pointing out the math of it, not suggesting any sort of guide.

WN


Check out one of my most recent arcs:
457506 - A Very Special Episode - An abandoned TV, a missing kid's TV show host and more
416951 - The Ms. Manners Task Force - More wacky villains, Wannabes. things in poor taste

or one of my other arcs including two 2010 Player's Choice Winners and an2009 Official AE Awards Nominee for Best Original Story

 

Posted

Quote:
Originally Posted by Talen Lee View Post
On the other hand, it's entirely possible that someone has a negative opinion of the arc and doesn't want to deal with an arc author throwing a tantrum about it. Lords knows that there are arc authors who do that. There are arc authors who think anything short of a 5-star is somehow harsh criticism.

...

Basically, I am disinclined to imagine conspiracy and griefing. It's much, much easier, in my opinion, to presume that between the flawed star system and the nature of the writing in the AE, that there are simply people who have now seen your arc and didn't like it.
I was thinking about what Talen Lee said earlier and I did some quick math.

If the system is doing typical mathematical rounding, it takes a real effort to move something with 100 ratings. One or two "1" votes out of 100 does not move anything. Assuming you received nothing but 3,4 and 5 star ratings and they averaged out exactly to a 4-star rating. Ten "1" votes still doesn't move it, it still rounds to a 4-star rating.

Therefor this only suggests one of the following must be true:
A- The system has a script that is calculating the average other than just using standard mathematical rounding and the rating, then, is an accurate reflection of the way the system is intended to work.

B- Those with 100 plays and see their arcs moving between 4 and 5 stars is not the result of ONE 1-star rating but evidence that there has been many. Like stated above, assuming your arc received nothing but 3,4,5 stars and averaged out to a exactly 4-stars. The fact is that with 100 ratings, it would have taken "ratings assassins" 19 one-star ratings to move a 4 to round down to a 3.

C- Or, this is just that way your arc rates with players.
After doing the math above I am inclined to believe that everyone is getting 1 and 2 stars all the time but they are being offset by the number of 4 and 5 stars we get. No one person comes along and kicked a 5 star arc to a 4. It must be that you were getting 1's, 2's and 3's along the way to 100 plays. If you you are on the border between rounding up or down between 4 and 5 even a 4 rating would kick you down.

I am a believer that people are purposefully rating arcs 1's as I am that people are purposefully rating arcs 5's, all regardless of content, which can more or less can cancel each other out. So we can either believe the math which suggests that you have a story that isn't 5-star worthy to players, or it must be a widespread conspiracy to take you down.

This is not meant to defend the current system. I want Hof, DC, and Guest Authors off the main page and the ratings abolished or an improved ratings system implemented. I am just firmly against any more HoF arcs and especially lowering the bar or anyone getting a free pass into it. And I mean absolutely no disrespect to anyone here, but there are tons of quality arcs out there that have a grand total of 0 votes. So quality is not reflected by ratings, nor is HoF a reflection of quality.

The whole interface design and ratings system was a bad idea from the beginning. We have too many HoF pages as it is (and most of those aren't worthy of the title) all taking up prime real estate on the interface.


 

Posted

Quote:
Originally Posted by Wrong_Number View Post
I have spent more hours than I can guess promoting MA as have others. The devs need to get more involved to generate player interest.

WN
That line can sum up all 11 pages of our rhetoric. The devs need to be more involved.

Like the contest Dr. Aeon had recently. One a year? I mean there's got to be something more than this once a year event.

There needs to be some appeal to the player base and not the just authors to even enter these contests. New people have 0 chance with these contests.

My entry... Well I did the best I could do alone. Despite playing that arc 100's of times. Recently I went back through it with a friend and notice a horrible error that if I'd had proof readers or help would have caught.

Who's a new author suppose to ask for help? My only help came from a few authors but lets get real here. Your fellow authors have their own interests at heart. If they saw the bug they might not even tell you. Human nature and competition.

So of course the winner was somebody who got a ton of help on their arc. Not that it didn't deserve to win. There's just no chance for the average player to even compete in these dev choice contests.

Hold a contest that a normal, average, even new person can participate in. Something that appeals to the majority to get them back into AE for the right reasons.

As for myself. I have been playing 3 solid months and have made 4 arcs and got a 5th slot tied up with that Origins Pack thing. In only 3 months I've ran my AE course and am pretty much done until the next contest or reason to write again.

There's no point in writing new/good arcs at all now. My latest arc sits there unplayed and will sit there unplayed forever. So what's the point to write another outside of our own amusement.






If I keep messing with MA I won't even have a slot for next years contest.


 

Posted

Quote:
Originally Posted by Bad_Dog View Post
A- The system has a script that is calculating the average other than just using standard mathematical rounding and the rating, then, is an accurate reflection of the way the system is intended to work.

B- Those with 100 plays and see their arcs moving between 4 and 5 stars is not the result of ONE 1-star rating but evidence that there has been many. Like stated above, assuming your arc received nothing but 3,4,5 stars and averaged out to a exactly 4-stars. The fact is that with 100 ratings, it would have taken "ratings assassins" 19 one-star ratings to move a 4 to round down to a 3.

C- Or, this is just that way your arc rates with players.
After doing the math above I am inclined to believe that everyone is getting 1 and 2 stars all the time but they are being offset by the number of 4 and 5 stars we get. No one person comes along and kicked a 5 star arc to a 4. It must be that you were getting 1's, 2's and 3's along the way to 100 plays. If you you are on the border between rounding up or down between 4 and 5 even a 4 rating would kick you down.
Except that is not what is happening and it has been demonstrated time and time again since AE was in beta. The lower ratings have more weight than the higher ratings, which is why a single 1-star can knock an arc with 100-ratings and a 5-star average off of page three and send it falling hundreds of pages. For those of us who do monitor our ratings and tickets from them the evidence is right there when an arc gets a new rating, bumps up to the 5-star region, then the very next rating it gets knocks it right back down.

I really, really doubt that we have hundreds of arcs sitting right on 4.94 all of the time for this to be possible. If only the Devs would allow us to mouseover the rating and see the true average, or better yet the number of each rank of star ratings. I got a feeling they don't want to though for two reasons: they just simply don't want to place an resources on the issue and two if we really could see the raw numbers it'd give our arguments much more merit if they turn out to prove our accusations.


 

Posted

Quote:
Originally Posted by Lazarus View Post
Except that is not what is happening and it has been demonstrated time and time again since AE was in beta. The lower ratings have more weight than the higher ratings, which is why a single 1-star can knock an arc with 100-ratings and a 5-star average off of page three and send it falling hundreds of pages. For those of us who do monitor our ratings and tickets from them the evidence is right there when an arc gets a new rating, bumps up to the 5-star region, then the very next rating it gets knocks it right back down.

I really, really doubt that we have hundreds of arcs sitting right on 4.94 all of the time for this to be possible. If only the Devs would allow us to mouseover the rating and see the true average, or better yet the number of each rank of star ratings. I got a feeling they don't want to though for two reasons: they just simply don't want to place an resources on the issue and two if we really could see the raw numbers it'd give our arguments much more merit if they turn out to prove our accusations.
I totally agree, as I noted this as an option. It is possible they are using some sort of formula that is not using typical rounding/averages. In that case, we will never truly know what is going on without knowing the math behind the ratings we see. And without the basics of the formula, it's unlikely we will be able to guess how it works. I would like to know if they are indeed putting more weight on lower rating numbers, because I can't see a reason that would make sense in averaging your rating. Well, unless their intent is to keep players out of the HoF.

I merely propose based on the way "normal" averages work, math suggests that a conspiracy to assassinate ones 5-star status is unlikely to be successful and it's more likely that 2s and 3s are used for ratings more often that we might believe.

I notice that we have a ton of statistics on our info page in the game that shows how we are using alignments/tips, and another page for all the statistics for pvp. I would think we should have a similar feature for a published arc. It would be nice to see exactly how many people voted 1, 2, 3, etc


 

Posted

Quote:
Originally Posted by Bad_Dog View Post
I merely propose based on the way "normal" averages work, math suggests that a conspiracy to assassinate ones 5-star status is unlikely to be successful and it's more likely that 2s and 3s are used for ratings more often that we might believe.
It's not that they use a special formula, it's that the only rating that matters is an average of 5 stars. Anything below 5 won't get you a hall of fame and it won't get your arcs a good position on random searches. That is why lower votes than 5s are just different degrees of down-rating, and why a single 1-star vote is a death sentence for any new arc.


Winner of Players' Choice Best Villainous Arc 2010: Fear and Loathing on Striga; ID #350522

 

Posted

Quote:
Originally Posted by FredrikSvanberg View Post
It's not that they use a special formula, it's that the only rating that matters is an average of 5 stars. Anything below 5 won't get you a hall of fame and it won't get your arcs a good position on random searches. That is why lower votes than 5s are just different degrees of down-rating, and why a single 1-star vote is a death sentence for any new arc.
A truly excellent reason to abolish the HoF and the ratings system.


 

Posted

As far as 'easy to implement' changes go, what about something like granting an extra slot to players who have had an arc posted for a year, or just like another part of the yearly vet rewards.


 

Posted

They just need to let us buy more slots. MA Arc Slot Booster pack. 5 slots $9.99 or sth.

Eco.

EDIT: And the vet reward idea is genius.


MArcs:

The Echo, Arc ID 1688 (5mish, easy, drama)
The Audition, Arc ID 221240 (6 mish, complex mech, comedy)
Storming Citadel, Arc ID 379488 (lowbie, 1mish, 10-min timed)

Quote:
Originally Posted by Samuel_Tow View Post
[The Incarnate System is] Jack Emmert all over again, only this time it's not "1 hero = 3 white minions" it's "1 hero = 3 white rocks."

 

Posted

Quote:
Originally Posted by MrCaptainMan View Post
They just need to let us buy more slots. MA Arc Slot Booster pack. 5 slots $9.99 or sth.

Eco.

EDIT: And the vet reward idea is genius.
If they ever make an MA Booster pack it better include giving you a choose-able selection of existing contacts, (outside of AE), that makes your arc part of the missions players can choose from those contacts. Or something just as impressive.

I can't see any other reason to buy an MA booster pack. We have just about all the maps and enemy groups. A booster of just slots would fall flat, since we could buy slots all along and most people have already bought the number they wanted.


 

Posted

Quote:
Originally Posted by Lazarus View Post
For those of us who do monitor our ratings and tickets from them the evidence is right there when an arc gets a new rating, bumps up to the 5-star region, then the very next rating it gets knocks it right back down.

I really, really doubt that we have hundreds of arcs sitting right on 4.94 all of the time for this to be possible.
Actually, my belief (somewhat supported by observed behavior) is that I have several arcs right on the cusp of 4.48 or 4.49 or thereabouts. I believe this because they tend to bounce briefly up into 5 star territory, then after a few days fall back down into 4 star territory.

My theory is that ratings are rounded nearest, so 4.48 displays as 4 stars ... once in a while, someone stumbles across the arc and likes it enough to give it a 5 star rating, pushing the average up to 4.5001 which then displays the arc's rating at 5 stars.

While the arc's displayed rating is 5 stars, it will often appear on page 3 or 4 of the default search, and enjoy a brief blooming period where the arc gets increased player traffic. Some of the players will love the arc and give it 5 stars. Some players will kinda like it and give it 3 or 4 stars. Perhaps some players will hate it (or be deliberately griefing, as conjectured by other posters) and give it 1 star.

Regardless, the increased traffic results in an increased number of random ratings, which modifies the 4.5001 average both up and down, depending on the rating. For most arcs, this stochastic process eventually leads the average rating to fall down to 4.4999 or below at some point, at which point the arc is again displayed as "4 stars", the position on the default sort falls to page 200 or so, and the number of random plays falls to almost nothing. This puts the arc into a hibernation state where it is perhaps just one or two plays away from rising to 5 star territory again, but it may be quite awhile before those two plays occur due to it being quite difficult for players to stumble across the arc.

Anyway, my theory (partially supported by data, but not ironclad) is that there are a very large number of arcs just below the 4.5 line, at 4.48 or 4.49 or 4.495 or whatever, that are slumbering in this manner. There are a very few arcs soaring above the 4.5 line due to extremely positive response from the player base (or, for the cynics, due to ratings collusion), as represented by a higher average result in the random ratings sequence. These "5 star" arcs (but truthfully they are probably around 4.6 to 4.75, or some may even be closer to 4.52) are the ones being identified in this thread as likely candidates for Hall of Fame.

But I think it is quite possible for one of the arcs "sleeping" at 4.49 to suddenly make the jump to Hall of Fame. Indeed, the most recent arc to become HoF took this route, switching from a 4-star arc with 999+ plays (not tracked by this thread currently and not on anyone's radar) to a 5-star arc with 999+ plays (Hall of Fame) overnight.

Anyway, not quite sure where I was going with all this math, but that's my thinking on how the ratings system currently works.

Regarding my opinion of what should be changed, based on my theory, the major flaw in the current system is that only arcs near the front page of the default search get a significant number of random plays: namely, guest arcs, DC arcs, HoF arcs and "5 star" arcs (really 4.51+ average) with a large number of ratings.

My belief is that the reasons for this are:
(a) Arcs on the front page are easiest for casual players to find. (Partly laziness, partly the arcs on the front pages had a dev think they were good, so usually can't suck too badly.)
(b) It's hard for casual players to find high quality story-related arcs using the current interface. (If you skip past the first few pages, there are way more pure farms or badly written arcs than there are worthwhile arcs.)

I came to these conclusions earlier this year, and it's my opinion that the main improvement we need is, a better way for casual players to find good story arcs. This is the main reason why I started the "Contact Tree for MA Story Arcs" thread. Props to Arrowrose for starting the "Excellent Arcs" thread, as well, with similar intent.

Best of all would be if Paragon Studios would assign some programmer resources to addressing this problem -- improving the search interface, adding a "If you liked this arc, you might also like arcs A, B and C", adding the ability to create favorites lists or "theme" lists and allow friends to see them -- nothing terribly groundbreaking, just basically stuff that amazon.com, iTunes, and other purveyors of digital media have already done.

Of course, it is up to Paragon Studios to decide what their priorities for developer resources are. But I believe some improvements along these lines would hugely improve the Mission Architect experience for story-oriented players. Mission Architect is the most boldly innovative feature I've seen in a MMORPG, and player-created content seems like a huge competitive advantage over other MMORPGs. Or it could be, if it were better supported. I think it's a shame that this feature is so underutilized right now.


@PW - Police Woman (50 AR/dev blaster on Liberty)
TALOS - PW war journal - alternate contact tree using MA story arcs
=VICE= "Give me Liberty, or give me debt!"

 

Posted

Quote:
Originally Posted by FredrikSvanberg View Post
It's not that they use a special formula, it's that the only rating that matters is an average of 5 stars. Anything below 5 won't get you a hall of fame and it won't get your arcs a good position on random searches. That is why lower votes than 5s are just different degrees of down-rating, and why a single 1-star vote is a death sentence for any new arc.
Bolded for emphasis. A four-star rating isn't bad...or at least it shouldn't be. Four stars is described as "excellent" in game. Assigning an arc a four-star rating should mean "I liked this arc, except for a few things (hopefully mentioned in comments.) It was worth my time. I'm glad I played it. If someone asks "should I play this arc?" I will say yes."

With our crappy rating and sorting system, assigning a four-star rating translates to "I hate you and I don't think your arc is worthy of ever being played again unless you're a prominent member of the AE community and heavily promote it."


Eva Destruction AR/Fire/Munitions Blaster
Darkfire Avenger DM/SD/Body Scrapper

Arc ID#161629 Freaks, Geeks, and Men in Black
Arc ID#431270 Until the End of the World

 

Posted

The problem is that the math being used is TOO simple.

It's just the raw average. It does not take into account modes or standard deviations. Nor does it take into account the number of ratings when doing the sort. Instead an arc with a 4.48 average and 10 votes gets placed dozens of pages ahead of an arc with a 4.47 average and 200 votes.

If they'd add options to sort the arcs by the amounts of each rating it received then arcs that have a large amount of 5-star votes but are being bogged down by a small number of 1-star votes would be able to rise to the top again.


 

Posted

I think one thing we could use is someone (I considered doing it, but haven't had the motivation to do one detailed enough) to collect what we think are the biggest problems in the MA including suggestions on fixing them into a single thread. It's something they've done in the bases forum - though hopefully the MA is a higher priority than bases.


 

Posted

Quote:
Originally Posted by Bubbawheat View Post
I think one thing we could use is someone (I considered doing it, but haven't had the motivation to do one detailed enough) to collect what we think are the biggest problems in the MA including suggestions on fixing them into a single thread. It's something they've done in the bases forum - though hopefully the MA is a higher priority than bases.
And just look how far they've come with bases.

oh, uhm, nvm.


 

Posted

Quote:
Originally Posted by Sumericon View Post
And just look how far they've come with bases.

oh, uhm, nvm.
To be fair, the Scrappers have one too, and it's a lot shorter than it used to be.

On the other hand, the Scrappers also have Arcanaville pestering the devs on their behalf, and a dev who actually reads the forums and has been known to actually listen to player feedback. We have...people replying to reasoned arguments with some nonsensical drivel about the latest exploit and a dev who treats the AE as his own personal playground.


Eva Destruction AR/Fire/Munitions Blaster
Darkfire Avenger DM/SD/Body Scrapper

Arc ID#161629 Freaks, Geeks, and Men in Black
Arc ID#431270 Until the End of the World

 

Posted

Quote:
Originally Posted by PoliceWoman View Post
Best of all would be if Paragon Studios would assign some programmer resources to addressing this problem -- improving the search interface, adding a "If you liked this arc, you might also like arcs A, B and C", adding the ability to create favorites lists or "theme" lists and allow friends to see them -- nothing terribly groundbreaking, just basically stuff that amazon.com, iTunes, and other purveyors of digital media have already done.
This is what we would get for 2/3s-ish of everything out there:

If you liked this arc, you might also like Lt. Farm, Ticket Farm, Farming Madness.....


 

Posted

Quote:
Originally Posted by Bad_Dog View Post
This is what we would get for 2/3s-ish of everything out there:

If you liked this arc, you might also like Lt. Farm, Ticket Farm, Farming Madness.....
If you played a farming arc, sure.


 

Posted

Quote:
Originally Posted by LaserJesus View Post
If you played a farming arc, sure.
I don't know about you but there's been many times I picked a random arc to play and found its just another farm mission. You don't have to look for a farm to find one.


 

Posted

December 1st update. Lost a few this month.

As of December 1st we now have the following breakdown of arcs with 100 or more plays and that are still rated at 5 stars. Of note this month is that 8 of the 10 arcs have over 200 plays.

Arcs Published Pre-15 - 7 (6 of these were published in April 09)
Arcs Published Post I-15 - 3

Total Number of arcs out of 69,296 arcs - 10

Put another way: .0001443% of the arcs in the system have more than 100 plays while still maintaining a 5 star rating.

Percentage of current HoF Arcs Published in 4/09 - 100%

History
May - 12 (11/1)
June - 14 (11/3)
July - 14 (11/3)
August - 11 (8/3)
September - 11 (7/4)
October - 10 (8/2)
November - 13 (10/3)
December - 10 (7/3)


WN


Check out one of my most recent arcs:
457506 - A Very Special Episode - An abandoned TV, a missing kid's TV show host and more
416951 - The Ms. Manners Task Force - More wacky villains, Wannabes. things in poor taste

or one of my other arcs including two 2010 Player's Choice Winners and an2009 Official AE Awards Nominee for Best Original Story

 

Posted

Quote:
Originally Posted by Bad_Dog View Post
I don't know about you but there's been many times I picked a random arc to play and found its just another farm mission. You don't have to look for a farm to find one.
This, distressing as it is, is quite true.


 

Posted

I would love it if someone would corner some devs at a con and ask them about this topic... and the 8 arc cap... and about plans to clean up the interface... and weed out the broken arcs... and... and... and...

Preferably someone who looks like they should be named "Guido".


 

Posted

A mini-update since I've never seen the numbers drop this low before.

As of right now there are only 6 arcs with over 100 plays and still at 5 stars. This is an all time low with the previous low being 10 (which was at the start of this month).

5 of the 6 arcs have over 200 plays. The 1 with just over 100 plays is one of the post I15 published.

The split is 3/3 with half the arcs coming from 4/09 (the first month MA was released) and the other half all published all after I15 with the oldest published in 11/09 and the newest 2/10.


WN


Check out one of my most recent arcs:
457506 - A Very Special Episode - An abandoned TV, a missing kid's TV show host and more
416951 - The Ms. Manners Task Force - More wacky villains, Wannabes. things in poor taste

or one of my other arcs including two 2010 Player's Choice Winners and an2009 Official AE Awards Nominee for Best Original Story