So, Let's Talk Ratings And Awards
By
Arctic_Princess,
Posted on: 2009-04-11 11:11 in Mission Architect
I pointed out how ridiculous the HoF requirements were with a couple of examples of voting distributions at the start of March (here)
[ QUOTE ]
Big example...
1500 people vote as follows:
100 1 star
200 2 stars
300 3 stars
400 4 stars
500 5 stars
The mean average would be c. 3.666 (=5500/1500). Rounded up that's 4.
So we'd have 900 people who rated it 4 or 5 out of 5 and they aren't enough to overide 300 who thought it less than average (if 3 is considered average).
So how many people would need to vote 5 stars just to move this arc into HoF territory?
Turns out that if an extra 500 voted, and all gave 5 stars, the mean average would just hit 4.0, so still not quite enough... so that'd give us the following distribution (which only hits a 4.0 mean):
2000 people voted as follows:
100 1 star
200 2 stars
300 3 stars
400 4 stars
1000 5 stars
[/ QUOTE ]
Effectively, for HoF considerations, a single vote of 1 cancels out 3 votes of 5 (1x1 + 3x5 = 4x4). SO effectively 3 people can love something and all it takes is one person (potentially with unrealistic expectations or griefing) to completely negate the favourable votes.
Now, given this, I don't think many people would call me cynical if I state the following:
I do not believe that any pre-I14 dev-created arc would rate well enough to surpass the Hall of Fame criteria if people voted on them with the same voting patterns and expectations that they are applying to player created MA arcs.
I think it's possible that 3 or 4 TFs/SFs/Trials might make it. But even that's not a definite.
Before you shout me down on this: Remember that it takes relatively few dissenting voices to rate something low to drop the mean average to <= 4. Do you know any arc that virtually everyone loves?
BTW - as an aside on this, if you rate average as being equivalent to dev-created arcs then what you are saying is that the 'average' score can only be attained by amateurs who manage to do the same level of content as the professional game developers. 
I actually believe that the level of most dev arcs should be more like a 4 rating.. a 3 should be a solid amateur arc.
BTW2 - marking 3 as being the same level as dev-arcs would really help stop the devs getting to HoF standards.
By my mohawk shall ye know me!
my toons
Funny: Ee-Ai-Ee-Ai-Oh! #3662 * The foul-mouthed Handyman! #1076 * City of Norms #132944
Serious: To Save A Single World (#83744) * Marketing Opportunity (#83747)
Its a litte early to tell how this will pan out atm, but I share your concerns about ratings and the HoF.
As it stands, I will wait to see how things go. The ratings at the moment are plagued with "my freinds gave me 5"(I have played a lot of 5 star arcs that barely scrape a 3) and, unfortunately, evidence of much more ganking than I would have predicted (and im pretty cynical).
Im finding I will, on average, get a far bigger bang for my buck by a 4 star, 20 votes arc, than a 5 star 1-10 vote arc. FAR bigger.
As you say, once a lot of votes start coming in,it gets very very hard to stay in the5 star bracket. I even supsect the one of the main reasons they do is the "follow the group" mentality of voting.
I don't like the tickets thing for ratings. Only getting tickets if the arc is rated 3 or more makes it seem like anything 3 or lower is either just okay, or really bad. I'm quite stingy when it comes to rating things usually. Out of 5, I'd say 2 isn't all that bad. 1 is, obviously just something I don't like, but I might see merit in something but just think there's a bit of room for improvement and mark it 2. 3 would be good, but nothing special and 4 would be something I really enjoyed. I'd hardly ever give out 5's unless I thought it was well done, memorable and I had a really good time, so much that I'd do it again. I'd much rather had a percentage score, or at least an out of 10 system. But hey, we don't so...
As for getting to HOF, I'd suspect it might take a good long while.
@craggy see me on Union for TFs, SFs (please!) or just some good ol fashioned teaming.
So from your maths if a fifth (thats 1 in 5 which is quite a lot really) of the people think that your mission os no more than avarage then you are upset that you dont get 5 stars. Eh if a fith of people think it is no more than average it does not deserve 5 stars.
Now if a twentieth (thats 1 in 20) of people felt it was no more than average i would expect it to be mid to high but still 4 stars.
This is the whole process where cream rises to the top scenario, I doubt that over time once some body has gotten all their supergroup members to give dross 5 stars and a few other people play it and realice how bad it is it will sink fast.
I am thinking long term with the MA (9months to a year) so if something has 1500 votes and is 4 stars it will most likely be enjoyable. if something has 1500 votes and has 5 stars it should be exceptional.
[ QUOTE ]
BTW - as an aside on this, if you rate average as being equivalent to dev-created arcs then what you are saying is that the 'average' score can only be attained by amateurs who manage to do the same level of content as the professional game developers.
I actually believe that the level of most dev arcs should be more like a 4 rating.. a 3 should be a solid amateur arc.
BTW2 - marking 3 as being the same level as dev-arcs would really help stop the devs getting to HoF standards.
[/ QUOTE ]
For your reference, my rough personal guide:
2 star - run of the mill paper/radio mish
3 star - most of the early hero arcs
4 star - Doc Buzzsaw or Seer Marion
5 star - Automatic Villainy with Tech Naylor, or the ITF
And I don't see why we shouldn't compare amateurs to professional devs. Doing the same level of content of say, a paper mish is something I can whip out in 5 minutes. When it comes to writing out a decent plot, we're pretty much on equal ground. The devs don't have any particular 'edge' to make them write more interesting arcs than what we could come up with ourselves.
A Penny For Your Thoughts #348691 <- Dev's Choice'd by Dr. Aeon!
Submit your MA arc for review & my arcs thread
Well my m aths may be weak, but if I got 6 votes of 5 stars (uhnless peoples feedback was a lie), to drop down to 4 stars on the fifth vote can only be a 1 star vote?
Now, Ive no problem with people voting me 3 stars, or even 2. They didnt like it, wasnt for them, no problem.
But a 1 vote? Have you seen the dross out there. Im n ot absolutely *certain* it was a gank vote, but im pretty suspicious.
tbh im not really that bothered or upset. I treat it as a gank vote. If someone had sent me feedback saying "sorry, i didnt like you arc for these reasons" and gave me 2 stars - I would have listened.
It wasnt a fifth of people thinking my arc was below average (im fine with that. Hey, im fine with half thinking that), its 1/7 (or about 1/10 now) saying my arc was rubbish (not below average, but plain rubbish) when everyone else is giving it 4 or 5 stars.
I just ignore outliers. Perhaps im being paranoid, but on the balance of probabilities, I see that as a gank vote.
I have seen even clearer evidence of gank voting on other peoples arcs, as well. Its particuarly nasty to gank vote an unrating arc (as I am almost certain happenedto Bovine Avengers arc).
I know everybodies tastes are different, but there are some well done, polished arcs that nobody in their right mind can give a 1 star vote for, even if its not to their taste. Or am I being unreasonable?
Personally, if I see a polished arc with something I dont like, I wouldnt give it 1 star. I would probably not rate with the veiw "Hey, I can see this is well execited, but my personal preferences mean I cant fairy vote on this".
EDIT: PS with regards to initial votes, I havent got my SG members to vote for arc in question yet, and only one freind has so far.
[ QUOTE ]
And I don't see why we shouldn't compare amateurs to professional devs. Doing the same level of content of say, a paper mish is something I can whip out in 5 minutes. When it comes to writing out a decent plot, we're pretty much on equal ground. The devs don't have any particular 'edge' to make them write more interesting arcs than what we could come up with ourselves.
[/ QUOTE ]
In the same way that a professional pilot has no edge over any other able-bodied adult.
They have years of practice and presumably have organised reviews and feedback. Hell the seminars, books, courses on creative writing possibly helped a little too...
Mind you they also have the ability to step outside limits imposed on us by the MA, so they do have additional tricks that we don't have.
The devs have no innate advantage at coming up with a story, just as a pilot has no innate advantage at navigation or flying, but in both cases years of training and practice, an organised support network and access to advanced tools ought to give them the upper hand.
By my mohawk shall ye know me!
my toons
Funny: Ee-Ai-Ee-Ai-Oh! #3662 * The foul-mouthed Handyman! #1076 * City of Norms #132944
Serious: To Save A Single World (#83744) * Marketing Opportunity (#83747)
Im interested to know what people reserve a 1 star for
[ QUOTE ]
Well my m aths may be weak, but if I got 6 votes of 5 stars (uhnless peoples feedback was a lie), to drop down to 4 stars on the fifth vote can only be a 1 star vote?
Now, Ive no problem with people voting me 3 stars, or even 2. They didnt like it, wasnt for them, no problem.
But a 1 vote? Have you seen the dross out there. Im n ot absolutely *certain* it was a gank vote, but im pretty suspicious.
tbh im not really that bothered or upset. I treat it as a gank vote. If someone had sent me feedback saying "sorry, i didnt like you arc for these reasons" and gave me 2 stars - I would have listened.
It wasnt a fifth of people thinking my arc was below average (im fine with that. Hey, im fine with half thinking that), its 1/7 (or about 1/10 now) saying my arc was rubbish (not below average, but plain rubbish) when everyone else is giving it 4 or 5 stars.
I just ignore outliers. Perhaps im being paranoid, but on the balance of probabilities, I see that as a gank vote.
I have seen even clearer evidence of gank voting on other peoples arcs, as well. Its particuarly nasty to gank vote an unrating arc (as I am almost certain happenedto Bovine Avengers arc).
I know everybodies tastes are different, but there are some well done, polished arcs that nobody in their right mind can give a 1 star vote for, even if its not to their taste. Or am I being unreasonable?
Personally, if I see a polished arc with something I dont like, I wouldnt give it 1 star. I would probably not rate with the veiw "Hey, I can see this is well execited, but my personal preferences mean I cant fairy vote on this".
EDIT: PS with regards to initial votes, I havent got my SG members to vote for arc in question yet, and only one freind has so far.
[/ QUOTE ]
See this is where you differ from me. I am taking the MA systeam as a world wide system with a player base in teh hundreds of thousands which over tim eiwll allow the good to rise and the bad to sink.
Yo a re complaing that 5 people gave you 5 stars and the 6th gave you a 1 star. You automatically think they are ganking you, maybe they did not like you colour scheme for your custom faction and gave it a poor rating for that alone harsh maybe, if they did not provide a comment or it got lost then you may never know. But to automatically jump to them ganking you is a leap.
Now if you have 600 vote and 599 were 5 stars and 1 was a 1 star i might have sympathy with you.
ninja edit
Did a quick search for cognito on the MA and could not find any missions to see for myself to provide feedback.
I see your point Dave, but I believe that assumes that by being an 'amateur' writer, they can't get access to similar training and experience a dev might have had. I'm sure there are a few people who have taken said seminars, books, and courses on creative writing, and people who gained other avenues of experience being hobbyist or even professional writers themselves, modders, dungeon masters, etc.
[ QUOTE ]
Mind you they also have the ability to step outside limits imposed on us by the MA, so they do have additional tricks that we don't have.
[/ QUOTE ]
I'll concede that players won't be able to make an exact replica of ITF with the current MA system (but they certainly can capture the spirit), but there's no special dev trickery used to make Automatic Villainy.
A Penny For Your Thoughts #348691 <- Dev's Choice'd by Dr. Aeon!
Submit your MA arc for review & my arcs thread
I actualy pretty much agree with you,Mr J. However, I at this point I was left with a vote which was presumably 5,5,5,5,5,5,1
Now, I *know* my arc is a marmite arc, and its had negative critique before. I also om of the veiw that it is hovering between 4 and 5 stars in my OWN head, so, I'm not thinking its uber or anything.
I am also pretty certain there is nothing that would cause ragequit on the first mission (at least).
Also, dueto immense help from the community, its polished. Proofread three times, every bio done, clues, all the "pollish" has been done.
And im left with 5,5,5,5,5,5,1. My conclusions are:
I know I will ignore the one, particuarly as I got no feedback. I also accept its a marmite arc.
It is highly likely, but not absolutely certain, that the voter got a pet peeve out of something. Thats fine. Not something I would give a 1 star for, but thats their choice I guess.
On the balance of probabilities, I see this as a gank vote. It may not be, it may be.
If I had something like 5,4,4,5,4,2 I would be fair more inclined to say, hold on, its not a good as I thought.
End of day, we have to interpret our results as we see fit. The problem with the voting system as it stands is that it can get mucked up by outliers, positive and negative.
As others have said, marmite arcs particularly suffer. There is a world of difference between ann arc cruising around 3 stars consistently, and one which scores half 1 and half 5.
Irrespective of my arcs, I know others who have almost certainly been ganked (I guess mine was around a 60% probability).
Im ay have completely missed the boat, there maybe people who are just trigger happy with a 1 star vote - personally, just a 5 stars needs to be stand out good, 1 star needs to be stand out bad. Trigger happy 1 starrers dont really help the system, however, at least in the early days.
[ QUOTE ]
Im interested to know what people reserve a 1 star for
[/ QUOTE ]
No story, no description of what the mission is about (objectives/tough or easy/purpose of me going there etc), mobs thrown together just for the sake of it . . . . . you get the idea 
Basically if you can make a mission more filled out than it in less than 5 minutes it deserves 1 start imo.
Its all about the description, if the description STATES that its for PL/farms/teams/low level/getting the publisher badge etc then i will judge it on those circumstances. I like to know what kind of mission im getting into beforehand.
@Damz Find me on the global channel Union Chat. One of the best "chat channels" ingame!
[ QUOTE ]
5/5 - for those that I want to be HoF
4/5 - for those that I would not object if they were to get a HoF
3/5 or lower - ...everything else.
[/ QUOTE ]
Do you know what? This is the best advice on how to vote. Ive been using this testing arcs to day and suddenly all my dilemas have been solved!
Id probably add 3/5 for stories I liked but wouldnt like to see clogging up the Devs Choice, 2/5 for a passable standard and 1/5 for crud.
Using this, my rating system has suddenly become clear.
I salute you sir!
A limit of 4+ rating with at least 1000 votes is no law or design pattern that cannot be easily changed. If the devs see that no or almost no arc will reach that limit, then I am quite sure they will adjust it, most likely changing the number of votes required.
That being said, I think the rating that Tangler suggests pretty much is what the devs had in mind, otherwise the 4+ would not have been stated IMHO.
[url="http://adingworld.wordpress.com/mission-architect-story-arcs/"][b]My Story arcs[/b][/url]: [i]The Siren Supremes[/i] ([b]1143[/b]), [i]The Missing Geneticist[/i] ([b]2542[/b]), [i]Elemental Jones[/i] ([b]263512[/b]), [i]The Soul Hunter[/i] ([b]294431[/b]), [i]Heart of Steel[/i] ([b]407104[/b]), [i]Project Serpens[/i] ([b]434082[/b])
Cognito:
I have given out one 1-star rating so far. It was for a mission called "My First Attempt" (or something like that), and the description was along the lines of "This is my first attempt, plz be nice to me".
I thought I would give it a try and provide some constructive feedback to the author. Turns out there was: no contact detail, no mission briefing (well, the briefing was "sdlkcj"), it was a Defeat All mission on the outdoor ruins map, and the entire map was spawned with a single custom mob, the design of which was... nothing special.
Needless to say, I wasn't particularly impressed, and gave it a 1-star rating along with some fairly direct feedback about _why_ I had given it a 1-star rating.
What really got to me, however, was the fact that 4 previous votes had given it an average 5-star rating. That just sooo yanked my chain :P
The Hall of Fame is supposed to be a long-term thing, to my mind anyway. However I do get the whole 'one person giving me a 1-star or not rating at all = general doom'.
Having said that, my personal criteria are as follows:
0 stars - absolute rubbish/uncompletable/incomprehensible OR a WIP arc that I am testing for a friend.
1 star - barely comprehensible in terms of story. Needs a lot of text editing. Mobs don't excite me (by that i generally mean Custom Mobs that add nothing to my personal experience)
2 star - obvious effort has been put in but the arc still lacks consistency. There could well be one thing that stands out as being great (and I will point out the 'keepers' in my feedback) but it is currently being overshadowed by bigger problems.
3 star - a good effort. Feels like a paper/radio/one-off mission but nothing to get too excited over. Nice to play and level up on.
4 star - Rather a lot of good stuff in there. Needs tightening up but otherwise worthy of the Hall of Fame. Minor problems that jar remain but are easily fixed.
5 star - It makes me laugh and/or cry (depending on flavour), clues are slick, editing is good, I could imagine my characters actually needing to go and do these things as Heroes or Villains. Also in this category are well-executed homages (that don't just use the names of the things they are pastiching - although I can take the odd exception) and experiences that feel like I am playing a different (and equally good) game.
I decided to not give out stars to the truly abhorrent because some missions I've played already have been dross. Yes this is personal opinion but, unless the author is being ironic (1337-speak for Freaks - shows signs of paying attention!) then txt spk and attrocious spelling and grammar on top of no plot, impossible enemies and a lack of thought means no stars at all from me. But plenty of comments!
Therefore I wholly support third-party advertising of arcs, player-created databases and all that because, well, if nothing it inspires creativity and, I like to think, will help create an additional community of creators. That's not to say Elite creators, just people who care about stories and entertaining their gaming peers.
You do realise that zero stars (so no rating) effectively does not get shown by the system?
The number next to the star rating is the number of ratings (not the number of plays).
In effect giving no stars to absolute rubbish and 1 star to a barely comprehensible story damages the public rating of the better arc but does not alter the rating of the rubbish arc.
By my mohawk shall ye know me!
my toons
Funny: Ee-Ai-Ee-Ai-Oh! #3662 * The foul-mouthed Handyman! #1076 * City of Norms #132944
Serious: To Save A Single World (#83744) * Marketing Opportunity (#83747)
So apparently the HoF req. are that you must have a 1000+ votes that must average out to 4+ rating. The incredible unlikeliness of ever getting that many votes aside, the devs have now just placed a standard as to arc ratings.
If anyone caught my ratings thread on the test boards, they may have known that I was pretty strict when it came to giving out those stars, but now instead of using the entire spectrum to rate arcs I now have to give:
5/5 - for those that I want to be HoF
4/5 - for those that I would not object if they were to get a HoF
3/5 or lower - ...everything else.
This is a pretty big deal to me, since in my mind a 3/5 was a good arc/marmite, but this system now means I have effectively doomed those arcs to obscurity by not giving them a 5/5, as well as making it pretty gosh-darn easy for malicious players to deny authors any merit.
I suppose my issue isn't that HoF has incredibly high standards in itself, but because it is the ONLY dev recognised standard we have for rating and judging, I now have to artificially inflate my ratings if I want to give author's any real props for their work in-game (the recognition and extra publish space, etc).
We really, really need more ways to get the message out to players about good arcs - IN-GAME - that do not fall into the 'well, it has lots of 5 stars' category. Tags, public awards, favourites, suggested arcs, community groups, seriously anything. Player's are dipping in blind to try a lot of these arcs and getting exactly what you would expect, and this just isn't going to go down well for all those players returning to try out this issue, only to play 5 'Go. Kil. Skulz.' arcs and call it a day.
A Penny For Your Thoughts #348691 <- Dev's Choice'd by Dr. Aeon!
Submit your MA arc for review & my arcs thread