So, how do you feel about the rating system?
By
Alari_Azure,
Posted on: 2009-05-10 18:28 in Mission Architect
The rating system is worthless. It should be a "all or nothing" type of system, where you can thumbs-up an arc or do nothing (important: no thumbs-down option). This removes the ability to grief while retaining the ability to actually play and recommend an arc. Combined with more robust search capabilities, this would allow for meaningful sorting of arcs. Good arcs would show up as having many ratings relative to plays; bad arcs would languish down where they belong.
I think that also, you should not be able to rate an arc until you've completed it. If you have to drop an arc due to poor design or too hard bosses, or whatever, it should be relayed in the form of anonymous comments, not a lower rating.
[ QUOTE ]
Any time there's any sort of material gain involved in these games there are people who will abuse the feature for their benefit and attack those who are in competition with them.
Giving arcs high ratings provides a very tangible benefit -- tickets. Giving arcs low ratings provides a less tangible detriment to the recipient -- a low rating -- but a tangible benefit to someone who is direct competition with the person receiving that low rating-- a higher relative rating.
I have seen quite a few instances of well-written arcs that are given zero or one stars (you can tell this when they start with a five which changes to a three on the second vote). These arcs function properly, have clearly stated goals that are attainable, are well-written and basically do what the description says.
The possibilities are: 1) someone accidentally rating it zero, or there is a bug that causes zero ratings if you just exit, 2) someone is intentionally running around rating arcs zero for whatever reason, 3) some aspect of the arc really ticked the person off and he zeroed it out of spite.
To my mind, if an arc delivers what the description promises it should get at least a three. The only time it should be zeroed or oned is when it uses some hideous gimmick (the infamous 5 chained AV spawns on a huge outdoor map) that makes it unplayable or excruciatingly frustrating, is socially or morally offensive while lacking any artistic merit, is so poorly written and constructed so as to be unintelligible, is a blatant farm with no story, and the like.
If we wanted honest ratings and comments about the arcs, the comments would be anonymous (to us, but traceable back to the original writer by the devs). There would be no tangible benefits from ratings (no tickets, no badges, no Hall of Fame, nothing except Dev's Choice to allow good arcs to become permanent additions to CoH). There would only be the rating, and nothing else. That still wouldn't make the ratings "fair," but it would remove the motivations for many abuses of the rating system.
Yes, there would still be people zeroing competing arcs out of spite to have a higher rating, voting cartels, vote swapping, etc., but it would just be for the ego boost and not to get tickets or badges.
But now that they've implemented this system, I don't see them retracting it. It should be adjusted in some manner. One change that should be made: if you never actually play the arc your vote should not be recorded.
Right now, if you click the PLAY button and then immediately quit the arc and vote, your vote is recorded. This is patently wrong. Players can just run through their "enemies" list and zero vote dozens of arcs in minutes. Similarly, you can easily run through your friends list and give them 25 tickets per arc every week (at least the last I read that was how it worked).
Your vote should only be recorded if you actually played at least part of the arc. How much you have to play is a good topic for debate. Basically, it should take long enough to make it not worth your while to grief or unfairly reward people. My guess is it should be one mission (considering that some arcs only have one mission).
Will people still grief others and unfairly reward their pals? Sure. But at least it will take longer, and that will make it self-limiting.
[/ QUOTE ]
Great post and I have to agree 100%.
I have 2 arcs currently that just made it to 5 stars and got pushed up to page 2.
Now rather they get shot with a zero or not, I do think the rating system as it stands "now" is borked, and I think alot of people on this thread have pointed out very good reasons why, it is.
I think I15 is suppose to help people search by taste's, and that will help people alot more to find, what they are looking for to play, including me.
But the zero needs to go IMO.
Good discussion Guys! 
First off, I'd like to object to a few assumptions in the OP. Just because your friend didn't receive any tickets does not mean he received a 0 rating. A rating of 1 or 2 stars wouldn't produce any tickets either. Second, I don't know how long you waited to look. The tickets may not show up immediately. In my own experience, the lag between rating and tickets can be huge, hours or even days. Third, it's not griefing to give someone a low rating. There are a lot of reasons why someone might have given your friend's arc a low rating. Maybe they just didn't like it. Everyone seems to assume that any rating below three stars must be griefing. Different people have different tastes. It's not a crime to dislike someone's arc.
As for the rating system itself, I think it's trying to do too much with too little. If the purpose of having a rating system is to sort the arcs by quality, then tying the ratings to badges and ticket rewards was a mistake. It's effectively bribing people to undermine the system.
A simple 5 star system also ignores the reality that there is no such thing as absolute quality in something as completely subjective as mission arcs. Imagine asking everyone on earth to rate all of the music the world has ever produced on a 5 point scale. What good would it do? How would someone who loves bluegrass music use that rating to find something they liked?
One system that might have been more useful would be "people who liked this mission also like..." type thing. It's not trivial to implement, but it would do the job pretty well. Another option is the "trusted reviewer". That's really in our hands, though the game could support it. I could review arcs, and post my reviews, and people who decided that they like the same kind of thing that I like could rely on my reviews to help them find new arcs to run. Someone who hates the stuff I like could use my reviews to make a list of things to avoid. It would be like movie reviews. Unfortunately the sheer volume of arcs, and the fact that they can change or vanish, make arc reviews more difficult and less useful than movie reviews.
Avatar: "Cheeky Jack O Lantern" by dimarie
[ QUOTE ]
First off, I'd like to object to a few assumptions in the OP. Just because your friend didn't receive any tickets does not mean he received a 0 rating. A rating of 1 or 2 stars wouldn't produce any tickets either. Second, I don't know how long you waited to look. The tickets may not show up immediately. In my own experience, the lag between rating and tickets can be huge, hours or even days. Third, it's not griefing to give someone a low rating. There are a lot of reasons why someone might have given your friend's arc a low rating. Maybe they just didn't like it. Everyone seems to assume that any rating below three stars must be griefing.
[/ QUOTE ]
Let me help you with your assumptions.
The only thing that having that happen to a friend of mine did was, start a debate between the 2 of us, that I felt would be a good topic for this board to discuss, rather they think the rating system is fair or not.
That is all, and it seems it is a good topic for discussion, just as I thought.
[ QUOTE ]
Correct. No vote counts as no vote and doesn't impact ratings. You have to click 1 star twice to vote zero and torpedo someone's arc (deservedly or not).
[/ QUOTE ]
So as long as I don't click on any of the stars AT ALL, I'm not unintentionally 0-starring?
If I've clicked on a rating, then decide I don't want to give any rating at all, is there a way at that point to do that?
[ QUOTE ]
The rating system is worthless. It should be a "all or nothing" type of system, where you can thumbs-up an arc or do nothing (important: no thumbs-down option). This removes the ability to grief while retaining the ability to actually play and recommend an arc. Combined with more robust search capabilities, this would allow for meaningful sorting of arcs. Good arcs would show up as having many ratings relative to plays; bad arcs would languish down where they belong.
I think that also, you should not be able to rate an arc until you've completed it. If you have to drop an arc due to poor design or too hard bosses, or whatever, it should be relayed in the form of anonymous comments, not a lower rating.
[/ QUOTE ]
1. Some arcs really, truly do deserve a 1-star, or a Thumbs-down. In your method, if I played an arc, and felt that it was just "medium", not actually good, but not bad, either, I would not be able to distinguish my opinion from someone who played it and honestly that it was lousy example of how to build an arc. My lack of a "thumbs-up" would be no different from someone who would give it an honest "thumbs-down".
2. I start an arc that an interesting description, something sounds like I may enjoy it. It's got a few spelling and grammar errors in the description, but not so many that I can't tell what the creator is saying. I enter the first mission and it's full of the toughest, baddest custom mobs the designer could create, completely impossible for me to defeat even a single one of them, saying things like "LAWLZ! SUXORZ 2 B U!", "LOOSER! NUB!! FOOLD U!!", you're saying I still have to somehow complete that piece of dung before I can give it the 1-star it deserves? There has to be an option to rate without performing ANY of the arc at all for those cases.
[ QUOTE ]
[ QUOTE ]
Correct. No vote counts as no vote and doesn't impact ratings. You have to click 1 star twice to vote zero and torpedo someone's arc (deservedly or not).
[/ QUOTE ]
So as long as I don't click on any of the stars AT ALL, I'm not unintentionally 0-starring?
If I've clicked on a rating, then decide I don't want to give any rating at all, is there a way at that point to do that?
[/ QUOTE ]
Nope, if you click on a rating and then decide you don't want to give any rating at all and so you remove that star, you will be giving it zero stars.
Don't count your weasels before they pop dink!
Tripp:
Any rating system relies, to some extent, on the assumption of good faith on the part of both creators and reviewers. That is not an assumption that can ever be made in this context. 
My characters at Virtueverse
Faces of the City
[ QUOTE ]
There has to be an option to rate without performing ANY of the arc at all for those cases.
[/ QUOTE ]
This is only true if the purpose of rating it one star is to punish the arc's creator. I submit that we will all be better off if instead of dropping an arc and rating it one star, we drop an arc and send feedback on why we dropped it.
As for arcs that truly deserve one star under the current system, the equivalent would be to play the arc and not vote it up, thus lowering its average votes per playthrough, or to not complete it, thus increasing the number of times the arc is dropped. Under the system I proposed, both of those statistics would be available to players to sort or search under.
Under my proposed system, any given opinion becomes equally weighted. If you didn't like the arc, you do not vote it up. The "score" changes from being a mean of star ratings to the number of votes out of all plays the arc received. If an arc is playable but not good, it may have many complete plays but few votes. If an arc is unplayable, it may have few complete plays and even fewer votes. The good arcs will have many complete plays and many votes. Under my proposed system, yes, the individual's vote does not hold the power to torpedo someone's arc, but the overall system is far more useful.
My system would not only report total number of votes; that would be useless without knowing how many times the arc was played and/or dropped.
[ QUOTE ]
Tripp:
Any rating system relies, to some extent, on the assumption of good faith on the part of both creators and reviewers. That is not an assumption that can ever be made in this context.
[/ QUOTE ]True. So very, very true.
Emberly,
I like your proposal. I've been thinking along similar lines.
Re: Rating System...
I think it comes down to one question: Is the purpose of the rating system to bring "quality" content to the top of the list?
If so, then the system is worthless. It clearly isn't getting the job done. How I feel about it isn't really relevant. It just straight-out doesn't work, yo. This... is a dead parrot.
Bonehead Move #1 was using a 5-star system. Especially after having witnessed the significant flaws in such a thing on these very forums (and then, I might add, removing it due to those flaws).
Bonehead Move #2 was linking badges to the ratings system.
It really needs to be scrapped entirely. They need to go with something more along the lines of Emberly's suggestion, and they need to not attach badges to the ratings system in any way, shape, or form.
Honestly, I gotta wonder how such a thing even made it past the initial discussion phase. "Let's Attach Badges to the Ratings" should've been tossed on the heap with "Let's Make All the MA Maps Pink" and "Let's Randomly Insert AVs Into People's Arcs."
I mean, honestly. What were they thinking? 
The Cape Radio: You're not super until you put on the Cape!
DJ Enigma's Puzzle Factory: Co* Parody Commercials
I'm guessing the ratings system (and search) was simply a much lower priority than actually making the system.
I mean, let's be fair... they got a LOT done, and a lot done right, for a monumental task that surpasses anything I've seen in any other MMO. It's not the first time an MMO has done user-created content (no matter what the ad copy says), but CoX has done so far better than any previous.
That said, they clearly have a lot of stuff to work on. The obvious and eternal problem with user content is connecting players to content they want. That's not an easy task, by any means.
The 'best' way to handle it is with a crack team of moderators, checking over every single arc with multiple ATs and evaluating it according to a thick book of analytical guidelines.
That clearly is never going to happen (nor should it), so the big challenge is what 'good enough' system to implement.
Personally, while I find the situation annoying, I'm honestly happy with their priorities; they've produced something amazing and cool that's fun to play with, and, hopefully, now they can bolt down the boring techy QoL functionality.
[ QUOTE ]
... Third, it's not griefing to give someone a low rating. There are a lot of reasons why someone might have given your friend's arc a low rating. ...
[/ QUOTE ]
It is griefing to give someone a low rating without even entering the mission. If casting a vote required some kind of time investment, even as small as five minutes, the number of potential griefings would be drastically reduced.
Right now someone could hack the client, the way they seem to have done for sending in-game email spam, and automatically grief every arc in the system.
By requiring that you spend some amount of time in the mission or complete at least the first mission before you can cast a vote, the devs would eliminate most of the opportunities for griefing.
How about missions are rated on the NUMBER of PLAYs. Of couse with the Devs trying to keep content down with their Devs Choice clogging up the list noobs are sure to try those fist. But I would rather like to try something a lot of others are trying. That tells me that it is enjoyed for some reason regargless of ratings.
The biggest problem right now has to be the discoverability of content. It's a shame that the ratings system is one of the only means by which of categorizing content on the MA.
I mean, lets look at Amazon.com (or any similar website) for example. There is a five star rating system for products, however there are also many other ways of discovering content on that site. There are categories which can be browsed, there is a robust search feature, there are user-made "lists", there are recommendations based on similar products. Reviews are a big part of the content on that site, but for example there are one-star reviews on say "The Grapes of Wrath" and it has really no effect on the book's placement on the site.
One of the problems with the MA is that ratings are too important, and they're really the only way of increasing (or decreasing) the visibility of an arc. Honestly, I'd prefer to see some sort of categorization when I open up the MA, rather than a list of Dev's Choice and some inflated 5-star arcs.
[ QUOTE ]
How about missions are rated on the NUMBER of PLAYs. Of couse with the Devs trying to keep content down with their Devs Choice clogging up the list noobs are sure to try those fist. But I would rather like to try something a lot of others are trying. That tells me that it is enjoyed for some reason regargless of ratings.
[/ QUOTE ]
Like, if it's short but gives massive xp/tickets?
My characters at Virtueverse
Faces of the City
My thoughts on what the rating system needs...
You should need to complete the arc to rate it.
This would have numerous benefits:
5 star cartels would actually have to play the missions they're rating, and may very well come to not want to give a 5 star by the end of it.
0 star cartels simply could not do this en masse, and most players who are liable to do this sort of behavior are likely too lazy to go through a full arc.
Insta-death arcs couldn't be finished and thus couldn't be rated. Remaining at 0 stars indefinitely.
Simply being "too difficult" wouldn't lose you stars necessarily, but it may keep you from getting many ratings.
And lets be honest, a rating on an arc you haven't finished is disingenuous no matter how bad the arc may be, or how much you may feel they deserve it. Just like a rating on any other form of media.
Add a count of times completed.
Even by itself this would help, although it would essentially obsolete the star system if the above wasn't done as well.
Regardless this would give another metric to rate arcs, something with a bad star rating but a lot of play throughs still warrants notice.
Add a count of times not completed.
Although not a black mark per say, this would help weed out those arcs that had high ratings, but either didn't deserve them or are possibly very difficult (for those who don't want the challenge, if the author didn't explicitly note this).
Allow anonymous *logged* feedback if you complete the arc.
Basically the only hitch here is giving authors a way to report harassment easily without knowing who is harassing them. Although this would only be for people who completed the arc, you would still allow IDed feedback for anyone. This would reduce harassment issues pretty much to nothing, since you couldn't spam messages without being put on ignore.
That is all 
Infatum on Virtueverse
[ QUOTE ]
[ QUOTE ]
How about missions are rated on the NUMBER of PLAYs. Of couse with the Devs trying to keep content down with their Devs Choice clogging up the list noobs are sure to try those fist. But I would rather like to try something a lot of others are trying. That tells me that it is enjoyed for some reason regargless of ratings.
[/ QUOTE ]
Like, if it's short but gives massive xp/tickets?
[/ QUOTE ]
Its piopular for a reason. Some maybe. But that would seem that if that was popular thats what the players want
But regardles the Devs Choice needs to go. MA is about the PLAYERS
[ QUOTE ]
And lets be honest, a rating on an arc you haven't finished is disingenuous no matter how bad the arc may be, or how much you may feel they deserve it. Just like a rating on any other form of media.
[/ QUOTE ]
Couldn't disagree more.
Honestly. 
The Cape Radio: You're not super until you put on the Cape!
DJ Enigma's Puzzle Factory: Co* Parody Commercials
Same here. Some missions are immediately obvious garbage, and I shouldn't have to play through five missions to rate it accordingly and try to warn off the next guy.
Arc 55669 - Tales of the PPD: One Hell of a Deal (video trailer)
Arc 64511 - The Wrecking Ball
Arc 1745 - The Trouble With Trimbles
Arc 302901 - HappyCorpse
Also, some arcs cannot be completed, due either to bugs or to overly difficult encounters. You can't in fairness tell someone they can't rate an arc they played 99% of only to have a glowy spawn in a wall on the last mission.
And for a while things were cold,
They were scared down in their holes
The forest that once was green
Was colored black by those killing machines
[ QUOTE ]
Same here. Some missions are immediately obvious garbage, and I shouldn't have to play through five missions to rate it accordingly and try to warn off the next guy.
[/ QUOTE ]
However, if you can only rate it if you play through it and the only way to get to hall of fame is to get ratings, then bad/unplayable trash will automatically be penalized simply by not being rated. The good ones will get played and rated (or at least the playable ones will get played) the bad ones will not get completed and not get rated.
Any other method is going to leave us where we are now - with folks being given the ability to grief/pump up an arc without even trying to play it.
Basically what most folks are proposing is that we replace low ratings with NO ratings - which then should be a searchable criteria.
Globals: @Midnight Mystique/@Magik13
Except that leaves "excellent but unadvertised" arcs in the same bin as "terrible crap that 600 people have already played and abandoned".
Under that system, I don't want to be the author of the "excellent but unadvertised" arc, and I don't want to be player #601 of the "terrible crap" arc. We need to do something, but I don't think this suggestion is it.
Arc 55669 - Tales of the PPD: One Hell of a Deal (video trailer)
Arc 64511 - The Wrecking Ball
Arc 1745 - The Trouble With Trimbles
Arc 302901 - HappyCorpse
[ QUOTE ]
[ QUOTE ]
Same here. Some missions are immediately obvious garbage, and I shouldn't have to play through five missions to rate it accordingly and try to warn off the next guy.
[/ QUOTE ]
However, if you can only rate it if you play through it and the only way to get to hall of fame is to get ratings, then bad/unplayable trash will automatically be penalized simply by not being rated. The good ones will get played and rated (or at least the playable ones will get played) the bad ones will not get completed and not get rated.
[/ QUOTE ]
I look forward to every HoF arc gaining a fifth mission (EDIT: well... or a fourth, an extra mission) as soon as they hit the HoF with the biggest outdoors map and filled with thugs/willpower AVs set to extreme. Just to make sure it can't be voted down from there.
"The arc is done, this is just a little something to make sure I don't get griefed. Thanks for playing!"
Players' Choice Awards: Best Dual-Origin Level Range Arc!
It's a new era, the era of the Mission Architect. Can you save the Universe from...
The Invasion of the Bikini-clad Samurai Vampiresses from Outer Space? - Arc ID 61013
Well, this is an interesting post... this like most other rating systems will always have the same problem... it's based on people's opinions and it will always be hard for people to agree on what is good and what is bad for example the movie ironman... alot of people loved it and there are some who thought it sucked... which to those of us who loved it (me)
cannot understand that but it is what it is... if you want people to respect your opinion then you should respect theirs not to like it
With any rating system you will always have to face... Bitter people, people that are jealous that they cannot write as good of an arc as you or people that just hate to see people have success and grieve votes, just to try and mess up their rating... and last but not last my favorate self-anointed saviors of coh/cov who are nothing but [censored] kissers trying to impress the devs, pointing out farm arcs... which is beyond funny to me since this game as farm aspects built in... like flasshback and so fourth... and in case some of you don't understand the concept of farming... if YOU have done a mission like.. an ITF for example with the same toon more than once then guess what.... YOU are farming it... and while some people try and justify why they are doing it... "well I need the influence" or "my base needs the prestige" or whatever it's still farming... it doesn't matter if you farm a mission once or 500 times farming is farming and for all those people who think farming should be removed from the game... they need to get a clue...
For this rating system to work people need to be responsible and not be quick to judge something to be something it might not be, just because you didn't take the time to read what the arc is about... most of the devs content can be deemed as a "farm" but you have to accept it because you can't rate non MA content... what it comes down to is people are going to do what they want and if you become a victim of someone wanting to be a jerk then there is nothing the devs can really do about that.... without taking the control we currently have with MA
It's better to save the Mystery, than surrender to the secret...