So, how do you feel about the rating system?
By
Alari_Azure,
Posted on: 2009-05-10 18:28 in Mission Architect
[ QUOTE ]
Well, if someone leaves a gibberish as a rating comment, it says a lot. Gibberish with one star rating means that rating was posted by an impatient person not playing for story, but for purple epic gear he or she can't find in coh. And probably asks people where are the raids here coming straight from grinder centric mmo's.
As for removing the rating system entirely...What would that achieve? As it stands now, it's next to useless (other than dev approval seal that does seem to warrant some quality). Removing rating won't change a thing, since again, no one would have any idea what stories are good and what are not good. I'll try to give it some thought and put a post up later if I come with any viable ideas.
P.S.
As for compulsory comments while rating: the assumption I made here is that rating an arc is *not* mandatory. And only when you do want to rate it you need to put couple words about stuff. So, there shouldn't be a problem with people spamming comment section just to close the rating window.
[/ QUOTE ]
My problem is linking stars to comments. If you rate you sure as hell should not have to comment. Not unless all comments are made completely 100% anonymous.
EDIT: and I think multiple people prior to your post have explained why the star system is useless for both finding good story arcs and for the whole hall of fame/badges/opening a mission slot. So again simply remove it.
I've seen only one good idea in this entire thread that can't be gamed and would make the star rating system relevant.
Blazara Aura LVL 50 Fire/Psi Dom (with 125% recharge)
Flameboxer Aura LVL 50 SS/Fire Brute
Ice 'Em Aura LVL 50 Ice Tank
Darq Widow Fortune LVL 50 Fortunata (200% rech/Night Widow 192.5% rech)--thanks issue 19!
[ QUOTE ]
[ QUOTE ]
The ratings system is fundamentally and irreparably flawed. The problem is not the abusers. The problem is the built-in incentives for abuse. Any solution that doesn't involve removing those incentives is not a solution, as the abusers will always find a way around whatever obstacle you implement
[/ QUOTE ]
Very well said. With me, this raises the question, "What is the incentive that is the most common reason for abuse?"
If it's the prestige of being listed on the first page, I think the fair solution is to make current default behavior create a first page at random amongst all the arcs. This way, there is absolutely no point to zero-star something as it won't make your arc list that much higher.
If it's for the badges, then change the badge requirements. However, I don't see how this would be causing so much of the proclaimed griefing.
If it's cause their a jerk, then there is nothing that can be done. Except, ban them after they bomb too many arcs and remove all the ratings they have given.
For me, the whole idea of bombing people's arcs is ludicrious. Not that it isn't done. But, that I cannot see where the enjoyment comes from. Then again, some people enjoy urinating in other's cheerios.
[/ QUOTE ]
As long as people can sort by ratings, there will always be an incentive to downrank some arcs to push others up, because it's 10 times easier to push something down.
But what's the alternative?
If you do something like "total positive votes" you get a "them as has, gets" scenario.
Proportion of total positive votes? Not much different from the current system, except that new arcs can get dogpiled and canned faster than established ones.
Disable rating sort entirely and just show a random selection? Possible. Possible.
Up with the overworld! Up with exploration! | Want a review of your arc?
My arcs: Dream Paper (ID: 1874) | Bricked Electronics (ID: 2180) | The Bravuran Jobs (ID: 5073) | Backwards Day (ID: 329000) | Operation Fair Trade (ID: 391172)
[ QUOTE ]
Very well said. With me, this raises the question, "What is the incentive that is the most common reason for abuse?"
[/ QUOTE ]
You pretty much summed it up. Rewards (tickets and badges), arc visibility, and the possibility of ruining somebody's day (whether as a retaliatory strike or just random griefing). Those are the incentives.
As to what to do about it? Well, had it been up to me, there wouldn't have been a rating system at all (at least not one players could see... it'd be there for dev purposes only). The first time I heard there was going to be a ratings system attached to MA, I cringed and predicted most of the problems we're now experiencing. It just seemed like a really bad idea to me... "bonehead" level bad.
I mean, even if the devs had somehow came up with a system that avoided most abuses, what would that've really accomplished for us? All it would've given us is a "mainstream" of arcs, right?
I don't know about you, but my tastes veer out of the mainstream quite frequently. I very often don't agree with the majority view on what's "good" and what isn't. I think most people are like that. Only the degree varies. Taste is highly subjective.
For this reason, I would've focused heavily on search functionality. On top of morality (which I may have had more options for), I would've had authors be able to set things like genre, level range, solo/team/both, game-focus or story-focus, and challenge level... basically all the stuff we see the community trying to accommodate right now on its own.
I might've even tried implementing some sort of recommendation system. Y'know, like a 'if you enjoyed that arc, you might enjoy...' type of thing. At the very least, players would've been able to search by author with but a click.
But that's me. My only condition now is, if we must have a ratings system, then it should passably accomplish what it's supposed to accomplish. The current one doesn't do that. It's completely meaningless.
Or, at the very least, we should have far more robust search functionality, so that we can just ignore the ratings completely, and let it be the weird little PVP thing it is.
In which case, I like your idea of the front page being randomly selected arcs.
The Cape Radio: You're not super until you put on the Cape!
DJ Enigma's Puzzle Factory: Co* Parody Commercials
[ QUOTE ]
[ QUOTE ]
Very well said. With me, this raises the question, "What is the incentive that is the most common reason for abuse?"
[/ QUOTE ]
In which case, I like your idea of the front page being randomly selected arcs.
[/ QUOTE ]
I like it too.
Blazara Aura LVL 50 Fire/Psi Dom (with 125% recharge)
Flameboxer Aura LVL 50 SS/Fire Brute
Ice 'Em Aura LVL 50 Ice Tank
Darq Widow Fortune LVL 50 Fortunata (200% rech/Night Widow 192.5% rech)--thanks issue 19!
[ QUOTE ]
Very well said. With me, this raises the question, "What is the incentive that is the most common reason for abuse?"[ QUOTE ]
In which case, I like your idea of the front page being randomly selected arcs.
[ QUOTE ]
I like it too.
[/ QUOTE ]
[/ QUOTE ]
[/ QUOTE ]
Thanks.
Perhaps my biggest complaint about the rating system, and MA in general, is visibility. The fact that the default search behavior favors the more popular arcs has long disturbed me; as we can already refine our search to target these. If people cannot get the arc they have spent hours making played, then people will generally tend to stop making arcs. This constant discussion about ratings griefing eliminates any argument against that.
I view the MA as a way to do something really cool with the game that you like, which you have been playing for some time; with someone else (or more) who obviously enjoys the same game. Being able to have a shot at getting your arc on the precious page one really enhances that chance (seriously, who decides to start on page 20+ all the time?). With the obvious effect of groupthink (feel free to argue) in play on these ratings, it is unfair to keep something on the front page which can easily be seen by what would instantly become one of the most common search methods used; the star rating method (the current indicator of quality, the most desirable feature currently being used by anyone who is searching for an arc).
The unfortunate flip side to this issue is that the very popular arcs have the very wonderful advantage of being able to show that this'll be something which you can finish; thanks to the number of people who have played it. A person who plays the MA needs to have an easy way to find a mission they know that they'll be able to complete. This might be the absolute biggest failing of the MA. Random visibility comes with the necessity of being able to create a page of arcs which the person knows that they'll be able to do. Being able to have a button which a person can push saying that they don't want to see anything which contains AVs can go a long ways towards that. Custom mobs should be another. Another preference would be one that adjusts with how the difficulty of the mob is set. Give people a choice to make it PG, and we're set. People can now easily use what is now the most popular feature of the game, and they can get what they want.
[ QUOTE ]
Perhaps my biggest complaint about the rating system, and MA in general, is visibility. The fact that the default search behavior favors the more popular arcs has long disturbed me; as we can already refine our search to target these.
[/ QUOTE ]
But, see, even this isn't true. You're not getting "the more popular arcs" on the front page right now. Not by a long shot.
Example:
Arc 1 has 15 ratings, all 5 stars. This isn't difficult to accomplish if the author happens to be in a moderately active SG.
Arc 2 has 100 ratings. Around half of those are 5 stars. The other half consists mostly of 4 to 2 star ratings, with a couple of 1s and 0s in there... which is just going to happen eventually if an arc gets played enough.
Obviously, Arc 2 is, by leaps and bounds, the more popular arc of the two. It has considerably more plays, and a lot more people 5-starred it.
But Arc 1 is the one that'll be at the top of the list, not Arc 2.
From reading the forums, I'm seeing that a lot of people are having success by searching for 4-star arcs with particular tags. That's how they're finding "the good stuff".
So, a 4-star rating has, in effect, become more of an indicator of quality than a 5-star rating.
I think that's laughably messed up.
The Cape Radio: You're not super until you put on the Cape!
DJ Enigma's Puzzle Factory: Co* Parody Commercials
[ QUOTE ]
[ QUOTE ]
Very well said. With me, this raises the question, "What is the incentive that is the most common reason for abuse?"
[/ QUOTE ]
You pretty much summed it up. Rewards (tickets and badges), arc visibility, and the possibility of ruining somebody's day (whether as a retaliatory strike or just random griefing). Those are the incentives.
As to what to do about it? Well, had it been up to me, there wouldn't have been a rating system at all (at least not one players could see... it'd be there for dev purposes only). The first time I heard there was going to be a ratings system attached to MA, I cringed and predicted most of the problems we're now experiencing. It just seemed like a really bad idea to me... "bonehead" level bad.
I mean, even if the devs had somehow came up with a system that avoided most abuses, what would that've really accomplished for us? All it would've given us is a "mainstream" of arcs, right?
I don't know about you, but my tastes veer out of the mainstream quite frequently. I very often don't agree with the majority view on what's "good" and what isn't. I think most people are like that. Only the degree varies. Taste is highly subjective.
For this reason, I would've focused heavily on search functionality. On top of morality (which I may have had more options for), I would've had authors be able to set things like genre, level range, solo/team/both, game-focus or story-focus, and challenge level... basically all the stuff we see the community trying to accommodate right now on its own.
I might've even tried implementing some sort of recommendation system. Y'know, like a 'if you enjoyed that arc, you might enjoy...' type of thing. At the very least, players would've been able to search by author with but a click.
But that's me. My only condition now is, if we must have a ratings system, then it should passably accomplish what it's supposed to accomplish. The current one doesn't do that. It's completely meaningless.
Or, at the very least, we should have far more robust search functionality, so that we can just ignore the ratings completely, and let it be the weird little PVP thing it is.
In which case, I like your idea of the front page being randomly selected arcs.
[/ QUOTE ]
Solid Post.
[ QUOTE ]
Quite frequently. I very often don't agree with the majority view on what's "good" and what isn't. I think most people are like that. Only the degree varies. Taste is highly subjective.
[/ QUOTE ]
Nobody goes there anymore, it's too crowded.
Seriously, guy, the majority is the majority because it's what most people agree with. If it wasn't what most people agree with it wouldn't be called the majority!
But the rating system isn't a reflection of majority opinion, there's a definite selection bias and a deliberate confounding factor.
Up with the overworld! Up with exploration! | Want a review of your arc?
My arcs: Dream Paper (ID: 1874) | Bricked Electronics (ID: 2180) | The Bravuran Jobs (ID: 5073) | Backwards Day (ID: 329000) | Operation Fair Trade (ID: 391172)
[ QUOTE ]
Seriously, guy, the majority is the majority because it's what most people agree with. If it wasn't what most people agree with it wouldn't be called the majority!
[/ QUOTE ]
Seriously, missy, even people generally in the majority will have differing tastes. Just because most people like X, doesn't mean they'll all like Y. Some will, some would rather dig on Z. 
A ratings system doesn't cater to specific tastes. And even people who're in the majority will have those. That's what I was saying. Everyone has preferences outside of the mainstream. Some more so than others. Hence "varying degrees".
[ QUOTE ]
But the rating system isn't a reflection of majority opinion
[/ QUOTE ]
Didn't say it was. I'm saying, even if it was, it'd only reflect a mainstream... the arcs that were most generally accessible. That's not necessarily an indicator of quality. Especially if you happen to be looking for a specific kind of experience, be it comedy, horror, extreme game challenge, or whatever else.
The Cape Radio: You're not super until you put on the Cape!
DJ Enigma's Puzzle Factory: Co* Parody Commercials
[ QUOTE ]
But the rating system isn't a reflection of majority opinion
[/ QUOTE ]
Didn't say it was. I'm saying, even if it was, it'd only reflect a mainstream... the arcs that were most generally accessible. That's not necessarily an indicator of quality. Especially if you happen to be looking for a specific kind of experience, be it comedy, horror, extreme game challenge, or whatever else.
[/ QUOTE ]
I agree 100%
[ QUOTE ]
[ QUOTE ]
Very well said. With me, this raises the question, "What is the incentive that is the most common reason for abuse?"[ QUOTE ]
In which case, I like your idea of the front page being randomly selected arcs.
[ QUOTE ]
I like it too.
[/ QUOTE ]
[/ QUOTE ]
[/ QUOTE ]
Thanks.
Perhaps my biggest complaint about the rating system, and MA in general, is visibility. The fact that the default search behavior favors the more popular arcs has long disturbed me; as we can already refine our search to target these. If people cannot get the arc they have spent hours making played, then people will generally tend to stop making arcs. This constant discussion about ratings griefing eliminates any argument against that.
I view the MA as a way to do something really cool with the game that you like, which you have been playing for some time; with someone else (or more) who obviously enjoys the same game. Being able to have a shot at getting your arc on the precious page one really enhances that chance (seriously, who decides to start on page 20+ all the time?). With the obvious effect of groupthink (feel free to argue) in play on these ratings, it is unfair to keep something on the front page which can easily be seen by what would instantly become one of the most common search methods used; the star rating method (the current indicator of quality, the most desirable feature currently being used by anyone who is searching for an arc).
The unfortunate flip side to this issue is that the very popular arcs have the very wonderful advantage of being able to show that this'll be something which you can finish; thanks to the number of people who have played it. A person who plays the MA needs to have an easy way to find a mission they know that they'll be able to complete. This might be the absolute biggest failing of the MA. Random visibility comes with the necessity of being able to create a page of arcs which the person knows that they'll be able to do. Being able to have a button which a person can push saying that they don't want to see anything which contains AVs can go a long ways towards that. Custom mobs should be another. Another preference would be one that adjusts with how the difficulty of the mob is set. Give people a choice to make it PG, and we're set. People can now easily use what is now the most popular feature of the game, and they can get what they want.
[/ QUOTE ]
These are pretty damn good too!
Blazara Aura LVL 50 Fire/Psi Dom (with 125% recharge)
Flameboxer Aura LVL 50 SS/Fire Brute
Ice 'Em Aura LVL 50 Ice Tank
Darq Widow Fortune LVL 50 Fortunata (200% rech/Night Widow 192.5% rech)--thanks issue 19!
[ QUOTE ]
[ QUOTE ]
But the rating system isn't a reflection of majority opinion
[/ QUOTE ]
Didn't say it was. I'm saying, even if it was, it'd only reflect a mainstream... the arcs that were most generally accessible. That's not necessarily an indicator of quality. Especially if you happen to be looking for a specific kind of experience, be it comedy, horror, extreme game challenge, or whatever else.
[/ QUOTE ]
I agree 100%
[/ QUOTE ]
Blazara Aura LVL 50 Fire/Psi Dom (with 125% recharge)
Flameboxer Aura LVL 50 SS/Fire Brute
Ice 'Em Aura LVL 50 Ice Tank
Darq Widow Fortune LVL 50 Fortunata (200% rech/Night Widow 192.5% rech)--thanks issue 19!
[ QUOTE ]
I've seen only one good idea in this entire thread that can't be gamed and would make the star rating system relevant.
[/ QUOTE ]
The star system could be made a little more relevant, but it would need an entire overhaul to do it. Just increasing the number of stars used as a base could help immensely. But, there still needs to be more to rating an arc than just assigning stars. If people are willing to put their playing time into running an arc, they shouldn't complain about taking a few more minutes to give more in depth feedback for establishing the basis of that rating. And, it doesn't have to be done with names attached. It could be kept anonymous.
I still think there should be a series of multiple choice criteria set for ratings that could be searched. Maybe make it so that if you weren't willing to do all that to rate an arc, then you wouldn't be able to give a rating at all, or it would only be counted as a partial rating. If you wanted to skip all the detail, then give it a 10 star rating would be reduced perhaps to 5 stars. And, if you only wanted to give it 1 or 0, then that would be another 5. That would at least give the people more incentive to provide the rest of the feedback. If they want to give their friends an automatic 10 stars, they could only give it a median number without the rest of the info.
This whole situation just makes me want to say this to the devs, "It is broken. Can you make it go?" 
No AV/EBs Deal with The Devil's Pawn-207266 Slash DeMento and the Stolen Weapons-100045 Meet the Demon Spawn-151099 Feedback
[ QUOTE ]
[ QUOTE ]
Perhaps my biggest complaint about the rating system, and MA in general, is visibility. The fact that the default search behavior favors the more popular arcs has long disturbed me; as we can already refine our search to target these.
[/ QUOTE ]
But, see, even this isn't true. You're not getting "the more popular arcs" on the front page right now. Not by a long shot.
Example:
Arc 1 has 15 ratings, all 5 stars. This isn't difficult to accomplish if the author happens to be in a moderately active SG.
Arc 2 has 100 ratings. Around half of those are 5 stars. The other half consists mostly of 4 to 2 star ratings, with a couple of 1s and 0s in there... which is just going to happen eventually if an arc gets played enough.
Obviously, Arc 2 is, by leaps and bounds, the more popular arc of the two. It has considerably more plays, and a lot more people 5-starred it.
But Arc 1 is the one that'll be at the top of the list, not Arc 2.
From reading the forums, I'm seeing that a lot of people are having success by searching for 4-star arcs with particular tags. That's how they're finding "the good stuff".
So, a 4-star rating has, in effect, become more of an indicator of quality than a 5-star rating.
I think that's laughably messed up.
[/ QUOTE ]
THIS.
THIS. A HUNDRED TIMES OVER, THIS.
The rating system doesnt favor popular arcs at all.
The fact that even a single onestar can send an arc careening into 3 or 4 stardom means that the vast majority of the time, the only arcs you will ever see on page one will have less than 30 plays total because nobody has gotten around to downrating them yet.
Even if an arc with 300-900 plays gets back to page 1, they quickly get sent flying back to 4star due to non-5 ratings the publicity will garner them.
Anyone who thinks this system favors the popular arcs is absolutely uninformed.
Want comedy and lighthearted action? Between levels 1-14? Try Nuclear in 90 - The Fusionette Task Force!
Arc ID 58363!
[ QUOTE ]
Anyone who thinks this system favors the popular arcs is absolutely uninformed.
[/ QUOTE ]
The Browser favors Arcs according to its' own bizarre logic. It actually seems like an odd combination of Star-rating/Number-of-votes/Arc-ID, but what is guaranteed is that Developer Choice and Hall of Fame arcs, proudly appear on the first page, and I have yet to find one of those that I have enjoyed as much as I have other more obscure ones. 
I believe that a Kheldian Gold Standard should be based on SO's, and for anything above that... there's Platinum!
Save Ms. Liberty (#5349) � Augmenting Peacebringers � The Umbra Illuminati
Save Ms. Liberty (#5349) � Augmenting Peacebringers � The Umbra Illuminati
[ QUOTE ]
Im wondering how people feel about the fairness of the rating system for the arcs?
Do you think that people who rate your arcs (a 0-5), should have their global name attached so you know who gave you what?
Or is it fair the way it is right now?
The reason I ask this is, a friend of mine who has a seemingly well liked arc (Kept a consistent 4 stars through about 40 plays or so), got a "0" rating while he was online.
He knew this because he saw that it had been rated and went to claim tickets, and there were none. (now granted something could have happened, but there was no feedback left regardless).
Now that got us into a debate, that I thought might be a great topic for here.
So do you feel its fair that when a player stars your arc, but does not leave feedback, you don't know who it was?
Or is it fair like it is now?
I personally see both sides of the story, and would like to see what others thoughts are on the rating system as it stands right now.
Discuss....
[/ QUOTE ]
Rating system is a waste of time.
[ QUOTE ]
but what is guaranteed is that Developer Choice and Hall of Fame arcs, proudly appear on the first page, and I have yet to find one of those that I have enjoyed as much as I have other more obscure ones.
[/ QUOTE ]
Amen
Under the current ratings system, it's very likely that the highest quality maps are all 4-star and nothing will ever stay in the Hall of Fame.
The 5-star maps are simply those that have been inflated by a handful of friends, and haven't been downrated yet. They typically have just a few ratings. Also, there are a few 5-star maps that are actually unplayable, and cannot be rated down for that reason.
I'd be really surprised if this was the devs intention of how things should look in the MA.