AE Ratings are inexcusably naive

Using the votes of a community to determine ratings and rankings for items submitted over time is not a simple thing. In fact, it is very complicated, and a lot of smart people have spent a lot of time and effort on figuring out how to do it well.

Here's an example of what I'm talking about:
http://blog.linkibol.com/post/How-to...-Proud-of.aspx

My issue here is not that the devs should have used any particular one of these algorithms; different situations demand different approaches, and the AE is not a link aggregator, nor a film rental service, nor a search engine, nor a major online retailer. But what it shares with all of these is that its value is extremely dependent on delivering to the user something they want, and that it does so by analyzing information given to it by other users. The major difference? AE's algorithm is dumb as rocks, and that is why it fails.

My intent with this post is not to chastise the devs for doing the simplest thing that could possibly work, given the time and resources available. My intent is to point out that (1) it does not work, (2) that something else could work better, and (3) that making it work better would be worth the effort. I have arguments to support all three of these, but this post is already reaching the limit of attention span so I will lay them out another time.

I don't understand what's wrong with the rating system. It displays the average rating and the number of votes. I guess I just don't understand why it needs to be more complicated than that.

I'm assuming what's being said, in a slightly roudabout way, is that the ratings system doesn't give enough information. All you see is an average of 1-5 stars, with no information behind it.

This has been discussed before, yes. You don't see comments (so you, looking for an arc, don't see that a group one-starred it for "tu much wurds storys iz dum no xps" as opposed to "It was internally inconsistent, had horrid spelling and grammar - you could take one word from each COH and COV arc, put them together randomly and have an arc that makes more sense." Alternately, seeing a five-star arc with no comments, versus one praising story and pacing, would let you know what to expect (though you can glean a bit of an idea by the description given.)

I hesitate to agree that "something needs to change now," given that it would affect every single arc currently rated. I wouldn't *mind* seeing more detailed ratings, mind you. And the extra detail available to the author to describe the arc ("Solo friendly, canon-related humor") does help somewhat in finding content - I don't tend to go just by stars. That, of course, is dependant on those being *filled out.*

Last note, this sounds more like a rant than a suggestion.

See my Guides and Tutorials

Time to respond and clarify.

M_B, my issue is not with the amount of information being provided about each arc, but goes significantly deeper. A well-built rating system does not make you go hunting for something that will appeal to you - it provides it without prompting. As for this being a rant, it's a fair cop. I was astounded on release of AE that the developers had taken a rating system which they knew had failed to provide any kind of meaningful information on their very own official forum, and ported it almost verbatim as the critical component of their flagship feature for the issue. And lest you think I am exaggerating, yes, I do consider the rating system to be the critical component of a user-generated content feature. If you can't provide fresh content that consistently beats Sturgeon's Law, you've created a place where effort goes to die, and that's unacceptable in a day and age where hard work has already been done to find solutions that prevent this from happening. I'm angry, yes, and it's because I feel the devs should have known better.

By way of evidence that the system as is does not work, I would like to point to the probability that an arc published at this very moment, regardless of how good it is, will ever make the top page of the arc listings for any amount of time. The AE should be connecting players with fresh, high-quality content. It does not. In fact, due to the implementation of HoF/DC, the top arcs on the front page of the AE are guaranteed to be old, likely broken, and not utilizing the latest features of the system. Individual player preferences are not taken into account, and the rewards tied to ratings encourage players and creators to subvert the ratings system.

(Incidentally, I am biased toward a player-centric rather than creator-centric perspective, for three reasons: first, I am myself more a player than a creator; second, players outnumber creators by a significant factor; and third, the developers are naturally biased toward the needs of creators because they are creators more than players - but on the other hand, creators are better served when their work is more likely to be seen, since this is the primary goal of most creators, and this is an area where AE fails creators as well as players.)

Spitting, you set up your case well. I agree that any user-generated content system lives or dies by the quality of its ranking system. But I'm not sure I'm understanding the entirety of your proposal. Is the limit of your suggestion to just add aging? I agree that would be a very important improvement.

If aging was incorporated, I assume it would apply to date of arc publication, not just ratings? If so, you'd want to think through how the system could be gamed by an author constantly doing small tweaks.

I'd also be up for experimenting with novel ways of arc promotion. Set aside some real estate on the side of the search screen for a Google Ads-style purchased promotion space. Let people bid inf to have links to their arcs appear there. Just as with Google, it'd be obvious that the sponsored links were paid for. Currently the only way for an author to promote an arc is in the forums, which most players don't read. You'd probably have to limit it to one ad per global account, to prevent someone like Fulmens from owning all the ad links forever. He'd do it just because it's a new inf sink!

EDIT: it might come across that I'm joking about the benefits of selling ad space. Most of us recoil from that idea, because we associate ads with spam. But when I think of a new author who really cares deeply about an arc he's written and has no way to promote it, no way to get it to stand out from the crap that is cluttering the system..... that would be deeply frustrating. It just seems like allowing people a a way to wave a virtual flag and say "Look over here! Try mine!" is a nice option to offer, as long as it's clearly marked (search engines have proven that people get mad at search results that invisibly add purchased results to the top of the list, hence the need for purchased ads to be clearly marked). For people deeply offended by this, the AE could offer an option (off by default) to disable showing ad links.

Sardan: Thank you. The reason why it's hard to discern the specifics of my proposal is because I don't have a specific method in mind for dealing with these issues. As I think I showed, tuning rating algorithms is demanding work, it's outside my area of expertise, and it's not my job. My point is that it is the devs' job, and that the job is not done. What I'm saying is (1) that there is a goal the AE rating system should be trying to achieve - that goal being to gather the best possible information from user feedback and use that feedback to present a continuously updating stream of good user-created content; (2) that the current rating system does not achieve this goal, and that this is due to its naive design; and (3) that by expending additional development effort on improving the rating system, the devs can make the AE player experience significantly better.

In short, I am pointing out that a problem exists, that it's more complex than the system we have can handle, that others have grappled with this problem before and found better solutions than the one we have, and assuming that the devs are competent enough to find an appropriate solution. That doesn't mean we shouldn't propose solutions - I'm just recusing myself from proposing a definitive solution on the grounds that I'm not competent to do so.

That said, my personal ideal solution would involve some level of tailoring recommendations to each player, but that's an extremely intensive undertaking. Even if we're going to have a single rating value for each arc, though, we can still get a much better heuristic than a simple average of ratings by factoring in additional data such as arc age, arc delta (is this arc very similar to a prior arc from the same creator, i.e. a fix and republish?), creator history (has this creator published good arcs in the past?), and voter history (how does this voter's vote on this arc compare to his voting record?). I would also, personally, remove the incentives and badges tied to high ratings because it encourages dishonest voting - but that's just me coming from a strongly player-centric view, and with the strong suspicion that creators would continue to create missions just because they want to create missions.