Star Rating = System Failure

PW's reviews, imo, are by far the most useful to the author. When I released my latest arc she was the only reviewer I asked to run the arc because I knew that even if I did not like what she said it would be useful and come from her intent on improving the arc. On the other hand I will honestly admit that her rating me 4 stars in game was upsetting and even had me thinking about giving up on writing further arcs. (OK, I admit I can be a drama queen).

Some may question why a 4 star rating from someone I respect so much was so disappointing. The answer is simple. In the current rating system any rating other than a 5 negatively affects the chance of an arc getting played. Though this is clearly not the intention of the rating system, it is a fact. While a 4 slightly hurts the chance of the arc being played, if you get a 3, 2 or 1 early on and you can pretty much write off all the work you did on that arc.

I feel that the result of the 5 star system is that many, many potential authors have their dreams of getting the arc they labor so hard on played crushed out of the gate. I also think that after this happening many just "walk away" from MA.

Honestly, the system as it is now it really only gives two actual choices. 5 stars = “I liked this arc enough to recommend it to others” and 1-4 stars = different weights of “I prefer people do not play this arc”. I would bet that the majority of the people that play MA do not know or, for some reason, don’t believe this is the case. Who can blame them? 4 stars according to the in game text associated with it are considered “Excellent”. Why would anyone think rating an arc as “Excellent” would hurt its chance of being played?

What I am saying here is nothing new, but since we now have a Dev dedicated to the MA system I thought it might be a good time to reopen the discussion.


WN

So 1 person 4-starred your arc and you are ready to quit MA altogether? I may have 5 starred less than 5% of the arcs I've run (I've 1 starred far more than that) as only exemplary arcs deserve 5 stars.

I also use the star system differently than you do, Wrong_Number.

I reserve 5 for ones that knock my socks off for w/e reason. And one for ones that are too horrible to leave unrated. most of my ratings are either - no rating or 2-4. I send a little feedback with every rating (other than no rating) though offering a short explanation of what I liked.

My rule of thumb

3 Stars = radio mission equivalent
4 Stars = Similar to a Dev Arc
5 Stars = hall of fame worthy


There is no hard and fast rules for starring.

My point is that no matter how you use the existing system, anything less than 5 stars means the arc is more likely to be buried 1000's of pages in where it will likely see very few or no plays ever again. It does not matter what any of us think the ratings should mean.

WN

When I look for arcs, I look in the 4 and 5 star section. When I feel lucky I check out 3 stars but an arc that got 2 stars or less have no chance of me playing it. Most of the arcs I enjoy tend to be 4 star arcs.

When I rate I reserve the 5 stars for the arcs that I really like a lot. Good arcs normally end up with 4 stars. I reserve the 3 star for arcs that try but somehow fail, the 2 for non farm bad arcs the 1 is reserved for farms.

What you are all not getting is that knocking an arc from 5 stars basically condemns the arc to never being played again. The irony of this is that we are left with a large percentage of five star arcs that are ticket farming missions (one of them was actually bold enough to announce it in their briefing dialogue). So, as the system stands at the moment, rating an arc less than five stars is basically saying it is worse than a farm.

How I rate arcs is something I developed awhile back based on my concern over this happening. I pretty much rate everything that makes an attempt at telling a story 5 stars.
This is by far the largest percentage of ratings that I hand out. My second largest rating, as a result of the farms becoming so commonplace in the 5 star ratings, is to one star the arc and report it as a farm. The last two ratings I hand out are 1 star, literally for an arc I would never like to see played again or no rating for obvious works in progress. To date, I can still count on two hands the number of story arcs I have given a 1 star to.

People are just going to have to lower their expectations. MA isn't "new" any more and too many people have had too many bad experiences with it. The most common opinion I've heard expressed from the general populace is that MA arcs are overwhelmingly low-quality, full of Mary Sues (and/or catgirls or vampires), etc. Throw in overwrought FUD from disgruntled farmers upset over low rewards just to rub salt in the wound.

Going forward, outstanding arcs are likely to get dozens of plays, not hundreds.

sorry yer feelin down.
in case you care, when i look fer arcs,i use keywords first and stars last.i always rate them on how much i enjoyed it.whether its a farm or not.
i stopped leaving comments when i found out its not anonomous but i dont mind tagging.
i dont have a solution for yer despondence,only posting so u get a cross section of community responses.
peace.

If you compare intent to reality, then yes, the system is a failure. The OP's (and many other authors') response to a 4-star rating is proof of that.

Four stars is considered "excellent." The intent is to inform the author, "You did a good job. I enjoyed this arc." The intent is to inform other players "This arc is above average. Playing it is a good use of your time."

The reality: 500 people rate your arc "Excellent," "Enjoyable," "A good use of your time," and the system considers it worse than every arc by an author with a second account.

A four-star rating should be a reward, not a punishment. The system punishes you for getting one. Fail.

I agree with the OP, and, in fact, wholly disagree with those who have called "bull" on her conclusion: anything less than 5 stars = no plays. I honestly believe they are speaking on opinion rather than actual personal experience. I agree with her because I've seen exactly that happen with my arcs.

My arcs that currently have five stars get an average of a play a day (once they pass a threshold, which seems to be about 20-30 plays). My arcs that currently have four stars get fewer than one play every six months. That's not opinion. That's is, in fact, exactly how it's happened for me, and, I'd suspect most other people.

I've had three arcs go from four star ratings to five star ratings (and yes, it took months for that one play that bumped it up to five stars to even happen). For all three arcs, the instant they hit five stars, they started getting about a play a day.

Two of the arcs maintained that rating. One of them lost it after a few days. The ones that maintained the rating get about a play a day. The one that lost it hasn't gotten a single play since it lost it - that was a few months ago.

I agree completely with Wrong Number. A rating of 5 stars = you are voting that the arc is worth other players taking a look at. A rating of less than 5 stars, EVEN 4 stars = you are voting that the arc is not good enough to ever get played again. And if your vote actually takes it to less than 5 stars, you have *ensured* it will not get played again. Heck, that's why even my scoring system is so strongly weighted toward giving 5 stars in game; there almost always has to be a major technical issue (like XP issues, an ubeatable foe) or the writing has to be TERRIBLE, not just average but actually very bad, before I'm willing to risk voting a five star arc into oblivion. And yes, in the rare cases where I did notice that my vote caused an arc with more than about ten plays to lose five star status (which is vary rare because most of the arcs people suggest that I look at really are good enough that I think other people should play them), I have changed my vote to return it's five star status. I'm not cruel, and I will not be the one responsible for condemning an arc to never be played. Heck, there's been times were the arc was 4 stars, and I ended up giving it 4 stars (again, rare - much of the time I've thought it was worth 5 stars) - and almost every time I've checked just to make sure a 5-star vote wouldn't bump it up to 5 stars. And if it would, I've just gone ahead and given it 5-stars.

It may not be the intent behind the rating system, but, as I've said, I agree completely with WN's assessment of the effect of the rating system.

What I don't know is if the effect of the system is exactly what the Dev's intended - if they wanted to ensure that the system was set up in a way to make sure that very, very, very few arcs actually got regular plays. That's always something you have to consider, because if its true - or at least if the effect of the system is acceptable to them (and, frankly I think it is), then the odds are very good no attempt will ever be made to overhaul the system.

Something I'm wondering is if part of this could be tweaked/averted with different math. Currently the system is using pure averages so it is exceptionally easy to fluctuate a score with just one rating.

What I propose is figuring out the median number of ratings is and base the average off of that number. Foe example, let's assume the median is 100 ratings. Every arc--rated or not--has every rating assigned a 3 star value. These silent 3 stars are replaced with real ratings as people play the arc and averaged in. With everything deviating towards 3 stars, singular ratings won't have as much of a crushing impact. Plus, this is only for the sake of calculating the order of appearance. What will also help is the ability to actually see the exact percentage of ratings an arc has received.

Oh, and the "More" search option should be expanded by default. While many here know the ins and outs of the search engine, most of the average players probably don't even know it exists.

The main point is not for the educated AE player who knows what the star ratings really mean. This is all about the casual player. If you only play one or two AE arcs a month, are you going to choose a 5 star arc, or a 4 star arc? I've come around to this way of thinking, and rate any arc I like as a 5 star.

While I think it is fine to discuss how the arc search, the ratings system, and so on could be changed to help aspiring architects get their work out to a larger audience, those of us with 4-starred arcs may have to bite on the bitter pill that perhaps no changes have been made to AE because it is WAI from where Paragon Studios stands. They simply may not feel motivated to make any changes.

I'm not saying I know WHY the devs would want it this way, but they didn't change it based on user comments back in beta, and it's been around a while with no changes or even any hint of future changes. They probably feel that players can now make their own content for play among friends, and architects can get ticket rewards from the ratings they receive, so all is well and good.

Maybe Dr. Aeon would have something to say about this. I don't know.

Regardless of where the devs stand on it, I agree with the premise of the OP. I have a few four-starred arcs that no one seems to play. Every once in a while I get a tell from someone - "fantastic arc! among the best i've seen in AE" - and then back to weeks of silence and no one playing my arcs.

I think it boils down to this: yes, the design is flawed from the standpoint of some architects and players, but the devs see it as functioning as intended and refuse to be dragged into making changes for a vocal minority.

WN, this is a great topic for discussion as we've already seen that everyone has a different standard of rating stars. In the spirit of making this productive, what solutions are on the table to solve the problem?

I saw Zamuel had a suggestion. Any others?

Get the DC and HoF arcs off the first two pages for starters. Move them to their own tab or something, just stop monopolizing the front page with the same dozen arcs for the last several months. All it does is make MA look stagnant to the more casual players.

If anything, the first page should be a random assortment chosen from the arcs in the system. My idea would be something like: 1 random Guest Author, 1 random HoF, 1 random DC, 4 random 5-star arcs, and 4 random 4-star arcs.

The search options need to be expanded by default. The little More button off to the side is not only easy to miss for your average user but also vaguely worded and it doesn't click for everyone what it means.

"My Level" is WORTHLESS. There needs to be a drop-down list with level ranges just like in player and market search. The current option is not only thwarted by arcs lazily set to 1-54 but also doesn't help anyone if they're looking to do content in a certain range and want to exemplar their high-level character for whatever reason.

The sorting of arcs should not just take ratings average, it should also take the quantity of ratings into account. If you're not going to show us the true ratings and keep with this simplified rounding, then an arc that is 4.3 with 200 ratings should not be listed 20 pages behind arcs that are 4.34 with only 2 ratings.

The Sigma Clipping method mentioned earlier would be a suitable alternative and would also make ratings griefing tougher as well since a single 1-star cannot torpedo your average by canceling out the last several 5-star ratings.

I really would like to be able to see stats on arcs. If something is 4-star average then I'd like to see how many of those actually were 4-star votes. If you can't give this to everyone then at least allow the author to see them for their own arcs.

WARNING: LONG, RAMBLING, FULL OF OPINION!

I myself have received arc reviews that rated some of my story arcs 4 stars, 3 stars, even 2 stars. What you take away from such a review is, of course, up to each individual person. I think all feedback is valuable, though; especially negative feedback.

For example, Venture recently gave one of my arcs 3 stars. In my opinion, it would be useless for me to rail at him and say stuff like, "You're so mean, why didn't you give me 5 stars? No one will play my arc now!" Aside from being nonconstructive, I'm sure he's built up an immunity to this kind of flame by now. Instead, what I prefer to take away from this experience is, "What can I change in my arc to make the next person give me more than 3 stars?"

GlaziusF said, "I've said several times that one individual rating doesn't matter. If you don't have enough ratings to get your arc noticed by the general public, you need more word of mouth and promotion than one vote is able to give you. If you do, one rating isn't going to mean much." I think this is true. But, using this reasoning, GlaziusF eventually gave up and started giving everyone 5 stars for everything (in game, anyway), even for things that he rates 2 stars on the forum. I don't agree with this approach, though.

It's not GlaziusF's one vote that matters. It's the next ten, next twenty votes after that. If you end up pressuring GlaziusF (or Venture, or me) into awarding you 5 stars, there is no guarantee that any of those next ten or twenty strangers will agree that your arc is worthy of being 5 starred.

If my new arc Poodle Skirt Girl against the Invaders from Mars deserves 2 stars because of its many flaws, it's not doing me a big favor to have one or two people 5-star it despite its 2-star-worthiness. The next ten or twelve random people who play Poodle Skirt Girl against the Invaders from Mars will go, "How the heck did this get up to 5 stars?" and probably award me some punitive 1-star or 2-star ratings. The MA Arc Finder superteam even has a game that works like this, "Make or Break", which tries to determine whether 5-star arcs really deserve to have 5 stars! A game that was, as far as I know, invented by this thread's original poster.

It would be better, in my opinion, for me to take GlaziusF's suggestions to try and improve Poodle Skirt Girl to be deserving of a 5-star rating, rather than just demand he give me a 5-star rating and hope the next group of people to play the arc will see the inherent beauty of Poodle Skirt Girl and charitably give me 5 stars too.

The best and most shining example of doing this is, in my opinion, Escalation. In my original review of Escalation I gave it 4 stars. Femfury took my feedback, along with the feedback of many other forum reviewers, and used it to improve the story arc. Now I can't praise Escalation highly enough. Escalation deservingly became a Dev Choice arc in November. In my opinion, this was due to the tireless efforts of its author to make it a better story arc.

It's anecdotal, I'll admit. But I think it's an example of how I think the feedback loop should work.