Feedback should be mandatory if you rate an arc!
By
3dent,
Posted on: 2009-04-19 16:53 in Mission Architect
[ QUOTE ]
One of my arcs was rated five stars after seven ratings, most of which I knew to be actual five star ratings because the ratings were from people I knew and who tested it for me, people who I trust greatly.
Two more ratings and it went down to four stars. I may be wrong but I am figuring, from the mathematics involved, that the two people who rated it next either gave one star and two stars, or both gave it one star.
[/ QUOTE ]
Once more the maths:
From what we've been told the HoF criteria use the mean average rating rounded up, so I am assuming that all in-browser ratings are mean average rounded up.
To have a 5-star rating in the browser after 7 ratings means that you need a mean average above 4, so you need a total of greater than of equal to 29, as 29=(4*7)+1 so 29/7 > 4.
But let's factor in that you said most of the initial 7 ratings were 5s, and assume that they actually were (and your friends weren't being kind and telling white lies and that they didn't rate an initial 5 to avoid hurting your feelings before going back and rerating it lower). Most of 7 is 4.. so lets assume 4 ratings of 5, that's 20 in total, leaving a total of 9 stars to get between the remaining 3 of 7... That's very possible, especially if your friends know you and either share similar tastes and know similar references (possibly more important if humour or references to other ingame characters are used).
To have a 4-star rating in the browser after 9 ratings you need to have a total between 28 and 36 inclusive, i.e. between 28=(3*9)+1 and 36=4*9.
So if you were only at a total of 29 (for your 5 stars after 7 ratings) then the 8th and 9th ratings could be as high as a 3 and a 4 and you would still drop to a 4-star mean average rounded up.
[ QUOTE ]
Thoughtlessness or simple one-star griefing?
[/ QUOTE ]
To maintain a mean average rounded up of 5 stars after 9 ratings you would need a total of at least 37 stars - i.e. (9*4)+1.
If the 8th and 9th votes had been 1-star (or lower) griefing then you would have had to have at least 35 points from the previous 7 votes. Possible... but it means that every single rating before the 8th gave full marks.
And if your arc is truly that good, then I still have faith that it will bear out in the long term.
BTW - they might take time, but I'd hope that sooner or later the devs do start spotting any true, recurring grief-voters and then their votes can probably be removed from the system. You'll notice that the system keeps track of past ratings that you've given to arcs.
[ QUOTE ]
Anyway, without some kind of feedback it is a little dispiriting for people who are putting a lot of effort into creating arcs they think people will enjoy and that will add, in some small way, to the game's content.
[/ QUOTE ]
True - but possibly nothing that can be done until the comments between EU and US are resolved. If the arc was picked at random, or because of it's position in a search, then I'd think that it's most likely that the player was on the US servers. I think this purely because they outnumber EU players so much that anyone randomly stumbling across an arc is more likely to be on the US servers.
And if that is the case they may even have left you comments but you never received them.
BTW - On the subject of ratings:
* A 2-star is described as 'Mediocre' on the arc info screen.
* A 3-star rating is 'Good'.
* A 4-star is 'Excellent'.
I've no idea what a 5-star is (I've placed some 5 -star votes, but for some reason it's not shown me them on the browser window).
If you treat a 2-star as mediocre/average and not as poor just because it's below the mean of the numbers 1 to 5 (and below the mean of the numbers 0 to 5) then 2 star ratings become somewhat more understandable. I wouldn't be surprised if these dev-provided descriptions become more common as people start noticing them more...
By my mohawk shall ye know me!
my toons
Funny: Ee-Ai-Ee-Ai-Oh! #3662 * The foul-mouthed Handyman! #1076 * City of Norms #132944
Serious: To Save A Single World (#83744) * Marketing Opportunity (#83747)
[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
Two more ratings and it went down to four stars. I may be wrong but I am figuring, from the mathematics involved, that the two people who rated it next either gave one star and two stars, or both gave it one star.
[/ QUOTE ]
Five votes of five, two votes of three, gives an average vote of (5 * 5 + 2 * 3 ) / 7 ~= 4.43, which rounds down to a rating of four stars.
[/ QUOTE ]
Hmmm. Perhaps I have misunderstood the system. I was under the impression that all ratings were rounded up. I will happily stand corrected, however.
[/ QUOTE ]
As far as I am aware we have never been told anything about how the ratings are aggregated except for the Ex Libris post stating that to make it into the HoF arcs needed a rating of an average of 5 rounded up after at least 1000 votes.
Some people at the time showed that they misunderstood what 'rounding up' is and some people still post about needing 4.5 or rounding to the nearest integer.
But, AFAIK, (presumably mean) average rounded up is all we have ever been told by rednames.
BTW - the devs have never clarified which average they mean. I guess that they probably mean the mean average as:
- it tends to be what most people refer to when saying 'average';
- mode and median really wouldn't seem to make much sense in this use!
By my mohawk shall ye know me!
my toons
Funny: Ee-Ai-Ee-Ai-Oh! #3662 * The foul-mouthed Handyman! #1076 * City of Norms #132944
Serious: To Save A Single World (#83744) * Marketing Opportunity (#83747)
Why, it's easy to test, all we need is to get 4 people, one with unused MA slot. They set up a "guinea pig" arc, and rate it certain way, say, 5,4,4 should be 5 stars if it's rounding up and 4 if it's down or nearest int. if it's 4 then we remove the arc, republiish it and rate it 5,5,4 it would be 4 stars if rounding down and 5 if nearest.
Of course, it would work only if rating is based on arithmetic mean, but that's a rather good bet.
Thanks for the lengthy response Dave, and I take the points you are making. I think the maths is actually fairly simple but I appreciate your effort in that post to lay it out clearly for everyone.
Anyway, I do not claim to be CoH's version of Ernest Hemingway so best for me not to continue discussing ratings for my own arc. There is often a tendency for one to regard one's own work too highly, something we all need to be careful of.
*Edited due to Dave's further response above.
[ QUOTE ]
Why, it's easy to test, all we need is to get 4 people, one with unused MA slot. They set up a "guinea pig" arc, and rate it certain way, say, 5,4,4 should be 5 stars if it's rounding up and 4 if it's down or nearest int. if it's 4 then we remove the arc, republiish it and rate it 5,5,4 it would be 4 stars if rounding down and 5 if nearest.
[/ QUOTE ]
Should be no need to republish - get one account to rerate so they switch between 4 and 5.
By my mohawk shall ye know me!
my toons
Funny: Ee-Ai-Ee-Ai-Oh! #3662 * The foul-mouthed Handyman! #1076 * City of Norms #132944
Serious: To Save A Single World (#83744) * Marketing Opportunity (#83747)
[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
[ QUOTE ]
Two more ratings and it went down to four stars. I may be wrong but I am figuring, from the mathematics involved, that the two people who rated it next either gave one star and two stars, or both gave it one star.
[/ QUOTE ]
Five votes of five, two votes of three, gives an average vote of (5 * 5 + 2 * 3 ) / 7 ~= 4.43, which rounds down to a rating of four stars.
[/ QUOTE ]
Hmmm. Perhaps I have misunderstood the system. I was under the impression that all ratings were rounded up. I will happily stand corrected, however.
[/ QUOTE ]
As far as I am aware we have never been told anything about how the ratings are aggregated except for the Ex Libris post stating that to make it into the HoF arcs needed a rating of an average of 5 rounded up after at least 1000 votes.
Some people at the time showed that they misunderstood what 'rounding up' is and some people still post about needing 4.5 or rounding to the nearest integer.
But, AFAIK, (presumably mean) average rounded up is all we have ever been told by rednames.
BTW - the devs have never clarified which average they mean. I guess that they probably mean the mean average as:
- it tends to be what most people refer to when saying 'average';
- mode and median really wouldn't seem to make much sense in this use!
[/ QUOTE ]
Indeed. Actually, one of the reasons I have always been under the impression that the scores were rounded up (such that a mean of 4.1 would merit a 5-star score on the MA display) is that it would alleviate some of the difficulty of any arc maintaining a 5-star score, which I think is very challenging indeed if rounding to the nearest.
That said, perhaps a better system overall would be to round up but allow half-star scores on the display (although maybe not when it comes to giving a rating after playing an arc). So an arc with a mean of 4.45 would show on the display as four and a half stars, an arc with a mean of 3.7 would show as four stars and an arc with a mean of 4.6 would show as five stars.
[ QUOTE ]
Should be no need to republish - get one account to rerate so they switch between 4 and 5.
[/ QUOTE ]
There is. If rating doesn't change, how else would you know if it wasn't changed because of rating algorithm or bbecause of lag/some bug related to re-voting? (Making a 3rd step with everybody rating it 1 won't work, maybe it was just voting from 4 to 5 or voting by that player or phase of the Moon. With a freshly published arc you could be a bit less paranoid.)
spent a week creating an arc and published it. silence.. ok, I admit I was disappointed, I sort of hoped for *some* feedback.
Logged in the other day , no messages waiting for me for feedback 
checked the MA to work on a second story and I notice that the arc has been played, and rated as a 1 star.
no feedback. I have no idea if someone really hated it or if its just a troll (or a farmer who found out that it isnt actually a farm arc 
)
thing is: to me the rating system is:
1 star: zero effort, no dialogue, almost zero story
2 star: looks like a random enemy group on a random map with a flimsy premise
3 star: average. basic intro, basic story, basic mobs
4 star: good idea. some effort but needs a few tweaks (difficulty, storyline etc)
5 star: nail on the head. its ready to play, its fun, its well put together. good pacing, good text and the player is kept informed.
so its a bit demoralising to receive 1 star and not have any idea what (if anything - and I'm not egotistical enough to believe that there is nothing) was wrong...
If this is a whole US feedback not being received in EU, they really need to fix it. make the comments attach to the mission and not sent to the player. Put a button in "my published arcs" to "view comments", or better yet, put a button in the main search link to view comments so you can decide to play it or not (a 5 star rating with comments of "k3wl" "1-50 in 1 mish. I 4r 1337"! would show me that even though its got 5 starsr, its not my type of mission).
[ QUOTE ]
Why, it's easy to test, all we need is to get 4 people, one with unused MA slot. They set up a "guinea pig" arc, and rate it certain way, say, 5,4,4 should be 5 stars if it's rounding up and 4 if it's down or nearest int. if it's 4 then we remove the arc, republiish it and rate it 5,5,4 it would be 4 stars if rounding down and 5 if nearest.
Of course, it would work only if rating is based on arithmetic mean, but that's a rather good bet.
[/ QUOTE ]
I can now state that: if the ratings badges are working correctly then the 'average rating' shown when browsing must be rounding up.
The last badge for ratings is for 2500 rating stars given to a single arc you published.
Today, The foul-mouthed Handyman! (arc id: 1076) finally hit went from 998 ratings to at least 1000 (I kept an eye on new ratings for an extra one past when I noticed that the count showed 999+).
So I know an arc has just hit 1000+ ratings, but I have not yet received the badge for an arc receiving 2500 rating stars so the total number of stars for this arc is <2500. This all means that the mean average star rating is <2.5 as 2.5 > (less than 2500)/(1000 or more).
But since not long after I14 went live, this arc has consistently shown as a 3 star rating on the browser screen. Therefore a rounded mean average of <2.5 is being shown as 3. Therefore the mean average must be rounded up, as nearest integer and rounding down would show the arc to be a 2-star.
By my mohawk shall ye know me!
my toons
Funny: Ee-Ai-Ee-Ai-Oh! #3662 * The foul-mouthed Handyman! #1076 * City of Norms #132944
Serious: To Save A Single World (#83744) * Marketing Opportunity (#83747)
Dave - your mind-blowing pimping posts aside, could you give a champing-at-the-bit would-be designer any tips on how to get people to try your arcs out? My beloved "Cancelled in Three" is still languishing at 45 plays
Cancelled in Three - MA ID #34608. Please play and comment! Top 10 placing in Projectionist's MA contest!
That wasn't actually a pimping post. More just showing the evidence that it does appear to be mean average rounded up that is used for the browser ratings.
I'll have a scan at when your arc was posted/what rating it has after I've finished this arc I'm playing (had to do 3rd mission twice as an object didn't spawn 1st time, and then dc'd 3 times halfway through last level of a final kill all ).
I trust that you've:
* posted your arc in City of Guides (ah yes - so you have).
* announced it in the announce your arc thread.
* asked for reviews in the applicable threads.
* added it to your sig (which you have).
I think that a couple of things really helped Handyman that most arcs won't have:
* It was a 'very short' arc, meaning that anyone who was looking for something short and quick could try it.
* It was in very early (though it still took 3 weeks to hit 1000 ratings - that's very fast IMO but still not overnight).
* Hopefully it's funny - so some may tell their friends about it.
* It's by me - a superstar in my own deranged mind 
Of these, I actually consider the first (the 'very short' description) to be the most important/influential on how many plays it has had. And the last to be a detracting factor to many EU forumites!
Now Handyman has done phenomenally well IMO - but it's an oddity.
Ee-Ai-Ee-Ai-Oh! is only just approaching 40 ratings and my other 2 serious arcs are hitting about 7 ratings each. If Handyman is doing so well and people liked it I'd have thought that another funny arc by the same author that's only 3 missions and is rating at 4 stars would get more attention... but no.
I'd actually say that Handyman is an odd-case and that 45 plays for your arc is doing well!!
Main thing is to be patient and consider comments and the number will rise...
EDIT: And Ee-Ai-Ee-Ai-Oh! has been doing better the past week or so when, if I get a comment on Handyman from an EU player I reply saying thanks, glad that they enjoyed it and if they have more time then they might enjoy Ee-Ai-Ee-Ai-Oh! If you have multiple arcs that have a similar feel/style it could be worth doing similar - though only if the arcs are similar enough in feel. I never try to push someone who enjoyed Handyman or EIEIO onto one of my serious arcs... well - I haven't so far anyway...
By my mohawk shall ye know me!
my toons
Funny: Ee-Ai-Ee-Ai-Oh! #3662 * The foul-mouthed Handyman! #1076 * City of Norms #132944
Serious: To Save A Single World (#83744) * Marketing Opportunity (#83747)
Ah, I dunno if my gentle ribbing is coming across the right way, but thanks for your answer, Dave - lots to ponder on there
Cancelled in Three - MA ID #34608. Please play and comment! Top 10 placing in Projectionist's MA contest!
I can give a one word review to most of the farms I've rated, but that would mean putting more work into the review than the author did to the mission.
Please fight My Brute: http://2hero.mybrute.com
Mission Architect 54161 - Michael Mundano, Megan Malloney and the Secret Senate.
Mission Architect 91838 - Constantinople Jones' Family Secret. A One Mission Story arc.
[ QUOTE ]
[ QUOTE ]
Two more ratings and it went down to four stars. I may be wrong but I am figuring, from the mathematics involved, that the two people who rated it next either gave one star and two stars, or both gave it one star.
[/ QUOTE ]
Five votes of five, two votes of three, gives an average vote of (5 * 5 + 2 * 3 ) / 7 ~= 4.43, which rounds down to a rating of four stars.
[/ QUOTE ]
Hmmm. Perhaps I have misunderstood the system. I was under the impression that all ratings were rounded up. I will happily stand corrected, however.