July 2014 gradings out

General discussions about ratings.
John Higgs
Posts: 385
Joined: Sun Mar 03, 2013 7:54 am

Re: July 2014 gradings out

Post by John Higgs » Mon Jul 28, 2014 10:08 pm

Thank you to the grading team for an excellent job.

Looking forward already to January 2015

8)
"I'm not the one who got it wrong. I'm the only one who got it right". Carrie Mathison.

Richard Haddrell

Re: July 2014 gradings out

Post by Richard Haddrell » Tue Jul 29, 2014 12:34 am

Richard Haddrell wrote:
stevencarr wrote:Almost everybody I know has had their grading go down. It is just a small sample, but have a lot of gradings gone down?
About the same as have gone up, at a guess. Mean grades, Jan 2014: Standard 132, Rapid 103. July 2014: Standard 133, Rapid 104. Last year's July figures were the same as this year's. I hope to publish more detail shortly on the SCCU website.
I have. See http://www.sccu.ndo.co.uk/grad.htm. It's selective, and no weighted averages I'm afraid.

MartinCarpenter
Posts: 3041
Joined: Tue May 24, 2011 10:58 am

Re: July 2014 gradings out

Post by MartinCarpenter » Wed Jul 30, 2014 10:41 am

Apologies since this might be rather tricky to do :) Is there is any way to see how much of an impact the strict 30 game count back has had on the standard deviations of grades?

Especially for people playing 20-40 games a season perhaps. I've a vague impression that is has stretched the 'normal error' for players that active from maybe +-5 to +-10. Hardly tragic if so of course.

Its obviously very easy to think of individual situations where it can have quite a dramatic effect but hard to tell how real it is in general.
(My grade might have stayed stable this season, but my 4 ten game chunks had a ~40 point grade spread between them so it'd have been very easy to have it varying rather a lot instead!).

Kevin Thurlow
Posts: 5802
Joined: Wed Apr 30, 2008 12:28 pm

Re: July 2014 gradings out

Post by Kevin Thurlow » Wed Jul 30, 2014 12:23 pm

"Especially for people playing 20-40 games a season perhaps. I've a vague impression that is has stretched the 'normal error' for players that active from maybe +-5 to +-10. Hardly tragic if so of course."

Is it really that low?

MartinCarpenter
Posts: 3041
Joined: Tue May 24, 2011 10:58 am

Re: July 2014 gradings out

Post by MartinCarpenter » Wed Jul 30, 2014 12:59 pm

Dunno, hence wondering :) I did mean 'normal' rather than extreme and obviously people who've stabilised so no juniors etc.

Don't think it can be much higher than +-10 though - +17 got you onto the most improved list (top 120) this season and that list is nearly all juniors.

Mike Gunn
Posts: 1014
Joined: Wed Apr 11, 2007 4:45 pm

Re: July 2014 gradings out

Post by Mike Gunn » Wed Jul 30, 2014 6:44 pm

Kevin Thurlow wrote:"Especially for people playing 20-40 games a season perhaps. I've a vague impression that is has stretched the 'normal error' for players that active from maybe +-5 to +-10. Hardly tragic if so of course."

Is it really that low?
A useful rule of thumb is that what Elo calls the "standard error" (S) is given by 24 divided by the square root of N (N = the number of games used in calculating the grade (G)). The standard error is defined so that the chance of a calculated grade actually lying in the interval G-S to G+S is 50%. (Obviously S is related to the standard deviation of the distribution.)

So the standard error for grades calculated on the last 30 games is about 4.4.

Richard Haddrell

Re: July 2014 gradings out

Post by Richard Haddrell » Wed Jul 30, 2014 10:39 pm

I don’t know about standard deviations, but there’s an aspect of the 30-game countback I have reservations about. It places too much weight on the exact dates of games around the cutoff point. Suppose my 30th and 31st games, counting backwards, are a good win and a horrible loss. They may have been played on consecutive days. But which of them happened to come first makes a difference of 3 points or more, one way or the other, to my grade. Can this be right?

This leads to the problem of wrong dates. The other week someone asked me to correct the date of a league match, which had been wrongly reported as played one month earlier than it really was. So I corrected it, and it promoted his game in this match to the top 30, supplanting another game which was slightly worse. His grade went up nearly half a point, causing it to round up rather than down. He knew it would, of course. He made no bones about the fact that it was the reason for his request. Not everyone would go to this length for a point on his grade, but he had every right. The match date was wrong. I daresay someone else in the match went down a point for equal and opposite reasons, but only one date was right.

That was one match. Fixing one is doable, but how many others are wrongly dated? I don’t know. I know people take dates from the fixture list sometimes, oblivious of rearrangements. Or mistype them, or miscopy them from illegible originals. Someone has just sent a congress played entirely in 2004. That one was easy because the computer rejected it, but others don’t leap out at you. There must be lots, and in principle they should be corrected when they come to light. It is perhaps fortunate not many do.

Then there’s the club championships where every game is dated 1st September. Correcting those would be impossible, but they will certainly be distorting grades.

We don’t have to use a rule that requires minute accuracy on dates. I would prefer a return to the old rule: use 30 games, but when you get to the earliest period you don’t take the most recent games in it. You take as many as you need, but notional games at the average score for the period. This fixes all those wrong-date problems at a stroke. I thought the current rule was an improvement when it was introduced, but I’ve changed my mind.

My opinion’s not official, but I hope the ECF wouldn’t reject it out of hand. Anyway, I wanted to say it. I’ll yield to standard deviations now.

Richard Haddrell
Grading Administrator, ECF

Roger de Coverly
Posts: 21291
Joined: Tue Apr 15, 2008 2:51 pm

Re: July 2014 gradings out

Post by Roger de Coverly » Wed Jul 30, 2014 11:54 pm

Richard Haddrell wrote: I thought the current rule was an improvement when it was introduced, but I’ve changed my mind.
It was introduced when rapid-play went six-monthly. Perhaps because very few people are bothered about rapid-play grades, there was no great controversy. But wasn't it a proposal of the "Howell" system when that was purely on paper? It got semi-implemented and then almost abandoned, but I do recall the July 2000 grading list being seriously bugged when a flag was set to only process half a year.

A debate about the theory of incorporating past period results for less active players is long overdue. It should have taken place back in around 1998 when elements of the "Howell" system were being implemented. If the ECF were prepared to publish "unofficial" performance grades alongside those used in calculation, then last 30 games is a sensible starting point, provided there's reasonable confidence in the dates recorded.

Mike Gunn
Posts: 1014
Joined: Wed Apr 11, 2007 4:45 pm

Re: July 2014 gradings out

Post by Mike Gunn » Thu Jul 31, 2014 9:22 am

I think Richard's suggestion would be an improvement on the existing arrangement and I support it.

The point about the standard error or deviation is this: it determines the accuracy of the calculated grade (following Elo's assumption that a player's strength plots like a normal distribution). If you calculate a grade based on 30 games the standard error is 4.4 but if you use 60 games the standard error is 3.1. It really makes sense to use as many games as possible when caculating grades (what's wrong with the last year if more than 30 games have been played?)

User avatar
Michael Farthing
Posts: 2069
Joined: Fri Apr 04, 2014 1:28 pm
Location: Morecambe, Europe

Re: July 2014 gradings out

Post by Michael Farthing » Thu Jul 31, 2014 9:31 am

Richard Haddrell wrote: I would prefer a return to the old rule: use 30 games, but when you get to the earliest period you don’t take the most recent games in it. You take as many as you need, but notional games at the average score for the period. This fixes all those wrong-date problems at a stroke. I thought the current rule was an improvement when it was introduced, but I’ve changed my mind.

My opinion’s not official, but I hope the ECF wouldn’t reject it out of hand. Anyway, I wanted to say it. I’ll yield to standard deviations now.

Richard Haddrell
Grading Administrator, ECF
Last night I wrote a long post setting out my arguments for this and using my own grading history as an example. I mentioned that my own example was illustrati9ve and not a grievance and expressed explicit support for you. I came to post it and discovered I had been logged oiff and the post disappeared into the ether. I had not the heart to re do it, but clearly a fairy godmather took pity on me, whispered it in your ear, and you have kindly replied.

Your solution is exactly what I was asking for. A point about the current system that you do not mention is the dramatic difference between these two scenarios:

A Jun 2013 Player Grade 130: Last game played Game: 30 June 2013
Jan 2014 Player Grade 140: 29 games
All new grading based on player grade 130

B Jan 2014 Player Grade 140: 30 games
All new grading based on player grade 140

This aspect of the system, where the player's base grade in calculations is based on the grade ruling at the time of the earliest game I find the most bizarre situation of all. Its main effect is, of course, on the player's opponents. However, it can also have a big effect on the player him/herself. For instance, our club is small with a wide range of grades. We make a conscious effort to be inclusive and to encourage games between people of widely differing standard. Consequently I play many games against opponents over 40 below me and similarly in the other direction. In both cases the base grade for calculating my next grade can be significantly affected by the situation described above (in January 2014 my estimate was that it cost me 6-7 points). Not a complaint or request for redress: but comment invited.

MartinCarpenter
Posts: 3041
Joined: Tue May 24, 2011 10:58 am

Re: July 2014 gradings out

Post by MartinCarpenter » Thu Jul 31, 2014 9:49 am

Its Richard's thing that worries me too really.

Take someone playing 20 games a season. Under the previous system they'd get those games + 10 games at the average of the twenty games played the previous season.

What they get now is those 20 games + 10 specific games by date countback. That obviously introduces more variability.

Its gets worse with people playing say 40 games, because then you're using 20 + 10 for each list, rather than either 40 or 20 + average from 40 scaled down to 10.

The other thing is that sets of 10 games tend to be quite clustered in time so the results achieved are not independent of each other! You can very easily get a very biased sample in either direction. One congress and a few games either side say.

iirc My 4 sets of 10 games making up my 40 games last season were something like: 185,160,170,200. I don't think this is untypical, but its obviously something that could need checking. That gives my January grade a potential +-10 pt variability based on which set of 10 games fell on the end of the season.

Not terrible, but not really desirable and no obvious motivation for it either.

Paul Dargan
Posts: 526
Joined: Sun May 13, 2007 11:23 pm

Re: July 2014 gradings out

Post by Paul Dargan » Thu Jul 31, 2014 11:35 am

Just to say I agree. Not just because I seem to start grading periods well and end them badly (LCC weekenders in December main culprit).

It cant be right that some dates are spot on and others (e.g. club champs) are all over the place - either we have a strict countback or we average the previous 6 months.

It also cant really be right that whether you won on the Sunday morning and lost in the afternoon of the local weekend swiss or vice-versa moves your grade by ~3 points.


Regards,

Paul Dargan

Richard Bates
Posts: 3338
Joined: Fri Nov 14, 2008 8:27 pm

Re: July 2014 gradings out

Post by Richard Bates » Thu Jul 31, 2014 6:54 pm

I strongly agree with using a weighted average, rather than simple countback, and for a separate (albeit related) reason from issues about dodgy dates etc.

The current system strongly encourages people to "game"/manipulate their grade if they have particularly extreme results around the cut-off point. I have heard several people say that they won't play near the end of a period if they have had very good results set to drop out of the system - i don't think the grading system should discourage people from playing. This is much less likely to happen under a weighted average system since unless somebody has been playing particularly well or badly over the six month period (provided they were reasonably active) they will rarely be in a situation where playing additional games leaves them either with nothing to gain (defending very good results) or nothing to lose (vice versa).

More generally i think the six monthly lists should be on a rolling yearly basis but that's a separate debate.

Brian Valentine
Posts: 574
Joined: Fri Apr 03, 2009 1:30 pm

Re: July 2014 gradings out

Post by Brian Valentine » Thu Jul 31, 2014 9:48 pm

The grading team will rightly consider Richard Haddrell's suggestion and also the idea of the rolling one year. Neither of these ideas is without merit, but I would like the team to anticipate the unintended consequences before making a change. I don't intend to answer every post made on this site, but I will try to log all the constructive points made for our deliberations.

Brian
Manager ECF Grading
Last edited by Brian Valentine on Fri Aug 01, 2014 10:15 am, edited 1 time in total.

Graham Borrowdale

Re: July 2014 gradings out

Post by Graham Borrowdale » Thu Jul 31, 2014 11:14 pm

I have to agree that a weighted average for the additional games seems to make more sense than the count back method.
I think my grade is a typical example, with 25 games in 13/14 plus 5 from 12/13, with the 5 old games at a much higher performance than the 25 newer ones. The consequence is that I need to start 14/15 with 5 games at least as good as the 5 from 12/13 just for my grade to stand still. This is unlikely to happen, so I must come to the conclusion that my latest published grade is too high. It will correct, of course, but only after inflating the grades of my next opponents. The impact will, I imagine, be a yo-yo effect.
Thanks to Brian and the team for taking these thoughts on board.

Post Reply