When is the new grading list coming out?

General discussions about ratings.
Matthew Turner
Posts: 3600
Joined: Fri May 16, 2008 11:54 am

Re: When is the new grading list coming out?

Post by Matthew Turner » Thu Aug 20, 2009 8:55 am

Many thanks to Howard Grist for such a long response.
His whole reply is based on the theory that the old system needed changing because players graded 10 poins more than their opponents only scored 57% rather than the 'predicted' 60%. I simply do not accept that this is a problem, because
1. The integrity of the grading system is still intact i.e. it ranks players correctly
2. It is a self-correcting 'error'

I am also not convinced that the grading actually predicts that a player who is graded 10 points more than their opponent is predicted to score 60%. Imagine two players have the same strength. It is quite unlikely that they will have the same grade, so one will be higher than the other, hence relatively speaking the higher player would be predicted to underperform. Natural variation means some players will be overgraded (relative to their strength) and some undergraded, the effect of this should mean that a ten point grading advantage should predict a score of something less than 60%. Evidence of a 57% score seems to be evidence the system is working correctly.

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: When is the new grading list coming out?

Post by Roger de Coverly » Thu Aug 20, 2009 9:04 am

Brian Valentine wrote:<The fact that it doesn't would indicate that there is a problem.>

There is a valid question on whether it diverges too much.
We might need to be cautious about the volume of data when looking at the graph. There are presumably many more games included in the data at +/- 10. Presumably it shows that players of the same grade score about 50%. A really inaccurate system would fail this test.

It might have been helpful to show graphs from different years - as to whether the divergence from the linear plot is static or increasing.

I come back to this paradox - If the grades are diverging from the mean, why didn't it show up on the "average grade" tests that have always been conducted. Is it perhaps the case that the theoretical dispersion is countered by the practical in that underperformance by the higher graded player would compress the list over time rather than expand it. So you got two forces pulling in opposite directions and you get a stable end result.

Compression and stability. 160 player plays against 150 players and scores 57%. Next year's grade is 157. The 150 players play mostly amongst themselves and score 50% - result compression.

Stability. 160 player plays 150 players and 170 players scoring 57% and 43%. No change to the grade - stability.

In terms of what a grading system is for, I would suggest that getting the ranking order of players correct has higher priority than its use as a result predictor. Retaining a reasonable degree of comparability over time also has to be fairly high on the desirables list.

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: When is the new grading list coming out?

Post by Roger de Coverly » Thu Aug 20, 2009 9:16 am

Matthew Turner wrote:
I am also not convinced that the grading actually predicts that a player who is graded 10 points more than their opponent is predicted to score 60%. Imagine two players have the same strength. It is quite unlikely that they will have the same grade, so one will be higher than the other, hence relatively speaking the higher player would be predicted to underperform. Natural variation means some players will be overgraded (relative to their strength) and some undergraded, the effect of this should mean that a ten point grading advantage should predict a score of something less than 60%. Evidence of a 57% score seems to be evidence the system is working correctly.
There's a similar statistical argument to this which says that a grade is only accurate to about 8 points over 30 games. In others words you have to be 8 points higher before you can be confident (in the statistical sense) that the higher graded player is better. This may be why the earliest versions of the grading system didn't tell people a number, instead putting them in an 8 point band 2a,2b etc.

Some theory is here

http://www.ecforum.org.uk/viewtopic.php ... 3258#p3258

Matthew Turner
Posts: 3600
Joined: Fri May 16, 2008 11:54 am

Re: When is the new grading list coming out?

Post by Matthew Turner » Thu Aug 20, 2009 9:27 am

Roger,
I appreciate I wasn't making a unique point, but it just seems to undermine the whole of the grading team's assumptions. If we accept that the grading system doesn't in fact predict that a player 10 points higher graded that their opponent will score (on average) 60%, then there isn't (as far as I know) any justification for the changes the grading team have made. That doesn't of course mean that the changes are incorrect.

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: When is the new grading list coming out?

Post by Roger de Coverly » Thu Aug 20, 2009 10:05 am

Matthew Turner wrote:Roger,
the grading team's assumptions.
After about three years, we now seem to have got to the grading team's central assumption - namely the S shaped graph and the interpretation placed on it. You now get further questions - does the shape of the S show any trends from one year to the next. If you go back in time before 2002 and graph events where you have a complete crosstable (the British Championship for example), do you get a similar S shape? My thought is that if the S shape is always present, then it's a feature and the justification for trying to remove it is reduced.

I don't know how far back in time the S was first generated, but the regarding exercise might have been less controversial had it been published every year as part of the annual statistical review.

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: When is the new grading list coming out?

Post by Roger de Coverly » Thu Aug 20, 2009 12:04 pm

Howard Grist wrote:If the people in the both lists includes a junior or two, and juniors are improving (or at least given points in a belief that they are) and the adults are staying the same (fairly central grading system principle), it is hardly surprising that for players in both lists the mean 2009 grade is higher than that in 2008.
I think we can accept that the stayers comparison needs filtering to eliminate the expected effect of the junior increments. However when I ran the equivalent test on 2008 published v 2007 published, the means stayed constant. Thus the "new" grades are "more inflationary" or perhaps "less deflationary" than the old ones.

It's claimed that the published grades have drifted, deflated or spread and thus need to be corrected. You would expect therefore to find evidence of this by comparing published grades from one year to the next. Such evidence is a little elusive. By contrast the assertion on "new" grades, that the top players are distancing themselves from the rest is easily verified from the published 2009 list. Even the "off-form" Mark H gained a point.

Another measure that might or might not give a combined indicator of grade reliability and player consistency would be to work out the (W-We) for each game for each player. The average of the absolute value (ignoring sign) is a metric of sorts of this. For what its worth, it seems to be about 0.1 per player per game in the last three British Championships (using the FIDE ratings). This was 0.15 in one of the recent Uxbridge events. I might speculate that this measure might increase as the grade and the age of the player reduced. Again one of the points is to give something that can be measured from one year to the next.

David Sedgwick
Posts: 5249
Joined: Mon Apr 09, 2007 5:56 pm
Location: Croydon

Re: When is the new grading list coming out?

Post by David Sedgwick » Thu Aug 20, 2009 1:06 pm

Roger de Coverly wrote:
Howard Grist wrote:I suppose the the most important point I have to make is that the method of calculating grades for both juniors and new players will be changing for the end of August list. The exact details of these changes are still being finalised.
This "making it up as you go along" approach is immensely unsatisfactory particularly when you are modifying a system with a reasonable fifty year track record. Why not just leave the system alone for another year and stop meddling until you have a coherent approach?
One day I might post a personal history of the grading fiasco and my (largely behind the scenes) efforts to prevent disaster.

For the moment, suffice it to say that, immediately prior to the April Council Meeting, I urged the Board to postpone the whole exercise for a year if they were not confident about how things were going to work out. To the best of my knowledge the Board did not even discuss the matter at its meeting on the morning of Council. At Council we were given assurances that everything was under control.

It's now too late to scrap things completely. Congress entry forms have been printed with the revised limits. Up and down the country the grading limits for League and other competitions will have been amended; the Counties Championships are a major case in point.

However, there are two things which could now be done quickly:

1. The 2009 "Old Grades" could and should be published. I understand that these have all been calculated and I hope (and pray) that these calculations have been done using the old methodology. Any League or Congress which preferred to use these figures rather then the "New Grades" would then be at liberty to do so.

2. The new methodology for calculating junior grades should be scrapped. Instead 2009 "New Grades" for juniors could be calculated by applying the approximate conversion formulae for adults to their 2009 "Old Grades". The resultant grades would be far from perfect, but they would be a great deal more sensible than those which we have at the moment.

User avatar
David Shepherd
Posts: 912
Joined: Fri Nov 23, 2007 3:46 pm

Re: When is the new grading list coming out?

Post by David Shepherd » Thu Aug 20, 2009 1:31 pm

David Sedgwick wrote:2. The new methodology for calculating junior grades should be scrapped. Instead 2009 "New Grades" for juniors could be calculated by applying the approximate conversion formulae for adults to their 2009 "Old Grades". The resultant grades would be far from perfect, but they would be a great deal more sensible than those which we have at the moment.
I am not sure I agree with this - one problem that I previously observed is that the junior grades are out of line with the adult grades. By applying the adult conversion factor the problem would not be solved. Maybe if the is approach is used it should be adult coversion factor plus a bit.

NB I suspect the higher the junior grade the more in line with an adult grade they are as they will tend to play in more adult competitions. For younger players many of their games are in a pool of junior players and the mechanism does not self correct.

From what I observe the ranking of adult grades tends to be reasonable and the ranking of juniors reasonable but only when the two mix problems are caused.

Matt Harrison
Posts: 129
Joined: Fri Apr 06, 2007 4:51 pm

Re: When is the new grading list coming out?

Post by Matt Harrison » Thu Aug 20, 2009 1:44 pm

On an experiment of one basis - my son's grade is not a million miles away from where it should be. But then he has played a large number of games (56 standard/36 rapidplay) mainly against adults, with any juniors he played being strong ones, he played very few ungraded players and he scored about 63% for the year as a whole.

His resultant grades are in line with each other (Rapidplay a few points higher as it always has been) and both about 10 points higher than expected, but this is entirely the junior increment.

It seems to me that the problems have generally come from situations already discussed (not many games, games against ungraded players, very high/low percentage scores). This to me indicates a programming issue not an method issue.

He did want to keep the old grades as he was going to improve 25 points anyway and this gain would be more respected by his clubmates. As it is he's gone up 41 points (junior bonus, plus a few points for the rerating) and will shoot up the board order for his club.

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: When is the new grading list coming out?

Post by Roger de Coverly » Thu Aug 20, 2009 1:48 pm

David Sedgwick wrote:
2. The new methodology for calculating junior grades should be scrapped. Instead 2009 "New Grades" for juniors could be calculated by applying the approximate conversion formulae for adults to their 2009 "Old Grades". The resultant grades would be far from perfect, but they would be a great deal more sensible than those which we have at the moment.
You may not even have to do that. The old approach was to take last year's performance and add a fudge factor aka junior increment. Were the reasons for dropping this approach convincing? You need a factor for a few reasons, combating deflation in next year's calculations, restricting access to rating limited competitions and prizes, ordering players for seeded swiss draws etc.

If you have players with both ratings lag (grades out of date with their strength) and volatility (no consistency of performance) then almost any "actual - expected" parameter is going to show problems. You may just have to accept this and just try to get it so that adults and juniors on about the same grade seem to be in approximately the right order.

In any event, I don't think it's satisfactory to have a grading system where you cannot measure performances until two months after the end of the season. That's fourteen months for some events. Isn't that a simple reason to reject a "treat all juniors as unrated" proposal?

Ola Winfridsson
Posts: 324
Joined: Fri Aug 14, 2009 12:26 pm

Re: When is the new grading list coming out?

Post by Ola Winfridsson » Thu Aug 20, 2009 2:49 pm

The most accurate solution would of course be to publish junior grades every three or four months (although I know this is unworkable with a centralized system where the adult grades are updated once a year) and scrap any "junior adjustments" to the grades.

User avatar
David Shepherd
Posts: 912
Joined: Fri Nov 23, 2007 3:46 pm

Re: When is the new grading list coming out?

Post by David Shepherd » Fri Aug 21, 2009 12:27 pm

Is a new list coming out - the grading list seems to be down.

User avatar
Carl Hibbard
Posts: 6028
Joined: Fri Dec 08, 2006 8:05 pm
Location: Evesham

Re: When is the new grading list coming out?

Post by Carl Hibbard » Fri Aug 21, 2009 12:42 pm

David Shepherd wrote:Is a new list coming out - the grading list seems to be down.
No and it should not be down, checking now...
Cheers
Carl Hibbard

User avatar
Carl Hibbard
Posts: 6028
Joined: Fri Dec 08, 2006 8:05 pm
Location: Evesham

Re: When is the new grading list coming out?

Post by Carl Hibbard » Fri Aug 21, 2009 1:05 pm

David Shepherd wrote:Is a new list coming out - the grading list seems to be down.
It's a hosting provider issue (and not me....) but is now under investigation!

Sorry :oops:
Cheers
Carl Hibbard

User avatar
Carl Hibbard
Posts: 6028
Joined: Fri Dec 08, 2006 8:05 pm
Location: Evesham

Re: When is the new grading list coming out?

Post by Carl Hibbard » Fri Aug 21, 2009 1:13 pm

David Shepherd wrote:Is a new list coming out - the grading list seems to be down.
It's back now - down for around an hour it seems, not too bad since it's been up for 87 days but could be better of course :(
Cheers
Carl Hibbard