GRADING ANOMALIES

[Neill Cooper]

Thanks for the comments on my graph. I should have descrbed my conclusions in more detail. I produced the graph as people had been discussing this data as to whether it was cyclical or not. I think the graph clearly shows a downward trend.

To reply to your points:
The graph is of the mean grade of all 10,000+ players and so the range should not be 0 to 260. But how steep or otherwise the graph is off less importance than that it shows a clear trend - with a fall of about 4 grading points in 10 years in the mean grade.

What it does not do is give a reason for the fall. That would need extra analyses. Possible causes: more (lowly graded) juniors being graded and so the average grade falling, some older players getting weaker, there being less strong overseas players in the list or there being systematic grade deflation. Sean's straw poll of players indicates that it might be grade deflation.

I attach a histogram of all the 2008 grades which shows a normal distribution apart from a high tail (overseas players, graded over 220?) and the cut-off at zero.

[Roger de Coverly]

with a fall of about 4 grading points in 10 years in the mean grade.

A trend which if you really insist on correcting, you could add back 4 points possibly at the rate of 1 point a year.

I believe Tim's point to be that you shouldn't correct a small drift with a 20 point inflation of the mean grade.

What does your distribution plot look like on the "new" grades?

We aren't seeing the whole picture in the published material. We've got 2008 grade and 2007 grade so we've got the change up or down for every player in both lists. We've got 2008 "new" grade but not 2007 "new" grade. So we cannot see for individual players whether the "new" grade change is up or down. My guess is that some players who were "down" on the current list are "up" on the "new" grades.

The grading team imply that they have calculations which show players overperforming against players rated above them and players underperforming against players rated below them. Presumably this isn't true for every player for every grading range - so some summaries of the results would be valuable. You would also imagine some of the statistical tests for significance had also been done. The allegation is that you are "more" likely to play more active players rated above you - again this cannot possibly be true for every player.

A "cook" on the "new" grades to make them go up is that the junior age increments have been changed. So there are two changes to the grades, one being the rebase and the other being the changed junior increments. Have the team run a sensitivity test with the old grades and the new increments and the rebased grades with the old increments?

[Peter Turner]

I'm fairly new to the forum and enjoying most of the debate, comments, explanations, diagrams etc about the 'New Grades'. A common theme is the 'under grading' of juniors, particularly those entering the system. As a former BCF/ECF Director of Junior Chess & Education and a 25 year involvement in encouraging youngsters to learn and to play chess I'd like to make some observations. When my own son started playing I soon realised that it would take at least 1 to 2 years to get an official grade and then later that as a junior his grade always significantly lagged his actual playing strength. Having an official grade is almost a 'rite of passage' to being accepted as a real chess player. Yet the system delays the moment when that occurs, I have always considered that, when a youngster joins the ECF, they should be awarded a temporary grade based on age. Almost a benefit of membership!! I assume that, with all the analysis taking place at the moment, it would be easy to assign a grade. This could be done on 6 month increments, average grade at 6 years, 6 1/2 years etc. The juniors would soon gravitate towards their actual strenghth but would have brought the expected grading points into the system. I believe the greater benefit would be to the youngster who would immediately feel accepted into the chess world. After all if they were to join a football team they would get all the 'gear' straight away and feel like a proper footballer rather than stand around for a year feeling like a second class citizen. This may be a load of rubbish but please be 'gentle' with your responses, I'm probably having a senior moment.

[Neill Cooper]

Histogram for new grades has a mean of 135 and standard deviation of 35.

Note added later: With the present grades over 4000 players are graded below 100. With the new grades there are less than 2000, most of whom are graded below 70 at present.

[Tim Spanton]

I'm fairly new to the forum and enjoying most of the debate, comments, explanations, diagrams etc about the 'New Grades'. A common theme is the 'under grading' of juniors, particularly those entering the system. As a former BCF/ECF Director of Junior Chess & Education and a 25 year involvement in encouraging youngsters to learn and to play chess I'd like to make some observations. When my own son started playing I soon realised that it would take at least 1 to 2 years to get an official grade and then later that as a junior his grade always significantly lagged his actual playing strength. Having an official grade is almost a 'rite of passage' to being accepted as a real chess player. Yet the system delays the moment when that occurs, I have always considered that, when a youngster joins the ECF, they should be awarded a temporary grade based on age. Almost a benefit of membership!! I assume that, with all the analysis taking place at the moment, it would be easy to assign a grade. This could be done on 6 month increments, average grade at 6 years, 6 1/2 years etc. The juniors would soon gravitate towards their actual strenghth but would have brought the expected grading points into the system. I believe the greater benefit would be to the youngster who would immediately feel accepted into the chess world. After all if they were to join a football team they would get all the 'gear' straight away and feel like a proper footballer rather than stand around for a year feeling like a second class citizen. This may be a load of rubbish but please be 'gentle' with your responses, I'm probably having a senior moment.

Sounds sensible to me - which probably means there's a flaw

[Tim Spanton]

To reply to your points:
The graph is of the mean grade of all 10,000+ players and so the range should not be 0 to 260. But how steep or otherwise the graph is off less importance than that it shows a clear trend - with a fall of about 4 grading points in 10 years in the mean grade.

Sorry, but the range should be 0 to 260 (or even wider), because that is the range of possible grades. By making a range of seven points you hugely magnify the downward slope. It's a basic deceptive technique described in stats primers, eg How To Lie With Statistics (not that I'm suggesting you did it deliberately).

[John Upham]

Sorry, but the range should be 0 to 260 (or even wider)

Can the significance (if any at all) of a rating of zero compared with -10 or +10 be explained?

AFAIK, zero is no more significant in denoting playing strength than a higher or lower value. Values less than zero are not published for fear of causing angst to those with such a rating.

For example, if you evaluated the playing strength of someone who did not know how to move the pieces (or say an inanimate object ) what would it be?

-50, 0 +10 or what?

It would seem that the ECF scale is rather more akin to the Celcius scale than the Kevin scale and that Elo is more akin to the Fahrenheit scale.

Would something with an absolute zero have some appeal?

It make add some sense to the rating scale if a baseline was defined...

[E Michael White]

I know you didnt mean this type of appeal but the opponent of a player with a grade of absolute zero would find it more difficult to make a 10.2 claim, as any random move could be considered as the zero player attempting to win by their normal means.

[Angus French]

Looking at Neill's graphs, what strikes me is that "current" grades appear to be normally distributed but "new" grades don't. The first chart approximates a bell curve but the second is sharper and draws (to my eyes, anyway) something more like a pyramid (albeit with tapers at the ends of the range). Am I right? Is this as it should be?

[Sean Hewitt]

Looking at Neill's graphs, what strikes me is that "current" grades appear to be normally distributed but "new" grades don't. The first chart approximates a bell curve but the second is sharper and draws (to my eyes, anyway) something more like a pyramid (albeit with tapers at the ends of the range). Am I right? Is this as it should be?

No, they are both normal distributions. They appear to have different standard deviations which, in laymans terms, changes how squashed the bell curve looks.

[Sean Hewitt]

To reply to your points:
The graph is of the mean grade of all 10,000+ players and so the range should not be 0 to 260. But how steep or otherwise the graph is off less importance than that it shows a clear trend - with a fall of about 4 grading points in 10 years in the mean grade.

Sorry, but the range should be 0 to 260 (or even wider), because that is the range of possible grades.

Wrong. Neill's graph shows the mean average grade (not possible grades) so the correct range is the minimum and maximum mean average grade (not possible grade). A mean grade of 0 or 260 is completely impossible and using such a scale would make no sense whatsoever - rather like you Tim.

[Sean Hewitt]

with a fall of about 4 grading points in 10 years in the mean grade.

A trend which if you really insist on correcting, you could add back 4 points possibly at the rate of 1 point a year.

I believe Tim's point to be that you shouldn't correct a small drift with a 20 point inflation of the mean grade.

You're correct, you could add back 4 lost points (or whatever the correction is) at the rate of 1 per year. But there are two important things to consider

Firstly, there is no evidence that the problem only started in 1998. That date is simply the earliest date that someone has posted on this forum the known mean average grade. The deflation problem goes back far further than that (best guess, early - mid 1970's) and is, as is well known, far larger than 4 points.

Secondly, once you know how many points deflation your grading system has suffered (and as I say, it is far more than 4 points), why one earth would you delay rectifying the problem? Doing it over 4 (or more) years means that you would be deliberately operating an inaccurate grading system for a considerable period of time. There would be no benefit to such tardiness.

[Roger de Coverly]

Firstly, there is no evidence that the problem only started in 1998. That date is simply the earliest date that someone has posted on this forum the known mean average grade. The deflation problem goes back far further than that (best guess, early - mid 1970's) and is, as is well known, far larger than 4 points.

The grading team should be able to go back to 1993 because the data is on the on-line server. They might also wish to note the relationship between consecutive lists as

Current year list = last year list + joiners - leavers.

You could work out the averages separately for
stayers defined as (current year list - joiners)
and joiners.

On the 2008 list, this comes out as a mean grade of 113.4 for stayers and 99.4 for joiners. The stayers had a mean of 113.2 on their 2007 grades so that's up 0.2. Standard deviations were 41.8 (42.7 on 2007 grades) and 56.9 for joiners. Combined, the mean is 111.4 with a standard deviation of 44.6.

Joiners range all the way from the zero rated up to international players playing their first Gibraltar , Liverpool, Isle of Man or Hastings. Standard deviation reduced slightly for established players and the mean went up very slightly.This is the opposite direction to the deflation and dispersion hypothesis.

One would need the 2007 list to calculate anything for leavers - but the small change in the mean from 2007 to 2008 suggests the profile of leavers as similar to joiners

The notion that deflation (if defined in the economics sense) goes back to the mid-seventies is in my view an unjustified assertion. I'm happy to accept that dispersion goes back that far because the grades of top players have increased and more players of low playing strength have been brought in at the base. There's a fair number of players around who have grades in the same range as in the seventies. Other players have grades which have dropped. It's my contention that to maintain a place in the top x%, you need to do more chess work in preparing for opponents and openings than used to be needed. I believe the grading system expects that players of the same grade should score 50% against one another. If some players improve and others don't, then the improved players will do better than 50% against the non-improvers. The non-improvers grades will fall without their absolute chess strength having worsened.

why one earth would you delay rectifying the problem?

For the very simple reason that you don't have to make any dramatic changes to well-established competitions. For example in the SCCU, there's a competition for juniors under the age of 14 and under the grade of 90. If no changes are made to the rules of this competition and the grading system changes go through as proposed, many of the potential entrants will not be eligible.


It's obvious isn't it that a grading system will have a number of inflationary and deflationary factors acting on it. These include but are probably not limited to

junior increments (deflationary if too small, inflationary if too large)
rules for estimating new players (deflationary if too low, inflationary if too high or given an explicit minimum)
results for more active players having less effect on the grade than the less active ( actually a loss costs the same for a player who plays 10 games a season as for one who plays 30, provided there are 2 previous seasons to average over)
40 point rule on improving players ( deflationary when they would have got a higher grade if they were new players)
40 point rule for elite players (inflationary because they get a grade + 10 score without having to work for it)
rapidly improving (or worsening) players
rules for inclusion of non-domestic results and players

I believe up to about 3 years ago, a general consensus would have been that the factors acting on the BCF grading system were broadly balanced over time. This could be evidenced by players having grades in the same range for long periods of time and observation of the mean and standard deviation of the published grades. Writing a review in July 2001, Bruce Holland comments on the statistical harvest that had been available since 1988. He also comments on the need to track the averages to monitor and control inflation. Nowhere does he even speculate that there had been 30 years of deflation.

[Tim Spanton]

Have found an old post of Howard Grist showing the avg in 1996 was 113.5.

So now we have:
1996 114
1999 116
2000 115
2001 114
2002 115
2003 114
2004 113
2005 114
2006 113
2007 112
2008 111

[Neill Cooper]

Sorry, but the range should be 0 to 260 (or even wider), because that is the range of possible grades.

I disagree. The graph is to show the data on it - which are mean grades. Whatever y-axes I used the slope of the linear regression line would still be 0.4 grading points a year. The significance of such a graph is not the visual steepness of the slope but the numerical slope. And the statistical significance is given by how close the points are to the linear regression line, not how steep the slope is.