GRADING ANOMALIES

[Neill Cooper]

Using the 1996 value (to 1 dp, and also a few other values I can find to that accuracy) I get the next plot. The red line is the trend line before the 1996 point is added, blue line after. The slope is reduced to only -0.265 grading points a year. More significantly the line is a much worse fit (correlation is now only -0.70 rather than -0.92) so one can be less certain of the trend being real.

Another way of stating this is to say that the 1996 point does not fit the rest of the data.

[Tim Spanton]

Another way of stating this is to say that the 1996 point does not fit the rest of the data.

Which could be regarded as surprising since it is almost the median figure, and if rounded to the nearest whole point, is the modal average

[Paul Stimpson]

This thread seems to have focused on the mean as an explanation for the existence or not of deflation, however like Sean said in an earlier post the mean cannot be used for this purpose. The shift in mean can only be a function of joiners and leavers to the system.

The real problem we need to look at is the problem with the performance prediction of the grading data, I would like to see Sean show more examples of that and try to understand why this model is not working.

The comparison of ECF to FIDE is a major Red Herring because as Roger pointed out FIDE have lowered the threshold over the years and only the best results counted so fixes to make this calculation work across all grades should be discounted.

[Sean Hewitt]

You're right (of course) Paul to point out that mean changes are just a function of leavers and joiners.

I'd love to be able to give more examples of the deflation in the system, but I'm not on the ECF grading team so I dont actually have access to data - historical or current. I was provided with data two years ago when I was asked to investigate the initial problem, but that data has long since been deleted (it was a huge file!).

It's all a bit irrelevant now though anyway, as the ECF decided last April to implement these new grades and like it or not they will become official in July.

[Sean Hewitt]

I just read with interest Roger de Coverley's recent post on the SCCU forum. For those who havent seen it, here it is

From Roger de Coverly
14.10.08
Richard,
Are there any plans to digitise old copies of the SCCU bulletin? [No. - rjh] To my mind, the central grading team might have jumped to fewer conclusions had they read past copies. For example from the March 1991 bulletin: "The BCF MB had rejected a proposal to deduct 4 points from everyone's grade to counter inflation."
Also this edition confirms that the age related junior increments were due to start "next season" - which would have been 1991-2. These replaced the plus 10 rule previously applying.
In the November 1990 issue, there is a long article describing debates with Roger Edwards and others about inflation. This article suggests that the mean grade had been increasing from 121 to 128 presumably from 1987 to 1990. This was blamed on the 10 point rule for juniors. It was also noted that the SCCU in particular didn't grade under 10 tournaments and it didn't include players with fewer than 5 games on the national list. There is also a suggestion that SCCU grades were about 10 to 15 points higher than the rest of the country.
Roger

Now, if Roger's SCCU bulletins are to be believed (and there is no reason for them not to be) the mean average of the grading system was between 121 and 128 in the years 1987-1990.

Of course, everyone should be aware by now that the mean average doesnt prove or disprove deflation. But the fact that the mean average has declined from 128 in around 1987 to 111 in 2008 should prevent the likes of Tim Spanton from claiming that the "constant mean of 120 since as long as I can discover shows that deflation never existed" in future.

Interestingly, on the same SCCU forum there is an old post of mine, dating back to 2006.

I examined the 1996 and 2006 grading lists, interested only in players who are in both lists so as to compare true like with like. I naturally excluded players who were juniors in 1996 on the premise that they would be expected to improve. This left me with 4815 players, so a pretty decent sample size. I looked at how much their grade had gone up (or down) in that 10 year period, and then banded the results as follows.

1996 grade Change in grade
200-217 4.9
190-199 1.77
180-189 -0.89
170-179 -1.8
160-169 -3.76
150-159 -2.46
140-149 -2.4
130-139 -3.74
120-129 -4.66
110-119 -5.29
100-109 -4.28
90-99 -8.43
80-89 -9.01
70-79 -12.19
60-69 -14.48
50-59 -14.87
0-49 -22.21

Note the trend - the lower the grade, the more they declined. Its almost linear!



If the theory that computers and better prepared strong players is to be believed, we should see an equal and opposite increase in strength somewhere (presumably amongst stronger players). It doesnt exist. Equally the theory doesnt explain why the decrease in playing strength increases the lower the grade - given that computers are available to everyone. In fact, I would have expected a bit of computer training / prep to perhaps add 10 points to a 50 graded player as there should be greater scope for improvement, but maybe only 0.1 to a 200 strength player. These results are the exact opposite of that.

[Roger de Coverly]

In the November 1990 issue, there is a long article describing debates with Roger Edwards and others about inflation. This article suggests that the mean grade had been increasing from 121 to 128 presumably from 1987 to 1990

.

It's amazing that the same information can be interpreted in two completely different ways. The point of my comment was that the BCF was concerned about inflation repeat inflation in 1990 and whilst deciding to reject the notion of subtracting 4 points from every grade, it did however decide to reduce the age related increment for juniors from a flat 10 to an age-related scale with a maximum of 10. This may have worked because the mean grade was down the 115/116 mark by the mid nineties. What we need though is a trace of the mean grade after the change of junior rules. 1994 and 1995 ought to be available because the grading data is on the online server.

One caveat though - grades before 1991 are pure performance, the rule was that you added 10 to the junior's published grade before using it in the following season. This of itself would change the averages (upwards) in the year in which it was introduced.

I have a theory that the influx of adult players in 1993 after the Short match had an impact which wasn't measured at the time. Most of them were pretty weak to start with, but were usually pitched into competitive play. So as a guess, the mean dropped between 1993 and 1994 (possibly beyond) partly because of the changed mix of players. Being non-junior improving players ( well some of them!), they also exerted a downward one-off force by taking points off established players.

I'm coming to the belief that the most powerful inflation or deflation force is the junior increment and that the "new" grades contain a massively inflationary junior increment to such an extent that a primary purpose of a grading system to rank players in order of playing strength has been broken.

I just don't believe that 150 players scoring 58% against 140 players and 42% against 160 players is any sort of powerful force on the dynamics of a grading system even if either true or statistically significant. The significance point applies also to Sean's grade comparison over time- what are the sample sizes involved?

If the theory that computers and better prepared strong players is to be believed,

Is`Sean trying to tell us that if someone banned Anand from using computers, then his chances against Kramnik would be unaffected? The point is that you have a group of players some of whom improve and the others don't, then those who improve maintain the same level of grade as other improvers whilst the non-improvers lose points. This can explain why the ECF president has lost 30 points whilst other players in the same grading range have not. It's a hard point for some players to accept, but if you last looked at some opening theory in 1986 you will not be able to maintain the same grade in 2008.

I could also refer the reader to David Robertson's account of Nigel Short's last round game at Liverpool - where the whole game had been previously analysed on Nigel's computer.

Other people must have historic runs of the BCF Newsflash, the SCCU bulletin or even the BCF board and council minutes. I would suggest that the histories be read before the grading team's extrapolations are believed.

[Roger de Coverly]

One caveat though - grades before 1991 are pure performance, the rule was that you added 10 to the junior's published grade before using it in the following season. This of itself would change the averages (upwards) in the year in which it was introduced.

One other point about rule changes over the years :- Another change for the 1991 or 1992 list was to change the definition of E rated ( minimum standard for inclusion) from 10 games in 3 years to 10 games in 3 years but at least 1 in the most recent season. This was done in part for the excellent reason that it froze the grades of deceased players but it would have the side-effect of accelerating the removal of players from the published list. It's not obvious how this change would affect the mean though.

[Sean Hewitt]

I'm coming to the belief that the most powerful inflation or deflation force is the junior increment and that the "new" grades contain a massively inflationary junior increment to such an extent that a primary purpose of a grading system to rank players in order of playing strength has been broken.

Finally. If you have a view on the source of the deflation, this must mean that you accept that deflation itself exists! Eureka. And you are absolutely right that the junior players (or, more accurately, rapidly improving players) are the source of the deflation. That is why my suggestion is to treat them as new players each season, thus surpressing the impact of their rapid improvement on the system.

I just don't believe that 150 players scoring 58% against 140 players and 42% against 160 players is any sort of powerful force on the dynamics of a grading system even if either true or statistically significant. The significance point applies also to Sean's grade comparison over time- what are the sample sizes involved?

As quoted in the post, the sample size was 4815 players - which is about 45% of the players who had a grade in 2006. That's more than enough to draw a meaningful conclusion I believe.

Is`Sean trying to tell us that if someone banned Anand from using computers, then his chances against Kramnik would be unaffected? The point is that you have a group of players some of whom improve and the others don't, then those who improve maintain the same level of grade as other improvers whilst the non-improvers lose points.

Sorry Roger, but thats nonsense. If Anand was banned from using a computer, his chances against Kramnik would of course be adversely affected. He would lose many rating points. But Kramnik would gain an equal number of points as Anand loses. If one group of players is losing rating points hand over fist because they are lazy, not preparing or not using a computer, then where are they going? Because, hard as I look, I cannot find an equal and opposite gain of grading points anywhere.

[Tim Spanton]

What a shame this wasn't properly investigated BEFORE it was decided to boost the mean by 20+ points.

[Roger de Coverly]

What a shame this wasn't properly investigated BEFORE it was decided to boost the mean by 20+ points.

If I believed in conspiracies rather than cock-ups, I would suggest it's a attempt to discredit the "traditional" system in order to advance the case for replacing it by an Elo style system.

How long it will take for the inflation to break through to the top level, I'm not sure. Monitoring my own performance shows a greater gain on "new" grades than old. Mind you, contrary to the opinion of the ECF grading team, I was underperforming against the IMs in Liverpool and overperforming against less taxing opposition.

[Tim Spanton]

What a shame this wasn't properly investigated BEFORE it was decided to boost the mean by 20+ points.

If I believed in conspiracies rather than cock-ups, I would suggest it's a attempt to discredit the "traditional" system in order to advance the case for replacing it by an Elo style system.

How long it will take for the inflation to break through to the top level, I'm not sure. Monitoring my own performance shows a greater gain on "new" grades than old. Mind you, contrary to the opinion of the ECF grading team, I was underperforming against the IMs in Liverpool and overperforming against less taxing opposition.

I'd like to thing you're right about the conspiracy theory but the alternative is much more plausible

[Sean Hewitt]

What a shame this wasn't properly investigated BEFORE it was decided to boost the mean by 20+ points.

Mean Grade 1987 = 128
'New' Mean Grade 2008 = 129

So where exactly is this 20+ point boost?

Or do you simply believe that if you tell the same lie often enough, it will become the truth?!

[Roger de Coverly]

Arithmetic average 2008 111.4 standard deviation 44.6
Arithmetic average 2008 ("new grades") 135.2 sd 35.4 - dispersion has been reduced

The 1987 population wasn't full of under 10's and players with grades under 50. In the pre-computer days, graders were apt to cut their workload by deeming events not strong enough to be graded.

Tim's original question

For what it's worth, my question is: How come this cure for deflation has bumped the average grade from 110 to 130?

From ECF website

50% of old grades are at least 110
50% of "new" grades are at least 135

So my answer to Tim is that the "cure" has introduced (rampant) inflation by making the junior increment far too high.

Here's a grandmaster on the subject of playing standards ( Matthew T)

I certainly think the top chess players have improved enormously over the past twenty years and probably higher FIDE ratings reflect this. It is therefore, probably true that in absolute terms the Liverpool EU Championships is the strongest event ever held in Britain.

If you want to know why the grading range has stretched between the 100 graded players and grandmasters, it's because the grandmasters study chess and the 100 graded players don't.

[Sean Hewitt]

Arithmetic average 2008 111.4 standard deviation 44.6
Arithmetic average 2008 ("new grades") 135.2 sd 35.4 - dispersion has been reduced
.

You mean deflation has been elimiated - which was the idea!

The 1987 population wasn't full of under 10's and players with grades under 50.

I have no idea how many players in 1987 were under 10. Nor, I suspect, do you. However, I do know how many there are now. 129 out of 10,262 players are either aged 10 or less, which is just over 1% of the graded population. I therefore wouldn't describe the current grading list as being "full of under 10's". Another of Rogers wild claims which has no basis in fact gets shot down in flames!

As for players graded under 50. Again, I have no idea how many there were in 1987, do you? I would expect it to be less than now, thanks to the effects of 20 years of deflation. Currently there are 789 such players, of whom 490 are adults.

So my answer to Tim is that the "cure" has introduced (rampant) inflation by making the junior increment far too high.

Show us the evidence for this conclusion. You seem to veer from one conclusion to another without ever being able to back any of your claims up with facts!

If you want to know why the grading range has stretched between the 100 graded players and grandmasters, it's because the grandmasters study chess and the 100 graded players don't.

Possibly. But this theory must address three questions satisfactorily to even be considered

1) Why have grandmasters stayed at same grading level whilst the lower graded players have decreased? If the assertion is correct, surely the weak players should have stayed at the same level whilst the grades of GMs increased?

2) Why are there not two groups of players, one increasing in grade whilst the other decreases in equal amounts? The evidence suggests a very small group have seen a small increase whilst a large group have seen a large decrease.

3) Why are the grades of lower level players also diverging. If you assertion is correct, the grades of lower level players should diverge from GM's but should stay stable relative to each other. Yet the eveidence is that the lower you are graded, the more deflation you have experienced. Why should the grades of two players, both graded under 120 but 25 points apart grow further apart as they decline if neither is studying chess?

[Roger de Coverly]

You mean deflation has been elimiated - which was the idea!

Well the historic series will have been destroyed and some of the players who have come into the list over the past 15 years are being awarded career best performances. We don't know, but suspect, that inflation has been introduced - If the ECF published new grades (2007) then we'd know whether or not new grades (2007) were about equal to new grades (2008) or several points lower.



I have no idea how many players in 1987 were under 10. Nor, I suspect, do you. However, I do know how many there are now. 129 out of 10,262 players are either aged 10 or less, which is just over 1% of the graded population.[/quote]

Including the rapidplay list, 751. Don't the rapidplay grades get used as estimates for the standardplay ones?

I will quote directly from the SCCU bulletin of November 1990. The article is headed BCF Grading : SCCU bias shock horror sensation and partly refers to findings that SCCU players had higher ratings than the rest of the country

The BCF computer has surveyed about 18,000 players in the published list. It split the scale into 10-point chunks, and counted how many players fell within each chunk. Results for the SCCU were markedly different from those for other Unions. The SCCU peaks araound the 140 mark, and the others around 125. This differential holds all the way up the scale. At 170-plus or so, half the players are SCCU. When you get over 200-210 the numbers are too small to draw conclusions. (Practically everyone over 230 is in the south, but that could reflect player-migration rather than grading bias.) At the other end of the scale, grades for the other Unions go down further than ours. The WECU has 50 players graded below 50; we only have 20-odd, despite our greater size. This could be because the WECU grades all its U11 events, and we certainly don't; which in turn could go some way towards explaining why SCCU grades are higher. Another possible source of bias is the fact that, by and large, SCCU graders don't enter data for new players with less than 5 games. The MCCU enters players with 1 game, and the other Unions, while not usually going as far as that, are less selective than us.

A later paragraph discusses inflation and the junior increment - which was a flat 10 points at this time.

Mr Edwards tell us that the evidence of Union bias emerged accidentally from an exercise designed to investigate grading inflation. It seems that published grades have increased nationally, on the average, by about 1.5 points per year over the last four years (which is as far back as the computer can go). The mean has risen, in that time from 121 to 128. The only reason he could suggest was that the 10-point addition for juniors might be unrealistic. In fact junior grades seem to go up by about 5.5 points a year, so the 10-point rule will be mildly inflationary. (U10s do go up 10 points but it tails off rapidly, as you'd expect, and U18s go up hardly at all. Mr Edwards did not think this could be the only reason for inflation.

Also

Maybe over-generous estimates for new players also come into it somewhere?

Show us the evidence for this conclusion

(rampant) inflation

The ECF could prove or disprove this thesis by publishing 2007 (new grades).

The other part is conjecture - that if you revalue junior 100 players to 150, they are not going to survive against adult 175 players revalued to 180. Where possible I've looked at some games of my potential next season opposition and I'm not impressed.

) Why have grandmasters stayed at same grading level whilst the lower graded players have decreased? If the assertion is correct, surely the weak players should have stayed at the same level whilst the grades of GMs increased?

Logically because there are "deflationary" forces affecting GM grades which have kept them stable in numeric terms. If you want to define improving GMs with static grades as deflation, that's up to you. However the revaluation is stating that the distance between GMs and "club" players is incorrect, I suspect it's right. Even at my level, if I can find the games, I will take 120's seriously. A number of them may have pet systems which they have studied and might potentially play to a 170 standard if allowed. Therefore I would select a variation which kept out of their pet analysis, thus improving my results.

Why are there not two groups of players, one increasing in grade whilst the other decreases in equal amounts? The evidence suggests a very small group have seen a small increase whilst a large group have seen a large decrease.

I expect because there are other forces at work. Your original contention was that if you had a grade of 175 20 years ago, you would now be down to about 155 and therefore should be revalued back to 175. I've been trying to point out that this is false and there are plenty of players (usually the people I play in Opens and the NCL) that were in the 170s both then and now. There are also players whose grades have collapsed down to the 150's. I've also contented that you need to know more than you used to in order to retain a 175 grade. You might also call this deflation and that the improvement in standards should have driven the improvers up towards the 200's. I could also note that the ECF system is not zero sum when less active and more active players meet. I'm not so sure about the large group/ very small group hypothesis. If I encounter someone of about my grade, I often find they were at about my grade for the available online history. The deflation hypothesis says they should all be collapsed 190s.

Yet the eveidence is that the lower you are graded, the more deflation you have experienced. Why should the grades of two players, both graded under 120 but 25 points apart grow further apart as they decline if neither is studying chess?

I'm not sure I understand this. Down at the sub-100 level I won't trust the grading system to produce sensible results because it ought to be easy, even for an adult player, to gain lots of playing strength. For what it's worth, I think a bit of selective reflation is necessary at the sub-100 level if only to get away from the zero grade. Various measures might be possible but shouldn't need wholesale revaluation of the entire system.

I would endorse the decision of the 1991 board - change the rules surrounding grading by all means but do not mess with the historic series of numbers