New grades (split from Editorial thread)

General discussions about ratings.
Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: New grades (split from Editorial thread)

Post by Roger de Coverly » Fri Jun 13, 2008 6:18 pm

So can anyone else come up with examples? Is there a particular area where it's more prevalent, or is my memory going?
I've struggled to find players that I've played during the last 15 years that have shown "unexplained" reductions in the their grades. Mine is almost unchanged and I'm just as many points below Mark Hebden (or Peter Sowray for that matter) as I ever was.

There's some sort of problem at the sub 100 level where you seem to get pockets of (adult) players stuck with really low grades. Howard Grist referred to this at his own club. What he didn't indicate was whether they really are that bad :( or whether they are all improving but only ever play against the same really low rated players. Whilst we have some confidence that a 190 player is much better (9:1) than a 150 player who is correspondingly better than a 110 player, it's difficult to have quite the same trust in the grades of 90 v 50 v 10. At that level, it surely cannot be that rare to improve 50 points or more in a year. Over time this starts to cause problems further up the system when the 75 player (who should be 100) plays against the real 100 player.

A change in recent years is a switch to using an estimation program for new players rather than the grader's guestimate. By its nature an estimation program assumes constant strength over the measuring period - a suspect assumption for the rapidly improving player. Graders usually did their work at the end of a season so may have been more influenced by recent results.

For no particular reason other than it might work empirically, I'd suggest that all new players be given a minimum grade (for their opponents) of say at least 50. I think this would inject points at the foot of the system. It might not be as inflationary as it sounds because some of the new players (juniors) probably only play for a couple of years.

I've never been completely clear what the Hewitt investigations consisted of. It looks as if the estimation program was run with no pre-existing grades and then the results from this program were compared against the actual published grades (and didn't fit). As well as this, shouldn't there have been an analysis which looked at the result of every game and grouped them both by the grade difference and the absolute value of the player's grades? You would then have a statement that 125 players scored as well against 100 players as 200 players against 175 (or not). Perhaps the win/draw/loss ratios would be different. Maybe there has been such an analysis - but I don't think it's been published.

Sean Hewitt

Re: New grades (split from Editorial thread)

Post by Sean Hewitt » Sat Jun 14, 2008 1:02 pm

Whilst I'm not going to get involved in the same old arguments, this is fresh manure from Roger that must be dealt with!
Roger de Coverly wrote:
Look at the individuals who were playing some years ago (say 10-15 years) and who are still playing now. Their grades, as a group, have gone down.
This is what hasn't been convincingly demonstrated. The sceptics are those active players whose grades today are not greatly different from those of 15 years or more ago. If you suggested that the standard of play has risen for a given grade over 15 years, I could accept this and it's the mature players who haven't done the work to keep up who have suffered. Is this deflation or as Tim S suggested is the revaluation an ego-boost for the lazy?
First Published on SCCU Forum 14/12/2006

I examined the 1996 and 2006 grading lists, interested only in players who are in both lists so as to compare true like with like. I naturally excluded players who were juniors in 1996 on the premise that they would be expected to improve. This left me with 4815 players, so a pretty decent sample size. I looked at how much their grade had gone up (or down) in that 10 year period, and then banded the results as follows.

'96 Grade '06 Grade
200-217................. + 4.90
190-199................. + 1.77
180-189................. - 0.89
170-179................. - 1.80
160-169................. -3.76
150-159.................. -2.46
140-149.................. -2.40
130-139.................. -3.74
120-129.................. -4.66
110-119.................. -5.29
100-109.................. -4.28
90-99..................... -8.43
80-89..................... -9.01
70-79..................... -12.19
60-69..................... -14.48
50-59..................... -14.87
0-49 ...................... -22.21

This tells us that below 190 every grading band has suffered deflation since 1996. We can also see that the lower graded the player, the worse the deflation is. This correlates with my previous findings.
Roger de Coverly wrote:Let's try a somewhat more realistic example which assumes the average strength of players is unchanged and there isn't a major influx of strong players.
I agree my example was absurd - but that was for a reason. To demonstrate absolutely that the mean average is meaningless!!
Roger de Coverly wrote:Every year 10 players give up and are replaced by 10 new players. All the new players are of 100 strength and are estimated as such. Let's suppose that the estimation process is changed so that they get an estimate of 80 instead of 100 ( remember they are still 100 strength). The average grade goes down because of the estimation process and the estimation has introduced deflation.
Of course, that is correct. But why on earth would you estimate someone known to be of 100 strength as 80? It certainly does not happen now.

Sean Hewitt

Re: New grades (split from Editorial thread)

Post by Sean Hewitt » Sat Jun 14, 2008 1:06 pm

Roger de Coverly wrote:I've never been completely clear what the Hewitt investigations consisted of.
That's blatantly obvious - and in the rush to claim that it was wrong you've never, in 2 years, bothered to ask either :oops:

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: New grades (split from Editorial thread)

Post by Roger de Coverly » Sat Jun 14, 2008 3:14 pm

Can we be clear about the Hewitt investigations?

Is it correct to state that ?

one year's grading data was selected
The new player estimation program was run
The results were plotted against the published list (and didn't fit)

From the grading paper
1. The player had a published SP grade in 2005
2. Only SP games played between 1/6/05 and 31/5/06 against players in the sample group would count
3. The player had a minimum of 5 games in the sample
4. The players games yielded a score between 15% and 85%
5. Steps 4 and 5 were repeated until all criteria were met
6. All players were assigned an arbitrary initial grade of 150

The sample produced 6302 players who played 61375 games between them.
So you've used one year of data on about half the published players and you expect us to believe it shows errors in the grading system going back decades.

Michele Clack
Posts: 437
Joined: Fri Feb 08, 2008 2:38 pm
Location: Worcestershire

Re: New grades (split from Editorial thread)

Post by Michele Clack » Sat Jun 14, 2008 9:12 pm

Quote:
I've struggled to find players that I've played during the last 15 years that have shown "unexplained" reductions in the their grades
.

I took up up chess in the early 80's having done no more than learn the moves at school a decade or so before. My first grade was 101 rising to 113 quite quickly and then gradually subsiding. My current grade is 102. My grade tends to fluctuate quite a bit but I have never got to 113 since. My point is that I am a far superior player now than I was in my first few years of playing club chess. My husband's grade has followed a similar pattern except that he peaked at 129 in the 80's. We both play a lot more chess now and this pattern has puzzled us for several years. We are from the Midlands and have been having this grading conversation with other midlanders long before the ECF looked into the grade levels. We also met some Welsh players last year who were convinced that their grades were far too high in comparison with English players.

So people in some parts of the country have felt there is something wrong with the grades for some time.

Looking at people staying roughly the same grade over the last few years and using it as evidence that there is no grade deflation presupposes that those people have not improved their play at all. This may be true but then again it may not.

Ian Kingston
Posts: 1071
Joined: Sat Apr 14, 2007 3:16 pm
Location: Sutton Coldfield
Contact:

Re: New grades (split from Editorial thread)

Post by Ian Kingston » Sat Jun 14, 2008 10:36 pm

Michele Clack raises an interesting point: will the new grades take account of regional differences? Are there any regional differences? Does anyone have any statistics on this?

Matthew Turner
Posts: 3600
Joined: Fri May 16, 2008 11:54 am

Re: New grades (split from Editorial thread)

Post by Matthew Turner » Sun Jun 15, 2008 8:16 am

I am convinced that deflation is occuring, so why can't Roger see it? Well surely the answer is fairly simple, at 190 his grade may be deflate by 10 points or so, if this occured over a decade it would be almost imperceptible. If Roger is playing a lot and improving, his grade may stay static even whilst deflation occurs. I suggest that Roger compares his ECF grade and his FIDE rating to see if there is any trend in the relationship between the two.
I have observed in recent years that the juniors that I teach always gain grading points in adult events. It is not unusual for a junior to struggle to 50% and a 70 performance in a junior tournament and follow up winning a minor event, U100, without breaking sweat. This suggests to me that the problems with the grading system are quite focused and I am not convinced that Howard and Sean are on the right lines in attempting to fix them.
An interesting idea would be to apply Howard's grading alterations just to juniors. This would be hugely populist, since the juniors would be happy that their grade had gone up and adults would be happy that juniors had less chance of 'taking their prize money'. If adults were under-graded then they would still have the opportunity to 'feed off' the higher junior grades.
I genuinely feel that Howard, Sean and others have put in some good work here, but I can see the new grades coming out and being mired in confusion. Given that Howard and Sean seemingly disagree on a number of important points it is difficult to see a unambiguous rationale coming out for the changes.

Roger de Coverly
Posts: 21301
Joined: Tue Apr 15, 2008 2:51 pm

Re: New grades (split from Editorial thread)

Post by Roger de Coverly » Sun Jun 15, 2008 9:57 am

I suggest that Roger compares his ECF grade and his FIDE rating to see if there is any trend in the relationship between the two.
ECF circa 175 (+/- 10)
FIDE circa 2100 (+/- 30)

Fewer games in the FIDE pool of course.

I've known for years that the traditional 8*ECF+600 = FIDE is broken at the 2000 and a bit level. It's quite easy to see why - it's because in the 1990s when the FIDE ratings only went down as far as 2000, you only got on the international list in the first place with your best performances. So in ECF terms if you can obtain a 175 grade by mixing 190 performances in tournaments with 160 performances, in 90s FIDE you had a system which ignored the 160 performances and counted the 190 performances, thereby entering the list at 190.

I suspect that over the last 20 years, my positional understanding has improved even if my tactical depth hasn't. However I still don't do any better against FMs and IMs. Tim Spanton's earlier point was that you have to put in some effort even to maintain parity. Presumably GM's still prepare things - even if it's just thinking up some ideas for specific opponents. So I draw the conclusion that FMs, IMs and GMs have to needed to improve to maintain parity of their ratings.

There's a philosophical problem here - if the absolute standard is increasing but the relative standard stays constant, should the absolute values in the grading list increase?

Matthew T raises the issue that junior player grades may be too low. He could be right - I would suspect a combination of the estimation program and 40 point rule as a cause by bringing players in a too low a level and by restricting their ability to break out from this low level. It must be more common for players to improve from 50 to 100 over a season, than from 150 to 200.

The overall effect of the Howard Grist recalculations is inflationary as currently proposed. I say this because on the new "corrected" list the statistic sum(new grades)/number(players) is going to exceed the statistic sum(old grades)/number(players). The players being the same in both cases.

The recalculation may demonstrate (and correct) some regional imbalances - if there's also an adult/junior imbalance it may attack these as well. We will have to wait for publication for these issues to be visible.

I've mentioned my suspicions about the estimation program and earlier Matthew T suggested it could be made less "right". In the old days of local grading, each local grader would collect results and calculate grades for their local events. The BCF would only receive something like event , player code,total points 1000 , total games 10. The local grader would therefore know the "local" grades. New players were mostly estimated locally. So in the same season, you could get a junior player scoring 50% against adult players graded 100 being given an estimate of 100 for this performance whilst scoring 50% against juniors graded 50 and getting a 50 estimate. Nowadays the central program might give the new player a grade of 75 with a knock-on effect downwards on the adult players.

To summarise, if there is deflation, then it's a recent effect and confined for the time being to the lower reaches of the system and possibly confined to player pools consisting mainly of juniors.

abi&timadams
Posts: 48
Joined: Sun Aug 24, 2008 4:21 pm
Location: Northumberland

Re: New grades (split from Editorial thread)

Post by abi&timadams » Sat Aug 30, 2008 4:55 pm

Tim and I organise the Scarborough and Northumberland Congresses and we are concerned at the timing of the production of the new 'corrected' grades. We believe it is imperative to make clear that these grades are not going to be used until Congress grading bands have changed accordingly, in the season 2009-10. On the ECF grading page it states:

A FURTHER REVISION (of grades) will appear in late August 2008 in time for the start of the new season. The late-August version will be official for all ECF Standardplay events throughout the season. The printed list, available at the start of September, will be identical to it.

Because the old current grades are going to be published alongside the new corrected grades the second sentence surely needs clarification. We have already received enquires concerning the validity of the new grades for this season and it is possible that players are delaying their entry until the new grades are published, which is not necessary.

RichardPalliser
Posts: 18
Joined: Wed May 14, 2008 2:46 am

Re: New grades (split from Editorial thread)

Post by RichardPalliser » Mon Sep 01, 2008 7:50 pm

I remain a little confused by the junior correction, and how best to proceed. When I was a junior, regional/county events were rarely graded. Undoubtedly junior events featuring juniors who are new to the game and/or have played few games against non adults are hard to grade. Could we thus move to not grading junior events with an average grade of, say, under 80? Or, if that will disappoint and affect too many juniors, perhaps a separate junior list could be produced for players up to age 11?

Two talented juniors in York have seen their grades rise from 117 to 149 and 73 to 117 this past year. A fantastic achievement, but allowing for the new correction they become 178 and 158 respectively! Looking at the corrected new grades of people they will often now play against, their current grades strike me as being about accurate; the 'corrected' grades rather misleading and too high. Perhaps this is just a regional issue, though, and some north-south further correction needs applying?

Simon Crowley
Posts: 1
Joined: Sat Aug 30, 2008 10:48 am

Re: New grades (split from Editorial thread)

Post by Simon Crowley » Mon Sep 01, 2008 10:07 pm

I am a bit 'miffed' about these new ratings?
I have a new 08-09 rating of 155, but my 'new' rating is 171! as 155 is the highest i have ever been, i feel a little daunted by the prospect of 'living up' to this rating, for a start i will have to play in the open sections (unless tournaments are going to reflect these increases by raising the grading sections ie minor u150 major u185 open etc)? considering my highest rating until 2 yrs ago was 132 (and i have been playing graded chess for 20 yrs) i really do find a rating of 171 a lot to live up to, or lets say i think im going to be made to look a bit silly!! lol :-)

Can i just ask you guys, is this rating going to be used as this seasons rating to calculate next? is that how it works? so if play to 150 this season i will get a rating of 160-161 next season?
thanks for any replys in advance :-) simon crowley :)

Paul McKeown
Posts: 3732
Joined: Thu Apr 12, 2007 3:01 pm
Location: Hayes (Middx)

Re: New grades (split from Editorial thread)

Post by Paul McKeown » Wed Sep 03, 2008 2:55 pm

Before I start, I would like to say that I only very occasionally contribute to online fora due to the prevalence of various forms of nutters that inhabit them. (If my memory serves me right, Homer Simpson was once advised by Bart, "Don't argue with idiots, they only drag you down to their level and then win on experience.") Nevertheless, I feel this topic to be sufficiently important me that I feel I should brave the dangers of posting! And this one looks relatively tame anyway....!

A. Method of Correction (Part I)

Firstly, let us assume (for the sake of argument and for now) that there is a real need for a correction to the current grading system. In that case I feel that it should not be carried out in the manner currently proposed.

Why?

Because the old numbers and the new numbers look alike, but nevertheless represent a severe break in the meaning conveyed by any particular number

This is confusing for players, coaches, team captains, etc. - I currently have a reasonable idea as to the level of opposition I can expect from a 157 player and from a 192 player. Next season I really won't have much of a clue. Nor will anyone else.

And what about games collections which include the players BCF/ECF grades? I know people who simply exclude players below a certain level for instance, because they want to learn from those whom they perceive as good players, not bad players. 210 (to name a number) will not be worth what it once was...

And what about future historians? In 20 years time, no one will remember the adjustment, but they may well still use the numbers as if the adjustment had not taken place. Surely this is simply leaving them a mantrap?

Never mind what will happen to pub bores - young whippersnappers of the future will be less impressed with the bores' former 150 or 180 or whatever grades than they ought to be!

The problem is simply that old and new grades are discontinuous, but are still represented by numbers that look similar. (Bit like government unemployment statistics one might think...)

May I make a suggestion apply the suggested adjustment to grades then add a 300 point increment to all grades, so that the scales look dissimilar and it will always be instantly clear whether people are referring to old or new grades.

Either that or move to a four digit scale, or indeed to an Elo based system.

B. Method of Correction (Part II)

Let us assume again for the sake of argument that there is sufficient reason to want to correct the grades.

Should one not rigorously investigate the causes of the dysfunction of the grading system and then implement measures to avoid future dysfunction and methods to track it? And shouldn't that be done before taking any action as drastic as one off correction?

It is perhaps the case that the cause(s) of the dysfunction have been rigorously established, in which I apologise at once. But it doesn't feel like that...

C. The Assumption that Grades have Deflated over Time

Hmmm.... I thought that the Elo system had neither an absolute scale nor was it invariant over time. There was simply a class interval of 200 points and a median value of 1500 (or whatever)... And that comparison between lists - well that carried a government health warning?

(And that leaves aside any dodgy ideas that one can assume Richard Clarke's system to have a genuine equivalence with Arpad Elo's...)

Leaving all that aside - it is nice after to ask how the team that won the Premiership last season would have done against the team that won the old First Division in 1970 say...

A chess rating is an ESTIMATE of a statistical parameter not the actual value. So it can never be known for certain that a player is 1700 Elo player. Sometimes he plays as a 1600 player sometimes as an 1800 player. Assume that a given player is 100% certain a 1900 player (an impossibility of course) who plays a player with a 1700 rating. Elo tables predict a 76% result for the stronger player. But what if the weaker player plays 50% of the time as a 1600 player and 50% of the time as an 1800 player? Then the stronger player will achieve 50% of his anticipated score against a 1600 and 50% of his anticipated score against an 1800, which is 0.5 x (0.64 + 0.85) = 74.5%. In otherwise, because of the uncertainty in the playing strength of the weaker player, the stronger player will not achieve his "theoretical" Elo score.

Actually that is an example taken from the well-known monograph, A Comprehensive Guide to Chess Ratings, by Professor Mark E. Glickman, a well respected academic mathematician who has carried out a great deal of research into chess rating systems. The paper can easily be found on the internet for those that are interested.

This example can be found on page 11 of the monograph, where Glickman explains the example in a rather more rigorous way and suggests that the stronger player should never actually achieve his predicted Elo score against the weaker player.

Which is perhaps exactly the effect that has been found by studying the ECF database...?

Perhaps the authors of the study would like to comment - and I will certainly respect the advice of experts if they say that this "Glickman effect" has been taken into account. But in that case I would suggest that any change should be made without making everyone confused in the future...

Regards.
Paul.

David Robertson

Re: New grades (split from Editorial thread)

Post by David Robertson » Wed Sep 03, 2008 5:36 pm

Paul: one downside of believing all forums are dominated by nutters is that you deprive yourself of discussions that are entirely intelligent. Over the past two years, starting initially on the following site

http://www.atticuschess.org.uk/forum/ph ... m.php?f=21

and taken up in due course here too, most if not all the points you raise have been thrashed out in great professional detail. Read through the vast material if you possess a parallel life :roll:

Of course those responsible for this exercise have diagnosed the causes of the problem. One may dispute the diagnosis, even some outcomes, but not the process that has been undertaken.

I'd go so far as to say that it has been the open and intelligent debate on the matter, on the Atticus forum and here, that has enabled ECF grading officers to proceed expeditiously with a degree of confidence.

Some of us hoped, when opening the debate two years ago, that a regrading exercise, if proved necessary, would run in parallel with a shift to Elo grades. That remains my objective. But one step at a time. Let's debate what is now emerging and gain confidence in it. Thereafter, use the new grades as the basis for a formal grading adjustment by 2010. And by then the community might be ready to accept a move to Elo. Let's hope so

David

Paul McKeown
Posts: 3732
Joined: Thu Apr 12, 2007 3:01 pm
Location: Hayes (Middx)

Re: New grades (split from Editorial thread)

Post by Paul McKeown » Wed Sep 03, 2008 6:40 pm

most if not all the points you raise have been thrashed out in great professional detail
- David Robertson

David - I haven't went through the threads on Atticus, however I have went through the threads here (on the ECF forum). Nowhere that I can see has anyone mentioned that a higher rateed player's actual score against a lower rated opponent should be somewhere between 50% and the Elo predicted score, depending on how accurately estimated the two ratings are. This is something that Mark Glikman pointed out in his paper and, if correct, would tend to make rating systems appear "compressed" or "deflated". In which case, why should one attempt to correct the proper result of a rating system?

Nor David, have I seen anyone propose a correction in a way which doesn't lead to the inevitable confusion between pre and post grades by changing the scale. (Except for people inevitably suggesting a move to another system, namely an Elo system.)

I'm intensely sceptical that any proposed change will be for the better, but am willing to be convinced, but not by arguments of the nature, oh it's already all been discussed. Naturally I have followed this forum from time to time... it's just that I don't post often.

Sean Hewitt

Re: New grades (split from Editorial thread)

Post by Sean Hewitt » Thu Sep 04, 2008 10:51 am

I dont think the ECF have seriously considered a move towards ELO ratings for two reasons. Firstly, the argument goes that the Clarke system is simpler than ELO (which it is) and players like to calculate their own grading performance. Secondly, no evidence has been presented that ELO is actually better than the Clarke system.

There was a discussion to move to a different scale, specifically along Welsh lines which use the Clarke system but produce grades which look alot like ELO ratings! This was dismissed for two reasons that I recall. Firstly, if the grades looked like ELO ratings then there could be confusion amongst the two. Secondly, and more importantly, it was felt that players wanted to compare year on year performance. Hence this year publishing the new grades so that people can see next year what their improvement or decline is. Of course, the new grades could simply have had 500 added and achieve that same benefit but the aim of the exercise was to "correct" deflation so that a 100 grade today is worth the same as 100 was in say 1980. Whether this aim was worthwhile is of course a matter for debate. Personally I don't think its a big issue either way.

What's interesting though is the misconception of what the new grade is. I have read people saying my new grade is xxx, but I'm not as good a player as that! Of course, the new grade is not saying that such a player is as good as an xxx was last year. What it is trying to say is that the player is of the same standard as an xxx was 25 years or more ago.

Regarding junior increments. Its worth remembering that the junior increment is supposed to reflect the average expected improvement in the junior in the next 12 months. So a juniors grade just after the publication of the grading list should appear inflated. Wait until next April though. If these junior grades still appear over graded the increment was too high. If however they are kicking our butts, then it was probably about right!! Given that Howard says the new grades give a good fit to expected scores, I fear we may still hate playing these kids come the end of the season.

Post Reply