Grading volatility

General discussions about ratings.
Post Reply
User avatar
Christopher Kreuzer
Posts: 8781
Joined: Fri Aug 06, 2010 2:34 am
Location: London

Grading volatility

Post by Christopher Kreuzer » Sat Feb 23, 2013 5:09 pm

Has there been much discussion about grading volatility? I've recently been calculating grading averages in several matches I've played in during both the first half and the second half of this season, which required looking up both grades this half-season and last half-season, and while some people's grades remained much the same, I did notice some rather large grade swings of around 10 points. Whether this is normal or not, I don't know. You would expect some large grade swings, but do you get more than normal with 6-monthly lists as opposed to 12-monthly lists, and can the volatility itself induce more volatility? i.e. If you play someone who is undergraded or overgraded due to this effect, does that not affect your grade as well? Or should it all even out? (i.e. damped oscillations or not?).

Richard Bates
Posts: 3338
Joined: Fri Nov 14, 2008 8:27 pm

Re: Grading volatility

Post by Richard Bates » Sun Feb 24, 2013 7:54 am

Logically volatility should increase because fewer games makes for a less accurate sample. I think statistically 30 games is considered a "minimum" level for sampling purposes, but it should not be considered an 'ideal' number. There is also the consideration that some people genuinely play at a different strength in different parts of the year - eg. because their relative balance between evening league chess, weekenders and long tournaments is different. This is a problem for a performance based system because creates a situation where they are constantly playing with the 'wrong' grade.

Of course these problems wouldn't apply had the grading team instead opted for a yearly rolling list.

Roger de Coverly
Posts: 21291
Joined: Tue Apr 15, 2008 2:51 pm

Re: Grading volatility

Post by Roger de Coverly » Sun Feb 24, 2013 10:46 am

Christopher Kreuzer wrote: I did notice some rather large grade swings of around 10 points. Whether this is normal or not, I don't know. You would expect some large grade swings, but do you get more than normal with 6-monthly lists as opposed to 12-monthly lists, and can the volatility itself induce more volatility? i.e. If you play someone who is undergraded or overgraded due to this effect, does that not affect your grade as well? Or should it all even out? (i.e. damped oscillations or not?).
A crude investigation would be to calculate for every player appearing in consecutive lists, the average change in grade, offsetting pluses against minuses and the average absolute change in grade. You would compute separate statistics by grade category. Without having done the calculations, I might half expect X rated players to show the highest changes. The different mix of games in six monthly periods introduces an extra variability factor not present when the grade is calculated over a calendar year.

As far as opponents are concerned, you might expect it to even out, in that you are as likely to meet a player 10 points over as one 10 points under.

Ian Thompson
Posts: 3543
Joined: Wed Jul 02, 2008 4:31 pm
Location: Awbridge, Hampshire

Re: Grading volatility

Post by Ian Thompson » Sun Feb 24, 2013 11:34 am

I would expect to see increased volatility not so much because grades are now published every 6 months, but because they are now calculated on a rolling 30 games average for most players. If a good win replaces a bad loss in the calculation, or vice versa, one game alone could change your average grade by more than 6 points in the most extreme case. Have a couple of those is a row and there's your 10 points change in grade from one list to the next. Alternatively a run of 3 or 4 less extreme good results replacing 3 or 4 bad results, or vice versa, would have a similar effect.

Clive Blackburn

Re: Grading volatility

Post by Clive Blackburn » Sun Feb 24, 2013 3:20 pm

I can't believe that nobody is suggesting that grading volatility is a direct result of the membership scheme! :shock:

User avatar
Carl Hibbard
Posts: 6028
Joined: Fri Dec 08, 2006 8:05 pm
Location: Evesham

Re: Grading volatility

Post by Carl Hibbard » Sun Feb 24, 2013 3:21 pm

Clive Blackburn wrote:I can't believe that nobody is suggesting that grading volatility is a direct result of the membership scheme! :shock:
:roll:
Cheers
Carl Hibbard

Roger de Coverly
Posts: 21291
Joined: Tue Apr 15, 2008 2:51 pm

Re: Grading volatility

Post by Roger de Coverly » Sun Feb 24, 2013 6:41 pm

Roger de Coverly wrote: A crude investigation would be to calculate for every player appearing in consecutive lists, the average change in grade, offsetting pluses against minuses and the average absolute change in grade. You would compute separate statistics by grade category.
So here are some calculations.

The data is taken from the January 2013 published list, so it's a comparison between January 2013 and July 2102.

The first selection was to isolate all records for standard play grades where players had a published grade in both lists. This totals 9947.

The spilt of this by category was

Code: Select all

X 333
A 2075
B 1607
C 2270
D 2301
E 1361
The total of grade change, defined as Jan grade minus July grade was 2332 and the total of absolute change, ignoring pluses and minuses was 37448.

Spilt by category this comes out as

Code: Select all

X			1056	 2726
A			1535	11649
B			 273	 6457
C			-292	 6484
D			 184	 6734
E			-424	 3398
which translates as a table of averages

Code: Select all

X			3.2	8.2
A			0.7	5.6
B			0.2	4.0
C			-0.1	2.9
D			0.1	2.9
E			-0.3	2.5
It's only six month's of data, but can we conclude
(a) some players who play a lot gain grading points
(b) others who play a lot have less stable grades than those who don't ?

If it's accepted that X grades are likely to be volatile, does using a countback approach introduces additional volatility into A and B grades as well ?

I also did Adults only and Juniors only (defining adult as not having a Junior age)

Adults

Code: Select all

X	234	0.5	6.5
A	1857	0.1	5.2
B	1487	-0.1	3.8
C	2095	-0.3	2.8
D	1976	-0.2	2.7
E	1129	-0.4	2.1
Juniors

Code: Select all

X	99	9.4	12.1
A	218	6.4	9.0
B	120	4.1	6.6
C	175	1.7	4.0
D	325	1.7	4.6
E	232	0.1	4.5

John McKenna

Re: Grading volatility

Post by John McKenna » Mon Feb 25, 2013 1:37 am

Volatile – characterised by rapid change/frivolously changeable/light-hearted, lively/dangerously unstable; explosive

Taking Roger’s theoretical advice above I did a little crude practical investigation myself using a microscope instead of a telescope. My yardstick was that BCF/ECF grades have always been and probably remain accurate to within only 8 grading points. Using a small sample of players with whom I am familiar – 18 regulars of my club – I found the following.

Cat. X = 1, A = 4, B = 5, C = 5, D = 3 (X & As are all also tournament players, Bs,Cs & Ds are league, apart from 1 D.)

Points +/- for players (nos.) July 2012 to Jan. 2013 grading lists:
+16(1), +13(1), +5(1), +4(1), +3(2), +2(3), +1(1), -1(3), -2(1), -3(2), -5(1), -8(1)

2 players exceeded an 8-point grading difference between July & Jan lists.

The player with +16 grading points had returned to playing otb after a gap of about 4 years. On the latest list he is 163 C - having been mostly 170+ (A-C) for a number of years prior to 2008. On the July 2012 list he returned with a grade of 147 D based on (13 games in 6 months) and an initial grade by local graders that was, I believe, his last published grade minus 20 points (5 points times 4 for that number of years of inactivity). I expect he will return to about 170 on future lists.

(Might I suggest that the local graders’ estimate was a tad conservative in his case and that their method of estimating returning players’ grades could result in under estimation in many cases these days?)

The other player was +13 but I would describe his results and play as highly ‘volatile’ (frivolously changeable). He went from 133 E (1 game in 2002) in 2003 to 149 D in July 2010 without any graded games from 2003 to 2008 at all! Then his grade went: 141 C, 144 C, 112 X! and 125 A in the latest list. I expect he will return to about 140 on future lists.

So, I think it can be understood why those two changes were exceptional once the personal circumstances of the players are known. The other 16 players’ gains/losses were unexceptional. So, all is in order and approximately correct in my club’s ECF gradings, as far as they go.

But, I must add that none of the 18 players above are juniors (although one was until recently) and it is in the grades of the junior group that one should look for really exceptional changes that could well involve a lot of underestimation and high ‘volatility’ (dangerously unstable; explosive). That is despite them all being treated as new players each year, since 2009, and the aged-based 5 or 10 point enhancements to their grades that are too crude and too general, but still significantly better than nothing.

Note that in Roger’s final two tables above (Adults & Juniors) all the figures in cols. 3/4 are significantly better for the juniors –

(Adults: 0.5/6.5, 0.1/5.2, -0.1/3.8, -0.3/2.8, -0.2/2.7, -0.4/2.1)
(Juniors: 9.4/12.1, 6.4/9.0, 4.1/6.6, 1.7/4.0, 1.7/4.6, 0.1/4.5)

That could be because they are improving so rapidly that their grades are lagging or because they are underrated to start with each year. Who knows?

“An underrated player truly victimises his opponents regardless of the outcome of the game…” (A. Elo – The Rating of Chessplayers, 1978)

Post Reply