GRADING ANOMALIES

[E Michael White]

................. but I'm not so sure about multiple games (despite this long thread based in a lighthouse). ..."

Brian - My way of looking at multiple games is: -

If the object of the grading system is to rank players from results obtained against a group of players the information input should be averaged equally over players and not over games. The effect of not doing this will not be too great in most cases but can inject or cancel grading points.

A numerical example could be :- Roger who has been 175 for 3 years played 15 games earlier in the rating year at 165 and Bob who has been 135 for 3 years plays 15 games at 125. Neither wishes to suffer the ignominy of dropping 10 points so they drop out of league chess and play a match of 60 games and agree the result at 54-6 to Roger; the result many graders would expect.

Rogers new grade is (15*165+54*185+6*85)/75=173
Bobs new grade is (15*125+54*125+6*225)/75=133

So each player retrieves their position to a loss of only 2 points. This result arises because both Bob and Rogers prior ratings appear to be positively biased mid season. If Cn is the contribution from the nth game. A players estimated new grade G is currently calculated as :-

G=w1C1+w2C2+w3C3……………..+wnCn , where all w are 1/n and n is the total games played.

Breaking down Cn = gn + bn + rn - en, where gn= the postulated prior true grade for opponent no n, bn is the bias in that grade, rn is the result discriminant ie +50, 0 or -50, and en is a performance fluctuation error factor. Normally graders assume both b1 and e1 will average to zero over all games. However if the first 4 games are played against the same opponent, whilst the es may average to zero b1=b2=b3=b4 so these become 4b1 causing some overall aberration.

A messy way to adjust this would be to replace c1+c2+c3+c4 by Cm=(c1+c2+c3+c4)/4 and calculate G as
G=(Cm+c5+c6+……………+cn)/(n-3)

Which says calculate a rating from the match and treat it as one game. I am not suggesting this should be done but it helps to understand where sources of error can come from and throws light on the lighthouse keepers' results. For individual 2 player matches this will not have much effect but when a new tournament comes along like the 4NCL or Basmans event, which largely pair players of near equal grade together the additional games can cause the same effect as a single match.

[Robert Jurjevic]

Hello Roger,

Your and your opponent's ECF grade with respect of each game is updated using the following formulae:

Code: Select all

a2 = a + ka*(q - p)/na;
b2 = b + kb*((100 - q) - (100 - p))/nb = b - kb*(q - p)/nb;

where 'a' is your grade (from previous season), 'b' grade (from previous season) of your opponent, 'p' your expected performance in the game (expected performance of your opponent in the game is then '100 - p'), 'q' your actual performance in the game (actual performance in the game of your opponent is then '100 - q'), 'a2' your new corrected grade (due to the game), 'b2' your opponent's new corrected grade (due to the game), 'na' the game count taken into calculation of your grade (if you played 30 or more games in the season then 'na' is the number of games you have played in the season), 'nb' the game count taken into calculation of grade of your opponent (if your opponent played 30 or more games in the season then 'nb' is the number of games your opponent has played in the season).

What makes ECF grading systems different is a choice of factors 'ka' and 'kb' and function 'p = f(d)'.

Total grade change with respect of each game is:

Code: Select all

(a2 - a) + (b - b2) = ((kb*na + ka*nb)*(q - p))/(na*nb);

We will examine two types of grading systems, grading systems for which it holds that 'ka + kb = 1' and grading systems for which it holds that 'ka + kb = 2'.

Total grade change with respect of each game for 'ka + kb = 1' grading systems is

Code: Select all

(a2 - a) + (b - b2) = (na + nb)/(2*na*nb) * (q - p) = c1 = c2/2;

Total grade change with respect of each game for 'ka + kb = 2' grading systems is

Code: Select all

(a2 - a) + (b - b2) = (na + nb)/(na*nb) * (q - p) = c2 = 2*c1;

!
As

Code: Select all

(na2 + nb2)/(na2*nb2) = (na1 + nb1)/(2*na1*nb1);

if 'na1 = na2/2' and 'nb1 = nb2/2', though 'c1(na1,na2) = c2(na1,nb1)/2' one could argue that 'c1(na1,na2) = c2(na2,nb2) = c2(na1/2,nb1/2)', i.e., that the total grade change with respect of each game for 'ka + kb = 1' grading systems for 'na' and 'nb' games equals the total grade change with respect of each game for 'ka + kb = 2' grading systems for 'na/2' and 'nb/2' games, consequently making the grade correction independent of the 'ka + kb' and thus denying any intrinsic difference between 'ka + kb = 1' and 'ka + kb = 2' grading systems (i.e., you can convert 'ka + kb = 2' into 'ka + kb = 1' grading system by halving 'na' and 'nb').

Reductio ad absurdum is a logical argument often used in mathematics. You basically start from negative statement of the statement you want to prove to be true and if by assuming that the negative statements is true you arrive at a contradiction (fallacy) you can claim that your original (positive) statement is true.

Say you want to prove that kingfisher can fly. You start assuming that kingfisher cannot fly and say that (assuming that kingfisher cannot fly) you arrived at a conclusion that the earth is flat (which is a contradiction or fallacy) you can claim that kingfisher can fly.

You can prove that square root of two is not a rational number (is irrational number) by assuming that it is a rational number and (taking that assumption as true) getting that say '1 = 2' (a contradiction).

I will use reductio ad absurdum logical argument to prove that statement 'c1(na1,na2) = c2(na2,nb2) = c2(na1/2,nb1/2)' is false, i.e., that statement 'c1(na1,na2) /= c2(na1/2,nb1/2)' is true.

!
If 'c1(na1,na2) = c2(na2,nb2) = c2(na1/2,nb1/2)' then 'na1/2' and 'nb1/2' (which must be whole numbers) may not be whole numbers (which is a contradiction), so, 'c1(na1,na2) = c2(na2,nb2) /= c2(na1/2,nb1/2)'.

!
So, there is an intrinsic difference between 'ka + kb = 1' and 'ka + kb = 2' grading systems and their corrections 'c1' and 'c2' are genuinely different in quality, as they cannot be made equal by choosing suitable 'na' and 'nb'.

For the cases where you have played against your opponent a match of 'na = nb = n' games total grade change with respect of each game for 'ka + kb = 1' grading systems is

Code: Select all

(a2 - a) + (b - b2) = 1/n * (q - p) = c1 = c2/2;

For the cases where you have played against your opponent a match of 'na = nb = n' games total grade change with respect of each game for 'ka + kb = 2' grading systems is

Code: Select all

(a2 - a) + (b - b2) = 2/n * (q - p) = c2 = 2*c1;

If you have played against your opponent a match of 'na = nb = n' games in each game scoring 'q' (either 0%, 50% or 100%) total grade change with respect of all 'n' games for 'ka + kb = 1' grading systems is

Code: Select all

n * ((a2 - a) + (b - b2)) = n * (1/n * (q - p)) = q - p;

If you have played against your opponent a match of 'na = nb = n' games in each game scoring 'q' (either 0%, 50% or 100%) total grade change with respect of all 'n' games for 'ka + kb = 2' grading systems is

Code: Select all

n * ((a2 - a) + (b - b2)) = n * (2/n * (q - p)) = 2 * (q - p);

!
Answering the question which are 'ka + kb = 1' or 'ka + kb = 2' grading systems better should be the same as answering the question if you have played against your opponent a match of 'n' games in each game scoring 'q' (either 0%, 50% or 100%) should total grade change with respect of all 'n' games be 'q - p' or '2 * (q - p)'.

Kind regards,

[Roger de Coverly]

Neither wishes to suffer the ignominy of dropping 10 points so they drop out of league chess and play a match of 60 games and agree the result at 54-6 to Roger; the result many graders would expect.

Alternatively they play in some congresses and manage to perform according to their grade. It's always been possible to lose bad results at the start of a season by getting decent results at the end of it. For matter if you find off form opposition with highish grades so much the better.

For the cases where you have played against your opponent a match of 'na = nb = n' games

I really don't know why you keep going on about 2 player matches. It's really not what the ECF grading system is about. The aim is to rank players who have scored similar results against similar spreads of opponents.

In the arbitrary example where everyone plays exactly 30 games in a season, I can turn the ECF system into your system by the simple expedient of redefining the minimum number of games for a one season grade to 60. The reason the ECF have never done this is the perception that we don't want our grading measurement of the most active players distorted by results played years ago. The oldest games included in the August 2008 grades (for >=30) went back to June 2007 and these grades were still in use in July 2009. Introducing additional averaging would backdate this another year to June 2006 and this is just not going to happen no matter how many formulae you write or graphs you draw or anomalies you discover in the treatment of two player matches.

[E Michael White]

Alternatively they play in some congresses and manage to perform according to their grade. It's always been possible to lose bad results at the start of a season by getting decent results at the end of it. For matter if you find off form opposition with highish grades so much the better.

Exactly. My point was that 2 player matches can be distorting as can groups of out of form players playing in an all play all or a late season swiss open. The lighthouse problem is similar and can be reduced by reducing the ECF K factor by increasing the 30 game rule. This effectively waters down the accumulated bias unless these lighthouse keepers play another match.

I am not suggesting anything should be done in the formula to try to fix this but graders seem to have spent a lot of time trying to fix a junior problem believing it to be the only problem. I think we agree players like to calculate their grades with a simple formula after each event; its really up to the grading team to identify the issues and perform tweaks to fix them but it would be advisable to publicise their intended approach before they do.

[Robert Jurjevic]

Hello Roger,

I really don't know why you keep going on about 2 player matches. It's really not what the ECF grading system is about. The aim is to rank players who have scored similar results against similar spreads of opponents.

The reason is simple, I want to find a method for assessing the difference between 'ka + kb = 1' and 'ka + kb = 2' grading systems. Examining their difference on real ECF data may be difficult (about 12,000 players playing about 200,000 games a year). One is (or at least should be) allowed to analyze simple examples of matches between two players (even they do not happen in practice) trying to judge which system produces more logical results.

This reminds me of the following: there is a calculation method for adding two numbers, and the method gives a wrong result in some cases, say the method gives '1 + 1 = 1'; I criticize the method saying that '1 + 1 = 2' and you say why bother as adding two equal numbers in practice is very rare; the reason why I want to examine those simple rare examples unlikely to happen in practice is because they may reveal a flaw in the calculation method which may be causing non-negligible addition errors when applying the method in real-life system. Hope that this makes some sense.

So, may I kindly ask you, if you have played against your opponent a match of 'n' games in each game scoring 'q' (either 0%, 50% or 100%), assuming that 'n >= 30', should, according to your judgement, total grade change '(a2 - a) + (b - b2)' with respect of all 'n' games be 'q - p', '2 * (q - p)' or neither?

('a' is your grade before the match, 'b' is your opponent's grade before the match, 'p' is your expected performance before the match (expected performance of your opponent before the match is then '100 - p'), 'q' is your actual performance in each game of the match (either 0%, 50% or 100%) (actual performance in each game of the match of your opponent is then '100 - q' (either 0%, 50% or 100%)), 'a2' is your new grade after the match, 'b2' is your opponent's new grade after the match, 'n' is the number of games played in the match)

Thanks.

Kind regards,

[Roger de Coverly]

The reason is simple, I want to find a method for assessing the difference between 'ka + kb = 1' and 'ka + kb = 2' grading systems.

Perhaps I should leave you to analyse this in peace. It has little or no relevance to the real world design of the ECF grading system.

[Brian Valentine]

If Cn is the contribution from the nth game. A players estimated new grade G is currently calculated as :-

G=w1C1+w2C2+w3C3……………..+wnCn , where all w are 1/n and n is the total games played.

Breaking down Cn = gn + bn + rn - en, where gn= the postulated prior true grade for opponent no n, bn is the bias in that grade, rn is the result discriminant ie +50, 0 or -50, and en is a performance fluctuation error factor. Normally graders assume both b1 and e1 will average to zero over all games. However if the first 4 games are played against the same opponent, whilst the es may average to zero b1=b2=b3=b4 so these become 4b1 causing some overall aberration.

Michael,
Well explained!

For completeness the grader's would expect gn+rn = true grade of player, summed over a large number of games. I'll need to think a bit more about your solution.

However I see this issue as very similar, and maybe of a lesser problem, to the junior issue. A junior's grade (as is a new entrant) has it's gn's based on a weighted average of the adults in their community. This can be seen in my formulation of our matrix presentation of the grader's iteration. No dat are published to investigate this, but I suspect many juniors playing a large number of games may have their grade based on very few rated players. IMHO this situation is worse as any residual bias is then multiplied up to get from graded players to games played.

There is some science to make my suposition - network science. The network here is the players as nodes linked to their opponents. The theory is that very few nodes dominate the network. One example is google's domination of the internet, but my favourite is the Kevin Bacon game. Here a film actor scores by the minimum number links, obtained by appearing in the same film as another actor, an actor is from Kevin Bacon. After Roger's heroic patient contribution to this thread I propose this is call the de Coverley game. How many opponents is a player away from RdC last season? I gather you score 1.

[Roger de Coverly]

gn= the postulated prior true grade for opponent no n, bn is the bias in that grade,

So you can look on the prior season grade as "estimate of strength" plus/minus "bias"

That's another way of approaching the lighthouse keeper anomaly. If we treat them as new players, we want them to have the same grade because of their drawn match but we don't know what absolute level it should be. If we try to anchor it to their past grades we don't know whether it should be the grade of the higher player, the grade of the lower player or somewhere in between. One or both players has a bias in their grade. If they are 25 points apart and the higher rated player is 25 points too high, then grading after 30 games of their match moves the bias to the previously lower rated player.

An improving player has a negative bias, so when you sum negative bias over the whole system, you get a dilution (deflation) of the grades of everybody. On this formulation junior increments are positive bias. 40 point rules are a bias as well ,both positive and negative depending on context. If a 180 player plays a GM at 230, the 180 player "sees" 230 less 10 whilst the GM "sees" 180 plus 10.

but I suspect many juniors playing a large number of games may have their grade based on very few rated players.

If they continue with the notion of discarding prior year experience, you could end up with the grades of an entire junior league depending on a handful of games possibly by just one player against adult opposition. That's always been an issue with nearly closed player sets, but previously you had the previous season's grades to give a consistency anchor.

[E Michael White]

However I see this issue as very similar, and maybe of a lesser problem, to the junior issue. A junior's grade (as is a new entrant) has it's gn's based on a weighted average of the adults in their community. This can be seen in my formulation of our matrix presentation of the grader's iteration. No data are published to investigate this, but I suspect many juniors playing a large number of games may have their grade based on very few rated players. IMHO this situation is worse as any residual bias is then multiplied up to get from graded players to games played.

Yes the bias in the graded opponents previously published grades will be carried through to juniors, usually to all juniors even if each adult doesn’t play all of them. Exceptions will be isolated groups of juniors who do not play outside that group. The proportions of each adult grade, which makes up the grade related part of the final result for juniors, turn up in the final columns of the expression F^N in my post of 18 Aug 2009 (, here) . In the matrix approach to graders junior iterations:-

G(N+1)= F x G(N) + P with final values G(N) = F^N x G(0) + (SUM( F^R: R=0,n-1))xP

the estimate for a junior comprises an adult grade related bit of F^NxG(0) and a grade independent bit (SUM( F^R: R=0,N-1)) x P, net of 40 rule and refeeding back steps.

The F^N part for a junior is w1 x g1 + w2 x g2+……. Wn x gn across all graded adult players, but in this instance fortunately for the graders the sum of w1+w2+ ..wn = 1, which in turn means that the bias in each juniors grade resulting from adult bias, is less than or equal to the most biased adult graded opponent and more than or equal to the least biased adult graded opponent. That gives plenty of scope for a high positive bias junior to appear much higher graded than a high negative bias junior.

For example using the fixture pattern of this cross table of results.

Code: Select all

\	a      b    c    f    Y    Z
a	X    0.5    1    1		 
b	0.5    X              1	 
c	0           X         1	 
d	0                X         1
Y	       0    0         X	 
Z	                 0         X

a,b,c,d being ungraded juniors, Y and Z being graded adults.

which gives an F matrix of

Code: Select all

    	a   	b   	c   	d   	Y   	Z
a	0.00	0.33	0.33	0.33	0.00	0.00
b	0.50	0.00	0.00	0.00	0.50	0.00
c	0.50	0.00	0.00	0.00	0.50	0.00
d	0.50	0.00	0.00	0.00	0.00	0.50
Y	0.00	0.00	0.00	0.00	1.00	0.00
Z	0.00	0.00	0.00	0.00	0.00	1.00

and F^32 of

Code: Select all

    	a   	b   	c   	d   	Y   	Z
a	0.00	0.00	0.00	0.00	0.67	0.33
b	0.00	0.00	0.00	0.00	0.83	0.17
c	0.00	0.00	0.00	0.00	0.83	0.17
d	0.00	0.00	0.00	0.00	0.33	0.67
Y	0.00	0.00	0.00	0.00	1.00	0.00
Z	0.00	0.00	0.00	0.00	0.00	1.00

showing that player “a”, who did not play rated players Y or Z will have inherited their bias in the proportions:-

a:bias = .67 x Y:bias+ .33 x Z:bias

I don’t think this bias is as significant as the pyramiding effect that can happen when juniors are rated on a smaller number of established historic grades as in the new method.

[Brian Valentine]

Michael,
I thought we were talking about the same matrix, albeit labelled differently. Can you confirm your Y and Z rows? In my version they are formed the same way as the a-d rows, as the proportion of total games by row player against column player.

[E Michael White]

Brian
If I remember correctly, your N matrix had graded players at the top. Also your matrix includes games between graded players. I defined my F matrix to cover just the new starter iteration process, in which according to Richard Hadrell, grades of graded opponents do not change from one iteration to the next.

Rows in F for graded players are 0 except for diagonal elements which are 1. P is a column matrix where P(j) = 100 x ( % score of player j )-50 for ungraded players. Rows in P for graded players are 0. These two conditions ensure grades of already graded players are not changed by the process. My matrix equation is then G(N+1)=F x G(N) + P. I couldn’t find out exactly how the new basis works so didn’t build in the 40 rule or extend the model.

[Brian Valentine]

Michael,
You are right in than my N matrix had rated players on top. I relabelled your Y and Z as A&B then Renamed a to d as c to f. Otherwise the results were copied, that is how I noticed the discrepancy. My approach ignored the 40 point rule although I do have something more sophisticated coming along.

I do have a rather rambling piece modelling how the bias feeds through, which I plan to write up. The essential feature is in my N matrix already. I think we agree that all the grades in your match are finally based on linear combinations of the starting grades of Y&Z. However my development shows the bias in posterior grades to be:
of prior Y of prior Z
Y is 0.714285714 0.285714286
Z is 0.571428571 0.428571429

a is 0.714285714 0.285714286
b is 0.571428571 0.428571429
c is 0.857142857 0.142857143
d is 0.714285714 0.285714286

I think the difference is that my N is the ultimate new grades whereas yours is at an intermediate stage - the final prior grades input into the final round. In the final stage Y&Z get feedback in that the inputs for a-d are based entirely on Y&Z. Any initial bias in Y&Z gets geared up at that point. Now I see we are not on the same page I'll speed up my paper.

[David Shepherd]

I have always had a feeling that grading disparities arose not only between adults and juniors, but also between say the North and South of the country and also mainly league players and mainly tournament players although the differences were not large. Although these pools overlap maybe in a sense each seperate pool has the characteristics of an individual player with a different bias.

[Alex Holowczak]

I have always had a feeling that grading disparities arose not only between adults and juniors, but also between say the North and South of the country and also mainly league players and mainly tournament players although the differences were not large. Although these pools overlap maybe in a sense each seperate pool has the characteristics of an individual player with a different bias.

That makes a lot of sense. Rather than just having one grading pool, there are probably several subpools within the pool. Birmingham-based players play mainly themselves, rather than London-based players, and vice versa. The grading points will therefore be passed around their respective subpools, rather than uniformly around the universal pool. That's probably the main problem for juniors who don't branch out into League chess; the points just get passed around the junior subpool, which distorts it from the adult subpool. This just happens to be the easiest distinction to notice. This could also explain why some people didn't notice the alleged stretching as much as others.

[Brian Valentine]

David,
This is indeed possible and fits under this theme of bias. Unfortunately the data to investigate this is not in the public domain and possibly the required information is not even captured.

With time on my hands at the London Classic, I begun wondering about the situation of Adams and Short (A&S). Although they wont be concerned about their ecf grades they are an example of a dislocated community within the list.

Their ecf grades over the last few years (apart from a small number of 4NCL games) are based on big international events heavily affected by foreign players: Liverpool, and Staunton Memorials for instance. The foreign players are treated very similarly to juniors for grading purposes. These overseas guy's grades are heavily influenced by Adams' and Short's grades which are used to fix theirs. In the following year they play again and A&S grades are based on these previous grades. A&S grades appear heavily influenced over a long period by any potential biases in their grades brought forward from earlier periods.

The same issue applies in other "communities", although it is then much more difficult to isolate and identify.