Some thoughts on anti-cheating systems

Discuss anything you like about chess related matters in this forum.
Post Reply
User avatar
JustinHorton
Posts: 7540
Joined: Mon Aug 04, 2008 10:06 am
Location: Somewhere you're not

Re: Some thoughts on anti-cheating systems

Post by JustinHorton » Sat May 30, 2020 4:30 pm

Roger Lancaster wrote:
Sat May 30, 2020 4:10 pm
The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results.
I am not entirely happy that this is a correct conclusion, but I do not see how I challenge it without at least in some way discussing the case concerned.
Roger Lancaster wrote:
Sat May 30, 2020 4:10 pm
Similarly, when people join Lichess, they are invited to supply a national or international grade/rating. Many newbies either won't have, or won't supply, this item of information so one has to ask - how does Lichess arrive at expected outcomes? Or, to put it another way, how do they know how well a totally unknown person plays chess?
This doesn't just apply to unknown players. One of my points during my struggle with chess.com was that they had no accurate idea of how well I played the form of chess in which I'd been taking part.
"Do you play chess?"
"Yes, but I prefer a game with a better chance of cheating."

lostontime.blogspot.com

Roger Lancaster
Posts: 971
Joined: Tue Mar 17, 2015 2:44 pm

Re: Some thoughts on anti-cheating systems

Post by Roger Lancaster » Sat May 30, 2020 4:33 pm

JustinHorton wrote:
Sat May 30, 2020 4:30 pm
Roger Lancaster wrote:
Sat May 30, 2020 4:10 pm
The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results.
I am not entirely happy that this is a correct conclusion, but I do not see how I challenge it without at least in some way discussing the case concerned.
Roger Lancaster wrote:
Sat May 30, 2020 4:10 pm
Similarly, when people join Lichess, they are invited to supply a national or international grade/rating. Many newbies either won't have, or won't supply, this item of information so one has to ask - how does Lichess arrive at expected outcomes? Or, to put it another way, how do they know how well a totally unknown person plays chess?
This doesn't just apply to unknown players. One of my points during my struggle with chess.com was that they had no accurate idea of how well I played the form of chess in which I'd been taking part.
Justin, thanks for showing sensitivity on the first point. On the second, as I think you will already have worked out, I agree.

John McKenna
Posts: 4137
Joined: Tue May 17, 2011 2:02 pm

Re: Some thoughts on anti-cheating systems

Post by John McKenna » Sat May 30, 2020 4:48 pm

Roger L>The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes.<

Don't know about that because -

The z-M test is a commonsense criterion expressed in probabilistic terms. In general, a performance (actual) may be regarded as exceptional if the probability of its is less than 10%, or better yet, less than 5%. (A. Elo)

E.g. an exception(al) performance over 9 games for a player with K=32 would be a gain (or loss) of 62 rating points at a 10% probability threshold and a gain (loss) of 79 rating points at a 5% probability threshold.
To find a for(u)m that accommodates the mess, that is the task of the artist now. (Samuel Beckett)

Matthew Turner
Posts: 2905
Joined: Fri May 16, 2008 11:54 am

Re: Some thoughts on anti-cheating systems

Post by Matthew Turner » Sat May 30, 2020 4:53 pm

Roger Lancaster wrote:
Sat May 30, 2020 4:10 pm

I'd be a tad more cautious than Matt and, since this is something I seem to have omitted from my earlier post in an effort to keep it to a manageable length, I'll try to explain why here. Some while ago, a previously completely unknown teenager emerged on the English chess scene playing to a standard where his ECF grade, converted to ELO, was a shade over 2400. Naturally, questions were asked. Further discussion on this forum was ruled prejudicial and the topic closed. I urge everyone not to reopen it as I cite it only as an extreme example of the relevant type of occurrence.

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results
You could have saved yourself some time and just written the last sentence. If you have the Pgns for a player’s games in an event you can perform a Regan test, if you don’t, you can’t.

Roger Lancaster
Posts: 971
Joined: Tue Mar 17, 2015 2:44 pm

Re: Some thoughts on anti-cheating systems

Post by Roger Lancaster » Sat May 30, 2020 4:55 pm

John McKenna wrote:
Sat May 30, 2020 4:48 pm
Roger L>The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes.<

Don't know about that because -

The z-M test is a commonsense criterion expressed in probabilistic terms. In general, a performance (actual) may be regarded as exceptional if the probability of its is less than 10%, or better yet, less than 5%. (A. Elo)

E.g. an exception performance over 9 games for a player with K=32 would be a gain (or loss) of 62 rating points at a 10% probability threshold and a gain (loss) of 79 rating points at a 5% probability threshold.
John - put, perhaps more succinctly, one can't decide whether someone's performance is exceptional or not without knowing whether that person is a 7-year-old novice or Magnus Carlsen out to have fun under an anonymous username.

Roger Lancaster
Posts: 971
Joined: Tue Mar 17, 2015 2:44 pm

Re: Some thoughts on anti-cheating systems

Post by Roger Lancaster » Sat May 30, 2020 5:03 pm

Matthew Turner wrote:
Sat May 30, 2020 4:53 pm
Roger Lancaster wrote:
Sat May 30, 2020 4:10 pm

I'd be a tad more cautious than Matt and, since this is something I seem to have omitted from my earlier post in an effort to keep it to a manageable length, I'll try to explain why here. Some while ago, a previously completely unknown teenager emerged on the English chess scene playing to a standard where his ECF grade, converted to ELO, was a shade over 2400. Naturally, questions were asked. Further discussion on this forum was ruled prejudicial and the topic closed. I urge everyone not to reopen it as I cite it only as an extreme example of the relevant type of occurrence.

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results
You could have saved yourself some time and just written the last sentence. If you have the Pgns for a player’s games in an event you can perform a Regan test, if you don’t, you can’t.
Maybe it's just the phraseology but I'm not sure we agree. If one has all a player's game scores in a particular event, but no more information about that player, the Regan test will tell you that he played to such-and-such a level but no more than that because it doesn't have the further information needed to determine the player's expected level and, consequently, what deviation there might be between the two.

John McKenna
Posts: 4137
Joined: Tue May 17, 2011 2:02 pm

Re: Some thoughts on anti-cheating systems

Post by John McKenna » Sat May 30, 2020 5:05 pm

JustinHorton wrote:
Sat May 30, 2020 4:20 pm
John McKenna wrote:
Sat May 30, 2020 4:16 pm
4NCL OTB is a social as well as a competitive team league.

4NCL O/l is an asocial and therefore highly likely to be more competitive than the OTB league.
What?
Have you not heard of the infamous 4NCL OTB morning game after the social the night before??

4NCL O/l is a hunting ground for young wolf-warriors, by contrast.

MartinCarpenter
Posts: 2604
Joined: Tue May 24, 2011 10:58 am

Re: Some thoughts on anti-cheating systems

Post by MartinCarpenter » Sat May 30, 2020 5:18 pm

Kevin Thurlow wrote:
Sat May 30, 2020 4:20 pm
I assume I told Lichess my FIDE-rating, if they asked, but they still estimated me as 1500 when I started playing. I even got congratulated by a club colleague for beating an 1800...
There’s actually a non trivial danger here. There’s plenty of clubs who have gone online just now and are playing as closed groups with everyone starting at 1500.

The resulting grades are obviously entirely unreliable to misleading.

Matthew Turner
Posts: 2905
Joined: Fri May 16, 2008 11:54 am

Re: Some thoughts on anti-cheating systems

Post by Matthew Turner » Sat May 30, 2020 5:28 pm

Roger Lancaster wrote:
Sat May 30, 2020 5:03 pm
Matthew Turner wrote:
Sat May 30, 2020 4:53 pm
Roger Lancaster wrote:
Sat May 30, 2020 4:10 pm

I'd be a tad more cautious than Matt and, since this is something I seem to have omitted from my earlier post in an effort to keep it to a manageable length, I'll try to explain why here. Some while ago, a previously completely unknown teenager emerged on the English chess scene playing to a standard where his ECF grade, converted to ELO, was a shade over 2400. Naturally, questions were asked. Further discussion on this forum was ruled prejudicial and the topic closed. I urge everyone not to reopen it as I cite it only as an extreme example of the relevant type of occurrence.

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes. In this case, because the teenager had no previous 'form', there was no legitimate basis for calculating or even estimating expected outcomes. Without expected outcomes, there was no way of arriving at the difference between these and the actual outcomes. No data equates to no results
You could have saved yourself some time and just written the last sentence. If you have the Pgns for a player’s games in an event you can perform a Regan test, if you don’t, you can’t.
Maybe it's just the phraseology but I'm not sure we agree. If one has all a player's game scores in a particular event, but no more information about that player, the Regan test will tell you that he played to such-and-such a level but no more than that because it doesn't have the further information needed to determine the player's expected level and, consequently, what deviation there might be between the two.
Well I just suggest you have a think about it. Work out if any pgns for a whole event exist in the case you are talking about. Pick a rating you want to use to start off and use your imagination to work out what the result will be.

Roger de Coverly
Posts: 18644
Joined: Tue Apr 15, 2008 2:51 pm

Re: Some thoughts on anti-cheating systems

Post by Roger de Coverly » Sat May 30, 2020 5:33 pm

Roger Lancaster wrote:
Sat May 30, 2020 4:10 pm

The significance of this case was that statistical methods alone, far from determining that he was a cheat, could not even put a probability on this being true. That's because all statistical methods of which I am aware rely on differences between actual outcomes and expected outcomes.
There was a result in one event which was so bad as to cast doubt on the legitimacy of all the others.

For all we know lichess and chess.com ban players for attempted sandbagging.

Matthew Turner
Posts: 2905
Joined: Fri May 16, 2008 11:54 am

Re: Some thoughts on anti-cheating systems

Post by Matthew Turner » Sat May 30, 2020 5:38 pm

Roger,
What like actually looking at their terms and conditions?

From Lichess

See the following for some examples of what you shouldn’t do.

2. Artificially inflating or deflating your rating. This is where a User purposefully loses, or has arranged with an opponent to win. As a result, the User’s rating will artificially increase or decrease.

Roger Lancaster
Posts: 971
Joined: Tue Mar 17, 2015 2:44 pm

Re: Some thoughts on anti-cheating systems

Post by Roger Lancaster » Sat May 30, 2020 5:47 pm

Matthew Turner wrote:
Sat May 30, 2020 5:28 pm

Well I just suggest you have a think about it. Work out if any pgns for a whole event exist in the case you are talking about. Pick a rating you want to use to start off and use your imagination to work out what the result will be.
Over-the-board events, if one can be reasonably sure that a player isn't cheating, are one thing - one can then assess 'true playing strength' from someone's moves. However, I was talking about moves played in an online tournament where - for all Lichess et al know - someone could, for example, have had a grandmaster looking over their shoulder and guiding them. Those games wouldn't form a sound basis for expected playing strength.

Matthew Turner
Posts: 2905
Joined: Fri May 16, 2008 11:54 am

Re: Some thoughts on anti-cheating systems

Post by Matthew Turner » Sat May 30, 2020 6:30 pm

Roger,
If I understand you correctly, you are now talking about a hypothetical example which has nothing to do with anything that is happening in British Chess or anything that has happened historically. You’ve got me, I don’t know.

John McKenna
Posts: 4137
Joined: Tue May 17, 2011 2:02 pm

Re: Some thoughts on anti-cheating systems

Post by John McKenna » Sat May 30, 2020 6:47 pm

I'd like to repeat - if we all could check our online anti-cheating credit/debit scores with the various chess orgs it might be a place to start!?

It may also be worth repeating (not sure about everyone seeing everyone else's details but in at least one Scandinavian country that is possible regarding pay, I believe) -
David Guthrie wrote:
Sat May 30, 2020 2:53 pm
The session last night was very interesting.

I don't understand why GDPR prevents the 4NCL publishing the Z-scores of all the players in the 4NCL online. Any publication should come with a disclaimer that the Z-scores are statistics only and whilst they may create a very strong statistical inference that computer assistance was used, they don't prove that it was.

The players names, ratings, and the game scores are already in the public domain. As I understand it, these are the main inputs that Regan uses to calculate the Z-scores, and then he just some fancy analysis of them, using a method that may or may not be in the public domain as well. There is a legitimate interest in processing this data to detect cheating, and presumably it could be replicated by anyone understood Regan's method sufficiently well to replicate it.

Alternatively, if publishing Z-scores breaches GDPR, then why doesn't publishing ECF Grades, FIDE ratings, Chess.com Move Accuracy Scores, or LiChess centipawn loss scores, for any players who are named by their real name? As they are all essentially the same thing i.e. numbers calculated on the basis of raw game data.
To find a for(u)m that accommodates the mess, that is the task of the artist now. (Samuel Beckett)

Richard Bates
Posts: 3067
Joined: Fri Nov 14, 2008 8:27 pm

Re: Some thoughts on anti-cheating systems

Post by Richard Bates » Sat May 30, 2020 7:09 pm

John McKenna wrote:
Sat May 30, 2020 6:47 pm
I'd like to repeat - if we all could check our online anti-cheating credit/debit scores with the various chess orgs it might be a place to start!?

It may also be worth repeating (not sure about everyone seeing everyone else's details but in at least one Scandinavian country that is possible regarding pay, I believe) -
We could ditch any need for a grading system as well! "Z-score" would be the new test of playing strength ;)

OK (and note I haven't paid anything but the most cursory attention to what it actually is) so there is the problem potentially of people playing deliberately bad moves in completely winning positions, without actually jeopardising the chances of a positive result...

Post Reply