Some thoughts on anti-cheating systems

[Matthew Turner]

Roger,
Let’s say I measure the heights of everyone in the 4NCL they will be a normal distribution. If I get them to stand in height order and I measure them it will still be a normal distribution. I think you are confusing the polynomial distribution that we start off with, where successive results do need to be independent with the normal distribution that we end up with.

[Matthew Turner]

Roger,
I have a titled player account at Lichess and a normal account at chess.com.
I could give you access to my chess.com account which has an established history of playing at a reasonably high level and you could attempt to use your computer to mimic that level of play or push it higher. You would be cheating and you might not get caught, you could get away with cheating, but so what?

Sorry, Matt, but you'll have to explain how that is in any way relevant to the issue of calculating 'expected' performances from no data.

You are are saying that an unknown player can come in with a very high rating by cheating from the start and this makes cheat detection impossible (or cheat detection that relies on statistical evidence). OK so they can, by simply taking over my account. What does it matter?

[Adam Raoof]

Adam, in my original post I made a point of saying: "I don't for one moment doubt the integrity of Professor Regan, nor do I doubt that he has taken a scholarly approach to this. Whether, in 10 years time, it will still be regarded as the Holy Grail strikes me as rather less certain".

That's still my position and the slight qualification is because I don't doubt that, over the next decade, Ken will further refine his algorithm - in which case, with hindsight, the present version won't be seen as perfect. I was less complimentary about the Lichess and Chess.com equivalents largely on the grounds that, if they were assembled with comparable rigour to Professor Regan, there would be no need to hide them from view.

You could say the same thing about the science of fingerprinting, or DNA. It doesn't have to be flawless to meet the burden of proof.

[Adam Raoof]

I did actually ask Ken whether Lichess and Chessdot used his algorithm as the basis for their detection software.

(3) Do the websites chess.com and lichess use your algorithm as the basis for their own anti-cheatingefforts? If not, do you think what you offer is better than the efforts of those platforms? – Adam Raoof

"Their tools include statistical tests. I do not have any licensing deals with them. I do not try to compete on their turf because they avail far more information. I have known this ever since a long talk with ICC’s Daniel Sleator in Nov. 2006 and a longer partnership with one of their sysops.
My work has two levels: (1) A quick “screening test” which gives quick feedback about a tournament in progress but does not make statistical judgments, and (2) a “full test” which does but is subject to manual judgment. I let Chess.com use my screening information during the PRO League’s 2017 and 2018 seasons and acted as backup on a few of their sanctions. The screening test has no “special sauce” especially since I made it completely data-driven in Jan. 2019---it follows simple rules that could be a weekly homework problem in an undergraduate statistics course. Two differences of my full test are (i) that it does full predictive analytics and (ii) it operates with quantities that demonstrably conform well to normal distribution, so that z-scores rather than more generic “p-values” can be given up-front."

[Roger de Coverly]

They are reliable enough to use in court.

Not as sole evidence though in any practical tests.

[Adam Raoof]

I did actually ask Ken whether Lichess and Chessdot used his algorithm as the basis for their detection software.

(3) Do the websites chess.com and lichess use your algorithm as the basis for their own anti-cheatingefforts? If not, do you think what you offer is better than the efforts of those platforms? – Adam Raoof

"Their tools include statistical tests. I do not have any licensing deals with them. I do not try to compete on their turf because they avail far more information. I have known this ever since a long talk with ICC’s Daniel Sleator in Nov. 2006 and a longer partnership with one of their sysops.
My work has two levels: (1) A quick “screening test” which gives quick feedback about a tournament in progress but does not make statistical judgments, and (2) a “full test” which does but is subject to manual judgment. I let Chess.com use my screening information during the PRO League’s 2017 and 2018 seasons and acted as backup on a few of their sanctions. The screening test has no “special sauce” especially since I made it completely data-driven in Jan. 2019---it follows simple rules that could be a weekly homework problem in an undergraduate statistics course. Two differences of my full test are (i) that it does full predictive analytics and (ii) it operates with quantities that demonstrably conform well to normal distribution, so that z-scores rather than more generic “p-values” can be given up-front."

So that tells me that I can rely less on sites being data-driven (ironically, because they rely on human intervention and other parameters) than Ken's algorithm - and that alone demonstrates why it is a mistake for organizations to hitch their wagons to big sites to "host" events, their whole ethos has evolved separately from over the board chess and has to meet very different demands.

[IM Jack Rudd]

A random sample of a Normal distribution will be Normal, but a nonrandom sample probably won't be, and a nonrandom sample drawn entirely from one side of the bell certainly won't be.

[Roger Lancaster]

Roger,
Let’s say I measure the heights of everyone in the 4NCL they will be a normal distribution. If I get them to stand in height order and I measure them it will still be a normal distribution. If I get them to stand in height order and only measure the first hundred that will be a normal distribution too. I think you are confusing the polynomial distribution that we start off with, where the result do need to be independent with the normal distribution that we end up with.

Does that mean that we agree that, if successive events aren't independent [as is clearly the case in Carlsen-Anand but, I have suggested, occurs very frequently] then that requires adjustments to be made - in the manner quite properly described by Professor Regan - which are empirical? In the Carlsen-Anand case, the three moves might be respectively weighted N by 1, [N+1] by 0.5 and [N+2] by 0.25 but there's no outside reference point as to what these weightings should be so it's all down to Ken Regan's skill and expertise which here appears to be unique. As I've said several times elsewhere, my doubts principally concern the anti-cheating mechanisms of others.

[Matthew Turner]

A random sample of a Normal distribution will be Normal, but a nonrandom sample probably won't be, and a nonrandom sample drawn entirely from one side of the bell certainly won't be.

Sorry, jack my mistake of course you are correct there.
Matt

[NickFaulks]

Roger,
“ The essence of a normal distribution is that it relies on successive events being random or independent,”
This is just wrong

No it isn't. Matthew, you really need to reacquaint yourself with the Central Limit Theorem.

[John McKenna]

So, what is really being asked for here - a way to protect innocents, while purging Judas Iscariots??

That might be done with some difficulty for 4NCL & ECF online sideshows but the vast majority of playing on online platforms will remain a gigantic free-for-all.

Why should those platforms go out of their way to satisfy a small minority of their clients? (Apart from their elite clients, of course.)

Viz.

I did actually ask Ken whether Lichess and Chessdot used his algorithm as the basis for their detection software.

(3) Do the websites chess.com and lichess use your algorithm as the basis for their own anti-cheatingefforts? If not, do you think what you offer is better than the efforts of those platforms? – Adam Raoof

"Their tools include statistical tests. I do not have any licensing deals with them. I do not try to compete on their turf because they avail far more information. I have known this ever since a long talk with ICC’s Daniel Sleator in Nov. 2006 and a longer partnership with one of their sysops.
My work has two levels: (1) A quick “screening test” which gives quick feedback about a tournament in progress but does not make statistical judgments, and (2) a “full test” which does but is subject to manual judgment. I let Chess.com use my screening information during the PRO League’s 2017 and 2018 seasons and acted as backup on a few of their sanctions. The screening test has no “special sauce” especially since I made it completely data-driven in Jan. 2019---it follows simple rules that could be a weekly homework problem in an undergraduate statistics course. Two differences of my full test are (i) that it does full predictive analytics and (ii) it operates with quantities that demonstrably conform well to normal distribution, so that z-scores rather than more generic “p-values” can be given up-front."

So that tells me that I can rely less on sites being data-driven (ironically, because they rely on human intervention and other parameters) than Ken's algorithm - and that alone demonstrates why it is a mistake for organizations to hitch their wagons to big sites to "host" events, their whole ethos has evolved separately from over the board chess and has to meet very different demands.

[Matthew Turner]

Roger,
Let’s say I measure the heights of everyone in the 4NCL they will be a normal distribution. If I get them to stand in height order and I measure them it will still be a normal distribution. If I get them to stand in height order and only measure the first hundred that will be a normal distribution too. I think you are confusing the polynomial distribution that we start off with, where the result do need to be independent with the normal distribution that we end up with.

Does that mean that we agree that, if successive events aren't independent [as is clearly the case in Carlsen-Anand but, I have suggested, occurs very frequently] then that requires adjustments to be made - in the manner quite properly described by Professor Regan - which are empirical? In the Carlsen-Anand case, the three moves might be respectively weighted N by 1, [N+1] by 0.5 and [N+2] by 0.25 but there's no outside reference point as to what these weightings should be so it's all down to Ken Regan's skill and expertise which here appears to be unique. As I've said several times elsewhere, my doubts principally concern the anti-cheating mechanisms of others.

So what we are saying is that 3 or 4 matches, or non matches might be clustered together. I just don’t think this makes any difference over a sample size of say 100. Yes, I agree that we shouldn’t make a decision based on one game (using purely statistical evidence) for this very reason, but everyone agrees that.

[E Michael White]

Is Ken Regan a stats chap? I thought he was a computer science prof!

[NickFaulks]

Is Ken Regan a stats chap? I thought he was a computer science prof!

That is indeed relevant.

[John McKenna]

Ken: Yea, it'd be interesting to see how much is “commonality”. I mean it's more the mindset. One thing that impressed me at a New England sports statistics conference that I attended last September, which incidentally was co-chaired by Mark Glickman of the Glicko rating system and the USCF rating statistician, was the idea that it's not just statistics. It's analytics. It's the way that quantified, structured data is influencing alot of our thought processes, not just compiling aggregate statistics.

https://www.chess.com/blog/SamCopeland/ ... -ken-regan