Some thoughts on anti-cheating systems

Discuss anything you like about chess related matters in this forum.
MartinCarpenter
Posts: 2680
Joined: Tue May 24, 2011 10:58 am

Re: Some thoughts on anti-cheating systems

Post by MartinCarpenter » Mon Jun 01, 2020 1:24 pm

NickFaulks wrote:
Mon Jun 01, 2020 11:46 am
The technical reason is that the Central Limit Theorem, in its simplest form, is based on adding independent variables. Whether you get Q6 of a maths exam right is not independent of whether you get Q7 right - they will have a clear tendency to go together.

In practical terms, let's suppose we find a mean of 70/100 and a sd of 20. Wouldn't you expect a cluster in the 90-100 range, these being the children who can do sums? There will be a long tail at the other end, depending on whether you get punished for guessing wrong.
That last is slightly out - we're much closer to examining the question of what happens if the same child does the same (or similar) exam over and over.

So raw ability is (kind of!) taken out of the equation, the worries are more things like getting disconcerted by having made an earlier mistake.
(I would actually be surprised if someone hadn't done roughly this experiment with exams.).

Roger Lancaster
Posts: 1068
Joined: Tue Mar 17, 2015 2:44 pm

Re: Some thoughts on anti-cheating systems

Post by Roger Lancaster » Mon Jun 01, 2020 2:03 pm

Paul McKeown wrote:
Mon Jun 01, 2020 1:12 am

It's a neural net, which needs to be trained by examining games:
There's probably a very simple answer to this question but, if Paul is correct in stating that Lichess use a neural net, would I be correct in thinking that the statistical approaches discussed here wouldn't apply, which would open up a rather different discussion?

John McKenna
Posts: 4327
Joined: Tue May 17, 2011 2:02 pm

Re: Some thoughts on anti-cheating systems

Post by John McKenna » Mon Jun 01, 2020 2:23 pm

Matthew Turner wrote:
Mon Jun 01, 2020 10:30 am
NickFaulks wrote:
Mon Jun 01, 2020 9:40 am
Adam Raoof wrote:
Sun May 31, 2020 1:49 pm
I did actually ask Ken

.....

(ii) it operates with quantities that demonstrably conform well to normal distribution, so that z-scores rather than more generic “p-values” can be given up-front."
That sums up the whole discussion. Most people on this forum seem to believe that this demonstration exists and is convincing, but I cannot find anyone who has actually seen it.
Nick,
I have a question for you. The idea is that chess is modeled in the same way as a multiple choice exam. Would you accept that if I took results from the 2004 GCSE Maths exam (in terms of marks rather than grades) that they would accurately approximate to a normal distribution?
Matt
Before Roger L and others go off into a discussion of computer science (analytics next is it?) I'll just add a couple of notes (from Wikipedia) on the statistical one

"The Chi-square test is intended to test how likely it is that an observed distribution is due to chance. It is also called a "goodness of fit" statistic, because it measures how well the observed distribution of data fits with the distribution that is expected if the variables are independent."

And -

"Chi-square Test for Normality. The chi-square goodness of fit test can be used to test the hypothesis that data comes from a normal hypothesis."

(Arpad Elo used the above test in his work and I'm sure Ken Regan is aware of it even if it has escaped our attention up to now.)

I'd like to ask if anyone here has any formal (or even an informal) relationship (apart from playing) with any online gaming platform?
Last edited by John McKenna on Mon Jun 01, 2020 2:43 pm, edited 4 times in total.
To find a for(u)m that accommodates the mess, that is the task of the artist now. (Samuel Beckett)

Joseph Conlon
Posts: 123
Joined: Thu Jun 06, 2019 4:18 pm

Re: Some thoughts on anti-cheating systems

Post by Joseph Conlon » Mon Jun 01, 2020 2:27 pm

Roger Lancaster wrote:
Mon Jun 01, 2020 2:03 pm

There's probably a very simple answer to this question but, if Paul is correct in stating that Lichess use a neural net, would I be correct in thinking that the statistical approaches discussed here wouldn't apply, which would open up a rather different discussion?
Neural nets are often used with the aim of producing statistical significances; one example is in particle physics, where neural nets are trained on particle collisions, and used to evaluate the statistical significance of certain signals (for example, that a Higgs boson event with a certain decay channel occurred).

Matthew Turner
Posts: 3136
Joined: Fri May 16, 2008 11:54 am

Re: Some thoughts on anti-cheating systems

Post by Matthew Turner » Mon Jun 01, 2020 2:43 pm

John McKenna wrote:
Mon Jun 01, 2020 2:23 pm

(Arpad Elo used the above test in his work and I'm sure Ken Regan is aware of it even if it has escaped our attention up to now.)
The two things are very closely linked and there is quite a bit about the ELO system on Ken Regan's blog, Godel lost letter.
The Elo system is underpinned by the idea that 95% of performances are within 2 standard deviations of the mean (400 points)

NickFaulks
Posts: 6079
Joined: Sat Jan 02, 2010 1:28 pm

Re: Some thoughts on anti-cheating systems

Post by NickFaulks » Mon Jun 01, 2020 2:48 pm

MartinCarpenter wrote:
Mon Jun 01, 2020 1:24 pm
That last is slightly out - we're much closer to examining the question of what happens if the same child does the same (or similar) exam over and over.
But that wasn't the question I was asked.

Anyway, I've had enough of this. We have recently been introduced to the concept of scarce dependency, which I think must be borrowed from computing, and now a dependence matrix has popped out of the box. I shall be impressed if anyone in the audience can give an explanation of how that works.
If you want a picture of the future, imagine a boot stamping on a human face — forever.

John McKenna
Posts: 4327
Joined: Tue May 17, 2011 2:02 pm

Re: Some thoughts on anti-cheating systems

Post by John McKenna » Mon Jun 01, 2020 2:53 pm

Well, Matt, Godel scotched any hope that you can prove much if anything at all - even wth arithmetic. I guess it all comes down how much faith you're prepared to put in any system.

Don't blame Nick F for throwing in the equivalent of the Mystery of the Turin Shroud...

https://www.huffpost.com/entry/shroud-o ... _n_5960832
To find a for(u)m that accommodates the mess, that is the task of the artist now. (Samuel Beckett)

Matthew Turner
Posts: 3136
Joined: Fri May 16, 2008 11:54 am

Re: Some thoughts on anti-cheating systems

Post by Matthew Turner » Mon Jun 01, 2020 3:09 pm

I have some sympathy with Nick.
In my opinion, the strength of Ken Regan's program is that the concept is actually very easy to understand. As he has refined it then it has got more complex and and less easy for a non-specialist (like myself) to understand.
You get a z score by adding up the move matches taking away the expected results and dividing by sigma (the standard deviation)
That works for tossing a coin where the results are independent, but but isn't 100% accurate for chess.
I think it is accurate enough if you have enough games, so maybe it would have been better had Ken Regan hadn't refined it further.
How does a dependence matrix work, I don't know exactly, but in essence when you are adding up the move matches and you have a number in a row rather than adding 1+1+1 maybe you add 1+0.85+0.8

User avatar
Michael Farthing
Posts: 1965
Joined: Fri Apr 04, 2014 1:28 pm
Location: Morecambe, Europe

Re: Some thoughts on anti-cheating systems

Post by Michael Farthing » Tue Jun 02, 2020 10:37 am

As a matter of interest perhaps for those who have discussed whether a normal distribution can be expected in exam results I can say that that it cannot necessarily. I speak as an ex exam officer with a national exam board. Part of my job (pre GCSE) was to decide where grade boundaries should be drawn and I was supplied with a distributon curve for the exam in question, together with the same for previous years and the decision taken in those past cases. All sorts of factors could alter the distribution and I remember one particular exam where the scewed tail at the bottom of the distribution curve for the project work was dramatically different from the previous year. The person presenting me with the figures was quite visibly annoyed with me that I wouldn't make a decision - he was wanting to start the processing. The syllabus was one I had not previously dealt with and I went to consult colleagues, who similarly scratched their heads in puzzlement until eventually one said, "Oh there's been a change - you no longer automatically fail if you don't submit a project." Which provided a convincing explanation - poorer candidates no longer bothered handing in a scrappy bit of paper which was barely a bona fide attempt at a project - though it didn't make the job of making a fair decision any easier. But other things affect it too - for example a particularly hard question or one that has not been written as carefully as might be can easily produce a distortion. The top end is quite likely to have a much smaller tail simply because you can't get a mark above 100%. - there tends to be more room at the bottom end. But that too depends on the syllabus, the character of the questions and the marking scheme.

John McKenna
Posts: 4327
Joined: Tue May 17, 2011 2:02 pm

Re: Some thoughts on anti-cheating systems

Post by John McKenna » Tue Jun 02, 2020 11:30 am

The practical example of a procedure that uses statistics to help grade (including determining failing grade?) given above could also be of interest from a theoretical standpoint.

However, without seeing the distribution curves, and the data they were based on, it is just an interesting anecdote, but one that is still superior to idle speculation about the possible distribution of exam results.

One thing is clear though - the education system itself has always been and continues to be SKEWED!

There's much less room at the top - and that is a reflection of the society that fosters it. (Perhaps that's why some people resort to cheating.)
To find a for(u)m that accommodates the mess, that is the task of the artist now. (Samuel Beckett)

Mike Gunn
Posts: 785
Joined: Wed Apr 11, 2007 4:45 pm

Re: Some thoughts on anti-cheating systems

Post by Mike Gunn » Tue Jun 02, 2020 12:07 pm

As a university teacher I used to plot the distribution of exam marks and I very rarely got a normal distribution. My distributions more commonly resembled a two humped camel with maxima in the mark ranges 40-55 and 70-85. When we fed the marks into university marks system it would calculate mean marks and standard deviations for every subject and my strange marks would turn out to be satisfactory from the university administration's point of view. No other teacher I knew did similar plots and so although I suspect this effect was widespread it seemed to escape general notice. I now have emeritus status at the university and receive all the departmental emails. This summer all students at the university did their exams online (and open book). There is some discussion that marks will be adjusted to correspond to earlier expected distributions by changing the mean mark and standard deviation of each subject. I anticipate that this process will result in a few surprises.

Matthew Turner
Posts: 3136
Joined: Fri May 16, 2008 11:54 am

Re: Some thoughts on anti-cheating systems

Post by Matthew Turner » Tue Jun 02, 2020 1:00 pm

Just to clarify, I was talking about a Maths exams and Ken Regan was linking his analysis to multiple choice exams. Of course there are all sort of other factors when the marking is subjective.

John McKenna
Posts: 4327
Joined: Tue May 17, 2011 2:02 pm

Re: Some thoughts on anti-cheating systems

Post by John McKenna » Tue Jun 02, 2020 1:08 pm

Mike G>... There is some discussion that marks will be adjusted to correspond to earlier expected distributions by changing the mean mark and standard deviation of each subject. I anticipate that this process will result in a few surprises.<

Both pleasant and unpleasant surprises can it be assumed?
To find a for(u)m that accommodates the mess, that is the task of the artist now. (Samuel Beckett)

User avatar
Michael Farthing
Posts: 1965
Joined: Fri Apr 04, 2014 1:28 pm
Location: Morecambe, Europe

Re: Some thoughts on anti-cheating systems

Post by Michael Farthing » Tue Jun 02, 2020 5:04 pm

Mike Gunn wrote:
Tue Jun 02, 2020 12:07 pm
As a university teacher I used to plot the distribution of exam marks and I very rarely got a normal distribution. My distributions more commonly resembled a two humped camel with maxima in the mark ranges 40-55 and 70-85. .
The measurement of this camel hump property I was taught has both a name and a moment! It is called kutosis and is the fourth moment after mean, deviation and skew.

Just tought I'd add some totally useless information to lighten the mood.

David Sedgwick
Posts: 4131
Joined: Mon Apr 09, 2007 5:56 pm
Location: Croydon

Re: Some thoughts on anti-cheating systems

Post by David Sedgwick » Tue Jun 02, 2020 5:14 pm

Michael Farthing wrote:
Tue Jun 02, 2020 5:04 pm
Just tought I'd add some totally useless information to lighten the mood.
I hope that you don't do that when you are chairing the online ECF Finance Council Meeting on Saturday 18th July. It needs to finish in time for Council members to play their games in the ECF Online Counties Championships in the evening.

Post Reply