Specification for ECF LMS API ?

General discussions about grading.
Roger de Coverly
Posts: 18092
Joined: Tue Apr 15, 2008 2:51 pm

Re: Specification for ECF LMS API ?

Post by Roger de Coverly » Mon Aug 26, 2019 12:24 am

Brian Towers wrote:
Mon Aug 26, 2019 12:20 am
How about using the format of that Excel file as one of the acceptable tournament report formats?
I thought Swiss Manager had already been customised to produce an ECF compatible grading file output. That presumes the organiser had been bothered enough to ensure the ECF grading codes were present.

Brian Towers
Posts: 1216
Joined: Tue Nov 18, 2014 7:23 pm

Re: Specification for ECF LMS API ?

Post by Brian Towers » Mon Aug 26, 2019 10:07 am

Roger de Coverly wrote:
Mon Aug 26, 2019 12:24 am
Brian Towers wrote:
Mon Aug 26, 2019 12:20 am
How about using the format of that Excel file as one of the acceptable tournament report formats?
I thought Swiss Manager had already been customised to produce an ECF compatible grading file output. That presumes the organiser had been bothered enough to ensure the ECF grading codes were present.
Exactly. However if the role of regional graders is to be phased out (which seems to be the direction of travel) then arbiters/organizers should be able to continue as currently by emailing the Excel file, just to one central email address rather than to their local grader. Alternatively they could log in to the ECF's LMS system and "upload file" or equivalent.
Ah, but I was so much older then. I'm younger than that now.

Alex Holowczak
Posts: 8907
Joined: Sat May 30, 2009 5:18 pm
Location: Oldbury, Worcestershire
Contact:

Re: Specification for ECF LMS API ?

Post by Alex Holowczak » Mon Aug 26, 2019 11:27 am

Brian Towers wrote:
Mon Aug 26, 2019 10:07 am
Alternatively they could log in to the ECF's LMS system and "upload file" or equivalent.
That's how it works in Ireland, I understand. Arbiters just push the button in their favourite pairing software, log in to the rating server, and upload the file. The role of the central administrator then becomes just tidying up, rather than actually doing the donkey work of the processing.

Ian Thompson
Posts: 2124
Joined: Wed Jul 02, 2008 4:31 pm
Location: Awbridge, Hampshire

Re: Specification for ECF LMS API ?

Post by Ian Thompson » Mon Aug 26, 2019 11:36 am

Alex Holowczak wrote:
Mon Aug 26, 2019 11:27 am
Brian Towers wrote:
Mon Aug 26, 2019 10:07 am
Alternatively they could log in to the ECF's LMS system and "upload file" or equivalent.
That's how it works in Ireland, I understand. Arbiters just push the button in their favourite pairing software, log in to the rating server, and upload the file. The role of the central administrator then becomes just tidying up, rather than actually doing the donkey work of the processing.
But how do they ensure players have been correctly identified? That's the fundamental issue the ECF needs to solve. The details of how results are reported are unimportant in comparison.

Alex Holowczak
Posts: 8907
Joined: Sat May 30, 2009 5:18 pm
Location: Oldbury, Worcestershire
Contact:

Re: Specification for ECF LMS API ?

Post by Alex Holowczak » Mon Aug 26, 2019 12:16 pm

Ian Thompson wrote:
Mon Aug 26, 2019 11:36 am
Alex Holowczak wrote:
Mon Aug 26, 2019 11:27 am
Brian Towers wrote:
Mon Aug 26, 2019 10:07 am
Alternatively they could log in to the ECF's LMS system and "upload file" or equivalent.
That's how it works in Ireland, I understand. Arbiters just push the button in their favourite pairing software, log in to the rating server, and upload the file. The role of the central administrator then becomes just tidying up, rather than actually doing the donkey work of the processing.
But how do they ensure players have been correctly identified? That's the fundamental issue the ECF needs to solve. The details of how results are reported are unimportant in comparison.
At the moment, the system is terrible at ensuring players are correctly identified.

Let's take the Leyland Congress, which is ongoing. Let's say their organiser generates the grading file tonight. He'd then check it against the grading list on 22nd July, and send it off to Matthew. I expect they will perfectly match up people with grading codes, and the checker will pick up any mistakes made to that point and corrections will be made. Existing players aren't the problem.

The problem is new players. Leyland will be using the grading list published on 22nd July, and so their "new" player may actually be an existing player who has already played in tournaments over the past 5 weeks. (Imagine a busier time of the year, say February or March, and this is more of an issue given there are large numbers of junior tournaments at that time, and thus large numbers of new players.) There's actually nothing we can do at the moment about that, other than generate the duplicate ID when Matthew processes it, and then check the feedback we get and instruct Matthew to merge players who were, in fact, not new after all.

This is much more labour intensive than other systems. As part of the upload process, you check your submission against the live data, so any new players who aren't new get detected at that stage. So far from being a problem, it's actually much more efficient than the current process if it's done properly.

Paul Cooksey
Posts: 436
Joined: Fri Oct 21, 2016 4:15 pm

Re: Specification for ECF LMS API ?

Post by Paul Cooksey » Mon Aug 26, 2019 12:31 pm

if only the ECF had another more reliable way of identifying players...

John McKenna
Posts: 3722
Joined: Tue May 17, 2011 2:02 pm

Re: Specification for ECF LMS API ?

Post by John McKenna » Mon Aug 26, 2019 1:15 pm

To find a for(u)m that accommodates the mess, that is the task of the artist now. (Samuel Beckett)

Ian Thompson
Posts: 2124
Joined: Wed Jul 02, 2008 4:31 pm
Location: Awbridge, Hampshire

Re: Specification for ECF LMS API ?

Post by Ian Thompson » Mon Aug 26, 2019 1:23 pm

Alex Holowczak wrote:
Mon Aug 26, 2019 11:27 am

At the moment, the system is terrible at ensuring players are correctly identified.
It would be hard to disagree with that.
Alex Holowczak wrote:
Mon Aug 26, 2019 11:27 am
As part of the upload process, you check your submission against the live data, so any new players who aren't new get detected at that stage. So far from being a problem, it's actually much more efficient than the current process if it's done properly.
Certainly an improvement, but not a solution.

It's fine if it correctly says my new player Joe Bloggs is actually an existing Joe Bloggs, so I'm to use their existing code.

It's not fine if it incorrectly says my new player Joe Bloggs is actually an existing Joe Bloggs, so I'm to use their existing code, when they are actually two different people. I had a case this season where a new player started playing for a club and was incorrectly matched with a former player from the same club with the same name. Not common, but it happens.

The most common situation at the moment is that the system gives several possible matches with existing players and then requires manual investigation to determine if any of them are the right match.

How will this be automatically dealt with, for example:

Code: Select all

New Player - Pin _100235: Khan, Shahbaz   (4172 - Imperial College London)
Possible Matches:
  30%  127530G  Khan, S   Club - 9006 - Cosmopolitan Banks
  27%  216600J  Khoon, K   (Grade - 104S in 2002) Club - 4172 - Imperial College London
  26%  214374E  Kan, S   Club - XXXX - Club Unknown
  20%  322402J  Khan, Saim  xx/xx/xxxx Clubs - 3398 - Q Elizabeth School Barnet, 0280 - London         *
  20%  320417A  Khan, Seth  xx/xx/xxxx (Grade - 32R in 2018) Club - 1300 - Oxfordshire Juniors
  20%  309952A  Khan, Sana  xx/xx/xxxx (Grade - 57R in 2018) Clubs - 2HFS - Hallfield School, 2WAJ - Warwickshire Juniors
New code generated - 326285G
(I've blanked out the DoBs for the players who had them.)

Neill Cooper
Posts: 1239
Joined: Sun Feb 24, 2008 4:43 pm
Location: Croydon
Contact:

Re: Specification for ECF LMS API ?

Post by Neill Cooper » Mon Aug 26, 2019 3:40 pm

Alex Holowczak wrote:
Mon Aug 26, 2019 12:16 pm
The problem is new players. Leyland will be using the grading list published on 22nd July, and so their "new" player may actually be an existing player who has already played in tournaments over the past 5 weeks. (Imagine a busier time of the year, say February or March, and this is more of an issue given there are large numbers of junior tournaments at that time, and thus large numbers of new players.) There's actually nothing we can do at the moment about that, other than generate the duplicate ID when Matthew processes it, and then check the feedback we get and instruct Matthew to merge players who were, in fact, not new after all.
The software does spot such duplicates! It happened a lot when I submitted the rapidplay and standard play grading files for the Briant Poulter (Surrey Secondary School) League. A new player would often appear in both files and on the second file processed the match would be spotted automatically by the software. Obviously in this case the name, date of birth and club (School) were identical so it was a 100% match. Similarly I submitted a grading file this July and just the exact match of name and Date of Birth (70% match) was enough for him to be given the correct grading code, only recently created.

So yes, it would be better if the grader could check against live data. But the software is already doing a pretty good job of that.

Alex Holowczak
Posts: 8907
Joined: Sat May 30, 2009 5:18 pm
Location: Oldbury, Worcestershire
Contact:

Re: Specification for ECF LMS API ?

Post by Alex Holowczak » Mon Aug 26, 2019 5:18 pm

Ian Thompson wrote:
Mon Aug 26, 2019 1:23 pm
How will this be automatically dealt with, for example:

Code: Select all

New Player - Pin _100235: Khan, Shahbaz   (4172 - Imperial College London)
Possible Matches:
  30%  127530G  Khan, S   Club - 9006 - Cosmopolitan Banks
  27%  216600J  Khoon, K   (Grade - 104S in 2002) Club - 4172 - Imperial College London
  26%  214374E  Kan, S   Club - XXXX - Club Unknown
  20%  322402J  Khan, Saim  xx/xx/xxxx Clubs - 3398 - Q Elizabeth School Barnet, 0280 - London         *
  20%  320417A  Khan, Seth  xx/xx/xxxx (Grade - 32R in 2018) Club - 1300 - Oxfordshire Juniors
  20%  309952A  Khan, Sana  xx/xx/xxxx (Grade - 57R in 2018) Clubs - 2HFS - Hallfield School, 2WAJ - Warwickshire Juniors
New code generated - 326285G
(I've blanked out the DoBs for the players who had them.)
It's not up to me to say how it will be dealt with in whatever the ECF does!

However, what the FIDE system does is when you upload the file, you're presented with a list of all of the possible matches, and you select from the list which one it is. If it isn't, then you choose that option. Then you move on to the next one, and proceed down the list. It's easy.

Alex Holowczak
Posts: 8907
Joined: Sat May 30, 2009 5:18 pm
Location: Oldbury, Worcestershire
Contact:

Re: Specification for ECF LMS API ?

Post by Alex Holowczak » Mon Aug 26, 2019 5:27 pm

Neill Cooper wrote:
Mon Aug 26, 2019 3:40 pm
Alex Holowczak wrote:
Mon Aug 26, 2019 12:16 pm
The problem is new players. Leyland will be using the grading list published on 22nd July, and so their "new" player may actually be an existing player who has already played in tournaments over the past 5 weeks. (Imagine a busier time of the year, say February or March, and this is more of an issue given there are large numbers of junior tournaments at that time, and thus large numbers of new players.) There's actually nothing we can do at the moment about that, other than generate the duplicate ID when Matthew processes it, and then check the feedback we get and instruct Matthew to merge players who were, in fact, not new after all.
The software does spot such duplicates! It happened a lot when I submitted the rapidplay and standard play grading files for the Briant Poulter (Surrey Secondary School) League. A new player would often appear in both files and on the second file processed the match would be spotted automatically by the software. Obviously in this case the name, date of birth and club (School) were identical so it was a 100% match. Similarly I submitted a grading file this July and just the exact match of name and Date of Birth (70% match) was enough for him to be given the correct grading code, only recently created.

So yes, it would be better if the grader could check against live data. But the software is already doing a pretty good job of that.
That's not the issue. The system is good at that. Let's pick a random secondary school child in London, who is new to the game, but has played a bit online so is good enough for the school team. They are venturing out into their first competitive tournaments, let's say they have the following schedule:
- Poplar Rapidplay, 7th September, organised by Norman Went
- Golders Green Rapidplay, 14th September, organised by Adam Raoof
- ECF Secondary Schools Tournament, 15th September, organised by Neill Cooper
- North London Schools Grand Prix 1, 21st September, organised by Angela Eyton
- CCF Autumn Rapidplay, 28th September, organised by Scott Freeman

They enter all of these, and the organiser looks them up on the August masterlist, where they can't find the player, so they assume he is new. All five organisers send off their grading files to Matthew as soon as they can. There is no update to the masterlist during this period, and so they get reported as new players. Matthew doesn't do any prior inspection of grading submissions, so he will process the files and send you some feedback. At that point, it's up to everyone except Norman - who we'll assume got in first! - to read this feedback and observe that the player is a duplicate. Sadly, there is no guarantee that every grader will do this (I'm sure those on this list will!), and so a duplicate will be created, and not identified until the September update is published. But some duplicates can go months without being noticed, and in that time, we've clogged up the grading system with four grading references we can't use anymore.

There are still tales told in the West Midlands of the seven versions of of Craig Whitfield that appeared on a published grading list when this happened back when he was starting out.

Neill Cooper
Posts: 1239
Joined: Sun Feb 24, 2008 4:43 pm
Location: Croydon
Contact:

Re: Specification for ECF LMS API ?

Post by Neill Cooper » Mon Aug 26, 2019 10:22 pm

Alex, has it changed since Richard Haddrell was grading administrator?

In 2013 I sent him 2 parallel grading files. That evening he processed both of them. Each player was duplicated with the same pin in both files.
First one:
Sheet 2 Cell A20: New Player - Pin 152: yyyy, xxx
Sheet 2 Cell A20: Possible Matches:
Sheet 2 Cell A20: 44% etc
Sheet 2 Cell A20: New code generated - 29394xx

Second one:
Sheet 2 Cell A20: New Player - Pin 152: yyyy, xxx
Sheet 2 Cell A20: Possible Matches:
Sheet 2 Cell A20: 80% 29394xxx
Sheet 2 Cell A20: Code to be used is 29394xx

So when the second file was procesed the software recognised that the player was on the first file submitted.

My imperssion was that whilst graders, when running the checker program, do not know of the most recent additions to the mastermasterlist, they just have the monthly masterlist, the grading software has its own mastermasterlist which is updated after every grading file is processed.

Perhaps we should check with the grading administrator?

Ian Thompson
Posts: 2124
Joined: Wed Jul 02, 2008 4:31 pm
Location: Awbridge, Hampshire

Re: Specification for ECF LMS API ?

Post by Ian Thompson » Mon Aug 26, 2019 11:03 pm

Alex Holowczak wrote:
Mon Aug 26, 2019 5:18 pm
Ian Thompson wrote:
Mon Aug 26, 2019 1:23 pm
How will this be automatically dealt with, for example:

Code: Select all

New Player - Pin _100235: Khan, Shahbaz   (4172 - Imperial College London)
Possible Matches:
  30%  127530G  Khan, S   Club - 9006 - Cosmopolitan Banks
  27%  216600J  Khoon, K   (Grade - 104S in 2002) Club - 4172 - Imperial College London
  26%  214374E  Kan, S   Club - XXXX - Club Unknown
  20%  322402J  Khan, Saim  xx/xx/xxxx Clubs - 3398 - Q Elizabeth School Barnet, 0280 - London         *
  20%  320417A  Khan, Seth  xx/xx/xxxx (Grade - 32R in 2018) Club - 1300 - Oxfordshire Juniors
  20%  309952A  Khan, Sana  xx/xx/xxxx (Grade - 57R in 2018) Clubs - 2HFS - Hallfield School, 2WAJ - Warwickshire Juniors
New code generated - 326285G
(I've blanked out the DoBs for the players who had them.)
It's not up to me to say how it will be dealt with in whatever the ECF does!

However, what the FIDE system does is when you upload the file, you're presented with a list of all of the possible matches, and you select from the list which one it is. If it isn't, then you choose that option. Then you move on to the next one, and proceed down the list. It's easy.
Well it wouldn't be easy with the player above. I'd have no way of knowing, without asking the league secretary and/or the club whether Khan, Shahbaz might or might not be Khan, S, and I also wouldn't rule out the possibility of him being Kan, S with a misspelt name.

Obviously, Kan,S should never have been allowed onto the grading list without any of date of birth, sex, club, area or visible indication of the event(s) in which they played.

Alex Holowczak
Posts: 8907
Joined: Sat May 30, 2009 5:18 pm
Location: Oldbury, Worcestershire
Contact:

Re: Specification for ECF LMS API ?

Post by Alex Holowczak » Tue Aug 27, 2019 10:27 am

Ian Thompson wrote:
Mon Aug 26, 2019 11:03 pm
Alex Holowczak wrote:
Mon Aug 26, 2019 5:18 pm
Ian Thompson wrote:
Mon Aug 26, 2019 1:23 pm
How will this be automatically dealt with, for example:

Code: Select all

New Player - Pin _100235: Khan, Shahbaz   (4172 - Imperial College London)
Possible Matches:
  30%  127530G  Khan, S   Club - 9006 - Cosmopolitan Banks
  27%  216600J  Khoon, K   (Grade - 104S in 2002) Club - 4172 - Imperial College London
  26%  214374E  Kan, S   Club - XXXX - Club Unknown
  20%  322402J  Khan, Saim  xx/xx/xxxx Clubs - 3398 - Q Elizabeth School Barnet, 0280 - London         *
  20%  320417A  Khan, Seth  xx/xx/xxxx (Grade - 32R in 2018) Club - 1300 - Oxfordshire Juniors
  20%  309952A  Khan, Sana  xx/xx/xxxx (Grade - 57R in 2018) Clubs - 2HFS - Hallfield School, 2WAJ - Warwickshire Juniors
New code generated - 326285G
(I've blanked out the DoBs for the players who had them.)
It's not up to me to say how it will be dealt with in whatever the ECF does!

However, what the FIDE system does is when you upload the file, you're presented with a list of all of the possible matches, and you select from the list which one it is. If it isn't, then you choose that option. Then you move on to the next one, and proceed down the list. It's easy.
Well it wouldn't be easy with the player above. I'd have no way of knowing, without asking the league secretary and/or the club whether Khan, Shahbaz might or might not be Khan, S, and I also wouldn't rule out the possibility of him being Kan, S with a misspelt name.

Obviously, Kan,S should never have been allowed onto the grading list without any of date of birth, sex, club, area or visible indication of the event(s) in which they played.
I think it's pretty easy to conclude that it's "None of the Above", or if it is one of the above, then it really doesn't matter.

It clearly isn't the bottom three, and the top 3 are all ungraded and have been for a while, if not ever. Does it matter if they need to be merged subsequently because one of them is Shahbaz? Not really. Again, players like this aren't the problem - it's the new players who play multiple events between masterlists.

Brian Valentine
Posts: 434
Joined: Fri Apr 03, 2009 1:30 pm

Re: Specification for ECF LMS API ?

Post by Brian Valentine » Fri Sep 20, 2019 4:37 pm

I think I have enough extra information to give an update.

Those in the know will have seen a consultation I have started with the graders about what should be held in the live player list. Suffice to say some good points have been made and I'm in discussions with various parties how to best meet them. There is still time for graders to respond to that mail, but I hope to have an updated proposal in a few days. Behind my rose tinted glasses, I am hopeful a live list overcomes most problems mentioned in this thread. Ideas on what to do with "new" players are not so advanced yet.

The other main issue is how the ECF LMS system will automatically upload results. There is a big meeting mid October and I hope that some specification will be available then. When that is settled I expect to start talking to John Upham and other interested parties.

I'm not sure why there is concern over the registration system. From the workload point of view I expect the local registrar (who may or may not be the grader) to have new work. Some of the detail will be that now given on the top page of a grading submission. Other entries might help everyone with information aiding game fee issues and voting register. Nevertheless we do need a schedule for when we expect a grading submission as a control as not everyone will submit monthly, at least to start with. I will be consulting graders and organisers on this issue in the near future.

As for the future of graders, I think they will be around for a while. I hope their ongoing workload will be eased over time to take up a more audit role rather than paper pushing.

Taking off those rose tinted glasses, I think that graders and others contact me by email if they have issues, rather than expect me to keep up to date on ecforum.

Brian Valentine
Manager ECF Grading

Post Reply