Duration
2017-2019
Principal Investigator
Pablo Romero Fresco (UVigo, GALMA)
Participants
Sabela Melchor Couto (University of Roehampton, GALMA)
Funding body
Canadian Radiotelevision Telecomunications Commission
Budget
€45,000
Supporters
Background and description
An Accuracy Standard for captioning has been part of the CRTC’s Standards for Quality in Closed Captioning since 2012, but the method of assessment defined there has proved to be ineffective and unworkable. As a result, the Commission started a new proceeding on this issue. In December 2015, members of the English Broadcasters Group (EBG) met with members of the Captioning Consumer Advocacy Alliance (CCAA), in response to CRTC Broadcasting Notice of Consultation 2015-325-2, to address aspects related to a closed captioning quality standard for live television programming in Canada. The EBG and CCAA reached agreement on a common purpose, namely to improve the quality of live captioning in Canada. One of the outcomes of the discussions was to recognize that an appropriate system for measuring accuracy needs to be found and it was agreed that the NER model should be evaluated as a potential system of assessing caption accuracy in Canada. This ultimately led to a CRTC supported two-year trial of a Canadianized NER model.
In order to conduct this trial, which is similar to the ones led by GALMA in the UK and Spain, we certified 10 candidates as NER evaluators by assessing their ability to use the NER model for the review of six ten-minute excerpts of live captions from TV programmes belonging to three genres (news, chat shows and sports). The objective was to ensure that the candidates obtained an inter-rater agreement of less than 0.09 as compared to the external evaluation.
The NER-certified evaluators are now conducting the two-year trial that is intended to results in the adoption of the NER model as the official tool to assess the quality of live captions in Canada
