Christie Christelis (Technology Strategies International)
Deborah Fels (Ryerson University)
Pilar Orero (UAB)
Pablo Romero-Fresco (UVigo, GALMA)
Canadian Broadcasting Accessibility Fund
€95, 245 euros
Television is an almost ubiquitous medium that provides entertainment and vital information to millions of households in Canada. Closed captioning of live television programs, which is particularly challenging, extends the accessibility of these programs to the Deaf, deafened, hard-of-hearing and hearing communities.
An Accuracy Standard for captioning has been part of the CRTC’s Standards for Quality in Closed Captioning since 2012, but the method of assessment defined there has proved to be ineffective and unworkable. As a result, the Commission started a new proceeding on this issue. In December 2015, members of the English Broadcasters Group (EBG) met with members of the Captioning Consumer Advocacy Alliance (CCAA), in response to CRTC Broadcasting Notice of Consultation 2015-325-2, to address aspects related to a closed captioning quality standard for live television programming in Canada. The EBG and CCAA reached agreement on a common purpose, namely to improve the quality of live captioning in Canada. One of the outcomes of the discussions was to recognise that an appropriate system for measuring accuracy needs to be found and it was agreed that the NER model should be evaluated as a potential system of assessing caption accuracy in Canada. This ultimately led to a CRTC supported two-year trial of a Canadianised NER model.
One of the challenges recognised in search for system to evaluate live captioning quality and accuracy was that there was a lack of substantive research into consumers’ subjective reactions to, and preference for, different forms of live captioning in Canada, which was considered an important factor limiting the ability to improve accessibility in live television programming. The CCAA, with EBG support, decided to pursue an independent research initiative aimed at establishing users’ preferences and responses to different aspects of live captioning in Canada. A proposal to conduct such research was submitted to the Broadcasting Accessibility Fund (BAF). The application for funding was successful and the project, titled Understanding User Responses to Live Closed Captioning in Canada, was formally launched in November 2016.
This project was born out of the need to gain insight into consumers’ subjective reactions to, and preferences for, different aspects of live captioning in Canada, in order inform a broader understanding of how to improve the quality of live captioning in Canada. The project was funded by the Broadcasting Accessibility Fund (BAF) and governed by a Steering Committee comprising of members of the Captioning Consumer Advocacy Alliance (CCAA), broadcasters and captioners.
The scope of the project included a qualitative research phase, in which the research design was undertaken, a quantitative research phase, in which Canadian deaf, hard-of hearing and hearing captioning consumers responded to an online survey, and a reporting and dissemination phase.
In total, 550 responses were received for the survey, 330 from the deaf and hard-of-hearing communities and 220 from the hearing community. Respondents to the survey were asked a series of questions on: viewing behaviour, the importance of live captioning attributes, their satisfaction and experience of captioned live TV segments, and demographics. The questionnaire was available in English and ASL.
Live captioning is important across all genres, but news/weather achieved the highest importance rating across the four genres included in the survey. Satisfaction with the general state of live captioning in Canada is moderately high, although respondents from the deaf and hard-of-hearing communities are significantly less satisfied with the quality of live captioning in Canada than those from the hearing community.
Captioning speed was ranked as the most important attribute amongst the hearing and hard-of-hearing respondents, followed by captions not obscuring important information on screen, the meaning of what was said not being changed, and that there should be little or no delay between what is said on screen and the captions appearing. Respondents from the deaf community prioritised low delay and important onscreen information not being blocked by captions above all other live captioning attributes. Captioning every word spoken on screen was their third most important attribute, and significantly more important than maintaining the integrity of meaning. Caption speed was the fourth most important attribute to deaf respondents.
Overall, respondents across all hearing groups were ambivalent about the quality of the live captioning of video segments presented to them, with almost as many respondents registering a sense of dissatisfaction as respondents who registered satisfaction across the different genres and captioning methods. There was also little to differentiate between video segments in terms of overall satisfaction ratings.
When evaluating specific captioned live video segments, more than half of all respondents rated ‘Caption Legibility’, ‘Spelling and Grammar’ and ‘Placement of Captions’ highly, while ‘Speaker Identification’ (where relevant), ‘Delay’ and ‘Caption Speed’ received the lowest performance ratings.
In the key driver analysis, ‘Delay’ emerged as the most important driver of viewer satisfaction across all video segments, regardless of hearing status (i.e. hearing vs. hard-of-hearing vs. deaf). ‘Caption Speed’ and the correct ‘Spelling and Grammar’ vied for second place, but this was dependent on genre of the captioned segment.
This survey of was the first of its kind in Canada and has provided some important insights into user responses to live closed captioning in Canada. The responses recorded in this survey reaffirmed that there are three pivotal captioning attributes of importance to users, namely, delay, caption speed and accuracy in some shape or form, and that performance against some of these variables is not as high as users would want. Although the survey does not identify a clear path for navigating this difficult terrain, it does suggest that if delay and caption speed are to be kept at acceptable levels, an approach that maintains the integrity of the meaning of the spoken word may prove to be an effective trade-off.