Reliability of judging in DanceSport

Article


Premelč, J., Vuckovic, G., James, N. and Leskošek, B. 2019. Reliability of judging in DanceSport. Frontiers in Psychology. 10. https://doi.org/10.3389/fpsyg.2019.01001
TypeArticle
TitleReliability of judging in DanceSport
AuthorsPremelč, J., Vuckovic, G., James, N. and Leskošek, B.
Abstract

Purpose The aim of this study was to assess the reliability and validity of the new judging system in DanceSport. Methods Eighteen judges rated the 12 best placed adult dancing couples competing at an international competition. They marked each couple on all judging criteria on a 10 level scale. Absolute agreement and consistency of judging were calculated for all main judging criteria and sub-criteria. Results A mean correlation of overall judging marks was 0.48. Kendall’s coefficient of concordance for overall marks (W = 0.58) suggesting relatively low agreement among judges. Slightly lower coefficients were found for the artistic part [Partnering skills (W = 0.45) and Choreography and performance (W = 0.49)] compared to the technical part [Technical qualities (W = 0.56) and Movement to music (W = 0.54)]. ICC for overall criteria was low for absolute agreement [ICC(2,3) = 0.62] but higher for consistency [ICC(3,3) = 0.80]. Conclusion The relatively large differences between judges’ marks suggest that judges either disagreed to some extent on the quality of the dancing or used the judging scale in different ways. The biggest concern was standard error of measurement (SEM) which was often larger than the difference between dancers scores suggesting that this judging system lacks validity. This was the first research to assess judging in DanceSport and offers suggestions to potentially improve both its objectivity and validity in the future.

KeywordsPsychology, DanceSport, ballroom dance, judging system, reliability, validity, aesthetic sports
PublisherFrontiers Research Foundation
JournalFrontiers in Psychology
ISSN1664-1078
Publication dates
Print07 May 2019
Publication process dates
Deposited21 May 2019
Accepted15 Apr 2019
Output statusPublished
Publisher's version
License
Copyright Statement

Copyright © 2019 Jerneja Premelc, Goran Vuckovic, Nic James and Bojan Leskošek. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use,distribution or reproduction is permitted which does not comply with these terms.

Additional information

Article number = 1001
** From Frontiers via Jisc Publications Router
** History: collection 2019;
received 12-02-2019;
accepted 15-04-2019;
epub 07-05-2019.
** Licence for this article: http://creativecommons.org/licenses/by/4.0/

Digital Object Identifier (DOI)https://doi.org/10.3389/fpsyg.2019.01001
LanguageEnglish
Permalink -

https://repository.mdx.ac.uk/item/8849y

Download files


Publisher's version
  • 82
    total views
  • 55
    total downloads
  • 1
    views this month
  • 0
    downloads this month

Export as

Related outputs

Expected Pass Turnovers (xPT) - a model to analyse turnovers from passing events in football
Peters, A., Parmar, N., Davies, M., Reeves, M., Sormaz, M. and James, N. 2024. Expected Pass Turnovers (xPT) - a model to analyse turnovers from passing events in football. Journal of Sports Sciences. https://doi.org/10.1080/02640414.2024.2379697
The influence of situational variables on the attacking process in football
Kim, J., James, N., Gwon, U., Kim, B., Parmar, N., Pocrnjič, M. and Vučković, G. 2024. The influence of situational variables on the attacking process in football. Kinesiologia Slovenica. 30 (1), pp. 127-140. https://doi.org/10.52165/kinsi.30.1.127-140
Evaluating goal threat in football using player and ball locations
Nordahl, O., Kim, J., James, N., Gwon, U., Parmar, N., Nicholls, S. and Vuckovic, G. 2024. Evaluating goal threat in football using player and ball locations. Kinesiologia Slovenica. 29 (3), pp. 26-48. https://doi.org/10.52165/kinsi.29.3.26-48
The importance of appearing trustworthy in the workplace: performance analysts’ perspectives
Nelson, L., James, N., Nicholls, S., Parmar, N. and Groom, R. 2023. The importance of appearing trustworthy in the workplace: performance analysts’ perspectives. Sport, Education and Society. https://doi.org/10.1080/13573322.2023.2288844
Performance analysis practice within Olympic and Paralympic sports: a comparison of coach and analyst experiences
Nicholls, S., James, N., Wells, J. and Parmar, N. 2022. Performance analysis practice within Olympic and Paralympic sports: a comparison of coach and analyst experiences. International Journal of Performance Analysis in Sport. 22 (3), pp. 343-351. https://doi.org/10.1080/24748668.2022.2054623
The attacking process in football: a taxonomy for classifying how teams create goal scoring opportunities using a case study of Crystal Palace FC
Kim, J., James, N., Parmar, N., Ali, B. and Vučković, G. 2019. The attacking process in football: a taxonomy for classifying how teams create goal scoring opportunities using a case study of Crystal Palace FC. Frontiers in Psychology. 10, pp. 1-8. https://doi.org/10.3389/fpsyg.2019.02202
The effect of the return of serve on the server pair’s movement parameters and rally outcome in padel using cluster analysis
Ramón-Llin, J., Guzmán, J., Llana, S., Martínez-Gallego, R., James, N. and Vučković, G. 2019. The effect of the return of serve on the server pair’s movement parameters and rally outcome in padel using cluster analysis. Frontiers in Psychology. 10, pp. 1-8. https://doi.org/10.3389/fpsyg.2019.01194
Using a situation awareness approach to identify differences in the performance profiles of the world’s top two squash players and their opponents
Murray, S., James, N., Perš, J., Mandeljc, R. and Vučković, G. 2019. Using a situation awareness approach to identify differences in the performance profiles of the world’s top two squash players and their opponents. Frontiers in Psychology. 10, pp. 1-11. https://doi.org/10.3389/fpsyg.2019.01036
A retrospective investigation on age and gender differences of injuries in DanceSport
Premelč, J., Vučković, G., James, N. and Dimitriou, L. 2019. A retrospective investigation on age and gender differences of injuries in DanceSport. International Journal of Environmental Research and Public Health. 16 (21). https://doi.org/10.3390/ijerph16214164
Determining unstable game states to aid the identification of perturbations in football
Kim, J., James, N., Parmar, N., Ali, B. and Vučković, G. 2019. Determining unstable game states to aid the identification of perturbations in football. International Journal of Performance Analysis in Sport. 19 (3), pp. 302-312. https://doi.org/10.1080/24748668.2019.1602439
The implementation of performance analysis and feedback within Olympic sport: the performance analyst’s perspective
Nicholls, S., James, N., Bryant, E. and Wells, J. 2019. The implementation of performance analysis and feedback within Olympic sport: the performance analyst’s perspective. International Journal of Sports Science & Coaching. 14 (1), pp. 63-71. https://doi.org/10.1177/1747954118808081
Using principal component analysis to develop performance indicators in professional rugby league
Parmar, N., James, N., Hearne, G. and Jones, B. 2018. Using principal component analysis to develop performance indicators in professional rugby league. International Journal of Performance Analysis in Sport. 18 (6), pp. 938-949. https://doi.org/10.1080/24748668.2018.1528525
Team performance indicators that predict match outcome and points difference in professional rugby league
Parmar, N., James, N., Hughes, M., Jones, H. and Hearne, G. 2017. Team performance indicators that predict match outcome and points difference in professional rugby league. International Journal of Performance Analysis in Sport. 17 (6), pp. 1044-1056. https://doi.org/10.1080/24748668.2017.1419409
Elite coaches’ use and engagement with performance analysis within Olympic and Paralympic sport
Nicholls, S., James, N., Bryant, E. and Wells, J. 2018. Elite coaches’ use and engagement with performance analysis within Olympic and Paralympic sport. International Journal of Performance Analysis in Sport. 18 (5), pp. 764-779. https://doi.org/10.1080/24748668.2018.1517290
Using a situation awareness approach to determine decision-making behaviour in squash
Murray, S., James, N., Perš, J., Mandeljc, R. and Vučković, G. 2018. Using a situation awareness approach to determine decision-making behaviour in squash. Journal of Sports Sciences. 36 (12), pp. 1415-1422. https://doi.org/10.1080/02640414.2017.1389485
The science of sport: Squash
Murray, S., Hughes, M., James, N. and Vučković, G. 2016. The science of sport: Squash. Wiltshire The Crowood Press Ltd..
Effects of rule changes on physical demands and shot characteristics of elite-standard men’s squash and implications for training
Murray, S., James, N., Hughes, M., Perš, J., Mandeljc, R. and Vučković, G. 2016. Effects of rule changes on physical demands and shot characteristics of elite-standard men’s squash and implications for training. Journal of Sports Sciences. 34 (23), pp. 2170-2174. https://doi.org/10.1080/02640414.2016.1216155
Skill acquisition in squash
Logan, O. and James, N. 2016. Skill acquisition in squash. in: The Science of Sport: Squash Wiltshire The Crowood Press Ltd.. pp. 172-185
Does SAQ training improve the speed and flexibility of young soccer players? A randomized controlled trial
Milanovic, Z., Sporiš, G., Trajković, N., Sekulić, D., James, N. and Vučković, G. 2014. Does SAQ training improve the speed and flexibility of young soccer players? A randomized controlled trial. Human Movement Science. 38, pp. 197-208. https://doi.org/10.1016/j.humov.2014.09.005
Analysis of dance postures in latin-american dances using annotation and Sagit tracking
Zaletel, P., Vučković, G., Dimitriou, L., James, N., Rebula, A. and Zagorc, M. 2011. Analysis of dance postures in latin-american dances using annotation and Sagit tracking. 6th International Scientific Conference on Kinesiology. Opatija, Croatia 08 - 11 Sep 2011 pp. 577
Determinants of Olympic fencing performance and implications for strength and conditioning training
Turner, A., James, N., Dimitriou, L., Greenhalgh, A., Moody, J., Fulcher, D., Mias, E. and Kilduff, L. 2014. Determinants of Olympic fencing performance and implications for strength and conditioning training. The Journal of Strength and Conditioning Research. 28 (10), pp. 3001-3011. https://doi.org/10.1519/JSC.0000000000000478
A time-motion analysis of turns performed by highly ranked Viennese waltz dancers
Prosen, J., James, N., Dimitriou, L., Perš, J. and Vučković, G. 2013. A time-motion analysis of turns performed by highly ranked Viennese waltz dancers. Journal of Human Kinetics. 37 (1). https://doi.org/10.2478/hukin-2013-0025
Bone mineral density, rib pain and other features of the female athlete triad in elite lightweight rowers
Dimitriou, L., Weiler, R., Lloyd-Smith, R., Turner, A., Heath, L., James, N. and Reid, A. 2014. Bone mineral density, rib pain and other features of the female athlete triad in elite lightweight rowers. BMJ Open. 4 (2), pp. 1-9. https://doi.org/10.1136/bmjopen-2013-004369
Graded shuttle run performance by playing positions in elite female basketball
Štrumbelj, B., Vuckovic, G., Jakovljevic, S., Milanovic, Z., James, N. and Erculj, F. 2015. Graded shuttle run performance by playing positions in elite female basketball. The Journal of Strength and Conditioning Research. 29 (3), pp. 793-799. https://doi.org/10.1519/JSC.0000000000000673
A new method for assessing squash tactics using 15 court areas for ball locations
Vučković, G., James, N., Hughes, M., Murray, S., Milanović, Z., Perš, J. and Sporiš, G. 2014. A new method for assessing squash tactics using 15 court areas for ball locations. Human Movement Science. 34, pp. 81-90. https://doi.org/10.1016/j.humov.2014.01.002
Relationship between repeated sprint ability and aerobic capacity in professional soccer players
Jones, R., Cook, C., Kilduff, L., Milanovic, Z., James, N., Sporiš, G., Fiorentini, B., Turner, A. and Vuckovic, G. 2013. Relationship between repeated sprint ability and aerobic capacity in professional soccer players. The Scientific World Press.
Fitness profiles of elite Portuguese rugby union players
Vaz, L., Morais, T., Rocha, H. and James, N. 2014. Fitness profiles of elite Portuguese rugby union players. Journal of Human Kinetics. 41 (1), pp. 235-244. https://doi.org/10.2478/hukin-2014-0051
Notational analysis of corner kicks in English Premier League Soccer.
Taylor, J., James, N. and Mellalieu, S. 2005. Notational analysis of corner kicks in English Premier League Soccer. in: Reilly, T., Cabri, J. and Araújo, D. (ed.) Science and Football V: the proceedings of the Fifth World Congress on Science and Football. Routledge. pp. 229-234
Comparative movement analysis of the winners and the losers of the rallies in squash.
James, N. and Vuckovic, G. 2009. Comparative movement analysis of the winners and the losers of the rallies in squash. in: Juhas, I. and Koprivica, V. (ed.) Book of abstracts: international scientific conference of theoretical, methodological and methodical aspects of competition and athletes’ preparation. University of Belgrade, Faculty of Sport and Physical Education. pp. 42-42
An automatic tracking analysis of the movement velocities of national level basketball guards, forwards and centres.
Vuckovic, G., Dezman, B., Perse, M., Kristan, M., Pers, J., Kovacic, S. and James, N. 2009. An automatic tracking analysis of the movement velocities of national level basketball guards, forwards and centres. in: Juhas, I. and Koprivica, V. (ed.) Book of abstracts of the international scientific conference of theoretical, methodological and methodical aspects of competition and athletes’ preparation. University of Belgrade, Faculty of Sport and Physical Education.
The efficiency and ergonomics of different data entry systems in real-time and lapsed-time computer notation systems
Hughes, M., Fuller, O., Murray, S., James, N. and Vuckovic, G. 2010. The efficiency and ergonomics of different data entry systems in real-time and lapsed-time computer notation systems. The 5th International Christmas Sport Scientific Conference: qualitative and quantitative research in sport science.. Szombathely, Hungary 12 - 14 Dec 2010
The use of Zone 14 as a strategic attacking area in the English Championship
Rees, G., James, N., Hughes, M., Taylor, J. and Vuckovic, G. 2010. The use of Zone 14 as a strategic attacking area in the English Championship. The 5th International Christmas Sport Scientific Conference: qualitative and quantitative research in sport science.. Szombathely, Hungary 12 - 14 Dec 2010
An objective method for depicting team performance in elite professional rugby union.
Jones, N., James, N. and Mellalieu, S. 2008. An objective method for depicting team performance in elite professional rugby union. Journal of Sports Sciences. 26 (7), pp. 691-700. https://doi.org/10.1080/02640410701815170
The influence of match location, quality of opposition, and match status on technical performance in professional association football.
Taylor, J., Mellalieu, S., James, N. and Shearer, D. 2008. The influence of match location, quality of opposition, and match status on technical performance in professional association football. Journal of Sports Sciences. 26 (9), pp. 885-895. https://doi.org/10.1080/02640410701836887
Reliability procedures for categorical data in performance analysis.
James, N., Taylor, J. and Stanley, S. 2007. Reliability procedures for categorical data in performance analysis. International Journal of Performance Analysis in Sport. 7 (1), pp. 1-11.
The statistical analysis of golf performance.
James, N. 2007. The statistical analysis of golf performance. International Journal of Sports Science & Coaching. 2, pp. 231-249. https://doi.org/10.1260/174795407789705424
Approach shot accuracy as a performance indicator for US PGA tour golf professionals
James, N. and Rees, G. 2008. Approach shot accuracy as a performance indicator for US PGA tour golf professionals. International Journal of Sports Science & Coaching. 3, pp. 145-160. https://doi.org/10.1260/174795408785024225
Notational analysis in soccer: past, present and future.
James, N. 2006. Notational analysis in soccer: past, present and future. International Journal of Performance Analysis in Sport. 6 (2), pp. 67-81.
An exercise protocol that simulates the activity patterns of elite junior squash.
Kingsley, M., James, N., Kilduff, L., Dietzig, R. and Dietzig, B. 2006. An exercise protocol that simulates the activity patterns of elite junior squash. Journal of Sports Sciences. 24 (12), pp. 1291-1296. https://doi.org/10.1080/02640410500497766
The role of notational analysis in soccer coaching.
James, N. 2006. The role of notational analysis in soccer coaching. International Journal of Sports Science & Coaching. 1 (2), pp. 185-198. https://doi.org/10.1260/174795406777641294
Human processes of control: tracing the goals and strategies of control room teams.
Patrick, J., James, N. and Ahmed, A. 2006. Human processes of control: tracing the goals and strategies of control room teams. Ergonomics. 49 (12-13), pp. 1395-1414. https://doi.org/10.1080/00140130600613042
Observational assessment of situation awareness, team differences and training implications.
Patrick, J., James, N., Ahmed, A. and Halliday, P. 2006. Observational assessment of situation awareness, team differences and training implications. Ergonomics. 49 (4), pp. 393-417. https://doi.org/10.1080/00140130600576272
The development of position-specific performance indicators in professional rugby union.
James, N., Mellalieu, S. and Jones, N. 2005. The development of position-specific performance indicators in professional rugby union. Journal of Sports Sciences. 23 (1), pp. 63-72. https://doi.org/10.1080/02640410410001730106
A comparison of individual and unit tactical behaviour and team strategy in professional soccer.
Taylor, J., Mellalieu, S. and James, N. 2005. A comparison of individual and unit tactical behaviour and team strategy in professional soccer. International Journal of Performance Analysis in Sport. 5 (2), pp. 87-101.
Game location effects in professional soccer: a case study.
Tucker, W., Mellalieu, S., James, N. and Taylor, J. 2005. Game location effects in professional soccer: a case study. International Journal of Performance Analysis in Sport. 5 (2), pp. 23-35.
Team performance indicators as a function of winning and losing in rugby union.
Jones, N., Mellalieu, S. and James, N. 2004. Team performance indicators as a function of winning and losing in rugby union. International Journal of Performance Analysis in Sport. 4 (1), pp. 61-71.
Possession as a performance indicator in soccer.
Jones, P., James, N. and Mellalieu, S. 2004. Possession as a performance indicator in soccer. International Journal of Performance Analysis in Sport. 4 (1), pp. 98-104.
The effects of creatine supplementation on cardiovascular, metabolic, and thermoregulatory responses during exercise in the heat in endurance-trained humans.
Kilduff, L., Georgiades, E., James, N., Minnion, R., Mitchell, M., Kingsmore, D., Hadjicharlambous, M. and Pitsiladis, Y. 2004. The effects of creatine supplementation on cardiovascular, metabolic, and thermoregulatory responses during exercise in the heat in endurance-trained humans. International Journal of Sports Nutrition and Exercise Metabolism. 14 (4), pp. 446-463.
Process tracing of complex cognitive work tasks.
Patrick, J. and James, N. 2004. Process tracing of complex cognitive work tasks. Journal of Occupational and Organizational Psychology. 77 (2), pp. 259-280. https://doi.org/10.1348/096317904774202171
Passing in the Soccer World Cup 2002.
Scoulding, A., James, N. and Taylor, J. 2004. Passing in the Soccer World Cup 2002. International Journal of Performance Analysis in Sport. 4 (2), pp. 36-41.
Behavioural comparisons of positional demands in professional soccer.
Taylor, J., Mellalieu, S. and James, N. 2004. Behavioural comparisons of positional demands in professional soccer. International Journal of Performance Analysis in Sport. 4 (1), pp. 81-97.
Analysis of strategies in soccer as a function of European and domestic competition.
James, N., Mellalieu, S. and Hollely, C. 2002. Analysis of strategies in soccer as a function of European and domestic competition. International Journal of Performance Analysis in Sport. 2 (1), pp. 85-103.
Educational strategies used in increasing fluid intake and enhancing hydration status in field hockey players preparing for competition in a hot and humid environment: a case study.
Dabinett, J., Reid, K. and James, N. 2001. Educational strategies used in increasing fluid intake and enhancing hydration status in field hockey players preparing for competition in a hot and humid environment: a case study. International Journal of Sport Nutrition and Exercise Metabolism. 11 (3), pp. 334-348.
Tactical use of the T area in squash by players of differing standard.
Vuckovic, G., Pers, J., James, N. and Hughes, M. 2009. Tactical use of the T area in squash by players of differing standard. Journal of Sports Sciences. 27 (8), pp. 863-871. https://doi.org/10.1080/02640410902926412
Performance analysis of golf: reflections on the past and a vision of the future.
James, N. 2009. Performance analysis of golf: reflections on the past and a vision of the future. International Journal of Performance Analysis in Sport. 9 (2), pp. 188-209.
Measurement error associated with the SAGIT/Squash computer tracking software.
Vuckovic, G., Pers, J., James, N. and Hughes, M. 2010. Measurement error associated with the SAGIT/Squash computer tracking software. European Journal of Sport Science. 10 (2), pp. 129-140. https://doi.org/10.1080/17461390903311927
Situation variable effects and tactical performance in professional association football
Taylor, J., Mellalieu, S., James, N. and Barter, P. 2010. Situation variable effects and tactical performance in professional association football. International Journal of Performance Analysis in Sport. 10 (3), pp. 255-269.
Training to break the barriers of habit in reasoning about unusual faults.
Patrick, J., Grainger, L., Gregov, A., Halliday, P., Handley, J., James, N. and O'Reilly, S. 1999. Training to break the barriers of habit in reasoning about unusual faults. Journal of Experimental Psychology: Applied. 5 (3), pp. 314-335. https://doi.org/10.1037/1076-898X.5.3.314