PhysioNet/Computing in Cardiology Challenge 2013: Top Scores

Over 60 teams participated in the 2013 Challenge on noninvasive fetal ECG. Below are the scores of the top-ranked teams. (Only 6 teams participated in event 3, and only 2 produced entries that could be scored.) Each team was allowed to submit up to 11 entries; the scores shown below are the best (lowest) received by each team for any of their entries.

Event 1 (Fetal heart rate estimation, evaluated using hidden test set C)

The score for each record was the mean squared error (in bpm2) between the fetal heart rate signals estimated from the reference and test annotations. The aggregate score was the mean of the scores for each record.

ParticipantScore
Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia and Rita Balocchi 187.091
Piotr Podziemski and Jan Gierałtowski 255.989
Rui Rodrigues 278.755
Luigi Yuri Di Marco, Alberto Marzo and Alejandro Frangi 380.853
Jakub Kuzilek and Lenka Lhotska 492.412
Joseph McBride, Brent McFerrin, Craig Towers and Xiaopeng Zhao (unofficial) 505.634
Aruna Deogire and Satish Hamde (unofficial) 580.284
Alessia Dessì, Danilo Pani and Luigi Raffo 684.158
Tian Wenlong (unofficial) 931.156
Joachim Behar, Julien Oster and Gari Clifford (unofficial) (179.439)

Event 2 (Fetal RR interval estimation, evaluated using hidden test set C)

The score for each record was the root mean squared difference (in ms) between corresponding reference and test RR intervals. The aggregate score was the mean of the scores for each record.

ParticipantScore
Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia and Rita Balocchi 20.975
Piotr Podziemski and Jan Gierałtowski 25.059
Luigi Yuri Di Marco, Alberto Marzo and Alejandro Frangi 26.085
Rui Rodrigues 28.201
Joseph McBride, Brent McFerrin, Craig Towers and Xiaopeng Zhao (unofficial) 29.680
Jakub Kuzilek and Lenka Lhotska 33.200
Akshay Dhawan (unofficial) 39.667
Andrea Fanelli, Giovanni Magenes and Maria Gabriella Signorini (unofficial) 42.859
Alessia Dessì, Danilo Pani and Luigi Raffo 47.990
Joachim Behar, Julien Oster and Gari Clifford (unofficial) (20.793)

Event 3 (Fetal QT interval estimation, evaluated using hidden test set C)

For each record, the median FQT interval was estimated. The aggregate score was the root mean squared difference (in ms) between the reference and test FQT intervals for all records.

ParticipantScore
Piotr Podziemski and Jan Gierałtowski 152.71
Joachim Behar, Julien Oster and Gari Clifford (unofficial) (153.07)

Event 4 (Fetal heart rate estimation, evaluated using open test set B)

The score for each record was the mean squared error (in bpm2) between the fetal heart rate signals estimated from the reference and test annotations. The aggregate score was the mean of the scores for each record.

ParticipantScore
Fernando Andreotti, Maik Riedl, Tilo Himmelsbach, Daniel Wedekind, Sebastian Zaunseder, Niels Wessel and Hagen Malberg 18.803
Jukka A Lipponen and Mika P Tarvainen 28.893
Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia and Rita Balocchi 33.952
Masoumeh Haghpanahi and David A Borkholder 50.063
Minnan Xu-Wilson, Eric Carlson, Limei Cheng and Srinivasan Vairavan 52.496
Ali Ghaffari, Seyyed Abbas Atyabi, Mohammad Javad Mollakazemi, Maryam Niknami and Ali Soleiman 63.750
Mantas Lukoševičius and Vaidotas Marozas 66.327
Martin Kropf, Günter Schreier, Robert Modre-Osprian and Dieter Hayn 82.438
Joachim Behar, Julien Oster and Gari Clifford (unofficial) (29.619)

Event 5 (Fetal RR interval estimation, evaluated using open test set B)

The score for each record was the root mean squared difference (in ms) between corresponding reference and test RR intervals. The aggregate score was the mean of the scores for each record.

ParticipantScore
Fernando Andreotti, Maik Riedl, Tilo Himmelsbach, Daniel Wedekind, Sebastian Zaunseder, Niels Wessel and Hagen Malberg 4.337
Jukka A Lipponen and Mika P Tarvainen 4.844
Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia and Rita Balocchi 5.098
Martin Kropf, Günter Schreier, Robert Modre-Osprian and Dieter Hayn 7.354
Mantas Lukoševičius and Vaidotas Marozas 8.239
Masoumeh Haghpanahi and David A Borkholder 9.062
Christoph Maier and Hartmut Dickhaus 9.326
Minnan Xu-Wilson, Eric Carlson, Limei Cheng and Srinivasan Vairavan 10.618
Vito Starc 10.920
Joachim Behar, Julien Oster and Gari Clifford (unofficial) (4.672)

How the entries were scored

Entries were scored using tach, mxm, ann2rr, and custom software developed for the challenge. For each event, the software processed the reference annotations and the test annotations (i.e., those generated by the participant’s entry) to obtain a score for each test record, then calculated an average score for all records in the test set.

Participants had access to training set A (both signals and reference annotations) and open test set B (signals only); these data sets remain freely available. Reference annotations for set B, and both the signals and annotations for hidden test set C, will be kept hidden, to allow fair comparisons between entries submitted during the challenge and those developed in followup studies by challenge participants as well as other interested researchers.

Note on unofficial participants

Teams listed above as "unofficial" participated in the Challenge and achieved excellent results.

Entries from the team of Joachim Behar, Julien Oster, and Gari Clifford were unofficial since their authors were among the challenge organizers, and had major roles in developing the challenge questions and data sets.

Other teams listed as unofficial were unable to attend CinC 2013 and discuss their work, a condition of eligibility for Challenge awards.