The following tables include the performance measures (as explained in the overview page) both for the supplied baseline systems and for the systems submitted by the participants. The performance measures are: mean F-measure for the samples (MF-samples); mean F-measure for the concepts (MF-concepts); and the mean average precision for the samples (MAP-samples). Unlike the development set tables, for the test set there is a fourth performance measure which is also the mean F-measure for the concepts, although considering only the concepts that were not seen during development.

In the tables, the values between the square brackets correspond to the 95% confidence intervals computed using Wilson's method.

At the bottom of the page you can also download files containing more complete raw results computed for the baseline systems and the participants' submissions.

Development set baseline results

MF-samples
(%)
MF-concepts
(%)
MAP-samples
(%)
baseline_oppsift 19.2   [18.3--20.2] 13.8   [11.6--19.0] 24.6   [23.7--25.8]
baseline_csift 18.6   [17.8--19.7] 10.7   [8.9--15.6] 24.2   [23.3--25.4]
baseline_rgbsift 18.5   [17.7--19.6] 13.0   [10.9--18.0] 24.3   [23.4--25.5]
baseline_sift 17.8   [17.0--18.9] 11.0   [9.2--15.8] 24.0   [23.0--25.1]
baseline_colorhist 16.1   [15.3--17.2] 8.0   [6.6--12.7] 22.1   [21.3--23.2]
baseline_gist2 14.8   [14.0--15.8] 7.2   [5.9--11.9] 21.3   [20.5--22.3]
baseline_gist 14.5   [13.7--15.5] 6.1   [5.1--10.7] 20.9   [20.1--21.9]
baseline_getlf 14.9   [14.2--16.0] 6.6   [5.3--11.3] 21.0   [20.2--22.1]
baseline_rand 6.2   [5.7--6.9] 4.8   [4.4--8.7] 10.9   [10.5--11.6]

Development set participants' results

Group and Run # MF-samples
(%)
MF-concepts
(%)
MAP-samples
(%)
TPT_6 51.3   [50.0--52.7] 45.0   [40.8--49.6] 50.4   [49.0--51.9]
TPT_4 50.7   [49.4--52.0] 42.5   [38.6--47.0] 48.9   [47.5--50.3]
MIL_1 34.6   [33.7--35.7] 35.2   [31.1--40.5] 44.5   [43.3--45.7]
MIL_4 34.0   [33.1--35.1] 34.7   [30.8--39.9] 43.8   [42.6--45.0]
MIL_2 34.3   [33.4--35.4] 33.9   [29.8--39.4] 43.1   [42.0--44.3]
UNIMORE_2 27.3   [25.8--29.0] 34.2   [29.9--39.8] 46.0   [44.6--47.4]
UNIMORE_5 33.3   [32.1--34.6] 33.7   [29.4--39.2] 47.9   [46.5--49.3]
UNIMORE_1 33.0   [31.9--34.3] 34.1   [29.9--39.7] 39.2   [38.1--40.4]
TPT_2 41.4   [40.1--42.8] 30.9   [27.4--35.9] 38.5   [37.3--39.9]
UNIMORE_6 33.0   [31.9--34.3] 34.1   [29.9--39.7] 46.0   [44.6--47.4]
RUC_4 31.6   [30.6--32.7] 33.4   [30.1--38.0] 41.2   [39.9--42.5]
MIL_5 34.0   [33.1--35.0] 33.4   [29.4--38.7] 42.2   [41.1--43.4]
RUC_5 31.0   [30.1--32.1] 32.7   [29.5--37.3] 40.5   [39.2--41.8]
MIL_3 34.2   [33.3--35.3] 33.4   [29.3--38.8] 42.5   [41.3--43.6]
UNIMORE_3 23.1   [21.6--24.8] 32.4   [28.3--37.9] 43.7   [42.3--45.1]
UNEDUV_3 22.5   [21.2--23.9] 31.5   [27.9--36.5] 27.1   [25.9--28.5]
UNEDUV_5 27.6   [26.8--28.6] 31.7   [28.3--36.5] 35.5   [34.1--36.9]
TPT_5 38.7   [37.7--39.7] 33.0   [29.2--38.2] 49.8   [48.4--51.2]
RUC_3 29.8   [28.9--30.9] 31.4   [28.6--35.7] 39.4   [38.1--40.8]
TPT_3 38.8   [37.8--39.9] 30.2   [26.7--35.2] 49.0   [47.6--50.4]
RUC_2 28.8   [27.8--29.9] 30.8   [28.1--35.0] 38.2   [36.9--39.6]
UNIMORE_4 26.8   [25.5--28.3] 31.7   [27.8--37.0] 39.7   [38.5--41.1]
UNEDUV_4 29.9   [28.7--31.3] 26.3   [23.7--30.7] 31.0   [29.8--32.3]
UNEDUV_1 25.0   [23.8--26.4] 27.5   [24.5--32.2] 32.8   [31.4--34.4]
RUC_1 28.8   [27.8--29.9] 26.6   [23.3--31.8] 36.1   [34.8--37.4]
UNEDUV_2 24.4   [23.2--25.7] 26.1   [23.6--30.4] 32.4   [31.0--33.9]
CEALIST_4 32.2   [31.2--33.3] 26.1   [23.1--31.1] 40.3   [38.9--41.7]
CEALIST_5 31.6   [30.6--32.7] 25.4   [22.5--30.3] 39.2   [37.9--40.6]
CEALIST_3 31.8   [30.8--32.9] 25.3   [22.5--30.2] 40.4   [39.0--41.8]
CEALIST_2 30.2   [29.3--31.3] 24.6   [21.7--29.5] 39.6   [38.3--41.0]
CEALIST_1 28.7   [27.7--29.7] 23.6   [20.8--28.5] 34.6   [33.3--35.9]
KDEVIR_1 25.3   [24.5--26.3] 21.1   [18.6--25.8] 28.7   [27.6--29.9]
URJCyUNED_3 27.9   [27.0--29.1] 19.8   [17.2--24.7] 32.6   [31.4--33.9]
MICC_5 22.7   [21.8--23.8] 21.4   [19.0--26.0] 29.1   [28.1--30.3]
MICC_4 22.4   [21.5--23.5] 21.0   [18.8--25.6] 29.2   [28.1--30.4]
URJCyUNED_2 27.7   [26.8--28.9] 19.7   [17.1--24.7] 32.2   [31.0--33.5]
MICC_3 22.3   [21.4--23.4] 21.0   [18.7--25.5] 29.0   [28.0--30.2]
URJCyUNED_1 27.4   [26.4--28.6] 19.2   [16.6--24.3] 32.0   [30.8--33.3]
MICC_2 23.3   [22.5--24.3] 20.7   [18.6--25.1] 29.0   [27.9--30.2]
MICC_1 20.4   [19.5--21.5] 20.3   [18.1--24.8] 28.7   [27.6--29.9]
KDEVIR_3 24.8   [24.0--25.7] 18.7   [16.5--23.4] 28.6   [27.5--29.8]
TPT_1 30.2   [29.3--31.2] 24.2   [20.9--29.6] 38.6   [37.4--40.0]
KDEVIR_6 24.5   [23.8--25.4] 18.4   [16.2--23.1] 28.3   [27.2--29.5]
KDEVIR_4 24.7   [24.0--25.6] 18.5   [16.2--23.2] 29.2   [28.1--30.5]
KDEVIR_2 25.0   [24.2--25.9] 19.2   [16.9--24.0] 26.4   [25.4--27.5]
KDEVIR_5 24.6   [23.9--25.5] 18.5   [16.3--23.2] 29.0   [28.0--30.3]
SZTAKI_1 10.4   [9.3--11.9] 17.7   [14.4--23.6] 32.9   [31.6--34.2]
INAOE_3 19.7   [19.0--20.7] 17.7   [15.7--22.1] 24.0   [23.1--25.1]
SZTAKI_2 9.8   [8.7--11.2] 17.1   [13.8--22.9] 32.7   [31.4--34.0]
THSSMPAM_3 17.0   [16.6--17.7] 13.0   [11.0--17.9] 20.9   [20.1--22.0]
THSSMPAM_2 17.0   [16.6--17.7] 13.0   [11.0--17.9] 21.7   [20.9--22.7]
LMCHFUT_1 12.2   [11.3--13.3] 13.6   [12.2--17.9] N/A
INAOE_1 21.3   [20.6--22.2] 9.0   [7.0--14.2] 21.5   [20.8--22.5]
THSSMPAM_1 18.2   [17.7--18.9] 13.7   [11.8--18.4] 16.3   [15.7--17.1]
INAOE_2 24.8   [23.9--25.8] 6.3   [4.7--11.5] 23.6   [22.7--24.7]
THSSMPAM_4 15.5   [15.1--16.1] 12.2   [10.2--17.2] 15.9   [15.3--16.7]
THSSMPAM_5 15.5   [15.1--16.1] 12.2   [10.2--17.2] 15.8   [15.2--16.7]
INAOE_4 15.9   [15.2--16.9] 11.7   [10.4--16.1] 17.9   [17.1--19.0]

Test set baseline results

MF-samples
(%)
MF-concepts
(%)
MF-concepts unseen in dev.
(%)
MAP-samples
(%)
baseline_oppsift 16.4   [15.7--17.1] 11.8   [9.8--16.3] 10.3   [5.9--28.3] 21.4   [20.7--22.3]
baseline_csift 16.2   [15.5--16.9] 10.5   [8.7--14.9] 10.8   [6.4--28.7] 21.2   [20.5--22.0]
baseline_rgbsift 15.8   [15.2--16.6] 11.7   [9.8--16.1] 10.5   [6.2--28.3] 21.2   [20.5--22.1]
baseline_sift 15.9   [15.3--16.6] 11.0   [9.1--15.5] 10.1   [5.9--27.9] 21.0   [20.3--21.9]
baseline_colorhist 13.9   [13.3--14.6] 8.0   [6.6--12.2] 9.6   [5.7--27.3] 19.0   [18.4--19.8]
baseline_gist2 12.9   [12.3--13.6] 7.8   [6.6--11.9] 8.2   [5.1--25.6] 18.2   [17.6--18.9]
baseline_gist 12.5   [12.0--13.2] 6.9   [5.8--10.8] 7.3   [4.4--24.8] 17.8   [17.2--18.5]
baseline_getlf 12.5   [11.9--13.1] 5.4   [4.4--9.3] 5.9   [3.5--23.5] 17.7   [17.1--18.4]
baseline_rand 4.6   [4.3--5.1] 3.6   [3.4--6.8] 2.3   [1.9--19.1] 8.7   [8.4--9.2]

Test set participants' results

Group and Run # MF-samples
(%)
MF-concepts
(%)
MF-concepts unseen in dev.
(%)
MAP-samples
(%)
TPT_6 42.6   [41.4--43.7] 34.1   [30.3--38.9] 45.1   [34.4--57.5] 44.4   [43.3--45.5]
TPT_4 41.8   [40.8--42.9] 33.7   [30.2--38.3] 45.3   [34.9--57.2] 43.2   [42.2--44.3]
MIL_1 33.2   [32.5--34.0] 32.6   [29.3--37.0] 33.8   [25.6--47.6] 42.1   [41.1--43.0]
MIL_4 32.4   [31.7--33.2] 32.3   [29.1--36.7] 35.8   [27.2--49.2] 41.4   [40.4--42.3]
MIL_2 32.7   [32.0--33.5] 31.8   [28.4--36.3] 31.4   [23.3--45.8] 40.7   [39.8--41.6]
UNIMORE_2 27.5   [26.4--28.7] 33.1   [29.4--37.9] 34.8   [26.0--48.8] 44.1   [43.1--45.2]
UNIMORE_5 31.5   [30.6--32.5] 31.9   [28.3--36.7] 31.9   [23.1--46.9] 45.6   [44.6--46.7]
UNIMORE_1 31.1   [30.2--32.0] 32.0   [28.5--36.7] 31.3   [22.9--46.2] 36.7   [35.9--37.6]
TPT_2 38.1   [37.1--39.1] 30.0   [27.2--34.1] 30.9   [24.5--43.8] 37.0   [36.0--38.0]
UNIMORE_6 31.1   [30.2--32.0] 32.0   [28.5--36.7] 31.3   [22.9--46.2] 44.1   [43.1--45.2]
RUC_4 29.0   [28.3--29.8] 30.4   [27.8--34.3] 32.8   [25.0--46.5] 38.0   [37.0--39.1]
MIL_5 31.7   [31.0--32.5] 30.9   [27.6--35.3] 30.2   [22.5--44.7] 39.7   [38.8--40.6]
RUC_5 28.3   [27.6--29.1] 29.6   [27.0--33.6] 31.5   [23.8--45.5] 37.6   [36.7--38.7]
MIL_3 31.8   [31.1--32.6] 30.2   [27.0--34.7] 29.5   [21.5--44.5] 39.6   [38.7--40.6]
UNIMORE_3 23.1   [21.9--24.3] 31.5   [27.9--36.3] 35.5   [26.7--49.2] 41.9   [40.9--43.1]
UNEDUV_3 23.1   [22.0--24.2] 31.3   [28.1--35.8] 43.2   [33.1--55.7] 26.6   [25.6--27.7]
UNEDUV_5 24.4   [23.8--25.1] 29.2   [26.7--33.1] 35.4   [27.7--48.2] 33.2   [32.2--34.3]
TPT_5 32.5   [31.7--33.3] 26.7   [23.8--31.2] 27.3   [20.2--42.2] 44.3   [43.3--45.5]
RUC_3 27.8   [27.1--28.6] 29.2   [26.8--32.9] 30.2   [22.9--44.3] 36.9   [35.9--37.9]
TPT_3 31.9   [31.1--32.7] 24.8   [22.0--29.1] 24.7   [19.2--38.8] 43.6   [42.5--44.7]
RUC_2 26.5   [25.8--27.3] 28.5   [26.3--32.2] 29.9   [22.8--43.9] 35.5   [34.5--36.6]
UNIMORE_4 24.1   [23.1--25.3] 29.5   [26.0--34.3] 28.0   [19.3--44.3] 36.2   [35.2--37.2]
UNEDUV_4 30.0   [29.0--31.1] 22.8   [20.9--26.5] 24.6   [19.5--38.5] 29.8   [28.9--30.9]
UNEDUV_1 23.0   [22.1--23.9] 25.0   [22.8--28.7] 31.7   [25.0--44.7] 30.3   [29.3--31.4]
RUC_1 25.4   [24.7--26.3] 23.9   [21.1--28.4] 22.7   [16.6--38.3] 32.4   [31.4--33.5]
UNEDUV_2 22.9   [22.0--23.8] 24.0   [22.3--27.5] 30.6   [24.7--43.1] 30.6   [29.6--31.7]
CEALIST_4 26.0   [25.4--26.8] 21.2   [18.9--25.4] 20.1   [14.7--35.7] 34.2   [33.2--35.2]
CEALIST_5 25.7   [25.1--26.5] 21.0   [18.7--25.2] 20.0   [14.6--35.7] 33.6   [32.6--34.6]
CEALIST_3 25.2   [24.6--26.0] 20.2   [18.0--24.5] 20.5   [15.0--36.1] 34.1   [33.1--35.1]
CEALIST_2 24.2   [23.6--24.9] 20.1   [17.9--24.3] 20.1   [14.7--35.7] 33.6   [32.6--34.7]
CEALIST_1 23.0   [22.4--23.7] 19.0   [16.8--23.2] 19.8   [14.5--35.4] 29.4   [28.5--30.4]
KDEVIR_1 22.2   [21.6--22.9] 18.0   [16.2--21.8] 17.3   [13.0--32.8] 26.1   [25.2--27.0]
URJCyUNED_3 24.1   [23.4--24.9] 17.3   [15.2--21.5] 14.8   [10.7--30.9] 28.1   [27.2--29.0]
MICC_5 20.0   [19.3--20.8] 18.0   [16.2--22.0] 18.6   [14.3--33.7] 26.2   [25.3--27.1]
MICC_4 20.0   [19.3--20.8] 18.0   [16.2--21.9] 18.6   [14.3--33.7] 26.1   [25.3--27.1]
URJCyUNED_2 23.8   [23.1--24.7] 17.2   [15.2--21.5] 14.6   [10.6--30.8] 27.6   [26.8--28.6]
MICC_3 20.0   [19.3--20.8] 18.1   [16.3--22.0] 18.5   [14.3--33.5] 26.1   [25.2--27.0]
URJCyUNED_1 23.7   [23.0--24.6] 17.1   [15.1--21.3] 14.6   [10.6--30.7] 27.6   [26.7--28.5]
MICC_2 20.4   [19.8--21.2] 17.5   [15.9--21.3] 17.0   [13.3--32.0] 26.1   [25.2--27.0]
MICC_1 18.7   [18.0--19.5] 17.3   [15.7--21.2] 17.6   [13.6--32.7] 25.9   [25.1--26.9]
KDEVIR_3 21.1   [20.6--21.7] 15.9   [14.4--19.7] 15.6   [12.0--31.0] 24.8   [24.0--25.8]
TPT_1 23.0   [22.4--23.7] 19.2   [16.6--23.8] 8.2   [6.6--24.2] 36.8   [35.8--37.8]
KDEVIR_6 20.8   [20.2--21.4] 15.7   [14.2--19.4] 15.0   [11.6--30.4] 24.3   [23.5--25.2]
KDEVIR_4 20.5   [20.0--21.1] 15.4   [13.9--19.2] 15.3   [12.0--30.5] 26.4   [25.5--27.3]
KDEVIR_2 20.7   [20.1--21.3] 14.8   [13.2--18.8] 12.6   [9.3--28.7] 23.5   [22.7--24.3]
KDEVIR_5 20.2   [19.7--20.8] 15.1   [13.7--18.8] 14.5   [11.4--29.8] 25.6   [24.8--26.6]
SZTAKI_1 9.5   [8.7--10.4] 16.4   [13.6--21.5] 16.7   [10.5--34.3] 28.2   [27.3--29.1]
INAOE_3 15.4   [14.9--16.0] 15.2   [13.6--19.0] 11.1   [8.4--27.2] 19.1   [18.4--19.8]
SZTAKI_2 8.8   [8.1--9.7] 15.1   [12.3--20.2] 16.0   [9.9--33.9] 28.0   [27.1--28.9]
THSSMPAM_3 14.8   [14.4--15.3] 12.7   [11.3--16.5] 11.1   [8.5--27.1] 15.9   [15.4--16.6]
THSSMPAM_2 14.8   [14.5--15.4] 12.7   [11.3--16.5] 11.1   [8.5--27.1] 16.1   [15.6--16.8]
LMCHFUT_1 11.0   [10.4--11.8] 12.1   [11.1--15.5] 11.3   [8.8--27.1] N/A
INAOE_1 16.9   [16.4--17.5] 6.9   [5.6--11.1] 5.1   [3.4--22.2] 17.5   [16.9--18.1]
THSSMPAM_1 11.8   [11.6--12.2] 10.0   [8.6--14.0] 6.6   [5.3--22.8] 12.0   [11.6--12.5]
INAOE_2 16.7   [16.1--17.3] 4.8   [3.7--8.8] 4.7   [3.3--21.7] 19.0   [18.3--19.7]
THSSMPAM_4 11.8   [11.6--12.2] 10.0   [8.6--14.0] 6.6   [5.3--22.8] 11.9   [11.6--12.4]
THSSMPAM_5 11.8   [11.5--12.2] 10.0   [8.6--14.0] 6.6   [5.2--22.8] 11.9   [11.5--12.4]
INAOE_4 6.2   [5.9--6.6] 3.4   [3.0--6.9] 2.3   [1.8--19.2] 8.3   [8.0--8.6]

Raw results

iclef13annot_results_baseline.zip
iclef13annot_results_CEALIST.zip
iclef13annot_results_INAOE.zip
iclef13annot_results_KDEVIR.zip
iclef13annot_results_LMCHFUT.zip
iclef13annot_results_MICC.zip
iclef13annot_results_MIL.zip
iclef13annot_results_RUC.zip
iclef13annot_results_SZTAKI.zip
iclef13annot_results_THSSMPAM.zip
iclef13annot_results_TPT.zip
iclef13annot_results_UNEDUV.zip
iclef13annot_results_UNIMORE.zip
iclef13annot_results_URJCyUNED.zip

Raw format description:

The results are text files in which each line corresponds to a type of performance measure. The first column is an identifier of the run, being of the form {GROUP}_{RUN#}, and the second column indicates the type of performance measure. The performances of type {PREC|RECL|F|AP}{samp|cnpt}-te, i.e., all of the ones that do not start with an 'm', are the precision, recall, f-measure or average precision for each sample or concept of the test set. The performances that start with 'm', are the mean of each corresponding measure, being the first value the actual mean, and the following two values the lower and upper limits of the 95% confidence intervals.