The results below are organized as follows:

  • each table below displays various solvers’ performance for individual problem instances for the given task and time limit
  • table values are normalized scores for each evaluated problem as outlined in Evaluation Criteria

overall

20sec

Problem lbp ibia-mar uai14-mar
Alchemy_11 0.0 100.0 0.0
CSP_11 0.0 100.0 0.0
CSP_12 77.5 100.0 100.0
CSP_13 0.0 100.0 0.0
DBN_11 0.0 100.0 18.5
DBN_12 0.0 100.0 86.2
DBN_13 0.0 100.0 0.0
DBN_14 0.0 100.0 100.0
DBN_15 0.0 100.0 0.0
DBN_16 0.0 100.0 68.0
Grids_11 0.1 100.0 0.0
Grids_12 36.5 100.0 100.0
Grids_13 13.2 100.0 0.0
Grids_14 0.0 100.0 0.0
Grids_15 0.0 0.0 64.7
Grids_16 0.0 0.0 18.4
Grids_17 0.0 0.0 0.0
Grids_18 0.0 0.0 0.0
ObjectDetection_11 0.0 100.0 99.8
ObjectDetection_12 0.0 100.0 99.1
ObjectDetection_13 0.0 97.1 95.1
ObjectDetection_14 0.0 96.2 90.5
ObjectDetection_15 0.0 100.0 98.1
ObjectDetection_16 0.0 98.8 97.2
ObjectDetection_17 0.0 98.6 96.3
ObjectDetection_18 0.0 100.0 84.6
ObjectDetection_19 0.0 92.3 54.9
ObjectDetection_20 0.0 96.3 57.3
ObjectDetection_21 0.0 100.0 94.8
ObjectDetection_22 0.0 100.0 100.0
ObjectDetection_23 0.0 97.9 95.6
ObjectDetection_24 0.0 97.6 92.7
ObjectDetection_25 0.0 100.0 98.9
ObjectDetection_26 0.0 98.3 97.3
ObjectDetection_27 0.0 98.4 96.8
ObjectDetection_28 0.0 100.0 100.0
ObjectDetection_29 0.0 98.4 99.9
ObjectDetection_30 0.0 100.0 96.9
ObjectDetection_31 0.0 93.2 98.9
ObjectDetection_32 0.0 100.0 99.3
ObjectDetection_33 0.0 100.0 97.3
ObjectDetection_34 0.0 100.0 100.0
ObjectDetection_35 0.0 100.0 99.9
ObjectDetection_36 0.0 97.3 99.9
ObjectDetection_37 0.0 100.0 99.6
ObjectDetection_38 0.0 100.0 99.9
ObjectDetection_39 0.0 98.3 99.7
ObjectDetection_40 0.0 100.0 100.0
ObjectDetection_41 0.0 100.0 0.0
ObjectDetection_42 0.0 100.0 100.0
ObjectDetection_43 0.0 99.0 99.9
ObjectDetection_44 0.0 100.0 100.0
ObjectDetection_45 0.0 100.0 99.4
ObjectDetection_46 0.0 100.0 100.0
ObjectDetection_47 0.0 100.0 99.9
ObjectDetection_48 0.0 99.1 99.9
ObjectDetection_49 0.0 100.0 100.0
ObjectDetection_50 0.0 100.0 0.0
ObjectDetection_51 0.0 100.0 100.0
ObjectDetection_52 0.0 100.0 100.0
ObjectDetection_53 0.0 99.1 99.8
ObjectDetection_54 0.0 100.0 99.9
ObjectDetection_55 0.0 97.5 99.9
ObjectDetection_56 0.0 100.0 99.9
ObjectDetection_57 0.0 100.0 99.0
ObjectDetection_58 0.0 100.0 100.0
ObjectDetection_59 0.0 97.8 100.0
ObjectDetection_60 0.0 100.0 97.0
ObjectDetection_61 0.0 100.0 99.9
ObjectDetection_62 0.0 100.0 100.0
ObjectDetection_63 0.0 98.4 99.9
ObjectDetection_64 0.0 100.0 100.0
ObjectDetection_65 0.0 100.0 99.1
ObjectDetection_66 93.4 100.0 100.0
ObjectDetection_67 0.0 97.8 99.9
ObjectDetection_68 0.0 100.0 100.0
ObjectDetection_69 0.0 100.0 99.2
ObjectDetection_70 78.3 100.0 100.0
ObjectDetection_71 0.0 100.0 100.0
ObjectDetection_72 0.0 99.8 0.0
ObjectDetection_73 0.0 100.0 58.9
ObjectDetection_74 0.0 100.0 99.9
ObjectDetection_75 0.0 100.0 100.0
Pedigree_11 0.0 0.0 0.0
Pedigree_12 0.0 0.0 0.0
Pedigree_13 0.0 0.0 0.0
Promedus_11 0.0 0.0 0.0
Promedus_12 0.0 0.8 0.8
Promedus_13 0.0 0.0 0.0
Promedus_14 0.0 1.5 7.4
Promedus_15 13.6 6.3 6.3
Promedus_16 0.0 22.5 22.5
Promedus_17 0.0 0.0 0.0
Promedus_18 0.0 15.0 18.0
Promedus_19 0.0 0.0 33.7
Promedus_20 0.0 0.0 0.0
Promedus_21 0.0 5.8 5.8
Promedus_22 33.6 18.1 18.1
Promedus_23 0.0 6.0 6.0
Promedus_24 0.0 29.2 29.4
Promedus_25 0.0 16.0 16.0
Promedus_26 0.0 7.7 7.7
Promedus_27 9.7 2.2 2.2
Promedus_28 0.0 0.0 0.0
Promedus_29 17.9 17.1 17.1
Promedus_30 18.6 14.2 14.2
Promedus_31 0.0 5.1 5.1
Promedus_32 0.0 8.7 8.7
Promedus_33 0.0 10.7 10.8
Promedus_34 0.0 0.8 0.8
Promedus_35 10.6 1.2 1.2
Promedus_36 0.0 1.2 1.2
Promedus_37 0.0 4.3 4.3
Promedus_38 0.0 4.1 0.0
Segmentation_11 0.0 100.0 100.0
Segmentation_12 0.0 100.0 100.0
Segmentation_13 0.0 100.0 100.0
Segmentation_14 0.0 100.0 100.0
Segmentation_15 0.0 100.0 100.0
Segmentation_16 0.0 100.0 100.0
linkage_12 0.0 0.0 94.4
linkage_13 0.0 0.0 0.0
linkage_14 0.0 98.7 92.1

1200sec

Problem lbp ibia-mar uai14-mar
Alchemy_11 96.8 100.0 100.0
CSP_11 78.3 100.0 100.0
CSP_12 77.5 100.0 100.0
CSP_13 75.1 100.0 100.0
DBN_11 17.5 100.0 100.0
DBN_12 84.9 100.0 100.0
DBN_13 0.0 100.0 100.0
DBN_14 100.0 100.0 100.0
DBN_15 0.0 100.0 100.0
DBN_16 68.0 100.0 100.0
Grids_11 0.1 100.0 100.0
Grids_12 36.5 100.0 100.0
Grids_13 13.2 100.0 100.0
Grids_14 0.0 100.0 100.0
Grids_15 23.4 96.9 100.0
Grids_16 0.0 95.9 100.0
Grids_17 0.0 91.2 100.0
Grids_18 0.0 91.4 100.0
ObjectDetection_11 87.0 100.0 100.0
ObjectDetection_12 80.6 100.0 100.0
ObjectDetection_13 82.1 99.8 99.6
ObjectDetection_14 76.1 100.0 100.0
ObjectDetection_15 91.6 100.0 100.0
ObjectDetection_16 89.3 100.0 100.0
ObjectDetection_17 90.8 100.0 100.0
ObjectDetection_18 52.9 100.0 100.0
ObjectDetection_19 9.1 100.0 100.0
ObjectDetection_20 35.4 100.0 100.0
ObjectDetection_21 82.4 100.0 100.0
ObjectDetection_22 86.1 100.0 100.0
ObjectDetection_23 83.5 100.0 100.0
ObjectDetection_24 84.6 100.0 100.0
ObjectDetection_25 94.8 100.0 100.0
ObjectDetection_26 91.3 100.0 100.0
ObjectDetection_27 93.2 100.0 100.0
ObjectDetection_28 98.7 100.0 100.0
ObjectDetection_29 93.7 100.0 99.9
ObjectDetection_30 86.1 100.0 100.0
ObjectDetection_31 91.5 99.9 100.0
ObjectDetection_32 86.5 100.0 100.0
ObjectDetection_33 86.4 100.0 100.0
ObjectDetection_34 98.3 100.0 100.0
ObjectDetection_35 97.3 100.0 100.0
ObjectDetection_36 97.7 100.0 100.0
ObjectDetection_37 96.6 100.0 100.0
ObjectDetection_38 92.7 100.0 100.0
ObjectDetection_39 88.6 100.0 100.0
ObjectDetection_40 86.5 100.0 100.0
ObjectDetection_41 0.0 100.0 100.0
ObjectDetection_42 96.5 100.0 100.0
ObjectDetection_43 96.2 100.0 100.0
ObjectDetection_44 94.3 100.0 100.0
ObjectDetection_45 93.5 100.0 100.0
ObjectDetection_46 0.0 100.0 100.0
ObjectDetection_47 0.0 100.0 100.0
ObjectDetection_48 0.0 100.0 100.0
ObjectDetection_49 0.0 100.0 100.0
ObjectDetection_50 0.0 100.0 100.0
ObjectDetection_51 96.1 100.0 100.0
ObjectDetection_52 94.8 100.0 100.0
ObjectDetection_53 94.6 100.0 100.0
ObjectDetection_54 97.5 100.0 100.0
ObjectDetection_55 97.2 100.0 100.0
ObjectDetection_56 96.7 100.0 100.0
ObjectDetection_57 95.8 100.0 100.0
ObjectDetection_58 92.2 100.0 100.0
ObjectDetection_59 93.4 100.0 100.0
ObjectDetection_60 92.3 100.0 100.0
ObjectDetection_61 97.0 100.0 99.9
ObjectDetection_62 97.6 100.0 100.0
ObjectDetection_63 96.7 100.0 100.0
ObjectDetection_64 96.8 100.0 100.0
ObjectDetection_65 96.3 100.0 100.0
ObjectDetection_66 93.4 100.0 100.0
ObjectDetection_67 92.8 100.0 100.0
ObjectDetection_68 92.4 100.0 100.0
ObjectDetection_69 92.5 100.0 100.0
ObjectDetection_70 78.3 100.0 100.0
ObjectDetection_71 88.6 100.0 100.0
ObjectDetection_72 57.0 100.0 100.0
ObjectDetection_73 56.1 100.0 100.0
ObjectDetection_74 97.9 100.0 99.9
ObjectDetection_75 98.4 100.0 100.0
Pedigree_11 0.0 0.0 0.0
Pedigree_12 0.0 0.0 0.0
Pedigree_13 0.0 0.0 0.0
Promedus_11 0.0 0.0 0.0
Promedus_12 6.1 0.8 0.8
Promedus_13 0.0 0.0 0.0
Promedus_14 0.0 4.7 4.7
Promedus_15 13.6 6.3 6.3
Promedus_16 23.8 22.5 22.5
Promedus_17 0.0 0.0 0.0
Promedus_18 17.7 18.0 18.0
Promedus_19 33.6 33.6 33.7
Promedus_20 0.0 0.0 0.0
Promedus_21 14.6 5.8 5.8
Promedus_22 33.6 18.1 18.1
Promedus_23 4.8 6.0 6.0
Promedus_24 29.5 29.2 29.4
Promedus_25 18.2 16.0 16.0
Promedus_26 8.5 7.7 7.7
Promedus_27 9.7 2.2 2.2
Promedus_28 0.0 0.0 0.0
Promedus_29 17.9 17.1 17.1
Promedus_30 18.6 14.2 14.2
Promedus_31 13.9 5.1 5.1
Promedus_32 16.0 8.7 8.7
Promedus_33 10.8 10.7 10.8
Promedus_34 8.5 0.8 0.8
Promedus_35 10.6 1.2 1.2
Promedus_36 10.6 1.2 1.2
Promedus_37 0.7 4.3 4.3
Promedus_38 2.4 4.1 4.1
Segmentation_11 38.1 100.0 100.0
Segmentation_12 100.0 100.0 100.0
Segmentation_13 50.8 100.0 100.0
Segmentation_14 89.9 100.0 100.0
Segmentation_15 48.6 100.0 100.0
Segmentation_16 80.7 100.0 100.0
linkage_12 6.7 91.2 98.5
linkage_13 0.0 94.0 68.8
linkage_14 86.0 99.6 100.0

3600sec

Problem lbp ibia-mar uai14-mar
Alchemy_11 96.8 100.0 100.0
CSP_11 78.3 100.0 100.0
CSP_12 77.5 100.0 100.0
CSP_13 75.1 100.0 100.0
DBN_11 17.5 100.0 100.0
DBN_12 84.9 100.0 100.0
DBN_13 0.0 100.0 100.0
DBN_14 100.0 100.0 100.0
DBN_15 0.0 100.0 100.0
DBN_16 68.0 100.0 100.0
Grids_11 0.1 100.0 100.0
Grids_12 36.5 100.0 100.0
Grids_13 13.2 100.0 100.0
Grids_14 0.0 100.0 100.0
Grids_15 23.4 96.9 100.0
Grids_16 0.0 95.9 100.0
Grids_17 0.0 91.2 100.0
Grids_18 0.0 91.4 100.0
ObjectDetection_11 87.0 100.0 100.0
ObjectDetection_12 80.6 100.0 100.0
ObjectDetection_13 82.1 99.8 100.0
ObjectDetection_14 76.1 100.0 100.0
ObjectDetection_15 91.6 100.0 100.0
ObjectDetection_16 89.3 100.0 100.0
ObjectDetection_17 90.8 100.0 100.0
ObjectDetection_18 52.9 100.0 100.0
ObjectDetection_19 9.1 100.0 100.0
ObjectDetection_20 35.4 100.0 100.0
ObjectDetection_21 82.4 100.0 100.0
ObjectDetection_22 86.1 100.0 100.0
ObjectDetection_23 83.5 100.0 100.0
ObjectDetection_24 84.6 100.0 100.0
ObjectDetection_25 94.8 100.0 100.0
ObjectDetection_26 91.3 100.0 100.0
ObjectDetection_27 93.2 100.0 100.0
ObjectDetection_28 98.7 100.0 100.0
ObjectDetection_29 93.7 100.0 99.9
ObjectDetection_30 86.1 100.0 100.0
ObjectDetection_31 91.5 99.9 100.0
ObjectDetection_32 86.5 100.0 100.0
ObjectDetection_33 86.4 100.0 100.0
ObjectDetection_34 98.3 100.0 100.0
ObjectDetection_35 97.3 100.0 100.0
ObjectDetection_36 97.7 100.0 100.0
ObjectDetection_37 96.6 100.0 100.0
ObjectDetection_38 92.7 100.0 100.0
ObjectDetection_39 88.6 100.0 100.0
ObjectDetection_40 86.5 100.0 100.0
ObjectDetection_41 0.0 100.0 100.0
ObjectDetection_42 96.5 100.0 100.0
ObjectDetection_43 96.2 100.0 100.0
ObjectDetection_44 94.3 100.0 100.0
ObjectDetection_45 93.5 100.0 100.0
ObjectDetection_46 0.0 100.0 100.0
ObjectDetection_47 0.0 100.0 100.0
ObjectDetection_48 0.0 100.0 100.0
ObjectDetection_49 0.0 100.0 100.0
ObjectDetection_50 0.0 100.0 100.0
ObjectDetection_51 96.1 100.0 100.0
ObjectDetection_52 94.8 100.0 100.0
ObjectDetection_53 94.6 100.0 100.0
ObjectDetection_54 97.5 100.0 100.0
ObjectDetection_55 97.2 100.0 100.0
ObjectDetection_56 96.7 100.0 100.0
ObjectDetection_57 95.8 100.0 100.0
ObjectDetection_58 92.2 100.0 100.0
ObjectDetection_59 93.4 100.0 100.0
ObjectDetection_60 92.3 100.0 100.0
ObjectDetection_61 97.0 100.0 99.9
ObjectDetection_62 97.6 100.0 100.0
ObjectDetection_63 96.7 100.0 100.0
ObjectDetection_64 96.8 100.0 100.0
ObjectDetection_65 96.3 100.0 100.0
ObjectDetection_66 93.4 100.0 100.0
ObjectDetection_67 92.8 100.0 100.0
ObjectDetection_68 92.4 100.0 100.0
ObjectDetection_69 92.5 100.0 100.0
ObjectDetection_70 78.3 100.0 100.0
ObjectDetection_71 88.6 100.0 100.0
ObjectDetection_72 57.0 100.0 100.0
ObjectDetection_73 56.1 100.0 100.0
ObjectDetection_74 97.9 100.0 99.9
ObjectDetection_75 98.4 100.0 100.0
Pedigree_11 0.0 0.0 0.0
Pedigree_12 0.0 0.0 0.0
Pedigree_13 0.0 0.0 0.0
Promedus_11 0.0 0.0 0.0
Promedus_12 6.1 0.8 0.8
Promedus_13 0.0 0.0 0.0
Promedus_14 0.0 4.7 4.7
Promedus_15 13.6 6.3 6.3
Promedus_16 23.8 22.5 22.5
Promedus_17 0.0 0.0 0.0
Promedus_18 17.7 18.0 18.0
Promedus_19 33.6 33.6 33.7
Promedus_20 0.0 0.0 0.0
Promedus_21 14.6 5.8 5.8
Promedus_22 33.6 18.1 18.1
Promedus_23 4.8 6.0 6.0
Promedus_24 29.5 29.2 29.4
Promedus_25 18.2 16.0 16.0
Promedus_26 8.5 7.7 7.7
Promedus_27 9.7 2.2 2.2
Promedus_28 0.0 0.0 0.0
Promedus_29 17.9 17.1 17.1
Promedus_30 18.6 14.2 14.2
Promedus_31 13.9 5.1 5.1
Promedus_32 16.0 8.7 8.7
Promedus_33 10.8 10.7 10.8
Promedus_34 8.5 0.8 0.8
Promedus_35 10.6 1.2 1.2
Promedus_36 10.6 1.2 1.2
Promedus_37 0.7 4.3 4.3
Promedus_38 2.4 4.1 4.1
Segmentation_11 38.1 100.0 100.0
Segmentation_12 100.0 100.0 100.0
Segmentation_13 50.8 100.0 100.0
Segmentation_14 89.9 100.0 100.0
Segmentation_15 48.6 100.0 100.0
Segmentation_16 80.7 100.0 100.0
linkage_12 6.7 91.2 97.5
linkage_13 0.0 94.0 70.1
linkage_14 86.0 99.6 100.0

Updated: