| A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Sparse judgements: TREC-AToMiC/AToMiC-Qrels-v0.2 | official | arxiv | Diff | ||||||||||||||||||||||
2 | T2M | M2T | T2M | M2T | T2M | M2T | ||||||||||||||||||||
3 | Corpus size | System | Setting | note | Img column | Text Column | MRR@10 | R@10 | R@1k | MRR@10 | R@10 | R@1k | MRR@10 | R@10 | R@1k | MRR@10 | R@10 | R@1k | MRR@10 | R@10 | R@1k | MRR@10 | R@10 | R@1k | ||
4 | small | Anserini | BM25 (default) | captions | page_title, section_title, hierachy, context_page_description, context_section_description | 0.654 | 76.70% | 92.33% | 0.745 | 82.94% | 93.24% | 0.636 | 74.69% | 90.75% | 0.719 | 80.37% | 91.91% | 0.02 | 2.01% | 1.58% | 0.03 | 2.57% | 1.33% | |||
5 | CLIP | openai | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.198 | 34.95% | 90.85% | 0.182 | 31.17% | 89.18% | 0.184 | 32.62% | 89.61% | 0.173 | 29.42% | 87.61% | 0.01 | 2.33% | 1.24% | 0.01 | 1.75% | 1.57% | |||
6 | CLIP | openai | ViT-L-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.349 | 53.98% | 95.88% | 0.246 | 39.54% | 91.17% | 0.326 | 50.62% | 94.95% | 0.226 | 36.49% | 88.75% | 0.02 | 3.36% | 0.93% | 0.02 | 3.05% | 2.42% | |||
7 | BLIP | encoder-only | blip-itm-base-coco | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.096 | 18.70% | 76.33% | 0.111 | 20.58% | 78.78% | 0.073 | 14.56% | 70.59% | 0.095 | 17.75% | 75.71% | 0.02 | 4.14% | 5.74% | 0.02 | 2.83% | 3.07% | |||
8 | BLIP | encoder-only | blip-itm-large-coco | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.123 | 22.91% | 78.41% | 0.138 | 24.25% | 83.26% | 0.092 | 17.78% | 71.42% | 0.115 | 20.56% | 80.03% | 0.03 | 5.13% | 6.99% | 0.02 | 3.69% | 3.23% | |||
9 | FLAVA | encoder-only | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.128 | 25.70% | 85.89% | 0.139 | 26.68% | 86.58% | 0.114 | 23.38% | 84.29% | 0.123 | 24.15% | 85.19% | 0.01 | 2.32% | 1.60% | 0.02 | 2.53% | 1.39% | |||
10 | OpenCLIP | laion400m_e32 | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.159 | 27.36% | 83.48% | 0.172 | 28.86% | 84.90% | 0.145 | 25.02% | 81.50% | 0.160 | 26.44% | 82.57% | 0.01 | 2.34% | 1.98% | 0.01 | 2.42% | 2.33% | |||
11 | OpenCLIP | laion2b_e16 | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.213 | 35.65% | 90.04% | 0.228 | 36.90% | 90.85% | 0.196 | 32.90% | 88.49% | 0.208 | 33.82% | 88.93% | 0.02 | 2.75% | 1.55% | 0.02 | 3.08% | 1.92% | |||
12 | OpenCLIP | laion2b_s32b_b82k | ViT-L-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.314 | 49.00% | 94.51% | 0.316 | 47.97% | 94.18% | 0.289 | 45.63% | 93.49% | 0.291 | 44.25% | 93.17% | 0.02 | 3.37% | 1.02% | 0.02 | 3.72% | 1.01% | |||
13 | OpenCLIP | laion2b_s32b_b79k | ViT-H-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.350 | 53.12% | 95.41% | 0.358 | 53.13% | 95.36% | 0.325 | 49.70% | 94.72% | 0.331 | 49.49% | 94.59% | 0.03 | 3.42% | 0.69% | 0.03 | 3.64% | 0.77% | |||
14 | OpenCLIP | laion2b_s39b_b160k | ViT-G-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.385 | 57.09% | 95.96% | 0.393 | 57.07% | 95.92% | 0.357 | 53.70% | 95.28% | 0.366 | 53.60% | 88.93% | 0.03 | 3.39% | 0.68% | 0.03 | 3.47% | 6.99% | |||
15 | base | Anserini | BM25 (default) | captions | page_title, section_title, hierachy, context_page_description, context_section_description | 0.279 | 42.19% | 72.99% | 0.451 | 60.40% | 82.84% | 0.287 | 42.32% | 70.97% | 0.431 | 57.89% | 80.73% | -0.01 | -0.13% | 2.02% | 0.02 | 2.51% | 2.11% | |||
16 | CLIP | openai | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.030 | 5.74% | 32.87% | 0.026 | 4.70% | 29.41% | 0.030 | 5.64% | 30.83% | 0.026 | 4.57% | 27.39% | 0.00 | 0.10% | 2.04% | 0.00 | 0.13% | 2.02% | |||
17 | CLIP | openai | ViT-L-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.074 | 13.04% | 50.54% | 0.045 | 7.82% | 38.25% | 0.069 | 12.25% | 47.34% | 0.044 | 7.72% | 34.67% | 0.01 | 0.79% | 3.20% | 0.00 | 0.10% | 3.58% | |||
18 | BLIP | encoder-only | blip-itm-base-coco | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.010 | 1.96% | 15.95% | 0.012 | 2.30% | 17.67% | 0.008 | 1.42% | 12.46% | 0.010 | 1.84% | 14.95% | 0.00 | 0.54% | 3.49% | 0.00 | 0.46% | 2.72% | |||
19 | BLIP | encoder-only | blip-itm-large-coco | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.015 | 2.88% | 20.71% | 0.016 | 2.91% | 21.26% | 0.011 | 2.20% | 16.15% | 0.013 | 2.44% | 18.15% | 0.00 | 0.68% | 4.56% | 0.00 | 0.47% | 3.11% | |||
20 | FLAVA | encoder-only | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.005 | 1.45% | 21.76% | 0.008 | 1.95% | 23.58% | 0.004 | 1.14% | 19.73% | 0.007 | 1.62% | 21.24% | 0.00 | 0.31% | 2.03% | 0.00 | 0.33% | 2.34% | |||
21 | OpenCLIP | laion400m_e32 | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.030 | 4.73% | 24.47% | 0.032 | 5.65% | 26.25% | 0.030 | 4.57% | 22.78% | 0.031 | 5.26% | 24.25% | 0.00 | 0.16% | 1.69% | 0.00 | 0.39% | 2.00% | |||
22 | OpenCLIP | laion2b_e16 | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.040 | 6.85% | 31.96% | 0.043 | 7.38% | 34.49% | 0.038 | 6.61% | 29.63% | 0.041 | 6.90% | 31.82% | 0.00 | 0.24% | 2.33% | 0.00 | 0.48% | 2.67% | |||
23 | OpenCLIP | laion2b_s32b_b82k | ViT-L-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.071 | 12.11% | 45.65% | 0.072 | 12.03% | 45.97% | 0.067 | 11.37% | 42.42% | 0.067 | 11.46% | 42.54% | 0.00 | 0.74% | 3.23% | 0.00 | 0.57% | 3.43% | |||
24 | OpenCLIP | laion2b_s32b_b79k | ViT-H-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.079 | 13.55% | 49.26% | 0.087 | 14.84% | 51.03% | 0.076 | 12.95% | 46.15% | 0.082 | 13.61% | 47.17% | 0.00 | 0.60% | 3.11% | 0.01 | 1.23% | 3.86% | |||
25 | OpenCLIP | laion2b_s39b_b160k | ViT-G-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.093 | 16.12% | 53.41% | 0.101 | 17.07% | 55.20% | 0.087 | 14.95% | 50.06% | 0.096 | 15.84% | 51.34% | 0.01 | 1.17% | 3.35% | 0.01 | 1.23% | 3.86% | |||
26 | Large | Anserini | BM25 (default) | captions | page_title, section_title, hierachy, context_page_description, context_section_description | 0.278 | 42.06% | 72.82% | 0.295 | 51.14% | 79.05% | 0.284 | 42.03% | 70.69% | 0.282 | 48.68% | 76.51% | -0.01 | 0.03% | 2.13% | 0.01 | 2.46% | 2.54% | |||
27 | OpenCLIP | laion400m_e32 | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.025 | 3.72% | 18.36% | 0.018 | 3.42% | 17.54% | 0.024 | 3.57% | 16.79% | 0.018 | 3.15% | 16.15% | 0.00 | 0.15% | 1.57% | 0.00 | 0.27% | 1.39% | |||
28 | OpenCLIP | laion2b_e16 | ViT-B-32 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.031 | 5.24% | 24.33% | 0.025 | 4.67% | 23.40% | 0.030 | 5.08% | 22.61% | 0.023 | 4.32% | 21.46% | 0.00 | 0.16% | 1.72% | 0.00 | 0.35% | 1.94% | |||
29 | OpenCLIP | laion2b_s32b_b82k | ViT-L-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.054 | 9.24% | 35.97% | 0.039 | 7.64% | 33.43% | 0.052 | 8.76% | 33.54% | 0.034 | 6.98% | 30.26% | 0.00 | 0.48% | 2.43% | 0.01 | 0.66% | 3.17% | |||
30 | OpenCLIP | laion2b_s32b_b79k | ViT-H-14 | pixels | page_title, section_title, hierachy, context_page_description, context_section_description | 0.062 | 10.81% | 40.46% | 0.051 | 9.83% | 38.82% | 0.060 | 10.31% | 37.78% | 0.047 | 9.07% | 35.55% | 0.00 | 0.50% | 2.68% | 0.00 | 0.76% | 3.27% | |||
31 | ||||||||||||||||||||||||||
32 | ||||||||||||||||||||||||||
33 | ||||||||||||||||||||||||||
34 | ||||||||||||||||||||||||||
35 | ||||||||||||||||||||||||||
36 | ||||||||||||||||||||||||||
37 | ||||||||||||||||||||||||||
38 | ||||||||||||||||||||||||||
39 | ||||||||||||||||||||||||||
40 | ||||||||||||||||||||||||||
41 | ||||||||||||||||||||||||||
42 | ||||||||||||||||||||||||||
43 | ||||||||||||||||||||||||||
44 | ||||||||||||||||||||||||||
45 | ||||||||||||||||||||||||||
46 | ||||||||||||||||||||||||||
47 | ||||||||||||||||||||||||||
48 | ||||||||||||||||||||||||||
49 | ||||||||||||||||||||||||||
50 | ||||||||||||||||||||||||||
51 | ||||||||||||||||||||||||||
52 | ||||||||||||||||||||||||||
53 | ||||||||||||||||||||||||||
54 | ||||||||||||||||||||||||||
55 | ||||||||||||||||||||||||||
56 | ||||||||||||||||||||||||||
57 | ||||||||||||||||||||||||||
58 | ||||||||||||||||||||||||||
59 | ||||||||||||||||||||||||||
60 | ||||||||||||||||||||||||||
61 | ||||||||||||||||||||||||||
62 | ||||||||||||||||||||||||||
63 | ||||||||||||||||||||||||||
64 | ||||||||||||||||||||||||||
65 | ||||||||||||||||||||||||||
66 | ||||||||||||||||||||||||||
67 | ||||||||||||||||||||||||||
68 | ||||||||||||||||||||||||||
69 | ||||||||||||||||||||||||||
70 | ||||||||||||||||||||||||||
71 | ||||||||||||||||||||||||||
72 | ||||||||||||||||||||||||||
73 | ||||||||||||||||||||||||||
74 | ||||||||||||||||||||||||||
75 | ||||||||||||||||||||||||||
76 | ||||||||||||||||||||||||||
77 | ||||||||||||||||||||||||||
78 | ||||||||||||||||||||||||||
79 | ||||||||||||||||||||||||||
80 | ||||||||||||||||||||||||||
81 | ||||||||||||||||||||||||||
82 | ||||||||||||||||||||||||||
83 | ||||||||||||||||||||||||||
84 | ||||||||||||||||||||||||||
85 | ||||||||||||||||||||||||||
86 | ||||||||||||||||||||||||||
87 | ||||||||||||||||||||||||||
88 | ||||||||||||||||||||||||||
89 | ||||||||||||||||||||||||||
90 | ||||||||||||||||||||||||||
91 | ||||||||||||||||||||||||||
92 | ||||||||||||||||||||||||||
93 | ||||||||||||||||||||||||||
94 | ||||||||||||||||||||||||||
95 | ||||||||||||||||||||||||||
96 | ||||||||||||||||||||||||||
97 | ||||||||||||||||||||||||||
98 | ||||||||||||||||||||||||||
99 | ||||||||||||||||||||||||||
100 | ||||||||||||||||||||||||||