AB
1
After reading this article and viewing the tool they released for browsing the smaller 12M image LAION-Aesthetics v2 6+ dataset, I was curious to know the services scraped by the larger 600M image LAION-Aesthetics v2 5+ dataset that Stable Diffusion's last 3 checkpoints were trained on. So I downloaded 96.4GB of Parquet files and queried them using DuckDB. The counts of images in the training dataset, grouped by hostname, are presented below. Hostnames with fewer than 100 images scraped are excluded to reduce the size of this dataset. CTRL/CMD+F is your friend. This work is provided as-is and you are free to use it as you see fit.
2
3
hostnameimage_count
4
cdn.shopify.com36,469,372
5
i.pinimg.com30,908,667
6
us.123rf.com12,756,063
7
www.specsserver.com11,961,144
8
render.fineartamerica.com11,103,045
9
i0.wp.com7,895,245
10
thumbs.dreamstime.com6,737,829
11
i1.wp.com6,372,786
12
i2.wp.com6,306,473
13
images.squarespace-cdn.com5,404,912
14
media.gettyimages.com4,398,538
15
images-na.ssl-images-amazon.com4,379,966
16
photos.smugmug.com4,144,982
17
ae01.alicdn.com3,651,639
18
ssl.c.photoshelter.com3,598,979
19
i.dailymail.co.uk3,480,377
20
static.wixstatic.com3,386,446
21
t0.gstatic.com3,184,876
22
i.ytimg.com2,926,226
23
s3.amazonaws.com2,507,429
24
i.ebayimg.com2,426,737
25
www.picclickimg.com2,161,006
26
s.yimg.com2,120,438
27
tse1.mm.bing.net2,094,722
28
m.psecn.photoshelter.com2,014,122
29
data.whicdn.com1,990,765
30
tse2.mm.bing.net1,979,720
31
tse3.mm.bing.net1,977,842
32
tse4.mm.bing.net1,971,243
33
www.gannett-cdn.com1,865,603
34
cdn11.bigcommerce.com1,780,222
35
cdn.vectorstock.com1,742,477
36
lh3.googleusercontent.com1,567,788
37
ctl.s6img.com1,540,649
38
img.freepik.com1,518,977
39
1.bp.blogspot.com1,378,063
40
m.media-amazon.com1,369,303
41
upload.wikimedia.org1,336,191
42
cdp.azureedge.net1,324,296
43
ak1.ostkcdn.com1,302,841
44
images.fineartamerica.com1,274,492
45
img0.etsystatic.com1,262,043
46
img1.etsystatic.com1,256,921
47
image.shutterstock.com1,239,541
48
media-cdn.tripadvisor.com1,232,173
49
storage.googleapis.com1,227,849
50
res.cloudinary.com1,218,917
51
encrypted-tbn0.gstatic.com1,132,283
52
s.hdnux.com1,078,376
53
2.bp.blogspot.com1,036,320
54
i.etsystatic.com1,036,028
55
t3.ftcdn.net1,021,414
56
t4.ftcdn.net1,019,486
57
3.bp.blogspot.com1,000,095
58
4.bp.blogspot.com984,590
59
media.istockphoto.com961,343
60
img.shopstyle-cdn.com956,743
61
ih1.redbubble.net936,580
62
secureservercdn.net928,803
63
a.1stdibscdn.com901,073
64
rlv.zcache.com899,648
65
imgc.artprintimages.com897,180
66
images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com849,957
67
ih0.redbubble.net844,526
68
cdn.xxl.thumbs.canstockphoto.com844,427
69
www.dhresource.com813,172
70
img01.ztat.net775,449
71
t2.ftcdn.net774,236
72
img.etsystatic.com773,945
73
t1.ftcdn.net773,581
74
a0.muscache.com740,522
75
lid.zoocdn.com734,721
76
st.depositphotos.com734,130
77
n.nordstrommedia.com728,710
78
st.hzcdn.com683,027
79
img.wfrcdn.com588,729
80
s7d9.scene7.com573,604
81
images.pond5.com559,827
82
media1.popsugar-assets.com544,789
83
secure.img1-fg.wfcdn.com539,153
84
imagesvc.meredithcorp.io531,495
85
t00.deviantart.net526,772
86
cdn.shoplightspeed.com523,455
87
di2ponv0v5otw.cloudfront.net516,604
88
iss-cdn.myrealpage.com515,025
89
s7d2.scene7.com514,174
90
s3-eu-west-1.amazonaws.com512,975
91
bt-photos.global.ssl.fastly.net511,375
92
secure.img1-ag.wfcdn.com505,374
93
p1.liveauctioneers.com503,077
94
images.thestar.com501,850
95
img.youtube.com499,223
96
pbs.twimg.com490,508
97
image.made-in-china.com488,927
98
rlv.zcache.co.uk486,406
99
sc01.alicdn.com483,831
100
www.cheatsheet.com479,343