600px-Robot_icon.svg.png Don’t Let the Bots Win 600px-Robot_icon.svg.png

Tipsheet by Liam Andrew (@mailbackwards) & Daniel Craigmile (@x110dc)

At NICAR you can learn to build bots (Slack bots, chat bots, scripts, crawlers, spiders, scrapers, and so on). These are great, but they also leave a trace on target sites. What is the experience for sites that are the target of bots-- whether they’re search engines, data scrapers, or malicious botnets? How can you treat bots as “users” of websites, and help them accomplish their goals? How can you build bots that are respectful towards the sites they crawl?

Bots are substantial users of news websites

Identifying bots

Managing bots

Building a bot-friendly site

Building a web-friendly bot