Preventing Unauthorized Image Crawling by Bots: A Clever Discovery and How to Address It
CK CK 3788 
Created at
Updated at  
 1,726 0 0

Today, while deploying our new service, we stumbled upon an intriguing method used by Amazonbot to crawl our site and randomly scrape images. By appending /?C=M;O%3DA to the folder name and accessing it, this bot was able to grab all the files within if the web server's indexing option was set to allow.

3.22.35.162 - - [28/Apr/2024:09:33:08 -0700] "GET /blahblah/?C=M;O%3DA HTTP/1.1" 403 220 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)"

With the rise of AI bots wandering the web and collecting data without permission, then building AI services based on this data, encountering a bot like Amazonbot, which indiscriminately scrapes content, is quite unusual.

Fortunately, for Apache web servers, simply adding the following options to the .htaccess file can effectively prevent such scraping:

Options -Indexes

Implementing this simple change can significantly enhance your site's performance and prevent unwanted traffic. If you've noticed a slowdown in performance or an increase in suspicious traffic, blocking indexing as mentioned above can make a big difference.

It's crucial to maintain control over your website's content and prevent unauthorized scraping, especially as bots become more sophisticated in their methods. By taking proactive measures like this, you can ensure the integrity of your site and protect your content from being misused.

Tags .htaccess .htaccess Configuration AI Bots Amazonbot Apache Server Bot Detection Data Privacy Image Scraping Web Crawling Web Performance Website Security Facebook X
Comments 0
Similar posts
  1. The Future of Programmatic Advertising: Privacy-First Strategies and Trade-offs
    513
  2. Viant Technology - a pioneer people-based digital advertising on the New Open Web
    1,215
  3. A prefect Apache server settings to avoid CORS error for HLS
    603
  4. Redis vs Memcached: Why Enterprise Tech Giants Choose Redis
    247
More in Linux
  1. Why SPF and DKIM Are Essential for Linux Email Servers?
    24
  2. Upgrading to GCC 4.9.4 in CentOS 7 requires installing a newer version manually
    579
  4. Force no caching in Apache by modifying the .htaccess file using appropriate headers
    2,668
  5. Beyond your conventional memory size with swap file instead of a partition
    1,138
  6. Gmail requires all senders to authenticate with either SPF or DKIM.
    679
  7. Bash script to install postfix and sasl for sending email
    1,192
  8. Guide for tmux shortcuts - split current pane, toggle pane
    1,294
  9. Tistory remote blogging example based in CLI & PHP
    2,706
  10. Generate Let’s Encrypt Wildcard SSL Certificate
    625
  11. Network diagnostic tool for IPv4/IPv6 - traceroute/traceroute6
    1,915
  12. chmod by type (directory and file permission)
    733
  13. Check network port is open or not on CentOS 7
    1,186
  14. Change MySQL port number for security purpose
    815
  15. CentOS warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory on CentOS 7
    1,070
  17. Set up an SMTP server on CentOS 7
    726
  18. How to mount windows shared folder on Centos 6.8 - mount error 12 = Cannot allocate memory to the mount.cifs(8)
    911
  19. Configuring IP Address on Centos 6.7
    786
  20. Adding new cron job for automated processing on every 10 minutes on Centos 6.7
    1,329
  21. Set timezone and time on Centos 6.7
    540
  22. How to optimize MySQL automatically in CentOS Linux
    1,006
  23. File conversion to UTF in Schell Script
    1,021
  24. Server status checking by Shell Script
    1,514
  25. How can I increase MySQL max connections?
    458
Recently updated
  1. Why 100% SEO Guarantees Better User Engagement and How to Achieve It in Your Service
    27
  3. Why SPF and DKIM Are Essential for Linux Email Servers?
    24
  4. How do I create animated background for Google Meet?
    19,731
  5. MaxMind GeoLite2 vs. DB-IP Lite vs. ip-api.com: Choosing the Right Free IP Geolocation Solution
    217
  6. Meet Valkey: Why Redis Changed Its License, Why Valkey is Better, and How to Use It as a Drop-in Replacement
    86
  7. How Rogue Web Scrapers Are Stealing My Content and Destroying My Server
    153
  8. Array of arguments passed to script in PHP
    85
  9. Invalid Auction Storms in Ad Tech
    93
  10. Building MRC-Compliant Instrumentation in Ad Tech: Implementation Steps and Key Challenges
    198
  11. The Two Clocks of Ad Server Architecture
    63
  12. Frequency Cap Architecture Overview
    509
  13. Collecting Proof-of-Play from 100,000 Screens
    141
  14. Vector Search in Relational Databases (RDBs): Trends & Usage Environments
    280
  15. What is RedisVL?
    105
  16. Designing a Skill Registry Architecture: Minimize Token Usage and Maximize AI Performance
    224
  17. TPU & GPU Cluster Management Tools
    234
  19. The Concept of Scaffolding in AI Agents
    222
  20. Beyond Static Pipelines: Architecting LLM Multi-Agent Systems for Autonomous Workflow Orchestration
    576
  21. AI-DLC vs. AI-SDLC: Model-Centric vs. System-Centric AI Development
    253
  22. Fine-Tuning Open-Weight LLMs: Training, Licensing, and Deploying Custom Models with Ollama
    328
  23. Inside the Next-Generation Streaming Architecture: Ultra-Low Latency, SGAI, and AI Codecs
    197
  24. Machine Learning: Training vs. Ranking
    213
  25. The Continuous Learning Machine: Designing a Loop That Actually Turns
    217
  26. The Next-Gen Low-Latency Standards: LL-HLS & MoQ
    189
  27. Designing a Data Curation System Around a Vector Database
    215
  28. Extracting Context from Live Video: The Architecture Got Simpler When We Stopped Building Features
    229
  29. The New Ad Stack Behind Live Sports Streaming: Inside SGAI, EABN, and the Standards Race to Fix Live Monetization
    649
  31. Designing MLOps Architectures to Handle Seasonal Data Drift in EV Fleet Models
    185
  32. From 15 to 40+ FPS: Accelerating PyTorch Object Detection on Automotive ECUs
    207
  33. Understanding Data Lineage
    252