Beginning in early 2024, McGill Libraries and IT Services faced an unexpected challenge: an overwhelming surge of automated web traffic directed at two of the Libraries’ major public platforms – eScholarship institutional digital repository for theses and open access materials, and AtoM, our archival collections site. What started as occasional slowdowns quickly escalated into multiple daily outages that disrupted access for students, researchers, and the public.
The cause was something many institutions face: AI‑driven web crawling. As artificial intelligence systems search for large amounts of online text to train their models, universities, libraries, and museums around the world have been hit with substantial increases in automated traffic. While McGill Libraries strongly supports open access, this rise has unintended consequences including bringing down systems making it difficult for legitimate users to reach scholarly content.
Traditional fixes such as blocking specific crawlers or adjusting firewall rules don’t solve the issue. At times, they even block helpful services like Google Scholar. It became clear the Libraries and IT Services needed to try something new to fix this issue.
In May 2025, IT Services proposed testing Azure Front Door, a content delivery network (CDN) designed to distribute cached copies of webpages across servers worldwide. By doing so, the system could absorb heavy traffic, reduce strain on McGill’s servers, and deliver content more quickly to users regardless of location.
The teams worked closely together to implement the solution, and the results were immediate. As Mutugi Gathuri, Software Developer at McGill Libraries, shared, “Our eScholarship site used to crash every hour (sometimes every 30 minutes!) under bot attacks, but now it hasn’t skipped a beat! Now our editors can focus on content, not firefighting. and users enjoy uninterrupted access. It’s been a fantastic collaboration and a real win for everyone.”
For Awais Mehmood Khalid, Libraries’ Programmer Analyst, the project showcased the strength of cross‑unit teamwork, “IT [Services] showed a keen understanding of our requirements and an unwavering willingness to help. Their solution absorbed the surge of AI‑driven bot traffic and restored performance across our platforms dramatically improving load times and overall resilience.”
Deepak Chauhan, Programmer Analyst at the Libraries, echoed the impact, “This helped us manage the traffic more efficiently and kept things running smoothly for real users. It made a noticeable difference in our site’s performance and stability.”
From the IT perspective, Graham Thorpe, Solutions Architect, noted the technical benefits, “[Azure Front Door] takes the load off our back-end servers, solving load-related issues, and brings the content closer to users making the sites more responsive. While some application tweaking was needed, the solution has been great from our perspective.”
This project stands as a success story of collaboration, problem‑solving, and dedication to keeping McGill’s digital scholarship accessible and secure.
Kudos to the teams who made this possible:
- IT Services: Matt Corks, Thomas Fline, Zachary McGibbon, Spiro Mitsialis, Graham Thorpe, Chung Yu
- McGill Libraries: Deepak Chauhan, Gagandeep Dhillon, Mutugi Gathuri, Awais Mehmood Khalid including the tireless testers Robin Desmeules, Anna Dysert, and Jennifer Innes.
Leave a Reply
You must be logged in to post a comment.