Big, Hungry, Orange RSS Alligators

178941 Resource 76096 21 21 1186858898411 1186925868728 Controlled Fighting Back Against Big, Hungry, Orange Alligators

Overly aggressive RSS feed aggregators can cause server performance problems; MyST Blogsite introduces new technology to detect (and reject) egregious offenders.

<h3>The Problem</h3><p>More than three years ago, MyST co-founder Bill French and I started referring to overly aggressive RSS feed readers as <i><a target="_blank" href="http://myst-technology.com/mysmartchannels/public/item/28885" title="First blog post: Big, Hungry, Orange RSS Alligators ">orange alligators</a></i>.  It was a pun that combined the color of the already ubiquitous RSS icons and the closeness of the words "aggregator" and "alligator".  The idea was that these alligators seemed insatiably hungry for our server bandwidth.  We lamented that while most RSS clients were well behaved (i.e., they used conditional requests, reasonable polling frequencies, identifiable user agent names, etc.), some were very <i>poorly</i> behaved and could [and did] cause serious server loading issues.</p><div style="border: 1px solid rgb(204, 204, 204); padding: 6px; float: right; margin-bottom: 8px; margin-left: 8px; width: 181px;"><img width="169" vspace="0" hspace="0" height="85" border="0" src="http://blogsite.com/styles/blogsite/Blogsite/content/orange-aligator.jpg" alt="Big, Hungry, Orange Alligators" /> <div style="font-weight: bold; font-size: 9pt; text-align: center;">Overly Aggressive<br />Feed Readers</div></div><p>What do I mean by "poorly behaved"?  For example, imagine a feed reader polling each of twelve feeds in a large blogsite every 30 seconds; further imagine the reader using unconditional requests and requesting cache refresh, thereby forcing the feed to be re-delivered, in its entirety, for each request.  This translates into 24 full-content requests per minute (that's a request every 2.5 seconds) just to service a single client!  Depending on the nature of the channel content, this can mean serving up many megabytes per minute of the same, comparatively slowly changing, content, over and over—big, hungry, orange alligators eating bandwidth sandwiches.  And it can be worse than just wasting bandwidth.  In the case of dynamic content, making unconditional, cache-refreshing requests is actually asking the server to <i>recreate</i> the dynamic content for each request.  This can lead to server overload conditions where the server simply can't satisfy the alligator's appetite, resulting in poor server performance for everybody, not just the alligator.</p><h3>The Solution</h3><p>We knew that the day would come when it was necessary to introduce server-side technology to detect and manage alligators.  For MyST Blogsite<sup>®</sup> servers, that day has come.  A little over two years ago, we introduced MyST SlimeGate™, a server-side technology that protects our servers against attacks by hackers and spammers.  Yesterday we introduced, to all MyST Blogsite servers, our new MyST AlliGate™ technology that protects against overly aggressive content consumers, most of which are poorly behaved RSS feed readers but could also include other types of web clients.</p><p>AlliGate monitors server requests watching for alligators.  AlliGate uses a variety of heuristics to identify alligators, but once an alligator is identified, its IP address is submitted to SlimeGate for "management".  Managed addresses may be dynamically blocked at the network firewall, preventing them from <i>any</i> communication with the server for a short period of time.  Once access is restored, the client may again access the server.  However, a subsequent offense by an managed IP address results in another firewall lockout, but this time for twice as long as the time before.  Also, once access is restored, if a managed IP goes for a while (currently 30 days) without further offense, the IP is completely forgiven.</p><p>While the heuristics used to detect alligators are somewhat complex, here are some recommendations that, if followed, will ensure that you never trigger AlliGate to "manage" your access:</p><ol><li>Do not poll a given feed more that five times per hour.  (Blogsite content is not more "real-time" than that anyway—in most cases, checking once per hour is perfectly adequate.)</li><li>Use web clients (e.g., RSS readers, browsers, etc.) that make conditional requests.</li><li>Do not force cache refresh on all requests.  (Refreshing cache occasionally is okay, but it should not be standard operating procedure.)</li></ol><h3>The Benefits</h3><p>Today, all MyST Blogsite servers are protected against hackers, referrer spammers, and now, orange alligators.  Our experience managing nefarious requests to hundreds of commercial blogsites is reflected in technologies like MyST SlimeGate and MyST AlliGate that ultimately benefit all of our clients.  This is another example of why our clients turn to us to manage and host their corporate blogsite.  Rather than "going it alone" and fending off hackers, spammers, and alligators, our clients leave the nitty-gritty logistics of operating a high-visibility blogsite to us.  This lets them wisely focus their attention, resources, and expertise on creating the best possible content for their site.  Its a partnership that just makes sense.</p> 9 application/xml false aggressive aggregators aggressive feed readers MyST AlliGate MyST SlimeGate orange alligators referrer spam RSS adoption Big, Hungry, Orange RSS Alligators Imagine a swamp that stretches to the horizon... http://myst-technology.com/mysmartchannels/public/item/28885 RSS Adoption Train--Next Stop: Mainstream The RSS Adoption train is picking up speed; if you're not already on board, it's time to buy a ticket. http://myst-technology.com/mysmartchannels/public/item/56043 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.5) Gecko/20070713 Firefox/2.0.0.5 127.0.0.1 127.0.0.1 aseidl false true false false true 174050 false false true false false false true 179103 Resource 174050 173239 21 1187054753522 1206884393953 Controlled Brilliant!

10 application/xml false xlint:ignore Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) 127.0.0.1 127.0.0.1 aseidl 68.40.167.222 false true false false true 76096 blog blogsite/Blogsite/web