Webpages are a treasure-trove of useful information for financial firms and software companies that are able to capture it using web crawling (or scraping) technology. Yet, for over 20 years, courts have struggled to draw the line between the usefulness of such information and the rights of the content owners and website operators from which that content is derived. Once a niche issue, the increased use of this technology has compounded the disputes related to it.
In particular, website operators have used the Computer Fraud and Abuse Act (CFAA) to prevent crawling of their websites. While recent judicial opinions have harmonized the rules for accessing websites without authorization, the courts diverge as to whether the CFAA prohibits accessing otherwise publicly available information for an unauthorized purpose. Moreover, new web crawling techniques are testing the limits of existing case law.1
