Edelman then assembled a database of Web sites for possible testing. He derived this list by automatically compiling URLs from the Yahoo index of Web sites, taking them from categories from the Yahoo index that differed significantly from the classifications that he had enabled in each of the blocking programs (taking, for example, Web sites from Yahoo's "Government" category). He then expanded this list by entering URLs taken from the Yahoo index into the Google search engine's "related" search function, which provides the user with a list of similar sites. Edelman also included and excluded specific Web sites at the request of the plaintiffs' counsel.
Taking the list of more than 500,000 URLs that he had compiled, Edelman used an automated system that he had developed to test whether particular URLs were blocked by each of the four filtering programs. This testing took place between February and October 2001. He recorded the specific dates on which particular sites were blocked by particular programs, and, using commercial archiving software, archived the contents of the home page of the blocked Web sites (and in some instances the pages linked to from the home page) as it existed when it was blocked. Through this process, Edelman, whose testimony we credit, compiled a list of 6,777 URLs that were blocked by one or more of the four programs. Because these sites were chosen from categories from the Yahoo directory that were unrelated to the filtering categories that were enabled during the test (i.e., "Government" vs. "Nudity"), he reasoned that they were likely erroneously blocked. As explained in the margin, Edelman repeated his testing and discovered that Cyber Patrol had unblocked most of the pages on the list of 6,777 after he had published the list on his Web site. His records indicate that an employee of SurfControl (the company that produces Cyber Patrol software) accessed his site and presumably checked out the URLs on the list, thus confirming Edelman's judgment that the majority of URLs on the list were erroneously blocked. Edelman forwarded the list of blocked sites to Dr. Joseph Janes, an Assistant Professor in the Information School of the University of Washington who also testified at trial as an expert witness. Janes reviewed the sites that Edelman compiled to determine whether they are consistent with library collection development, i.e., whether they are sites to which a reference librarian would, consistent with professional standards, direct a patron as a source of information.
Edelman forwarded Janes a list of 6,775 Web sites, almost the entire list of blocked sites that he collected, from which Janes took a random sample of 859 using the SPSS statistical software package. Janes indicated that he chose a sample size of 859 because it would yield a 95% confidence interval of plus or minus 2.5%. Janes recruited a group of 16 reviewers, most of whom were current or former students at the University of Washington's Information School, to help him identify which sites were appropriate for library use. We describe the process that he used in the margin. Due to the inability of a member of Janes's review team to complete the reviewing process, Janes had to cut 157 Web sites out of the sample, but because the Web sites were randomly assigned to reviewers, it is unlikely that these sites differed significantly from the rest of the sample. That left the sample size at 699, which widened the 95% confidence interval to plus or minus 2.8%.
Of the total 699 sites reviewed, Janes's team concluded that 165 of them, or 23.6% percent of the sample, were not of any value in the library context (i.e., no librarian would, consistent with professional standards, refer a patron to these sites as a source of information). They were unable to find 60 of the Web sites, or 8.6% of the sample. Therefore, they concluded that the remaining 474 Web sites, or 67.8% of the sample, were examples of overblocking with respect to materials that are appropriate sources of information in public libraries. Applying a 95% confidence interval of plus or minus 2.8%, the study concluded that we can be 95% confident that the actual percentage of sites in the list of 6,775 sites that are appropriate for use in public libraries is somewhere between 65.0% and 70.6%. In other words, we can be 95% certain that the actual number of sites out of the 6,775 that Edelman forwarded to Janes that are appropriate for use in public libraries (under Janes's standard) is somewhere between 4,403 and 4,783.
The government raised some valid criticisms of Janes's methodology, attacking in particular the fact that, while sites that received two "yes" votes in the first round of voting were determined to be of sufficient interest in a library context to be removed from further analysis, sites receiving one or two "no" votes were sent to the next round. The government also correctly points out that results of Janes's study can be generalized only to the population of 6,775 sites that Edelman forwarded to Janes. Even taking these criticisms into account, and discounting Janes's numbers appropriately, we credit Janes's study as confirming that Edelman's set of 6,775 Web sites contains at least a few thousand URLs that were erroneously blocked by one or more of the four filtering programs that he used, whether judged against CIPA's definitions, the filters' own category criteria, or against the standard that the Janes study used. Edelman tested only 500,000 unique URLs out of the 4000 times that many, or two billion, that are estimated to exist in the indexable Web. Even assuming that Edelman chose the URLs that were most likely to be erroneously blocked by commercial filtering programs, we conclude that many times the number of pages that Edelman identified are erroneously blocked by one or more of the filtering programs that he tested. Edelman's and Janes's studies provide numerous specific examples of Web pages that were erroneously blocked by one or more filtering programs. The Web pages that were erroneously blocked by one or more of the filtering programs do not fall into any neat patterns; they range widely in subject matter, and it is difficult to tell why they may have been overblocked. The list that Edelman compiled, for example, contains Web pages relating to religion, politics and government, health, careers, education, travel, sports, and many other topics. In the next section, we provide examples from each of these categories. 6. Examples of Erroneously Blocked Web Sites
Several of the erroneously blocked Web sites had content relating to churches, religious orders, religious charities, and religious fellowship organizations. These included the following Web sites: the Knights of Columbus Council 4828, a Catholic men's group associated with St. Patrick's Church in Fallon, Nevada, http://msnhomepages.talkcity.com/SpiritSt/kofc4828, which was blocked by Cyber Patrol in the "Adult/Sexually Explicit" category; the Agape Church of Searcy, Arkansas, http://www.agapechurch.com, which was blocked by Websense as "Adult Content"; the home page of the Lesbian and Gay Havurah of the Long Beach, California Jewish Community Center, http://www.compupix.com/gay/havurah.htm, which was blocked by N2H2 as "Adults Only, Pornography," by Smartfilter as "Sex," and by Websense as "Sex"; Orphanage Emmanuel, a Christian orphanage in Honduras that houses 225 children, http://home8.inet.tele.dk/rfb_viva, which was blocked by Cyber Patrol in the "Adult/Sexually Explicit" category; Vision Art Online, which sells wooden wall hangings for the home that contain prayers, passages from the Bible, and images of the Star of David, http://www.visionartonline.com, which was blocked in Websense's "Sex" category; and the home page of Tenzin Palmo, a Buddhist nun, which contained a description of her project to build a Buddhist nunnery and international retreat center for women, http://www.tenzinpalmo.com, which was categorized as "Nudity" by N2H2.
Several blocked sites also contained information about governmental entities or specific political candidates, or contained political commentary. These included: the Web site for Kelley Ross, a Libertarian candidate for the California State Assembly, http://www.friesian.com/ross/ca40, which N2H2 blocked as "Nudity"; the Web site for Bob Coughlin, a town selectman in Dedham, Massachusetts, http://www.bobcoughlin.org, which was blocked under N2H2's "Nudity" category; a list of Web sites containing information about government and politics in Adams County, Pennsylvania, http://www.geocities.com/adamscopa, which was blocked by Websense as "Sex"; the Web site for Wisconsin Right to Life, http://www.wrtl.org, which N2H2 blocked as "Nudity"; a Web site that promotes federalism in Uganda, http://federo.com, which N2H2 blocked as "Adults Only, Pornography"; "Fight the Death Penalty in the USA," a Danish Web site dedicated to criticizing the American system of capital punishment, http://www.fdp.dk, which N2H2 blocked as "Pornography"; and "Dumb Laws," a humor Web site that makes fun of outmoded laws, http://www.dumblaws.com, which N2H2 blocked under its "Sex" category. Erroneously blocked Web sites relating to health issues included the following: a guide to allergies, http://www.x- sitez.com/allergy, which was categorized as "Adults Only, Pornography" by N2H2; a health question and answer site sponsored by Columbia University, http://www.goaskalice.com.columbia.edu, which was blocked as "Sex" by N2H2, and as "Mature" by Smartfilter; the Western Amputee Support Alliance Home Page, http://www.usinter.net/wasa, which was blocked by N2H2 as "Pornography"; the Web site of the Willis-Knighton Cancer Center, a Shreveport, Louisiana cancer treatment facility, http://cancerftr.wkmc.com, which was blocked by Websense under the "Sex" category; and a site dealing with halitosis, http://www.dreamcastle.com/tungs, which was blocked by N2H2 as "Adults, Pornography," by Smartfilter as "Sex," by Cyber Patrol as "Adult/Sexually Explicit," and by Websense as "Adult Content."
The filtering programs also erroneously blocked several Web sites having to do with education and careers. The filtering programs blocked two sites that provide information on home schooling. "HomEduStation the Internet Source for Home Education," http://www.perigee.net/~mcmullen/homedustation/, was categorized by Cyber Patrol as "Adult/Sexually Explicit." Smartfilter blocked "Apricot: A Web site made by and for home schoolers," http://apricotpie.com, as "Sex." The programs also miscategorized several career-related sites. "Social Work Search," http://www.socialworksearch.com/, is a directory for social workers that Cyber Patrol placed in its "Adult/Sexually Explicit" category. The "Gay and Lesbian Chamber of Southern Nevada," http://www.lambdalv.com, "a forum for the business community to develop relationships within the Las Vegas lesbian, gay, transsexual, and bisexual community" was blocked by N2H2 as "Adults Only, Pornography." A site for aspiring dentists, http://www.vvm.com/~bond/home.htm, was blocked by Cyber Patrol in its "Adult/Sexually Explicit" category. The filtering programs erroneously blocked many travel Web sites, including: the Web site for the Allen Farmhouse Bed & Breakfast of Alleghany County, North Carolina, http://planet- nc.com/Beth/index.html, which Websense blocked as "Adult Content"; Odysseus Gay Travel, a travel company serving gay men, http://www.odyusa.com, which N2H2 categorized as "Adults Only, Pornography"; Southern Alberta Fly Fishing Outfitters, http://albertaflyfish.com, which N2H2 blocked as "Pornography"; and "Nature and Culture Conscious Travel," a tour operator in Namibia, http://www.trans-namibia-tours.com, which was categorized as "Pornography" by N2H2.
The filtering programs also miscategorized a large number of sports Web sites. These included: a site devoted to Willie O'Ree, the first African-American player in the National Hockey League, http://www.missioncreep.com/mw/oree.html, which Websense blocked under its "Nudity" category; the home page of the Sydney University Australian Football Club, http://www.tek.com.au/suafc, which N2H2 blocked as "Adults Only, Pornography," Smartfilter blocked as "Sex," Cyber Patrol blocked as "Adult/Sexually Explicit" and Websense blocked as "Sex"; and a fan's page devoted to the Toronto Maple Leafs hockey team, http://www.torontomapleleafs.atmypage.com, which N2H2 blocked under the "Pornography" category. 7. Conclusion: The Effectiveness of Filtering Programs Public libraries have adopted a variety of means of dealing with problems created by the provision of Internet access. The large amount of sexually explicit speech that is freely available on the Internet has, to varying degrees, led to patron complaints about such matters as unsought exposure to offensive material, incidents of staff and patron harassment by individuals viewing sexually explicit content on the Internet, and the use of library computers to access illegal material, such as child pornography. In some libraries, youthful library patrons have persistently attempted to use the Internet to access hardcore pornography.
Those public libraries that have responded to these problems by using software filters have found such filters to provide a relatively effective means of preventing patrons from accessing sexually explicit material on the Internet. Nonetheless, out of the entire universe of speech on the Internet falling within the filtering products' category definitions, the filters will incorrectly fail to block a substantial amount of speech. Thus, software filters have not completely eliminated the problems that public libraries have sought to address by using the filters, as evidenced by frequent instances of underblocking. Nor is there any quantitative evidence of the relative effectiveness of filters and the alternatives to filters that are also intended to prevent patrons from accessing illegal content on the Internet. Even more importantly (for this case), although software filters provide a relatively cheap and effective, albeit imperfect, means for public libraries to prevent patrons from accessing speech that falls within the filters' category definitions, we find that commercially available filtering programs erroneously block a huge amount of speech that is protected by the First Amendment. Any currently available filtering product that is reasonably effective in preventing users from accessing content within the filter's category definitions will necessarily block countless thousands of Web pages, the content of which does not match the filtering company's category definitions, much less the legal definitions of obscenity, child pornography, or harmful to minors. Even Finnell, an expert witness for the defendants, found that between 6% and 15% of the blocked Web sites in the public libraries that he analyzed did not contain content that meets even the filtering products' own definitions of sexually explicit content, let alone CIPA's definitions.