Table of Contents

Introduction

Crawl packages are pre-built crawls, designed and set up by the 80legs team, that crawl specific domains and retrieve specific information from those domains.  Crawl packages typically crawl websites with significant amounts of data, such as e-commerce sites, social networks, real estate sites, blogs and so on.

Purchasing a Crawl Package

To purchase a crawl package, log in to the Web Portal and go to the Marketplace tab.  You'll see a small selection of featured crawl packages on the Marketplace page.  Click on the "More" button to view all available crawl packages.

Once you've found a crawl package that interests you, click on the icon for it.  A popup will appear providing more details on the package.  Once you're ready to purchase, click on the "Buy Now" button to do so.  You'll then be able to select the crawl package by selecting "Live Data Feed" and clicking the "Buy Now" button again after making your selection.

After you've purchased a crawl package, it will become available in your account.  Go to the Crawl the Web tab, click on "My Crawls" and then "Crawl Packages".

Purchasing Archived Data

To purchase archived data, log in to the Web Portal and go to the Marketplace tab.  You'll see a small selection of featured crawl packages on the Marketplace page.  Click on the "More" button to view all available crawl packages.

Once you've found a crawl package that interests you, click on the icon for it.  A popup will appear providing more details on the crawl package.  Once you're ready to purchase, click on the "Buy Now" button to do so.  You'll then be able to select the specific archived data that you would like to purchase.

After you've purchased an archived data item, it will become immediately available in your account.  Go to the Crawl the Web tab, click on "My Crawls" and then "Archived Data".

Using a Crawl Package

Your crawl package should start producing data for you within a few hours.  Since you are purchasing access to a pre-built crawl, rather than the data itself, you'll need to wait some time for the crawl to run and start producing data.

As results become available, you can use the 80legs Downloader App to download the result files.  Results are typically in XML or CSV format, though there are some exceptions.