80legs Documentation
This wiki can provide you with information on 80legs as well as help on using its many features.
Table of Contents
Introduction
80legs lets you access the entire Internet and perform analysis on Internet-wide content quickly and easily. Our goal is to provide the fastest, most robust, and easiest way to analyze the entire web.
The 80legs platform provides three major components for web-scale application development:
- Web Portal: The portal allows you to customize a number of settings, including when a crawl (or "job") will run, how often it will run, which pages to crawl, and so on.
- 80apps: Use pre-built apps to get the content you want or write your own code to analyze any web page.
- API: The API lets you create and manage jobs programmatically. Code libraries in a variety of languages will be provided.
Getting Started
To start using 80legs, just follow these simple steps:
- Register and login at http://portal.80legs.com.
- Try running a sample crawl by following the instructions shown to you when you first login.
- Learn more about how to use 80legs by reading the documentation here.
If you require help along the way, you can submit a support ticket or contact us directly.
The web portal provides an easy-to-use interface that allows you start crawling the web right away. It will also provide tools for account management and other important services. Click here to view.
Each crawl is run as a unique "job" on the 80legs platform. You can specify crawl settings independently for each job. Click here to view.
The real power of 80legs is in the custom code you create to run on the system. Our customers can developer their own custom document analysis functionality to run across all the documents they crawl on the Internet. This functionality allows customers to analyze the web in any way they want. Click here to view.
The API is a simple programmatic interface that allows you to access and use 80legs from outside of the web portal. Click here to view.
When a crawl (or "job") completes a run, it will create a file which shows the results of your job. Click here to view.
80legs is free to use for crawls up to 10,000 pages! You can crawl up to 10,000 pages under the Basic Plan, or subscribe to the Plus or Premium plan for higher limits, added features, and lower restrictions. Our pricing makes it possible for everyone to easily scale up their web-scale analysis. Click here to view.
Check out the FAQ for answers to a wide variety of questions. Click here to view.
Want to learn how to best take advantage of 80legs? Some tips and best practices are included here to help you improve the performance of your crawls. Click here to view.
80legs has some limitations which prevent it from crawling certain websites. Many of these should be address in future updates. Click here to view.
Check the release log for information on the latest changes to 80legs. You can also find out what we have in store for upcoming releases. Click here to view.
Comments (0)
You don't have permission to comment on this page.