Release Log

Page history last edited by Shion Deysarkar 1 year, 8 months ago

Table of Contents

Release - 1.4 - 20 April 2010

Crawl Packages released. Purchase pre-configured crawls and start using 80legs in a few simple clicks.
Marketplace released. The 80app Store has been merged with the Marketplace. Products available: Crawl Packages, 80apps.

80app Store: pre-built 80apps available for purchase. Developers can sell their own 80apps to earn money.
80app Packs: each subscription plan now includes select pre-built 80apps from the 80app Store absolutely free!

Larger crawls now supported - max # of pages per job increased to 10 million
Implemented three types of crawls to support specific user needs - fast, comprehensive, and breadth-first
Can now crawl https:// pages
Better handling of only crawling on current domain
Crawler tries to fetch page more than once before returning NO_RESPONSE error
Crawls no longer hang and slow down the job when stuck on certain domains

Official release of the Java API - programmatically submit jobs, download results, etc.
Sample desktop application for using 80legs available (including source code)

New and easier-to-use job form in the web portal (as well as updated interface throughout the portal)

Job form switches certain fields to defaults in a more convenient way
Code section in portal provides more information
Results section in portal provides more information
Several performance improvements for crawling and back-end data store
Calls to constructors in 80apps are only charged once per 1000 URLs (note: 80legs is still free to use for now)

Smarter URL selection for larger crawls
Sandbox jobs run automatically and the user gets access to stdout from their 80app
Domain throttling information in the portal
Time estimates shown in the portal
Crawled results files additions:

Many different improvements to the handling of 80apps to help developers get started
Payment system beta (80legs will still be free for now, but we'll have our users add pretend money to their accounts)
Allow larger result sizes (1KB/page is still free with a 10KB/page max)
More settings for crawl control
More crawl status messages (see here)
Many more backend improvements based on user feedback from 0.80
Results returned faster after job completes
Improved 80app class loading time

This major release is the first time 80legs customers are able to use their own 80apps (custom code).

80apps initial release (first I80App release with parseLinks() and processDocument())
Option to analyze specific MIME types
Option to preserve query strings when crawling
Resulting crawl list shows status codes and other reasons for failing to crawl (e.g. robots.txt, DNS, etc)
Better handling of failed URLs
Sandbox server for testing custom code on your own machine using the 80legs framework.
Stop problem jobs automatically

Portal keeps you logged in until you're inactive for 30 minutes
Reduce duplicate pages crawled for small jobs (there are still quite a few, but reducing it further potentially increases the job runtime significantly)
Throttle based on parent domains and IP addresses
Option to only crawl on domains from seed list
Gzip/deflate crawling
Improved foreign language content selection (UTF-8)
Increase portal limit to 1M maxPagesToCrawl
Improved URL parsing

Centralized Robot/DNS higher performance.
Removed long pauses from some longer-running jobs
In the Portal, made seed lists writable when copying job
In the Portal, added help section on actions you can do on jobs (copy, cancel, delete)
Improved robots.txt handling - fixed empty disallow string
Fix for jobs sometimes not starting right and showing completed even though it has zero pages done
Refined crawler regular expressions (about: links not being ignored)
Fixed formatting in Portal result files
Cancel job bug in Portal fixed

Compress Portal results files
Set user-agent as 80bot (Fix for Alibaba.com)
In the Portal, take an old job and resubmit, allowing for parameter changes ("Copy Job" link)
Global robots.txt and DNS Manager
Fix for ending conditions for some crawls
Option to "Include crawled but not analyzed" pages in results
Add change log to external website and within portal
Add explanation for result file naming convention
Refined crawler regular expressions (some mailto: and javascript: links were not being ignored)