(by nicholas, last updated: jun 2013)
headers returned by server
?
?
url scheme (http/https/ftp/...)
duplication_types
?
issues this page has by id
meta-description from this page
total count of links on the page
all urls for all the links on the page
redirect_to_string
redirect location if 30x
last crawl date
word count
duplicate_page_ids
reference to duplicate pages
page_inlinks_count
inbound links count internal links count. total amount of links to this page in this domain.
host this page is found on
no idea, seems always empty
Active/inactive No idea what it stands for
reference to the histories stored on S3
simularity hash for the title
description_simhash
description simularity hash
returned status code
url of the page
No idea, is always empty
last updated timestamp
response time of the page
no idea
list of images on the page
duplicate_simhash
simhash to detect duplicate pages (body)
redirect location if 30x (duplicate of redirect_to_string)
port used to fetch page
opengraph_site_name
open graph site name if present
query of the url if present
??
potential_keywords
list of potential keywords
hash of cleaned body
This is always "Page" TODO: Ask william
classification of authority
page title
title_duplicate_pages
title of duplicate pages
opengraph type if present on page
url path
creation timestamp
absolute url
duplicate_pages
duplicate pages ids (?? duplicate_page_ids dupe?),
fragment of url
full text of the page
no idea TODO: Ask william
list of tags (duplicate of body, conversion, ...)
backlink count
list of inlinks
timestamp of last recheck
domain it belongs to
(??? always empty)
hash of body
list of links on the page with detail information
robot tags attached to this page (all, index, nofollow)...(??? how is this calculated)
description_duplicate_pages
list of duplicate description pages
total simhash
list of anchor texts on the page
