org.apache.nutch.tools.arc

Class ArcSegmentCreator

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.tools.arc.ArcSegmentCreator

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.Text,NutchWritable>, org.apache.hadoop.util.Tool

public class ArcSegmentCreator
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool, org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.Text,NutchWritable>

The ArcSegmentCreator is a replacement for fetcher that will take arc files as input and produce a nutch segment as output.

Arc files are tars of compressed gzips which are produced by both the internet archive project and the grub distributed crawler project.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG
static String	URL_VERSION

Constructor Summary

Constructors

Constructor and Description
ArcSegmentCreator()
ArcSegmentCreator(org.apache.hadoop.conf.Configuration conf) Constructor that sets the job configuration.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
void	configure(org.apache.hadoop.mapred.JobConf job) Configures the job.
void	createSegments(org.apache.hadoop.fs.Path arcFiles, org.apache.hadoop.fs.Path segmentsOutDir) Creates the arc files to segments job.
static String	generateSegmentName() Generates a random name for the segments.
static void	main(String[] args)
void	map(org.apache.hadoop.io.Text key, org.apache.hadoop.io.BytesWritable bytes, org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output, org.apache.hadoop.mapred.Reporter reporter) Runs the Map job to translate an arc record into output for Nutch segments.
int	run(String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

LOG

public static final org.slf4j.Logger LOG

URL_VERSION

public static final String URL_VERSION

See Also:

Constant Field Values

Constructor Detail

ArcSegmentCreator

public ArcSegmentCreator()

ArcSegmentCreator

public ArcSegmentCreator(org.apache.hadoop.conf.Configuration conf)

Constructor that sets the job configuration.

Parameters:

conf -

Method Detail

generateSegmentName

public static String generateSegmentName()

Generates a random name for the segments.

Returns:

The generated segment name.

configure

public void configure(org.apache.hadoop.mapred.JobConf job)

Configures the job. Sets the url filters, scoring filters, url normalizers and other relevant data.

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

Parameters:

job - The job configuration.

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

map

public void map(org.apache.hadoop.io.Text key,
       org.apache.hadoop.io.BytesWritable bytes,
       org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output,
       org.apache.hadoop.mapred.Reporter reporter)
         throws IOException

Runs the Map job to translate an arc record into output for Nutch segments.

Specified by:

map in interface org.apache.hadoop.mapred.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.Text,NutchWritable>

Parameters:

key - The arc record header.

bytes - The arc record raw content bytes.

output - The output collecter.

reporter - The progress reporter.

Throws:

IOException

createSegments

public void createSegments(org.apache.hadoop.fs.Path arcFiles,
                  org.apache.hadoop.fs.Path segmentsOutDir)
                    throws IOException

Creates the arc files to segments job.

Parameters:

arcFiles - The path to the directory holding the arc files

segmentsOutDir - The output directory for writing the segments

Throws:

IOException - If an IO error occurs while running the job.

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception