% Licensed to the Apache Software Foundation (ASF) under one or more
% contributor license agreements. See the NOTICE file distributed with
% this work for additional information regarding copyright ownership.
% The ASF licenses this file to You under the Apache License, Version 2.0
% (the ``License''); you may not use this file except in compliance with
% the License. You may obtain a copy of the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an ``AS IS'' BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.

\begin{changemargin}{1.5in}{0in}

\section{Overview}

The MetaCarta GTS appliance indexes documents and allows users to
search these documents based on both keywords and geographic
references. The MetaCarta Web Crawler allows system administrators
to configure connections and define jobs to ingest documents crawled
from web pages on the Internet.

This document specifies the means for crawling web pages, ingesting
content from those web pages, and maintaining the relationship between
your document index and those web pages.

\subsection{Assumptions}

This document assumes you have a basic level of familiarity with GTS
appliance administration. This document also assumes that you have a
basic understanding of the repositories to which you are trying to
connect. If you need more information about the MetaCarta GTS
appliance, please read the \documentref{MetaCarta GTS Administrator's
Guide} stored on the appliance at
\dirpath{/usr/share/doc/metacarta/AdminGuide.pdf}.

Throughout this document, we assume that your appliance is named \\
\url{metacarta.example.com}. 

\section{Installation}

The Web Crawler must be used with the MetaCarta Connector Framework,
described in the \documentref{Metacarta Connector Guide}. If you
already have the Connector Framework installed, you can install the
Web Crawler.  The Web Crawler is available as a field-test addon
ISO file from from Metacarta. Use the following steps to install the
Web Crawler to your appliance:

\begin{enumerate}

\item Confirm that the system is running correctly using
\command{check\_system\_health}.

\item Copy the disc image and md5sum supplied by MetaCarta to the
\dirpath{/isos} directory on your appliance. Rename the ISO file to
\dirpath{addon-web-conn.iso} when you copy it to your appliance.

\item Check the md5sum of the ISO file using the \command{md5sum} command.

\item Install the iso using the following command:

\command{upgrade\_control install /isos/addon-web-conn.iso}

The appliance will reboot once the installation is complete.

\item Upgrade your license file, if necessary, as described in
\documentref{MetaCarta Appliance Administrator's Guide}. You may need
to contact MetaCarta Customer Service to obtain the appropriate
license file. (Please see page \pageref{SupportContact} for contact
information.)
 
\end{enumerate}

\section{Configuration}

\subsection{Access to the Connector}

The administrator to the Web Crawler \input{access-input}


\section{Collecting Documents From The Web} % Retitle this, yo.

The Connector Framework manages retrieving documents from different
servers through \emph{jobs}. Jobs can be scheduled to run
regularly; each job connects to a server using a particular
set of credentials. Each job is tied to a \emph{repository
connection}. Repository connections contain information allowing the
connector framework to connect to a given repository. A repository
connection may be tied to an \emph{authority connection}, which
manages document security. While using the Web Crawler to connect
to web pages, no authority connection is needed; you can simply
create a repository connection.


\subsection{Creating Repository Connections}

In order to create jobs to ingest documents, you need to create a
repository connection. To do so, click ``List Repository
Connections'' on the sidebar menu. Then, when presented with the list
of repository connections, click ``Add a new connection.'' You will
see the following two tabs:


\input{sidebar}


\includegraphics[width=300pt]{web-edit-repository-tab1}

\includegraphics[width=300pt]{web-edit-repository-tab2}

Now you must provide a name, description, and connector type for your
new repository connection. The name should be unique, as you will use
it to select this connection later when defining jobs. The description
should explain the repository connection to you or another
administrator.  The connector type is the type of repository from
which you will get documents, in this case a Web Crawler. The
authority type is the type of authority from which you will get
authorization information. The Web Crawler does not use an
administrator-created authority; you should simply select ``Standard
(Kerberos)'' here.

Once you have filled in those tabs, click Continue to be taken to the
repository-specific options.

\input{web-edit-repo}

\subsection{Creating and Running Jobs}

To run a job, click ``Status and Job Management'' on the sidebar menu.
You can run or edit existing jobs from this menu.

To create a new job, click ``List All Jobs'' on the sidebar menu. Then, when
presented with the list of current jobs, click ``Add a new job.'' You
will be presented with two tabs, in which you must fill in the following
information:

\includegraphics[width=300pt]{web-edit-job-tab1}

\begin{itemize}

\item \textbf{Name:} The name of the job. You will use this to identify
the job later.

\item \textbf{Collection:} The collection name metadata for all
documents in this job. End users can use this name to select the set
of documents in this job. For more information on collection name
metadata, please see the \documentref{MetaCarta GTS Administrator's
Guide}.

\item \textbf{Document template:} \input{doctemplate}

\end{itemize}

\includegraphics[width=300pt]{web-edit-job-tab2}

\begin{itemize}

\item \textbf{Connection:} The name of the repository connection you
wish to use for this job. You select this from the list of repository
connections you have already made. You may have more than one job use
the same repository connection, but if you have two jobs crawl the same
documents, the documents will have the metadata and collection name
associated with whatever job crawled the document most recently. This
will cause unpredictable results when searching those collections,
searching for those documents, or trying to delete those collections.
We recommend never crawling the same document in two different jobs.

\item \textbf{Start method:} Whether you want to start this job the next
time jobs are scheduled to run (``Start when schedule window starts''),
immediately after you finish defining it (``Start even inside a schedule
window''), or not at all (``Don't automatically start this job'').

\item \textbf{Priority:} From 1 (highest) to 10 (lowest), the priority
this crawl should have if it must compete for resources with other
crawls on the appliance. You should not need to change this unless you
are running more than one crawl at the same time; if you are, assign a
higher priority to the crawls whose documents you want to be processed
preferentially before documents from other jobs.

\end{itemize}

After filling in those options, click ``Continue'' and you will be
presented with two additional repository-specific tabs.

\input{web-edit-job}

\subsection{\label{ManageJobs}Status and Job Management}

You can then look at the status of your job by clicking ``Status and 
Job Management'' on the sidebar. You will see a list of one or more jobs
much like this one:

\includegraphics[width=300pt]{web-jobs-list}

You can start any crawl you like immediately from this interface by
clicking ``Start'' next to the name of the crawl. This interface also
allows you to see how many documents have been crawled; this information
may help you structure and plan future crawls.

\note{Refresh this page by clicking the ``Refresh'' link at the bottom
of the page, not by clicking your browser's reload button.}

\section{Tuning Crawl Peformance}

During high broad, high intensity crawls, you may wish to tune your
crawl by changing the number of threads dedicated to ingestion. In the
file \dirpath{/etc/metacarta/agents.conf} on your appliance, you
can change the \texttt{com.metacarta.crawler}\linebreak\texttt{.threads} parameter from
its default value of 30 up to 80. Dedicating more appliance resources
to the crawl can help the crawl complete faster. However, it may
affect other aspects of appliance performance.

% Like what? We should at least give an overview, I think. Maybe
% "including search UI responsiveness" at least...


\section{Reports}

The Connector interface can generate two types of status reports, on
current crawl status, and four types of history reports, on past crawl
history.

\subsection{Status Reports}

The two types of status report are:

\begin{itemize}

\item Document Status, which lets you find information on individual
documents currently part of a job.

\item Queue Status, which lets you aggregate information about groups
of documents currently part of a job.

\end{itemize}

\subsubsection{Document Status}

This report was generated by selecting ``Web Crawler,'' selecting the document state ``Documents processed at
least once,'' selecting all possible document statuses, clicking
continue, selecting ``WebCrawlerJob,'' and clicking Go.

\includegraphics[width=300pt]{web-document-report}

\begin{itemize}

\item \textbf{Connection:} The repository connection from which to 
generate a report. You must select the repository connection and click
Continue to see all repository-specific options. %Are there any?

\item \textbf{Time offset from now:} Defaults to zero. Allows you to
see estimates of future status or, with negative numbers, a record of
past status.

\item \textbf{Document state:} Allows you to select documents that
have not yet been processed or documents that have been processed
at least once.

\item \textbf{Document status:} Allows you to choose one or more 
statuses of document to report on. The statuses you can choose are:

\begin{itemize}

\item Documents that are no longer active

\item Documents currently in progress

\item Documents currently being expired

\item Documents currently being deleted

\item Documents currently available for processing

\item Documents currently available for expiration

\item Documents not yet processable

\item Documents not yet expirable

\end{itemize}

\item \textbf{Document identifier match:} A regular expression
allowing you to see only certain document identifiers. A document's
identifier is its URL.

\item \textbf{Jobs:} The job or jobs for which you want to generate
a report.

\end{itemize}

You can sort this report by any of the returned fields; to do so,
click the field names.

\subsubsection{Queue Status}

This report was generated by selecting ``Web Crawler,'' selecting both
document states, selecting all possible document statuses, clicking
continue, selecting ``WebCrawlerJob,'' and clicking Go.

\includegraphics[width=300pt]{web-queue-report}

This form offers the same fields as the Document Status report with
one addition, the \textbf{Identifier class description}, which allows
you to group results based on a regular expression. In this case, the
identifier class description is just ``.*''. Each document is shown
individually as a group of one. If we set the identifier class
description to
\texttt{\^{}https?://([\^{}$\backslash\backslash\backslash$?/]*)},
documents would be grouped together if their URLs specified the same
server, and each group's documents would be analyzed separately from
the other documents selected. The ungrouped documents would be all
analyzed together in the first row of the results table.

\subsection{History Reports}

The four types of history report are:

\begin{itemize}

\item Simple History, which lets you list an ordered set of log events
based on chosen criteria

\item Maximum Activity, which lets you see the period of time in
which a certain event happened most often

\item Maximum Bandwith, which lets you see the period of time in
which the most bandwidth was used 

\item Result Histogram, which provides log information that would be
appropriate for constructing a histogram or other diagram

\end{itemize}

Each of these reports allows you to specify a connection, one or more
activities, a start time, an end time, an entity match, and a result code
match.  Some also allow you to specify an identifier class description
and a sliding window size. This section will show sample results for
each type of report and an explanation of the fields selected.

\subsubsection{Simple History}

This report was generated by selecting ``Web Crawler,'' 
clicking Continue, selecting ``document ingest,'' and clicking Go.

\includegraphics[width=300pt]{web-simple-history-report}

\begin{itemize}

\item \textbf{Connection:} The repository connection from which to generate
a report.

\item \textbf{Activities}: What crawler activities you would like to
see.  Your options are document ingest, document remove, fetch, job
abort, job continue, job end, job start, and job wait.

\item \textbf{Start time}: The earliest time in the crawler logs to be
considered for this query.  Choose ``Not specified'' for any field to
start at the beginning of the crawler's logs.

\item \textbf{End time:} The latest time in the crawler logs to be
considered for this query. Choose ``Not specified'' for any field 
to end at the current time.

\item \textbf{Entity match:} A regular expression to limit the
Identifier field. For the Web Crawler, the document identifiers are
the document URLs. If the entity match field in the example above had
been \texttt{www.example.com}, only document fetches with URLs from
that server would be shown.

\item \textbf{Result code match:} A regular expression to limit the
ResultCode field.

\end{itemize}

You can sort the history report by any of the returned fields; to do so,
click the field names.

\note{If you are not familiar with regular expressions,
there are a variety of tutorials available on the web,
including \url{http://gnosis.cx/publish/}\linebreak
\url{programming/regular_expressions.html} and
\url{http://perldoc.}\linebreak\url{perl.org/perlrequick.html}. If you
still have difficulty with these settings, please contact Customer Support
(see page \pageref{SupportContact}).}

\subsubsection{Maximum Activity}

This report was generated by selecting ``Web Crawler,''
clicking Continue, selecting ``document ingest,'' changing the Identifier
class description, and clicking Go.

\includegraphics[width=300pt]{web-maximum-activity-report}

This form offers two more fields than the previous form:

\begin{itemize}

\item \textbf{Identifier class description:} A regular expression that
determines how to group identifiers together. The setting in the example,
\texttt{()}, groups all documents together. Some other possibilities:

\begin{itemize}

\item \texttt{(.*)}: No groupings, all ingestions listed separately.

\item \texttt{(www.example.com)}: One group of documents whose URLs contain ``www.example.com,'' a second group of documents whose URLs do not contain ``www.example.com.''

\item \texttt{\^{}http://([\^{}$\backslash\backslash\backslash$?/]*)}:
Groups for each separate server.

\item
\texttt{$\backslash$.html?($\backslash$?|\$)|$\backslash$.pdf?($\backslash$?|\$)}
One group of files with the extention ``.html'' or ``.htm'', one group
or files with the extension ``.pdf'', and one group of all other
files.


\end{itemize}

\item \textbf{Sliding window size}: The search interval in minutes.

\end{itemize}

The report returned will have only one result per group with one or more
documents in it, if there is a clear highest activity rate, or a list of
all the results tied for highest activity rate if there are more than one.

\subsubsection{Maximum Bandwidth}

This report was generated by selecting ``Web Crawler,''
clicking Continue, selecting ``document ingest,'' changing the Identifier
class description, and clicking Go.

\includegraphics[width=300pt]{web-maximum-bandwidth-report}

This form offers the same fields as the maximum activity form, and
returns similar results; instead of tracking events per time window,
it tracks the window with the highest average bandwith usage, measured
in bytes per second. Again, the identifier class description has been
changed to a regular expression that will match all identifiers (and
thus in this case all documents).

\subsubsection{Result Histogram}

This report was generated by selecting ``Web Crawler,'' clicking Continue, selecting ``document ingest,''
altering the identifier class description to group documents into
groups based on server, and clicking Go.

\includegraphics[width=300pt]{web-activity-result-report}

This form adds one new field:

\begin{itemize}

\item \textbf{Result code class description:} A regular expression that
determines how to group result classes together; like Identifier class
descriptions but for result classes.

\end{itemize}

This report does not produce an actual histogram, but provides data that
could be used to generate histograms.  

\end{changemargin}
