Including or Excluding Name Patterns

SiteSurfer will include all pages or files it finds unless it is specifically told to exclude one, using either the Robots Protocol or SiteSurfer's own internal pattern matching.

SiteSurfer compares the URL of each page or file it finds with the patterns in this list, starting at the top and proceeding to the bottom. The first enabled pattern that matches the name causes the page to be explicitly included or excluded based on whether the Include flag is true or false, respectively.Comparison of files with patterns is not case sensitive. Since pages are included by default, "include patterns" are used to short circuit the pattern match, eithe for speed (e.g., the *.HTM and *.HTML patterns), or to include files which would otherwise be excluded by a later "exclude pattern."

The order of the patterns can be set using the and buttons. The current selection may be removed with the button, and a new pattern may be added by clicking on the button. Lastly, an already-defined pattern may be re-configured with the button.

The grammer for these patterns should be easy to understand for anyone familiar with DOS-style command-line expressions: The following are "terminal" characters in this grammer: *, ?, [, ], (, ), \ and ~.

• ? Matches any 1 character.
• * Matches any 0 or more characters.
• \ Escapes out any terminal character that follows, so it is taken literally.
• [c-g] Matches c and g and all values in between.
• [a,b,c] Matches a or b or c.
• ~ Negates whatever expression follows.
• (ab|bc|cd) Matches ab or bc or cd.

Examples:

• */serve.cgi\?1234
  Matches (among others): http://www.devtech.com/serve.cgi?1234
  
• (ftp|gopher)://generic.com
  Matches: ftp://generic.com and gopher://generic.com
  
• http://www.mycorp.com/~public/*
  Matches: any path on http://www.mycorp.com not under the public directory
  
• http://www.mycorp.com/users/\~doe/*
  Matches: any path on http://www.mycorp.com under the /users/~doe directory