Batch processing with ANZMP

Introduction

"Batch processing" allows grade 3 Eco Companion members to specify a list of geospatial metadata documents. The ANZMP geospatial metadata parser will then retrieve each SGML/XML file from across the network, interpret the metadata, validate the structure, check some content, and produce presentation documents. You will be advised by email when the process is complete, then you can view an overall report (with links to the individual documents and reports), and download the package of all documents to your own computer.

Geospatial metadata is structured information which concisely describes a collection of related data. It is important that the structure of each document is complete and that the content of certain elements is legitimate (according to the relevant standards) . This enables computer software to automatically read and reliably interpret the content of the document. Upon this base sophisticated searching facilities can be built. Geospatial dataset descriptions that are intended to be included in the Australian Spatial Data Diectory (ASDD) will need to pass the tests of validating metadata parsesr.

This ANZMP batch processing will enable you to properly manage your collection of dataset descriptions. The availability of this network service means that you never have to bother messing about with the installation of software. Rather, you can concentrate on the content of your documents.

ANZMP will produce consistent presentation documents directly from your SGML/XML data files. ANZMP will even add your own header and footer to each generated page. A small locality map is also produced for each document.

Preparing your configuration files

You will need to prepare two simple configuration files and place them on your Internet server where the ANZMP parser can retrieve them (i.e. not in a password protected directory).

Global configuration file

• This file specifies some things that can be applied to all of the presentation documents that will be produced.
  • topfragment=http://www.indexgeo.com.au/data/test/top.frg ... specify the URL of a file containing a fragment of HTML to be inserted at the top of the HTML pages (e.g. include banners, logos, and navigation bar).
  • tailfragment=http://www.indexgeo.com.au/data/test/tail.frg ... specify the URL of a file containing a fragment of HTML to be inserted at the bottom of the HTML pages (e.g. include copyright statement and navigation bar).
  • baseurl=http://www.indexgeo.com.au/data/test/ ... Specify the base URL for the final location of this batch of documents. This URL is required for the locality graphic which is embedded in the HTML dataset description.
  • bgcolor=#E2DBC2 ... specify the background colour for the HTML pages (e.g. #FFFFFF for white)
  • stylesheet=http://www.yourserver.org/css/metadata.css ... specify the CSS stylesheet to be used for the HTML pages.
• This configuration file must be called "anzmp.cfg".
• Rules and conventions are explained in the example file.
• Start with the example global configuration and edit it to suit.

Dataset list

• This file specifies the list of geospatial metadata documents that are to be processed.
• One entry per line, in the following comma-delimited format ...
  • URL,savename,applytop,applytail
  • URL ... the full Internet location of the SGML/XML metadata file (must have extension .sgml, .sgm, or .xml).
  • savename ... the short name to save the dataset as. If this is empty then the basename of the SGML file will be used.
  • applytop ... define which fragment of html to insert at the top of this HTML output file (mine, ec, none).
  • applytail ... define which fragment of html to insert at the tail of this HTML output file (mine, ec, none).
• This configuration file must be called "anzmp.lst".
• There is a limit of 500 datasets per batch (to enable production reports that are of manageable size).
• Rules and conventions are explained in the comments within the example configuration file.
• Start with the example dataset list and edit it to suit.

What happens behind the scenes

Schedule a batch job by filling out the online form that is available from your member services page. Supply the URLs of your two configuration files. The parser will retrieve them across the network and ensure that all is OK. If your configuration files are valid then your batch job will be scheduled and you will be advised that you will be contacted by automatic email when the processing is complete.

The Eco Companion server will check for scheduled batch jobs every hour. Your list of datasets will be processed by fetching each SGML/XML metadata document across the network and validating it with the ANZMP parser. An overview report will be produced which lists each dataset and summarises any problems with it. The overview report links directly to each HTML and text document and error report that is produced. You will also be shown the location of compressed packages containing all of your documents and reports which can be downloaded to your own computer for local use.

When the batch run has finished, you will be automatically sent an email message which provides the URL of your overview report. This is accessible only to you, in your private member preparation space.

You can schedule any number of batches, they will be automatically run in turn. Separate reports and download files will be generated for each batch.

Metadata management

### Discussion will be added soon.

The need for metadata validation

### Discussion will be added soon.

URL:http://www.indexgeo.com.au/ec/help/anzmp-batch.html
Last Modified: 20 March 2003