Welcome to trollduction’s documentation!

Trollduction is a configurable framework for satellite image batch production.

This documentation is a work in progress, but the most important bits should be present. The missing details will be added once noticed.

How does Trollduction work?

Trollduction builds on the principle that satellite image batch production is event based, and that different processing steps are chained together to produce the final products. This is why trollduction is a collection of independent elements of this chain, which are then communicating together through lightweight network messages.

The different elements provided are:
  • trollstalker: triggers an event message each time a file is put in given directory
  • l2processor: generates rgb/channel images when an appropriate event message is received
  • gatherer: gathers polar satellite granules together given an area of interest, and sends an event message when a matching group of granules has been gathered
  • aapp_runner: runs the AAPP software on raw hrpt files, when such an event message is received, to generate level 1 data
  • viirs_dr_runner: runs the CSPP software on Suomi-NPP RDR files to generate level 1 data
  • modis_dr_runner: runs the SPA software on EOS-Terra/Aqua PDS files to generate level 1 data
  • pps_runner: triggers processing of PPS main script once AAPP or CSPP is ready with a level-1 file

Setting things up cheat sheet

  1. Install other required packages:
    • libnetcdf-dev
    • libhdf5-dev
    • python-numpy
    • python-zmq
    • python-pyinotify

    and the Pytroll packages:

  2. Configure mpop
    • modify mpop.cfg to suit your needs
    • add configurations for satellites you are going to use (see the templates in mpop/etc/)
  3. Create Trollduction configuration files
    • use examples/trollstalker_config.ini_template as a template
    • use examples/l2processor_config.ini_template as a template
    • save config file to your chosen place without the _template ending
  4. Create Trollduction product configuration file
    • use examples/product_config_hrpt.xml_template as a template
    • save the file to the path defined in your l2processor_config.ini without the _template ending
    • define your output_dir in the xml file
    • topic from the trollstalker_config.ini must be included in the list of topics in l2processor_config.ini
  5. Create logging configurations for trollduction and trollstalker
    • use examples/l2processor_logging.ini_template and examples/trollstalker_logging.ini_template as templates
    • check the log filename (by default logs go to /tmp/ directory)
    • save these configs to the path defined in your l2processor_config.ini and trollstalker_config.ini without the _template ending
  6. Start posttroll/bin/nameserver
    • this will register the different subscribers on the network and relay the messages between different processes
    • ./nameserver
  7. Start trollduction/bin/trollstalker.py
    • file watcher that sends messages of new files available for processing
    • for example: ./trollstalker.py -c /path/to/trollstalker_config.ini -C noaa_hrpt
  8. Start Trollduction trollduction/bin/l2processor.py
    • ./l2processor.py -c /path/to/l2processor_config.ini -C noaa_hrpt
    • this should print your configuration and stop to wait for new messages

  9. Copy a suitable file to your data input directory

  10. Check the output directory for images

Indices and tables

Listener

Listener module for Trollduction.

class trollduction.listener.Listener(topics=None, queue=None)

PyTroll listener class for reading messages for Trollduction

add_to_queue(msg)

Add message to queue

create_subscriber()

Create a subscriber instance using specified addresses and message types.

restart()

Restart subscriber

run()

Run listener

stop()

Stop subscriber and delete the instance

class trollduction.listener.ListenerContainer(topics=None)

Container for listener instance

restart_listener(topics)

Restart listener after configuration update.

stop()

Stop listener.

XML read

XML reader for Trollduction system and product configuration files.

class trollduction.xml_read.Dataset(data, **attributes)

Dataset class.

copy(copy_data=True)

Return a copy of the data.

class trollduction.xml_read.InfoObject(**attributes)

InfoObject class.

get(key, default=None)

Return the requested info.

class trollduction.xml_read.ProductList(fname)

Class for reading and storing product lists.

check_groups()

Check area groups.

insert_vars()

Variable replacement

parse()

Parse product list XML file.

trollduction.xml_read.get_filepattern_config(fname=None)

Retrieves the filepattern configuration file for trollstalker, and returns the parsed XML as a dictionary. Optional argument fname can be used to specify the file. If fname is None, the systemwide file is read.

trollduction.xml_read.get_root(fname)

Read XML file and return the root tree.

trollduction.xml_read.parse_xml(tree, also_empty=False)

Parse the given XML file to dictionary.

Custom handler

For Panu

class trollduction.custom_handler.PanusTimedRotatingFileHandler(template, *args, **kwargs)

Like TimedRotatingFileHandler with a custom filename template.

doRollover()

do a rollover; If there is a backup count, then we have to get a list of matching filenames, sort them and remove the oldest ones.

getFilesToDelete()

Determine the files to delete when rolling over.

More specific than the earlier method, which assumed the date to be a suffix.