How to pass parameters to tools
This document proposes an updated method of passing parameters to tools when invoking them.
Current status
Normally tools are invoked from their tool pages with no arguments (i.e. parameters passed to them). There are exceptions where the tool is started with selected parameters. The best example of this is the indeed viewer that plots sensor measurement data on nees.org.
Indeed can be started from
- the indeed tool page with no arguments.
- a link to a data file in the Project Warehouse.
- a database viewer where the user can select multiple data files.
Current Implementation
This is implemented by a simple file mechanism.
Step 1: Add a list parameter to the "invoke" URL
The "invoke" URL of indeed (https://nees.org/tools/indeed/invoke) is augmented with a "list" parameter. The parameter can be set statically (Project Warehouse pages) or dynamically (database viewer).
Here is an example of a static link.
The list parameter contains one item which is the file name.
list=/nees/home/NEES-2005-0022.groups/Experiment-2/Trial-6/Rep-1/Corrected_Data/NEES-0022_RWN_Sensor_Corrected_Data.csv
Step 2: A web plugin processes the list argument and creates file.
A plugin specific to indeed (/www/neeshub/plugins/mw/indeed.php) has been created to process the list parameter. The plugin creates a file /apps/projects/workingdir/<user>/file where user is the name of the user. This means that the user must be logged into the website before they can run indeed. The plugin will take you to the login page if you are not already logged in. The format of the file is that each element of the list is written on its own line. It the file contains two parameters, there will be two lines to the file. The purpose here was to make parsing very simple since it's the tool that will be doing.
Step 3: The tool is invoked and reads the file.
It is up to the tool to check it the file /apps/projects/workingdir/<user>/file exists and to read its contents.
- indeed must first check if the file exists. If indeed is launched from the tool page, there will be no file. Also because of NFS delays, the file may not be accessible by the execution host when indeed is first launched. indeed waits and polls for the existence of the file. It will give up have some number of tries.
- indeed reads the file are processes each line as a date file name.
- indeed deletes the file to indicate that it has processed the data and so that the parameters are used by another invocation.
Problems with the current implementation
- The tool has to poll for the existence of the file even in cases where there may not be one.
- Read and write access is needed to read and delete the file. This means the file systems /apps must be mounted read/write.
- The file can be overwritten by other tools or invocations of indeed.
- Must create a plugin for each tool that wants to accept parameters.
New Implementation Proposal
The following proposal changes the current implementation.
Step 1: Add a list parameter to the "invoke" URL
This is the same as before.
Step 2: The invoke component processes the list argument and creates the file.
The argument processing is moved from plugins into the invoke tool component. This means that all tools can get arguments passed. If a tool doesn't accept parameters, it can simply ignore them.
- The name of the file is generated base on the file contents (md5 hash) to be unique.
- The arguments are written into the file as before. The format may be changed in conjunction with parsing routines in rappture. The file is written somewhere in /apps.
Step 3: The tool is invoked with the file name passed.
- The tool is invoked and the unique file name is passed, either via the command line, or an environment variable. This means that the tool immediately knows if it is being passed parameters or not.
- NFS will still require polling for the existence of the file.
- The tool does not delete the file. A middleware cleaner process will delete the old files. This means the file can be read from a read-only file system on the execution host.
Questions
- Who should handle encoding issues? If there's a space in an argument, the URL list will contain a %20. Should the tool convert this or the plugin?
- What should the file format be. It has to handle spaces, back slashes. I imagine most arguments will be file names.
