ToDo

ToDo List

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

Public web site

(empty)

Data model, processing

• Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
  • create a template overview page
  • add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
  • a facility to edit existing templates
  • a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
  • log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
  • user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

• Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
  • Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
  • check ownership of tables after uploading
  • verify that there is no restriction on the size of uploaded tables
  • verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
  • improve error messages, especially in the case of double records, missing

Progress:

• Design and implementation of the annotation table upload facility:
  • 07-10-2011, LV: Added the Description of the Data upload and Annotation Service. A breakdown of the work is also listed there.
  • 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
  • 16-09-2011, LV: First of upload facility (draft) version complete
  • 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
  • 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
  • 02-11-2011, LV: Added a description of the workflow to the user manual.

• Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
  • On hold while working on annotation metadata import.

• Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
  • On hold while working on annotation metadata import.

• Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
  • On hold while working on annotation metadata import.

Administration

• List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
  • 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

• add a project data table; assign projects to users or users to projects; link bird access to projects
  • On hold while working on annotation metadata import.

• Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

• Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

• new policy for addition of new users: disable e-mail function, become user and check functionality before notification
  • 14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.

Finished

• Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

• Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

• Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)

There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)

• Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

• Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

• Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

• BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

• Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

• Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

• QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

• Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

• Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

• List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

• Make a readonly copy of Flysafe Wiki on public server (september 2010)

• Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

• Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

• Add accelerometer calibration factors to uva_device (september 2010)

• Verify whether there are still users using Irods (WB, June 2011)
  • There are no users using the Irods service anymore (September 2011)

• add a description of the user's rights fields, instead of the database names for rights (may 2011)
  • A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

• make sure that access to weather data is properly restricted
  • In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

• Design and implement a datamodel for the output of the "tides and currents model":
  • 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
  • 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model unstruc' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
  • 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
  • 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
  • 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model Waddenfijn. This model has a resolution of 100m.
  • 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
  • 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
  • 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

Other

• How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
  • LV, 22-09-2011 Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.

• How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
  • LV, 22-09-2011 Raw data is not filtered at all, apart from the CRC8 checksum.