ToDo

ToDo List

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

Public web site

• Copy UvAGPS functions available on the flysafe server to the public server (JS, July 2011, medium priority)
  • In progress, expect to finished by 14-12-2011.

Data model, processing

• Enable uploading tables to the database (for example to our personal schemas). For example, we would like to flag gps points with behaviour classes based on our accelerometer data and then store this information so that it can be reused (in short annotate data). Is there any way you can facilitate this process. Otherwise analysis has to be rerun locally every time, which is very time consuming. (JS, april 2011, high priority). (Following meeting July 14 2011 JS & LV) Adding a new empty table can be done in pgAdmin, not clear how to fill the table. When JS is ready with a proposal for an annotation schema, the option of setting up an upload facility (similar to the metadata) will be explored.
  • In progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the user's own schema in which the data must be stored. (LV, August 2011)
  • First (draft) version complete (LV, 16-09-2011)

• Design and implement a datamodel for the output of the "tides and currents model" as soon as the data is available. (WB, July 2011, medium priority)
  • The schema "flow" has been added to the datbase for storing the flow grids and model results. (25-08-2011)
  • A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model unstruc' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis) (25-08-2011).
  • The test data from Deltares has been imported into the database automatically. (31-08-2011)
  • The automatic importing of data will be started as soon as Deltares has actual data available. (31-08-2011)
  • Two model output data sets have been supplied by Deltares. Import has been started for the model Waddenfijn. This model has a resolution of 100m. (21-09-2011). Expected finish is 23-09-2011.

• Write a manual which describes how to query the spatial database, for easily accessing the flow data for given (x,y)-coordinates and time. (LV, 22-09-2011)

• Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). (JS, July 2011, low priority)
  • Expect to be finished by 14-12-2011.

• Add direction to calculations in uva_tracking_speed view. (JS, July 2011, low priority)
  • Expect to be finished by 14-12-2011.

• Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, July 2011, low priority)
  • On hold while working on annotation metadata import.

• Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, June 2011, low priority)
  • On hold while working on annotation metadata import.

• Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, march 2011, very low priority)
  • On hold while working on annotation metadata import.

Administration

• List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, april 2011, high priority);
  • In progress; a web interface listing user's rights is being created as part of the administrative interface (july 2011), on hold for uploading annotation data.

• add a project data table; assign projects to users or users to projects; link bird access to projects

• Note: this account should have very limited rights, and should not be able to change the own password (otherwise one guest could make it impossible for other guests to see the site)

• Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, March 2011, low priority, should be improved by March 2012)

• Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

Other

Below is a list of open questions, once answered they will be removed and documented elsewhere.

• How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)

• How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)

Finished

• Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)

There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)

• Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

• Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

• Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

• BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

• Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

• Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

• QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

• Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

• Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

• List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

• Make a readonly copy of Flysafe Wiki on public server (september 2010)

• Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

• Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

• Add accelerometer calibration factors to uva_device (september 2010)

• Verify whether there are still users using Irods (WB, June 2011)
  • There are no users using the Irods service anymore (September 2011)

• add a description of the user's rights fields, instead of the database names for rights (may 2011)
  • A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

• make sure that access to weather data is properly restricted
  • In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)