ToDo

From ecology
Revision as of 10:49, 14 October 2011 by Lyklev (talk | contribs) (Administration)
Jump to: navigation, search

ToDo List

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

Public web site

  • Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
    • In progress, expect to finished by 14-12-2011.

Data model, processing

  • Enable uploading tables to the database (for example to our personal schemas). For example, we would like to flag gps points with behaviour classes based on our accelerometer data and then store this information so that it can be reused (in short annotate data). Is there any way you can facilitate this process. Otherwise analysis has to be rerun locally every time, which is very time consuming. (JS, high priority). (Following meeting July 14 2011 JS & LV) Adding a new empty table can be done in pgAdmin, not clear how to fill the table. When JS is ready with a proposal for an annotation schema, the option of setting up an upload facility (similar to the metadata) will be explored.
    • 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the user's own schema in which the data must be stored.
    • 16-09-2011, LV: First of upload facility (draft) version complete
    • 07-10-2011, LV: Added the Description of the Data upload and Annotation Service. A breakdown of the work is also listed there.
    • 13-10-2011, LV: add template creation form
    • 13-10-2011, LV: add template copy form (forced: owner, name, description)
    • 13-10-2011, LV: addition to template edit form: checkbox per column for upload inclusion (default on)
    • 13-10-2011, LV: finalise user documentation (mail to Judy/Willem when 90% finished) (iterate within SARA first)
    • 13-10-2011, LV: log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities


  • Design and implement a datamodel for the output of the "tides and currents model" as soon as the data is available. (WB, medium priority)
    • 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
    • 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model unstruc' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
    • 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
    • 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
    • 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model Waddenfijn. This model has a resolution of 100m.
    • 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
  • Write a manual which describes how to query the spatial database, for easily accessing the flow data for given (x,y)-coordinates and time. (LV)
    • 13-10-2011, LV: create example function; keywords: nearest neighbor, query, water-levels + gps data, selection in space and time
  • Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
    • 10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table gps.uva_tracking_speed_2d was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.
    • 13-10-2011, LV: Field "angle" must be renamed to "direction" and must be added to the gps.uva_tracking_speed_2d view. The unit must be degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees. Expected finish is 18-10-2011.


  • Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
    • On hold while working on annotation metadata import.
  • Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
    • On hold while working on annotation metadata import.
  • Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
    • On hold while working on annotation metadata import.

Administration

  • List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
    • 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.
  • add a project data table; assign projects to users or users to projects; link bird access to projects
    • On hold while working on annotation metadata import.
  • Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)
  • Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.
  • new policy for addition of new users: disable e-mail function, become user and check functionality before notification
    • 14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.

Other

Finished

  • Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)

There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)

  • Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)
  • Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.
  • Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.
  • BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)
  • Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).
  • Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)
  • QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)
  • Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)
  • Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)
  • List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)
  • Make a readonly copy of Flysafe Wiki on public server (september 2010)
  • Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)
  • Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)
  • Add accelerometer calibration factors to uva_device (september 2010)
  • Verify whether there are still users using Irods (WB, June 2011)
    • There are no users using the Irods service anymore (September 2011)
  • add a description of the user's rights fields, instead of the database names for rights (may 2011)
    • A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".
  • make sure that access to weather data is properly restricted
    • In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

Other

  • How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
    • LV, 22-09-2011 Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.
  • How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
    • LV, 22-09-2011 Raw data is not filtered at all, apart from the CRC8 checksum.