ecology - User contributions [en]

ToDo

2013-05-17T06:13:27Z

Lyklev: /* ToDo List */

== ToDo List ==
Note: the actions in this page have been moved to the [https://services.e-ecology.sara.nl/redmine Redmine] tracker.

== Issues ==

=== Urgent issues ===

* There is missing data for device 211, for February 2012.
** Find out why this data is missing, re-import the data
** Find out why no error was reported and fix the failed mechanism

* Matlab script wrapper fail for firmware versions 1.0.3.4 and 1.0.3.5
** Adjust matlab script wrapper to accept these firmware versions.

=== Medium priority ===

* Get the "projects" table working
** Make it work so that is causes ''fewer'' issues, not more. If this is not possible, streamline the old-fashioned way of creating users and granting access
** Give WB access to the projects table.

* Extension of the table "gps.uva_devices"
** JS will specify which new columns need to be added.

* New hardware specification
** Coen will write down a first draft specification of the new hardware.

=== Lower priority issues ===

* Userflag application
** There appears to be an application for flagging invalid GPS data. What is the status of this application? Who has access?

=== Change requests ===

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

(empty)

=== Data model, processing ===

* If there is an error during the processing of a GPS file, move that file to a "failed" subdirectory.
** ''Failed files are moved to a "failed" subdirectory in the same place as the failed file.''
* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the UvA-bits web site.
** ''Scripts and urls have been provided to Aart-Jan for the UvA-bits web site''

* Automatically and periodically generate power graphs for the GPS loggers; take the graph generating program from the Matlab script wrapper.
** ''Script is run periodically; plots are placed in a directory where the UvA-bits web site can read them from. Graphs are generated for all loggers. To protect the generated images so only authorized users can see them, only the UvA-bits machine can access these graphs. Access must be controlled by the UvA-bits machine.''

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** ''In progress, expect to finished by 14-12-2011.''
** ''LV: the uvagps functions are available through the functions tab and correctly filter gps devices.''

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

OLD: Description of the Data upload and Annotation Service

2012-10-09T10:14:57Z

Lyklev: /* Data upload and annotation service */

This page has been moved to [[Annotation tables]].

OLD: Description of the Data upload and Annotation Service

2012-10-09T10:14:07Z

Lyklev: moved Description of the Data upload and Annotation Service to OLD: Description of the Data upload and Annotation Service

=Data upload and annotation service=

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

With such a facility the file-based data, for example a comma-separated value file, could be easily uploaded via a web form and all data would automatically be inserted into a new database table without further user interaction.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

For such a system, users must be able to create new and view existing templates, and use these templates to upload file-based data. Such a system requires 3 modules:

1. Template viewer

The template viewer providesan overview of available templates, and shows details such as:
* the user who created the template
* the name of the template
* a description of the template

2. Template creator

The template editor allows a user to create new (edit existing) templates, defining all relevant aspects such as:

* a template name
* a list of columns, where each column has a name and a database type
* a description of the template and its purpose

3. Upload service

Once a template has been created, a flat file with corresponding data columns can easily be uploaded to the services. The user does not need any prior knowledge of the database and the contents of the file will be inserted into the database automatically.

Once a new table has been uploaded, it may be shared with other users and used to annotate existing data in the database. Such functionality requires access to and intermediate knowledge about database usage.

As a first step we will create and integrate the three modules described above, and provide documentation about typical usage schemes (i.e. creating a template, uploading a table to the database, sharing it with other users, editing the contents of a table, and annotating data.

Description of the Data upload and Annotation Service

2012-10-09T10:14:07Z

Lyklev: moved Description of the Data upload and Annotation Service to OLD: Description of the Data upload and Annotation Service

#REDIRECT [[OLD: Description of the Data upload and Annotation Service]]

ToDo

2012-10-09T10:12:00Z

Lyklev: /* Data model, processing */

== ToDo List ==

== Issues ==

=== Urgent issues ===

* There is missing data for device 211, for February 2012.
** Find out why this data is missing, re-import the data
** Find out why no error was reported and fix the failed mechanism

* Matlab script wrapper fail for firmware versions 1.0.3.4 and 1.0.3.5
** Adjust matlab script wrapper to accept these firmware versions.

=== Medium priority ===

* Get the "projects" table working
** Make it work so that is causes ''fewer'' issues, not more. If this is not possible, streamline the old-fashioned way of creating users and granting access
** Give WB access to the projects table.

* Extension of the table "gps.uva_devices"
** JS will specify which new columns need to be added.

* New hardware specification
** Coen will write down a first draft specification of the new hardware.

=== Lower priority issues ===

* Userflag application
** There appears to be an application for flagging invalid GPS data. What is the status of this application? Who has access?

=== Change requests ===

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

(empty)

=== Data model, processing ===

* If there is an error during the processing of a GPS file, move that file to a "failed" subdirectory.
** ''Failed files are moved to a "failed" subdirectory in the same place as the failed file.''
* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the UvA-bits web site.
** ''Scripts and urls have been provided to Aart-Jan for the UvA-bits web site''

* Automatically and periodically generate power graphs for the GPS loggers; take the graph generating program from the Matlab script wrapper.
** ''Script is run periodically; plots are placed in a directory where the UvA-bits web site can read them from. Graphs are generated for all loggers. To protect the generated images so only authorized users can see them, only the UvA-bits machine can access these graphs. Access must be controlled by the UvA-bits machine.''

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** ''In progress, expect to finished by 14-12-2011.''
** ''LV: the uvagps functions are available through the functions tab and correctly filter gps devices.''

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

Annotation tables

2012-08-14T14:50:11Z

Lyklev: /* Giving access to your schema */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* give other users access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema. Due to an inconsistency in PgAdminIII, changing this in PgAdminIII requires administrator privileges, although this should not be required. As a workaround, you must use SQL to give others access to your schema.

If you are not a database administator for the flysafe database, the menus do not show the possibility to modify the schema rights, even if you can. In this case, you have to modify the schema rights by using SQL.

Open the sql interface by clicking on the "SQL" button as shown in the figure. This will open the SQL interface.

[[File:annotation_sql.png]]

The SQL shown will be to create the object that is hightlighted. You can safely remove the code and type your own code.
To grant another user access to this schema, type

GRANT USAGE ON SCHEMA <your schema name> TO <user name>;

This is shown in the following figure, where user "judy" gets access to the schema "lyklev".

[[File:annotation_sql_2.png]]

Fill in your schema name and the user you want to give access to. Next, hit the "execute" button, as highlighted in the figure.

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2012-08-14T14:48:21Z

Lyklev: /* Using SQL */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* give other users access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema. Note that in the PgAdminIII menus, this can only be done by a database administrator. If you are not a database administrator, you must use SQL.

If you are not a database administator for the flysafe database, the menus do not show the possibility to modify the schema rights, even if you can. In this case, you have to modify the schema rights by using SQL.

Open the sql interface by clicking on the "SQL" button as shown in the figure. This will open the SQL interface.

[[File:annotation_sql.png]]

The SQL shown will be to create the object that is hightlighted. You can safely remove the code and type your own code.
To grant another user access to this schema, type

GRANT USAGE ON SCHEMA <your schema name> TO <user name>;

This is shown in the following figure, where user "judy" gets access to the schema "lyklev".

[[File:annotation_sql_2.png]]

Fill in your schema name and the user you want to give access to. Next, hit the "execute" button, as highlighted in the figure.

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2012-08-14T14:48:04Z

Lyklev: /* Using the menus in PgAdminIII */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* give other users access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema. Note that in the PgAdminIII menus, this can only be done by a database administrator. If you are not a database administrator, you must use SQL.

==== Using SQL ====

If you are not a database administator for the flysafe database, the menus do not show the possibility to modify the schema rights, even if you can. In this case, you have to modify the schema rights by using SQL.

Open the sql interface by clicking on the "SQL" button as shown in the figure. This will open the SQL interface.

[[File:annotation_sql.png]]

The SQL shown will be to create the object that is hightlighted. You can safely remove the code and type your own code.
To grant another user access to this schema, type

GRANT USAGE ON SCHEMA <your schema name> TO <user name>;

This is shown in the following figure, where user "judy" gets access to the schema "lyklev".

[[File:annotation_sql_2.png]]

Fill in your schema name and the user you want to give access to. Next, hit the "execute" button, as highlighted in the figure.

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2012-07-11T09:57:40Z

Lyklev: /* Introduction */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* give other users access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema. Note that in the PgAdminIII menus, this can only be done by a database administrator. If you are not a database administrator, you must use SQL.

==== Using the menus in PgAdminIII ====

# In PgAdminIII look for the database "flysafe", open the "schemas" item and locate your own schema. This is the schema with the same name as your login name, "lyklev" in this example.
# Right-click the schema; a menu will appear, as shown.
# Click "Properties".

[[File:Schema_user_rights_1.png]]

After you click "Properties", a dialog window will appear.

# Click the privileges tab (highlighted in the figure below).
# Select the user you want to give access to at the "Role" option. In this example, this is user "tijs".
# Select "usage", as highlighted.
# Click on "Add/Change" to give user "tijs" the "usage" right to your schema. A new line will appear in the list.
# Finally, click on "Ok" to store the new rights.

[[File:Schema_user_rights_2.png]]

==== Using SQL ====

If you are not a database administator for the flysafe database, the menus do not show the possibility to modify the schema rights, even if you can. In this case, you have to modify the schema rights by using SQL.

Open the sql interface by clicking on the "SQL" button as shown in the figure. This will open the SQL interface.

[[File:annotation_sql.png]]

The SQL shown will be to create the object that is hightlighted. You can safely remove the code and type your own code.
To grant another user access to this schema, type

GRANT USAGE ON SCHEMA <your schema name> TO <user name>;

This is shown in the following figure, where user "judy" gets access to the schema "lyklev".

[[File:annotation_sql_2.png]]

Fill in your schema name and the user you want to give access to. Next, hit the "execute" button, as highlighted in the figure.

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2012-07-11T09:56:07Z

Lyklev: /* Introduction */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

The upload service is located at:

https://services.flysafe.sara.nl/cgi-bin/flysafe/uva_admin/templates.cgi

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* give other users access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema. Note that in the PgAdminIII menus, this can only be done by a database administrator. If you are not a database administrator, you must use SQL.

==== Using the menus in PgAdminIII ====

# In PgAdminIII look for the database "flysafe", open the "schemas" item and locate your own schema. This is the schema with the same name as your login name, "lyklev" in this example.
# Right-click the schema; a menu will appear, as shown.
# Click "Properties".

[[File:Schema_user_rights_1.png]]

After you click "Properties", a dialog window will appear.

# Click the privileges tab (highlighted in the figure below).
# Select the user you want to give access to at the "Role" option. In this example, this is user "tijs".
# Select "usage", as highlighted.
# Click on "Add/Change" to give user "tijs" the "usage" right to your schema. A new line will appear in the list.
# Finally, click on "Ok" to store the new rights.

[[File:Schema_user_rights_2.png]]

==== Using SQL ====

If you are not a database administator for the flysafe database, the menus do not show the possibility to modify the schema rights, even if you can. In this case, you have to modify the schema rights by using SQL.

Open the sql interface by clicking on the "SQL" button as shown in the figure. This will open the SQL interface.

[[File:annotation_sql.png]]

The SQL shown will be to create the object that is hightlighted. You can safely remove the code and type your own code.
To grant another user access to this schema, type

GRANT USAGE ON SCHEMA <your schema name> TO <user name>;

This is shown in the following figure, where user "judy" gets access to the schema "lyklev".

[[File:annotation_sql_2.png]]

Fill in your schema name and the user you want to give access to. Next, hit the "execute" button, as highlighted in the figure.

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2012-07-11T09:53:15Z

Lyklev: /* Using SQL */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* give other users access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema. Note that in the PgAdminIII menus, this can only be done by a database administrator. If you are not a database administrator, you must use SQL.

==== Using the menus in PgAdminIII ====

# In PgAdminIII look for the database "flysafe", open the "schemas" item and locate your own schema. This is the schema with the same name as your login name, "lyklev" in this example.
# Right-click the schema; a menu will appear, as shown.
# Click "Properties".

[[File:Schema_user_rights_1.png]]

After you click "Properties", a dialog window will appear.

# Click the privileges tab (highlighted in the figure below).
# Select the user you want to give access to at the "Role" option. In this example, this is user "tijs".
# Select "usage", as highlighted.
# Click on "Add/Change" to give user "tijs" the "usage" right to your schema. A new line will appear in the list.
# Finally, click on "Ok" to store the new rights.

[[File:Schema_user_rights_2.png]]

==== Using SQL ====

If you are not a database administator for the flysafe database, the menus do not show the possibility to modify the schema rights, even if you can. In this case, you have to modify the schema rights by using SQL.

Open the sql interface by clicking on the "SQL" button as shown in the figure. This will open the SQL interface.

[[File:annotation_sql.png]]

The SQL shown will be to create the object that is hightlighted. You can safely remove the code and type your own code.
To grant another user access to this schema, type

GRANT USAGE ON SCHEMA <your schema name> TO <user name>;

This is shown in the following figure, where user "judy" gets access to the schema "lyklev".

[[File:annotation_sql_2.png]]

Fill in your schema name and the user you want to give access to. Next, hit the "execute" button, as highlighted in the figure.

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

File:Annotation sql 2.png

2012-07-11T09:48:30Z

Lyklev:

File:Annotation sql.png

2012-07-11T09:48:06Z

Lyklev:

Annotation tables

2012-07-01T22:38:13Z

Lyklev: /* Giving access to your schema */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* give other users access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema. Note that in the PgAdminIII menus, this can only be done by a database administrator. If you are not a database administrator, you must use SQL.

==== Using the menus in PgAdminIII ====

# In PgAdminIII look for the database "flysafe", open the "schemas" item and locate your own schema. This is the schema with the same name as your login name, "lyklev" in this example.
# Right-click the schema; a menu will appear, as shown.
# Click "Properties".

[[File:Schema_user_rights_1.png]]

After you click "Properties", a dialog window will appear.

# Click the privileges tab (highlighted in the figure below).
# Select the user you want to give access to at the "Role" option. In this example, this is user "tijs".
# Select "usage", as highlighted.
# Click on "Add/Change" to give user "tijs" the "usage" right to your schema. A new line will appear in the list.
# Finally, click on "Ok" to store the new rights.

[[File:Schema_user_rights_2.png]]

==== Using SQL ====

If you are not a database administator for the flysafe database, the menus do not show the possibility to modify the schema rights, even if you can. In this case, you have to modify the schema rights by using SQL.

Open the sql interface by clicking on the "SQL" button. This will open the SQL interface.

The SQL shown will be to create the object that is hightlighted. You can safely remove the code and type your own code.

The SQL query for granting access to your schema is:

GRANT USAGE ON <your schema name> TO <user name>;

Fill in your schema name and the user you want to give access to and select "execute".

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Usage

2012-06-29T13:15:29Z

Lyklev: /* Uploading UvAGPS log files using the Irods interface */

== Working with the database directly ==

Users who have direct access to the database can retrieve retrieve data directly using Psql or PgadminIII, or from programming environments such as Matlab or R. The data in the database is stored in different tables to reduce redundancy. To get useful information, tables can be combined ('joined' in database terms) in queries. This section contains information of how to connect to the database, and when connected, how to combine tables in queries.

* [[ Connecting to the UvA-BiTS database ]]

* [[ Connecting to the Flysafe database ]]

* [[ Useful Queries ]]

* [[ Queries for linking bird tracks to wind and flow data ]]

* [[ Annotation tables ]] for uploading GPS anntotation data into your own schema

Annotation tables

2012-03-08T09:02:04Z

Lyklev: /* Sharing tables with other users */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* give other users access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema.

# In PgAdminIII look for the database "flysafe", open the "schemas" item and locate your own schema. This is the schema with the same name as your login name, "lyklev" in this example.
# Right-click the schema; a menu will appear, as shown.
# Click "Properties".

[[File:Schema_user_rights_1.png]]

After you click "Properties", a dialog window will appear.

# Click the privileges tab (highlighted in the figure below).
# Select the user you want to give access to at the "Role" option. In this example, this is user "tijs".
# Select "usage", as highlighted.
# Click on "Add/Change" to give user "tijs" the "usage" right to your schema. A new line will appear in the list.
# Finally, click on "Ok" to store the new rights.

[[File:Schema_user_rights_2.png]]

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2012-03-08T08:57:22Z

Lyklev: /* Sharing tables with other users */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

=== Giving access to your schema ===

To give other users access to your tables, you must first give the other users access to your schema.

# In PgAdminIII look for the database "flysafe", open the "schemas" item and locate your own schema. This is the schema with the same name as your login name, "lyklev" in this example.
# Right-click the schema; a menu will appear, as shown.
# Click "Properties".

[[File:Schema_user_rights_1.png]]

After you click "Properties", a dialog window will appear.

# Click the privileges tab (highlighted in the figure below).
# Select the user you want to give access to at the "Role" option. In this example, this is user "tijs".
# Select "usage", as highlighted.
# Click on "Add/Change" to give user "tijs" the "usage" right to your schema. A new line will appear in the list.
# Finally, click on "Ok" to store the new rights.

[[File:Schema_user_rights_2.png]]

=== Giving access to your tables ===

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

File:Schema user rights 2.png

2012-03-08T08:55:52Z

Lyklev:

File:Schema user rights 1.png

2012-03-08T08:55:29Z

Lyklev:

Annotation tables

2012-03-08T08:55:03Z

Lyklev: /* Sharing data with other users */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema. The owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read the contents of the table. The contents are still protected, so they cannot be changed by other users.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser.

To give other users access to your tables, you must first give the other users access to your schema.

# In PgAdminIII look for the database "flysafe", open the "schemas" item and locate your own schema. This is the schema with the same name as your login name, "lyklev" in this example.
# Right-click the schema; a menu will appear, as shown.
# Click "Properties".

[[File:Schema_user_rights_1.png]]

After you click "Properties", a dialog window will appear.

# Click the privileges tab (highlighted in the figure below).
# Select the user you want to give access to at the "Role" option. In this example, this is user "tijs".
# Select "usage", as highlighted.
# Click on "Add/Change" to give user "tijs" the "usage" right to your schema. A new line will appear in the list.
# Finally, click on "Ok" to store the new rights.

[[File:Schema_user_rights_2.png]]

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

[[File:Table_user_rights_1.png]]

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2012-03-05T15:55:28Z

Lyklev: /* Sharing tables with other users */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing data with other users ==

New tables are created in the user's own database schema and the owner will be the user who uploaded the table. By default, both your schema and the tables contained in the schema are only accessible by you. If you want to share data with other users, you need to:
* give other users the right to use your schema;
* access the tables in your schema.

You can give other users access to your schema and the tables contained in your schema using a database administration tool, such as PgAdminIII. The following example shows how to give another user the usage privilege to your schema. The second step shows how to give another user read-only access to the recently uploaded table, so that the other user can read, but not edit the contents of the table.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser, which is the left pane in the window, as shown below.

[[File:Table_user_rights_1.png]]

First, you must give the other user the "usage" right of your schema.
Look for the database "flysafe", open the "schemas" item and locate your own schema. This is the schema with the same name as your login name.

Now that you have given access rights to your schema, other users can see the contents of your schema, but they cannot yet access tables. To give another user access to a table,
unfold your schema
and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

ToDo

2012-03-02T09:30:34Z

Lyklev: /* ToDo List */

== ToDo List ==

== Issues ==

=== Urgent issues ===

* There is missing data for device 211, for February 2012.
** Find out why this data is missing, re-import the data
** Find out why no error was reported and fix the failed mechanism

* Matlab script wrapper fail for firmware versions 1.0.3.4 and 1.0.3.5
** Adjust matlab script wrapper to accept these firmware versions.

=== Medium priority ===

* Get the "projects" table working
** Make it work so that is causes ''fewer'' issues, not more. If this is not possible, streamline the old-fashioned way of creating users and granting access
** Give WB access to the projects table.

* Extension of the table "gps.uva_devices"
** JS will specify which new columns need to be added.

* New hardware specification
** Coen will write down a first draft specification of the new hardware.

=== Lower priority issues ===

* Userflag application
** There appears to be an application for flagging invalid GPS data. What is the status of this application? Who has access?

=== Change requests ===

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

(empty)

=== Data model, processing ===

* If there is an error during the processing of a GPS file, move that file to a "failed" subdirectory.
** ''Failed files are moved to a "failed" subdirectory in the same place as the failed file.''
* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the UvA-bits web site.
** ''Scripts and urls have been provided to Aart-Jan for the UvA-bits web site''

* Automatically and periodically generate power graphs for the GPS loggers; take the graph generating program from the Matlab script wrapper.
** ''Script is run periodically; plots are placed in a directory where the UvA-bits web site can read them from. Graphs are generated for all loggers. To protect the generated images so only authorized users can see them, only the UvA-bits machine can access these graphs. Access must be controlled by the UvA-bits machine.''

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** ''In progress, expect to finished by 14-12-2011.''
** ''LV: the uvagps functions are available through the functions tab and correctly filter gps devices.''

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2012-02-07T10:46:21Z

Lyklev: /* Administration */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

(empty)

=== Data model, processing ===

* If there is an error during the processing of a GPS file, move that file to a "failed" subdirectory.
** ''Failed files are moved to a "failed" subdirectory in the same place as the failed file.''
* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the UvA-bits web site.
** ''Scripts and urls have been provided to Aart-Jan for the UvA-bits web site''

* Automatically and periodically generate power graphs for the GPS loggers; take the graph generating program from the Matlab script wrapper.
** ''Script is run periodically; plots are placed in a directory where the UvA-bits web site can read them from. Graphs are generated for all loggers. To protect the generated images so only authorized users can see them, only the UvA-bits machine can access these graphs. Access must be controlled by the UvA-bits machine.''

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** ''In progress, expect to finished by 14-12-2011.''
** ''LV: the uvagps functions are available through the functions tab and correctly filter gps devices.''

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2012-02-07T10:40:29Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

(empty)

=== Data model, processing ===

* If there is an error during the processing of a GPS file, move that file to a "failed" subdirectory.
** ''Failed files are moved to a "failed" subdirectory in the same place as the failed file.''
* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** ''In progress, expect to finished by 14-12-2011.''
** ''LV: the uvagps functions are available through the functions tab and correctly filter gps devices.''

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2012-02-07T10:37:22Z

Lyklev: /* Finished */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

(empty)

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** ''In progress, expect to finished by 14-12-2011.''
** ''LV: the uvagps functions are available through the functions tab and correctly filter gps devices.''

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2012-02-07T10:36:35Z

Lyklev: /* Public web site */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

(empty)

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

Annotation tables

2012-01-30T12:32:33Z

Lyklev: /* Introduction */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== CSV Files ===

The records that you want to upload must be stored in CSV (comma-separated value) files. The file must meet the following requirements:

* the first line of the file must contain the column names, separated with commas
* each next line must contain the data fields in the same order as the column names, separated with commas
* the column names must be unique
* spaces around column names and fields are always ignored
* in column names, there is no distinction between uppercase and lowercase.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_info_serial = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema and the owner will be the user who uploaded the table. By default, no other users can access the uploaded table. You can share your tables with other users using a database administration tool, such as PgAdminIII. The following example shows how to give another user read-only access to the recently uploaded table, so that the other user can read, but not edit the contents of the table.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser, which is the left pane in the window, as shown below.

[[File:Table_user_rights_1.png]]

Look for the database "flysafe" and unfold the schema with your
login name and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

UvAGPS

2011-12-07T16:31:49Z

Lyklev: /* UVA_TRACKING_SPEED or UVA_TRACKING_SPEED_LIMITED */

= UvAGPS Data Model =

For different versions of the UvAGPS firmware different tables are used. For firmware version v0.9.3.6 till v1.0.0.X the tables UVA_ENGINEERING_DATA and UVA_TRACKING_DATA are used. But in the new firmware versions 1.0.1.4 till 1.0.2.4 so much information was added that new tables were introduced. However for the most important tracking tables a view is created so that all similar tracking for all types of firmware can be seen in one view. That is the UVA_TRACKING view.

All tables, views and functions are also in the GPS schema.

Not everybody has access to all data. For most users access is limited to "limited" views that only give access to selected devices.

== UVA_TRACKING_DATA or UVA_TRACKING_LIMITED ==

Tracking data from UvA GPS tags for older firmware

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||int4||PK||NOT NULL ||
|-
|date_time||timestamp||PK||NOT NULL||GMT
|-
|latitude||float8||||||Decimal degrees
|-
|longitude||float8||||||Decimal degrees
|-
|h_accuracy||float8||||||Horizontal accuracy (m)
|-
|v_accuracy||float8||||||Vertical accuracy (m)
|-
|altitude ||float8||||||m
|-
|x_acceleration||Float8||||||x axis acceleration (m/s22) movement to right is + and to left is -
|-
|y_acceleration||Float8||||||y axis acceleration (m/s22) forward movement is + and backwards is -
|-
|z_acceleration||Float8||||||z axis acceleration (m/s22) upwards movement is + downwards is -
|-
|speed ||float8||||||m/s
|-
|course||float8||||||Degrees
|-
|pressure||float8||||||Pressure sensor (pascal)
|-
|temperature||Float8||||||Degrees celcius
|-
|solar_voltage||Float8||||||V
|-
|battery_voltage||Float8||||||V
|-
|userflag||Integer||||||Data flagged as unacceptable by user
|-
| “"location”"||geometry||||||postgreSQL geometry
|-
|}

== UVA_ENGINEERING_DATA ==

Engineering data from UvA GPS tags for older firmware

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||int4||PK||NOT NULL ||
|-
|date_time||timestamp||PK||NOT NULL||GMT
|-
|t_syncsts||smallint||||||0: not synced, 1: user synced, 2: Gps synced
|-
|t_entzt||smallint||||||0: UCT time,1: TZT time (timezone time)
|-
|gzp_currentrategps||int4||||||
|-
|gzp_currentratecom||int4||||||
|-
|gzp_currentratepwr||int4||||||
|-
|sbm_voltagesolarpanel||float8||||||
|-
|sbm_voltagebattery||float8||||||
|-
|sbm_chargemode||int2||||||
|-
|egzp_mainrategps||integer||||||
|-
|egzp_mainratecom||integer||||||
|-
|egzp_mainratepwr||integer||||||
|-
|smg_on||text||||||
|-
|smg_off||text||||||
|-
|smg_esavegpsrate||integer||||||
|-
|smz_on||text||||||
|-
|smz_off||text||||||
|-
|smz_esavecomrate||integer||||||
|-
|gpsnavigationmode||smallint||||||
|-
|gpsfixtimeout||smallint||||||
|-
|eh||text||||||
|-
|}

== UVA_DEVICE or UVA_DEVICE_LIMITED ==

To be able to track the different firmware versions and specific properties for the different devices the UVA_DEVICE table was added. This table also includes essential information needed to calibrate the accelerometers. For more information see [[UvAGPS|Accelerometer calibration]]

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
| device_info_serial || integer || PK || NOT NULL || Serial number of tag
|-
| firmware_version || text |||| NOT NULL || Firmware version installed on tag
|-
| device_mass || double |||||||Mass of tag (g)
|-
| x_s|| double |||||||Calibration sensitivity of accelerometer in the x-axis
|-
| y_s|| double |||||||Calibration sensitivity of accelerometer in the y-axis
|-
| z_s|| double |||||||Calibration sensitivity of accelerometer in the z-axis
|-
| x_o|| double |||||||Calibration offset of accelerometer in the x-axis
|-
| y_o|| double |||||||Calibration offset of accelerometer in the y-axis
|-
| z_o|| double |||||||Calibration offset of accelerometer in the z-axis
|-
|}

== UVA_COMM101 or UVA_COMM_LIMITED==

Communication table indicates status and result of communications with ground stations. Only for newer firmware.

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
| device_info_serial || integer || PK || NOT NULL ||
|-
| date_time || timestamp || PK || NOT NULL || timestamp of start communication
|-
| date_time_end || timestamp |||||| timestamp of end communication
|-
| date_time_utc || timestamp |||||| date_time in utc (if available)
|-
| date_time_end_utc || timestamp |||||| date_time_end in utc (if available)
|-
| action || text|||||| JOINED / RESCHEDULE ??
|-
| bytes_communicated || integer |||||| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text |||||| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || integer||||||milliseconds
|-
| number_identical_lines || integer |||||| Number identical lines in data block. Gives indication
|-
| non_ascii_char || integer |||||| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp |||||| derived from S0 hh,mi
|-
| timesynced || smallint|||||| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || smallint||||||DeviceMode (0-99)
|-
| vbat || float |||||| Battery voltage in V (0-~4.200V)
|-
|logb || integer |||||| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || smallint |||||| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || integer |||||| GpsIval, current (0-65535)
|-
|comii || integer|||||| ComIval, current (0-65535)
|-
|accii || integer|||||| AcceleroIval, current (0-65535)
|-
|swddd || integer|||||| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||||||timestamp of last fix
|-
|latitude||float||||||Latitude of last fix. Decimal degrees
|-
|longitude||float||||||Longitude of last fix. Decimal degrees
|}

== UVA_SETTINGS101 or UVA_SETTINGS_LIMITED ==

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines
If C0-C1 is parsed its values are compared with the previous value and if the settings are the
same no new records needs to be added
If uplink settings are parsed wait till the last setting has been seen, (DL command or EOF)
then store all the new settings in one record. Only for newer firmware.

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||int4||PK||NOT NULL ||
|-
|date_time||timestamp||PK||NOT NULL||Derive timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || smallint ||||||
|-
|timezone_time || smallint ||||||
|-
|swddd || integer|||||| SpecialUserSwitches
|-
|gpsnavmode || smallint|||||| GpsNavMode 0-7
|-
|gpsreceivermode || smallint |||||| GpsReceiverNode 0-3
|-
|gpsfixtime || smallint|||||| GpsFixTime 0-255
|-
|gps_ivaln_on_h || smallint|||||| hours Gps IvalN on
|-
|gps_ivaln_on_m || smallint|||||| minutes Gps IvalN on
|-
|gps_ivaln || integer|||||| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || smallint|||||| hours Gps IvalE on
|-
|gps_ivale_on_m || smallint|||||| minutes GpS IvalE on
|-
|gps_ivale || integer|||||| GpsIvalE
|-
|com_ivaln_on_h || smallint|||||| hours Com IvalN on
|-
|com_ivaln_on_m || smallint|||||| minutes Com IvalN on
|-
|com_ivaln || integer|||||| ComIvalN (0-65530)
|-
|com_ivale_on_h || smallint|||||| hours Com IvalE on
|-
|com_ivale_on_m || smallint|||||| minutes Com IvalE on
|-
|com_ivale || integer|||||| ComIvalE
|-
|accii || integer|||||| Accelero measurement interval
|-
|accbn || integer|||||| Accelero number of blocks
|-
|accsn || integer|||||| Accelero number of samples (0-65536)
|-
|f || smallint||||||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || smallint|||||| PdopMsk (3-250)
|-
|pac||smallint|||||| PaccMsk (10-100)
|-
|tdp ||smallint|||||| TdopMsk (3-250)
|-
|tac ||smallint|||||| TaccMsk (30-300)
|-
|ga || smallint|||||| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || smallint||||||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text|||||| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

== UVA_TRACKING_DATA101 or UVA_TRACKING_LIMITED==

The tracking dat for newer firmware.

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||int4||PK||NOT NULL ||
|-
|date_time||timestamp||PK||NOT NULL||
|-
|latitude ||float||||||Decimal degrees
|-
|longitude ||float||||||Decimal degrees
|-
|altitude ||integer||||||heigth above sealevel meters
|-
|pressure ||integer||||||in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature ||float||||||in celcius degrees
|-
|satellites_used ||smallint||||||satellites used for fix
|-
|gps_fixtime ||float||||||GpsFixTime in seconds
|-
|positiondop ||float||||||Position DOP in units
|-
|h_accuracy || float||||||Horizontal accuracy (units of meter)
|-
|v_accuracy || float||||||Vertical accuracy (units of meter)
|-
|x_speed ||float||||||Speed in x-axis (m/s)
|-
|y_speed ||float||||||Speed in y-axis (m/s)
|-
|z_speed ||float||||||Speed in z-axis (m/s)
|-
|speed_accuracy || float||||||Speed accuracy (m/s)
|-
|userflag||Integer||||||Data flagged as unacceptable by user if not 0
|-
|location || geometry |||||| lat/long as PostGIS geometry
|}

== UVA_ACC_START102 or UVA_ACC_START_LIMITED ==

General acceleration entry associated with a set of accelerations. Added in Firmware 1.0.2.7 because there line numbers were introduced for acceleration lines so that these lines can be tracked.

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||int4||PK||NOT NULL ||
|-
|date_time||timestamp||PK||NOT NULL||
|-
|line_counter||integer||||||Number of lines associated with this set of accelerations
|-
|timesynced||smallint||||||TimeSynced (0=not synced, 1=user set, 2=Gps Synced)
|-
|accii||integer||||||Accelero measurement interval
|-
|accsn||integer||||||Number of samples (0-65536)
|-
|f||smallint||||||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|}

== UVA_ACCELERATION101 or UVA_ACCELERATION_LIMITED ==

Acceleration entries if accelerometer was turned on. Only for newer firmware. Actual measurements have shown that the range of values for x,y and z are larger then indicated by the manufacturer. For more information about calibration and interpretation of accelerometer data see [[UvAGPS#Accelerometer_calibration|Accelerometer calibration]]

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||int4||PK||NOT NULL ||
|-
|date_time||timestamp||PK||NOT NULL||
|-
|index ||smallint||PK||NOT NULL||
|-
|x_acceleration ||smallint||||||acceleration in X-axis (-2666 till 2666)
|-
|y_acceleration ||smallint||||||acceleration in Y-axis (-2666 till 2666)
|-
|z_acceleration ||smallint||||||acceleration in Z-axis (-2666 till 2666)
|}

== UVA_ENERGY101 or UVA_ENERGY_LIMITED ==

Only for newer firmware.

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||int4||PK||NOT NULL ||
|-
|date_time||timestamp||PK||NOT NULL||
|-
|timesynced ||smallint||||||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||smallint||||||(0-99)
|-
|vsll ||float||||||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float||||||Battery Voltage in V (0-~4.200)
|-
|ssw ||smallint|||||| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||||||extended fields if special user switch is on
|-
|vsoo ||float|||||| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float||||||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float||||||charge current voltage
|-
|g ||smallint|||||| Gps battery threshold OK
|-
|c || smallint||||||Com battery threshold OK
|-
|m || smallint||||||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || integer|||||| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float|||||| in celcius degrees
|}

== UVA_TRACKING or UVA_TRACKING_LIMITED ==

View to show identical fields for older and newer firmware tracking data

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||int4||PK||NOT NULL ||
|-
|date_time||timestamp||PK||NOT NULL||
|-
|latitude||float||||||
|-
|longitude||float||||||
|-
|altitude||float||||||
|-
|pressure||integer|||||
|-
|temperature||float||||||
|-
|h_accuracy||float||||||
|-
|v_accuracy||float||||||
|-
|x_speed||float||||||meters/second
|-
|y_speed||float||||||meters/second
|-
|z_speed||float||||||meters/second
|-
|gps_fixtime||float||||||
|-
|userflag||integer||||||Flagged as inacceptable by user if not 0
|-
|satellites_used||smallint||||||
|-
|positiondop||float||||||
|-
|speed_accuracy||float||||||
|-
|location||geometry||||||
|}

== UVA_TRACKING_SPEED or UVA_TRACKING_SPEED_LIMITED ==

Additional view that computes with the x_speed, y_speed and z_speed components the vnorth, veast and vdown speed components as well as the 3D speed in meters/second.

In addition to the fields in the UVA_TRACKING view this view adds :

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|vnorth||float||||||- vx * sin(lat) * cos(long) - vy * sin(lat) * sin (long) + vz * cos(lat)
|-
|veast||float|||||| - vx * sin(lon) + vy * cos(lon)
|-
|vdown||float|||||| - vx * cos(lat) * cos(lon) - vy * cos(lat) * sin(lon) - vz * sin(lat)
|-
|speed||float|||||| sqrt(vnorth^2 + veast^2)
|-
|speed3d || float |||||| sqrt(vnorth^2 + veast^2 + vdown^2)
|-
|direction||float|||||| heading in degrees; 0 is north, 90 is east
|}

== UVA_WARNING102 or UVA_WARNING_LIMITED ==

Since firmware version 1.0.2.7 this table was added as catchall for tags that do not occur very often (C10, C20) or errors that occur during parsing.
For example all CRC failures can be selected with :

select * from gps.uva_warning102
where warning = 'CRC8 failure'
order by device_info_serial;

{|border=1
|'''Field'''||'''Type'''||'''Key'''||'''Constraints'''||'''Comments'''
|-
|device_info_serial||integer||PK||NOT NULL||
|-
|filename||text||PK||NOT NULL||UvAGPS log filename
|-
|linenumber||integer||PK||NOT NULL||linenumber in file
|-
|warning||text||||||The warming message or tag
|-
|line||text||||||The actual line in the file
|}

= UvAGPS functions =

== get_uvagps_track_speed ==

Compute distance (meters), time-interval, speed (meters/second) and heading (degrees) for a series of subsequent tracking entries with specific device_info_serial.
For the first entry in the sequence these values will always be missing because they are computed as the difference between subsequent rows.

Usage examples :

<pre>
select * from gps.get_uvagps_track_speed(119, '2008-01-01 00:00:00', '2010-01-01 00:00:00');

select * from gps.get_uvagps_track_speed(119);

select t.*, a.distance, a.interval, a.speed, a.direction from gps.get_uvagps_track_speed(119, '2008-01-01 00:00:00', '2010-01-01 00:00:00') a join gps.uva_tracking t using(device_info_serial, date_time);
</pre>

Distances and heading are calculated with spherical correction with the Haversine formula as described [http://http://www.movable-type.co.uk/scripts/latlong.html here].
The heading is the initial heading. PostGIS does not calculate the heading, so therefore we don't use PostGIS.

In these functions the records that have been flagged as invalid by setting the '''userflag''' are not taken into account and discarded.
If it is required get the speed for all records a additional boolean argument for '''get raw data''' can be given. i.e.

<pre>
select * from gps.get_uvagps_track_speed(119, '2008-01-01 00:00:00', '2010-01-01 00:00:00', true);

select * from gps.get_uvagps_track_speed(119, true);
</pre>

== get_uvagps_track_distance ==

Compute distance (meters) for UvAGPS tracks to arbitrary point in lat/long or geometry location

Example usage :

<pre>
select * from gps.get_uvagps_track_distance(119, 52.1897017, 6.1996606);

select * from gps.get_uvagps_track_distance(119, (select location from gps.uva_tracking where device_info_serial = 119 and date_time = '2009-05-07 19:55:08'));

</pre>

Here the distance is calculated with the PostGIS distance_sphere function which uses spherical correction.

== get_uvagps_track_distance_direction ==

Compute distance (meters) and direction (degrees) for UvAGPS tracks from arbitrary point in lat/long or geometry location

Example usage:

<pre>
select * from gps.get_uvagps_track_distance_direction(119, 52.1897017, 6.1996606);
</pre>

Distances and direction are calculated with spherical correction with the Haversine formula as described [http://http://www.movable-type.co.uk/scripts/latlong.html here]. The direction is the direction from the given point.

In this function the records where the '''userflag''' has been set are discarded. To see these records a additional boolean argument for '''show raw data''' should be added like :

<pre>
select * from gps.get_uvagps_track_distance_direction(119, 52.1897017, 6.1996606, true);
</pre>

= Accelerometer calibration =

The accelerometer measures the acceleration in 3D (20 times per second). Acceleration is caused by gravity (static component) and by a change in speed (dynamic component).
When the logger is static in horizontal position (solar cells up), gravity (green arrow) shows a heave (vertical acceleration, z) of +1 g (1 g = 9.81 m.s-2) while surge and heave are both 0. When the front of the logger is vertically pointing down (forward acceleration, x), surge is +1g and when it is rolled to the right (sidewards acceleration, y) sway is +1g. Due to gravity, the pitch of a logger is thus translated into the values of heave, surge and sway (or z, x, y).

However, the values are measured in mV instead of g. Before delivering the GPS-loggers, they have been calibrated at UvA. The acceleration (A) expressed in g is calculated from the measured value (M) and the calibration factors offset (''O'') and sensitivity (''S'').

Thus acceleration for heave (z), surge (x) and sway (y) is calculated as:
Az = (Mz-Oz)/Sz ; Ax = (Mx-Ox)/Sx; Ay = (My-Oy)/Sy ;

The values of ''O'' (in general ranging from 0-60, sometimes even 300) and ''S'' (ranging from 1300-1450) are available in the device table of the data base. If not available, use the factory defaults ''O'' = 0 and ''S'' = 1365.

{|
| [[File:accelero1.png|300px]]||[[File:accelero2.png|300px]]|
|}

In the figure above heave (z) is shown in green, surge(x) in red and sway (y) in blue, all values are expressed in g (on the y-axis).

= Data processing and filtering =
The raw log files are processed before being entered into the database. Each firmware version has a dedicated parser for the task. In certain cases data that is found in the log files is not stored in the database. This can occur for two reasons:

1. The data did not pass the CRC8 checksum (cyclic redundancy check). This is a standard procedure designed to detect accidental changes to raw computer data.

2. If the data has an invalid signature. For example, if a certain data format (eg numeric) or number of values (eg 4 digits) is expected but not returned, this record is not stored in the database.

When a line is deleted, the log file name, line number and the type of error is stored in the table gps.uva_warning102

ToDo

2011-11-28T11:09:28Z

Lyklev: /* Public web site */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** ''In progress, expect to finished by 14-12-2011.''
** ''LV: the uvagps functions are available through the functions tab and correctly filter gps devices.''

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-28T11:09:18Z

Lyklev: /* Public web site */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** ''In progress, expect to finished by 14-12-2011.''
** ''LV: the uvagps functions are available through the functions tab and correctly filter gps devices''

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

Showing login roles

2011-11-28T09:55:25Z

Lyklev:

By default, PgAdminIII only shows groups when modifiying the table privileges. If you want to share tables with individual users, you must change the PgAdminIII preferences to show users when modifying privileges. This is a PgAdminIII setting, so you need to adjust it in your own installation.

To make PgAdminIII show users:

# Select "Options" from the File menu. A settings window will appear.
# Go to the "Preferences" tab in the window, as highlighted in the figure below
# Set the "Show users for privileges" option, as highligted in the figure below.

[[File:Pgadmin_show_users_for_privileges.png]]

File:Pgadmin show users for privileges.png

2011-11-28T09:52:33Z

Lyklev: Image for showing how to make PgAdminIII show individual users instead of only group roles; relevant sections are highlighted.

Image for showing how to make PgAdminIII show individual users instead of only group roles; relevant sections are highlighted.

Showing login roles

2011-11-28T09:51:14Z

Lyklev: Created page with "By default, PgAdminIII only shows groups when modifiying the table privileges. If you want to share tables with individual users, you must change the PgAdminIII preferences to sh..."

ToDo

2011-11-21T14:33:40Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

* Improvements after meeting on 17-11-2011 (expected corrections on nov 28).
** Improve documentation, describing how to give individual users access to tables; this is a PgAdminIII feature, so describe where to look in the menus
** check ownership of tables after uploading
** verify that there is no restriction on the size of uploaded tables
** verify that mixing case in specified and actual column names in handled in a consistent way (for example 'nestid' vs. 'nestID')
** improve error messages, especially in the case of double records, missing

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-21T14:24:07Z

Lyklev: /* Finished */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-21T14:23:53Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-21T14:22:41Z

Lyklev: /* Finished */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "speed" was added to the gps.uva_tracking_speed view; the original speed was renamed to "speed_3d".

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-21T14:21:59Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed_2d view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-21T14:20:59Z

Lyklev: /* Finished */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Field "angle" was renamed to "direction" and was added to the gps.uva_tracking_speed_2d view. The unit is degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees.

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-21T14:20:04Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-07T12:57:44Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.
** 13-10-2011, LV: Field "angle" must be renamed to "direction" and must be added to the gps.uva_tracking_speed_2d view. The unit must be degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees. Expected finish is 15-11-2011.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-07T12:57:13Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.
** 13-10-2011, LV: Field "angle" must be renamed to "direction" and must be added to the gps.uva_tracking_speed_2d view. The unit must be degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees. Expected finish is 15-11-2011.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-07T12:56:04Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.
** 13-10-2011, LV: Field "angle" must be renamed to "direction" and must be added to the gps.uva_tracking_speed_2d view. The unit must be degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees. Expected finish is 18-10-2011.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-07T12:55:29Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).
** 02-11-2011, LV: Added a 'nicer', more consistent interface to the template management and data table upload system.
** 02-11-2011, LV: Added a description of the workflow to the user manual.

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.

* Write a manual which describes how to query the spatial database, for easily accessing the flow data for given (x,y)-coordinates and time. (LV)
Progress:
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.
** 13-10-2011, LV: Field "angle" must be renamed to "direction" and must be added to the gps.uva_tracking_speed_2d view. The unit must be degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees. Expected finish is 18-10-2011.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-07T12:53:09Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).

* Design and implement a datamodel for the output of the "tides and currents model":
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.

* Write a manual which describes how to query the spatial database, for easily accessing the flow data for given (x,y)-coordinates and time. (LV)
Progress:
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.
** 13-10-2011, LV: Field "angle" must be renamed to "direction" and must be added to the gps.uva_tracking_speed_2d view. The unit must be degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees. Expected finish is 18-10-2011.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-07T12:52:42Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
* Design and implementation of the annotation table upload facility:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the *user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).

* Design and implement a datamodel for the output of the "tides and currents model" as soon as the data is available. (WB, medium priority)
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.

* Write a manual which describes how to query the spatial database, for easily accessing the flow data for given (x,y)-coordinates and time. (LV)
Progress:
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.
** 13-10-2011, LV: Field "angle" must be renamed to "direction" and must be added to the gps.uva_tracking_speed_2d view. The unit must be degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees. Expected finish is 18-10-2011.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

ToDo

2011-11-07T12:51:31Z

Lyklev: /* Data model, processing */

== ToDo List ==

This is a dynamic list for Lykle Voort that changes with changing priorities. This list includes items predominantly related to the database that are pending and include a request data, status and person reporting problem (WB - Willem Bouten;JS - Judy Shamoun-Baranes; EB - Edwin Baaij). When items are solved they are moved to the FINISHED section.

=== Public web site ===

* Copy UvAGPS functions available on the flysafe server to the public server (JS, medium priority)
** In progress, expect to finished by 14-12-2011.

=== Data model, processing ===

* Create an annotation uploading facility. An annotation uploading facility enables users with database access to upload files with a previously defined structure to the database in the user's own schema. A table will typically be used to annotate GPS data. The columns of the table are created in 'templates', so to a template management system is to be created. This work consists of the following steps:
** create a template overview page
** add facilities to create a new template, either by starting from scratch or by copying an existing template from another user (where the new owner will be the user logged in)
** a facility to edit existing templates
** a facility to skip columns in uploaded files, in case not all columns are needed or in case some columns are copies of existing columns in other tables;
** log of upload service (who, when, which table) + overview page?? --> think/brainstorm on possibilities
** user documentation describing the workflow; (mail to Judy/Willem when 90% finished) (iterate within SARA first)

Progress:
** 07-10-2011, LV: Added the [[Description of the Data upload and Annotation Service]]. A breakdown of the work is also listed there.
** 15-08-2011, LV: Upload facility is in progress. The user will be presented an upload form allowing where the user can select a file to be uploaded and a table in the user's own schema in which the data must be stored.
** 16-09-2011, LV: First of upload facility (draft) version complete
** 24-10-2011, LV: Completed template creation and editing faclity (first draft, needs cosmetic improvements).

* Design and implement a datamodel for the output of the "tides and currents model" as soon as the data is available. (WB, medium priority)
** 25-08-2011, LV: The schema "flow" has been added to the datbase for storing the flow grids and model results.
** 25-08-2011, LV: A postgis table "grid_waddenzee" has been added to the database. This postgis table contains grid used by the flow model ''unstruc''' which was imported from OpeNDAP data provided by Deltares. You can view the grid using postgis-capable programs (e.g. Qgis).
** 31-08-2011, LV: The test data from Deltares has been imported into the database automatically.
** 31-08-2011, LV: The automatic importing of data will be started as soon as Deltares has actual data available.
** 21-09-2011, LV: Two model output data sets have been supplied by Deltares. Import has been started for the model ''Waddenfijn''. This model has a resolution of 100m.
** 28-09-2011, LV: Data sets appear to be incomplete, Deltares has been contacted. The 100m-model has data up to July 19th. Data has been imported from April 1st to July 18th to save disk space. The 300m-model has even less useful content, so this data was not imported into the database.

* Write a manual which describes how to query the spatial database, for easily accessing the flow data for given (x,y)-coordinates and time. (LV)
Progress:
** 17-10-2011, LV: Database functions have been added which find the flow velocities for a given location and time range. These functions can be used in queries which add local wind and flow velocities to the track data of a bird.
** 17-10-2011, LV: Example queries and documentation have been added to the user documentation.

* Add 2D speed to the speed calculations in the uva_tracking_speed view (rename speed to 3D_speed). Add direction to calculations.
Work ahead:
** ''10-10-2011, LV: Function was not renamed, because other views depend on this view; instead a new table '''gps.uva_tracking_speed_2d''' was created. This view has the field 'angle', which is the angle from north, positive towards east, in radians.
** 13-10-2011, LV: Field "angle" must be renamed to "direction" and must be added to the gps.uva_tracking_speed_2d view. The unit must be degrees, where north is 0 degrees, northeast is 45 degrees and east is 90 degrees. Expected finish is 18-10-2011.

* Automated upload and import UvAGPS metadata which should include a web form and webservice (WB, low priority)
** On hold while working on annotation metadata import.

* Add function for computation of elevation (and perhaps azimuth) of sun above the horizon for each gps point. This would provide users with the possibility to determine whether a point is measured during the day, at twilight (elevation lets users decide if they want civil or nautical twilight) or at night. An example script can be found in the R script crepuscule.R which is part of the maptools toolbox and would extract the value “solarpos”. Michael Kemp can provide more information on script usage and examples (m.u.kemp@uva.nl) day start, day end to mark whether a data point is at night or during the day (JS, low priority)
** On hold while working on annotation metadata import.

* Accelerometer data: based on the current data model, we cannot properly sort accelerometer measurements that are not coupled with a GPS fix, an extra field in the accelerometer table is probably needed. (JS, very low priority)
** On hold while working on annotation metadata import.

=== Administration ===

* List of access rights: user's rights must be set consistently to make sure that all users have the correct rights to see data (WB, high priority);
** 01-07-2011, LV: In progress; a web interface listing user's rights is being created as part of the administrative interface; on hold for uploading annotation data.

* add a project data table; assign projects to users or users to projects; link bird access to projects
** On hold while working on annotation metadata import.

* Bookkeeping: Change interface so that a user can select uvaGPS and then a particular tag (or tags with cntr shift for example) and range of dates. Currently a user can select uvaGPS and a range of dates and will receive all tags, many tags may not be relevant to the user, making it difficult to quickly assess the status (JS, low priority, should be improved by March 2012)

* Provide authentication scripts and methods to Aart-Jan for the Uva-bits web site.

* new policy for addition of new users: disable e-mail function, become user and check functionality before notification
** ''14-10-2011: new users have received a new e-mail with updated data, referring them to the UvA-Bits web site. Working on check-list for adding new users.''

=== Other ===

== Finished ==

* Adapt parser for new firmware (V 1.0.35) format (WB/EB, July 2011, high priority)

''Parsing of the output of firmware version 1.0.35 is working. Some differences between the documentation and the actual logger output were reported to EB. (Sept 5 2011)''

''There is an error when parsing the S0 lines; according to the manual there should be 16 fields, but there are 17. Waiting for a reply from EB (sept 1 2011)''

* Upload facility for Enram radar images. An Ftp-server has been set up for uploading Enram radar images from European weather radars. The total data volume is estimated to be around 15 TB, so files are migrated to grid storage after verifying that the upload is complete and correct. (LV, august 2011)

* Login once to the main web site and the phppsqladmin site (Wiki already shares the login) (WB, april 2011, not resolved - STOPPED). This item has proven to be difficult to resolve due to security issues.

* Guest account: we shoud have a guest account with very restricted access, but who can see for example birdview with the tracks of one or two ('donated') birds. (May 2011, Complete). User name = guest, password = guest, access provided to several aspects of the virtual lab and 1 gull trajectory.

* BirdView: I noticed that the upload option for kml files from a local directory includes kml only, can we extend the support to kml and kmz files? The http upload does actually support both. (solved July 2011)

* Import UvAGPS log files via standard storage facility: Data can be uploaded manually via winSCP, iRODS, and digital drop box. Dropbox is becoming the standard for all new stations and linked to the database services (implemented May 2011).

* Quality control: Users with write rights can flag data points manually in the database through pgAdmin. Data cannot be queried but must be viewed through the data viewer to edit. Currently data is manually flagged by J. Shamoun-Baranes (solved April 2011)

* QC & functions: When only a subset of data (say data approved from QC) is queried, the distance and direction functions work on the subset, there is an option to run these functions on all data - but not by default (April 2011)

* Implement generate KMZ service for UvAGPS on public server (implemented May 2011 through virtual lab)

* Metadata: Excel files are either sent to SARA and then uploaded to the database or uploaded manually (March 2011)

* List of user's rights: a script was written to quickly produce an overview of all users and their rights (12-05-2011)

* Make a readonly copy of Flysafe Wiki on public server (september 2010)

* Improve UvA GPS parser to log errors such as CRC8 in database (september 2010)

* Implement service to create graphs for UvAGPS devices according to Willem's Matlab scripts (october 2010)

* Add accelerometer calibration factors to uva_device (september 2010)

* Verify whether there are still users using Irods (WB, June 2011)
** There are no users using the Irods service anymore (September 2011)

* add a description of the user's rights fields, instead of the database names for rights (may 2011)
** A description of the meaning of the user's rights has been added to the flysafe wiki, under "administration".

* make sure that access to weather data is properly restricted
** In weather data two types are distinguished: ECMWF and ECMWF_RNLAF (more recent) data. Access to any type of weather data has to be granted explicitly. (September 2011)

== Other ==

* How often is the FlySafe database updated with raw data uploaded via irods or winscp? (JS, July 2011)
** LV, 22-09-2011 ''Irods is no longer used by users to upload data. Winscp is almost never used since Dropbox became available.''

* How is raw data filtered if at all, in addition to the CRC8 Checksum and the invalid signature, before entering the database? If data is being filtered out based on expert judgment, this must be documented and records should be flagged in the database rather than removed. (JS, July 2011)
** LV, 22-09-2011 ''Raw data is not filtered at all, apart from the CRC8 checksum.''

Annotation tables

2011-11-03T14:34:18Z

Lyklev: /* Uploading the annotation file, joining the data */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"searching for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_serial_info = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema and the owner will be the user who uploaded the table. By default, no other users can access the uploaded table. You can share your tables with other users using a database administration tool, such as PgAdminIII. The following example shows how to give another user read-only access to the recently uploaded table, so that the other user can read, but not edit the contents of the table.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser, which is the left pane in the window, as shown below.

[[File:Table_user_rights_1.png]]

Look for the database "flysafe" and unfold the schema with your
login name and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2011-11-03T11:57:18Z

Lyklev: /* Annotation of GPS track data */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"looking for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_serial_info = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema and the owner will be the user who uploaded the table. By default, no other users can access the uploaded table. You can share your tables with other users using a database administration tool, such as PgAdminIII. The following example shows how to give another user read-only access to the recently uploaded table, so that the other user can read, but not edit the contents of the table.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser, which is the left pane in the window, as shown below.

[[File:Table_user_rights_1.png]]

Look for the database "flysafe" and unfold the schema with your
login name and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

Annotation tables

2011-11-03T11:55:40Z

Lyklev: /* Creating a database table by uploading a CSV file */

= User annotation tables =

== Introduction ==

In a highly database oriented environment with different types of users, some users prefer the possibility of uploading file-based data instead of interacting directly with the database.

For the creation of a new table, the automated system should know the number and type of columns in the uploaded file. This means that some sort of template must exist in the system before a table can be uploaded and inserted automatically. Since most users will typically upload more than one table with the same column names and types, a template has to be created only once and can be reused many times by the same user or by a group of users.

These pages describe the different components in the annotation service:

* the template management pages, for creating and modifying templates;
* the upload form, for uploading files into new database tables.

Furthermore, these pages describe how tables can be used and shared with other users.

== Creating a new table template ==

To create a new table template, go to the page 'Templates', which shows an overview of existing templates. Follow the link called "Create a new template".

Before you can define the fields of the template, you must give the template a name and a description.

* Choose the name such that you can easily find it back.
* The description should be a description of what the template is for, not what the template will contain (as this will be visible when you define the columns). It has a mimimum of 40 characters.

You can start defining columns after you click on "Create template". This will also save the template name and description.

== Defining and modifying template columns ==

Each column in a template definition tells the system which columns to create when you upload a new file into the database. So the names of the columns are used as table column names later on. This means that for each column:

* the name must be a valid table column name, that is, it must start with a letter and it must contain no spaces or special characters;
* the type must a valid database type;
* the checkbox "primary key" defines whether the column will be used as a primary key.

The primary key can be one column or a combination of columns. In a database table, the primary key field or combination of each table row must be unique. For the analysis of GPS tracks, a good combination would be the 'datetime' and the 'device_info_serial' fields. Every database table must have at least one column marked as a primary key.

To add a new template column, fill in the values on the last or only (and empty) row of the overview, then click on the button "Add column" next to it.

To modify existing columns, edit the fields of the columns, then click on "Update columns" below the overview.

To delete an existing column, click on the "Delete column" button next to the row.

== Editing an existing template ==

If you want to edit an existing template:

# Go to the template overview page by selecting the "Templates" tab in the menu.
# Click on the "edit" link next to the template you want to edit. You can only edit your own templates.
# Edit the fields in the same way as when creating a new template.

== Creating a database table by uploading a CSV file ==

In the current version, you can create database tables from CSV files.
Before you can upload a file, you must have created a template in which
each column corresponds with a column in your CSV file.

To upload a file into a new database table, go to the "Upload table"
page in the Annotation management pages. This will show you an upload
form.

First, select the file you want to upload by clikcing on the "browse"
button.

Second, enter a name for the new database table. This must be a valid
database table name, so it must start with a letter, contain no spaces or
special characters, and be no longer than ? characters.

Then select the template which will be used for creating the new table.
Currently, all templates from all owners are shown.

Finally, click the "Upload file" button. This will start uploading the
file, create a new table from the specified template and import the data
from the file into the new table.

If any error occurs during the importing of the contents of the file, the
table is deleted again. The upload system will try to indicate where the
error is in the file.

== Annotating GPS track tables ==

=== Introduction ===

One of the purposes of uploading tables is to have some system to make personal annotations to GPS track data. It is not desirable to have personal annotations in the tracking data tables; instead, personal annotations should be stored in personal tables. Since users can have personal schemas, this is the logical place to put personal files. After creating the new tables, the annotations must be combined with the GPS track data.

=== Definition of the annotation tables ===

If a bird track must be annotated, there must be a way to idenfity records belonging to one particular bird track. The preferred way is to match the 'datetime' and the 'device_info_serial' fields, so annotation tables should have at least these two columns. This means that templates for annotation tables should have at least these two columns. It is also a good idea to make these fields primary keys, as this will speed up the joining later on. Your template will contain your annotation field too.

As a minimal example for adding a comment to a GPS track, a template should contain:

* a 'datetime' field of the type 'timestamp';
* a 'device_info_serial' field of the type 'integer';
* a 'comment' field, which in this example will be of the type 'text'.

=== Uploading the annotation file, joining the data ===

With the above example, suppose there is a very simple file with annotations for just two records. The file looks like this:

"datetime","device_info_serial","comment"
2011-05-21 12:10:45,533,"on nest"
2011-05-21 12:25:42,533,"looking for food"

When this file is uploaded to the database, a new table is created. Suppose that this table is "user.annotation", the results can be combined in one query:

SELECT td.datetime, td.longitude, td.latitude, an.comment
FROM gps.uva_tracking_data td
LEFT JOIN user.annotation an
ON td.datetime = an.datetime AND
td.device_info_serial = an.device_info_serial
WHERE device_serial_info = 533;

== Sharing tables with other users ==

New tables are created in the user's own database schema and the owner will be the user who uploaded the table. By default, no other users can access the uploaded table. You can share your tables with other users using a database administration tool, such as PgAdminIII. The following example shows how to give another user read-only access to the recently uploaded table, so that the other user can read, but not edit the contents of the table.

Start PgadminIII and connect to "services.flysafe.sara.nl". Then
look in the server browser, which is the left pane in the window, as shown below.

[[File:Table_user_rights_1.png]]

Look for the database "flysafe" and unfold the schema with your
login name and unfold the "tables" item in the tree. This will
look like the following figure; all tables you upload will appear
in this schema. In this example "test8" is a table that must
be shared with other users.

Right-click on the table that you want to share with other users.
A menu will pop-up as shown in the following figure; in this
example the table "test8" was right-clicked.

[[File:Table_user_rights_2.png]]

From the popup menu, select "Properties". A window like the one below will be shown
with various options for the selected table.

# Select the tab "privileges"
# In the "role" field, select the user you want to share the table with; ''note: if you see only groups and no users, you have to change your PgadminIII preferences; this is described in [[showing login roles]]''.
# Select the privileges you want to give this user on this table. To read a table, a user must have the "SELECT" privilege; to change contents, the user must also have the "UPDATE" privilege.
# Click the "Add/Change" button. The user with all privileges will appear in the small window on the top.
# Click "Ok" to save the changes and close the window, or repeat the process to share the table with more users.

[[File:Table_user_rights_3.png]]

In the example above, the table "test8" is shared with user "tijs", who will be able to read the table ("SELECT"), but not update the table ("UPDATE" is not checked).

== Annotation of GPS track data ==