ecology - User contributions [en]

How to access the e-Ecology database

2024-05-30T08:11:05Z

Stacy shinneman: /* Setting up an ODBC connection (Windows) */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

The database is no longer accessible via the phpPgAdmin web service but via a new PgAdmin portal. You can access the database through an ODBC (see below) or via the PgAdmin portal following these instructions:

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS
4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology
5. Fill in your username and password in the “Connection" tab

6. Press “Save"

Once you create a connection to the database you can view the database structure in the left panel. You will have to write your own SQL queries to retrieve data since this service no longer has a GUI to build queries.

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[https://www.postgresql.org/ftp/odbc/releases]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-21T12:25:31Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

The database is no longer accessible via the phpPgAdmin web service but via a new PgAdmin portal. You can access the database through an ODBC (see below) or via the PgAdmin portal following these instructions:

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS
4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology
5. Fill in your username and password in the “Connection" tab

6. Press “Save"

Once you create a connection to the database you can view the database structure in the left panel. You will have to write your own SQL queries to retrieve data since this service no longer has a GUI to build queries.

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-20T08:35:29Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

The database is no longer directly accessible via the PgAdmin app, so you will need to access the database either through an ODBC connection (see below) or through a new web portal by following these instructions:

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS
4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology
5. Fill in your username and password in the “Connection" tab

6. Press “Save"

This new system does not have a query builder like the old database web portal so you will need to create your own queries.

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-20T08:29:10Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

The database is no longer directly accessible via the PgAdmin app, so you will need to access the database either through an ODBC connection (see below) or through a new web portal by following these instructions:

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS
4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology
5. Fill in your username and password in the “Connection" tab

6. Press “Save"

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-20T08:28:28Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

The database is no longer directly accessible via the PgAdmin app, so you will need to access the database either through an ODBC connection (see below) or through a new web portal by following these instructions:

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS

4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology

5. Fill in your username and password in the “Connection" tab

6. Press “Save"

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-20T08:25:43Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

The Phgpadmin database is no longer active so you will need to access the database either through an ODBC connection (see below) or through a new web portal by following these instructions:

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS

4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology

5. Fill in your username and password in the “Connection" tab

6. Press “Save"

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-20T08:25:29Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

The Phgpadmin database is no longer active so you will need to access the database either through an ODBC connection (see below) or by using a new web portal by following these instructions:

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS

4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology

5. Fill in your username and password in the “Connection" tab

6. Press “Save"

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-20T08:24:23Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

The Phgpadmin database is no longer active so you will need to access the database either through an ODBC connection (see instructions below) or by using a new web portal by following these instructions:

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS

4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology

5. Fill in your username and password in the “Connection" tab

6. Press “Save"

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-20T08:22:14Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials

2. Click on "Add New Server”

3. In "General" tab fill out the fields as following:
Name: UvA-BiTS

4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology

5. Fill in your username and password in the “Connection" tab

6. Press “Save"

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2023-07-20T08:16:10Z

Stacy shinneman: /* How to access the database */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Through the web portal'''

1. Go to pgadmin.e-ecology.nl and log in with your e-Ecology credentials
2. Click on "Add New Server”
3. In "General" tab fill out the fields as following:
Name: UvA-BiTS
4. In the "Connection" tab fill out the fields as following:
Host name / address: pub.e-ecology.nl
Port: 5432
Maintenance DB: eecology
5. Fill in your username and password in the “Connection" tab
6. Press “Save"

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pub.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pub.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pub.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

E-Ecology Documentation and Guidance

2021-01-22T09:58:46Z

Stacy shinneman:

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURF Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[How to access the e-Ecology database]]
::* [[Reference systems for GPS coordinates and altitudes]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::'''''This service is currently not available.''''' Access meterological and radar data and overlay with GPS bird tracking data.
:: [[BirdSim]]: 
:::'''''This service is currently not available.''''' More dynamic version of BirdView that allows the user to scroll through weather data in time and space.

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users. '''''This service is currently not available.'''''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[How to access the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

Reference systems for GPS coordinates and altitudes

2021-01-22T09:37:49Z

Stacy shinneman:

All of our UvA-BiTS GPS trackers use the Ublox Max-6G GPS receiver to collect vertical and horizontal position data.

The GPS unit measures the distance from the satellites to the device and then it calculates the distance relative to the WGS84 ellipsoid (World Geodetic System WGS84). GPS coordinates in the database are calculated relative to the WGS84 datum.

The altitude measurements provided in the database are altitude above a geoid that approximates global mean sea level rather than altitudes relative to the WGS84 ellipsoid. (A good explanation about the difference between ellipsoids and geoid is available here: [https://www.esri.com/news/arcuser/0703/geoid1of3.html])

Be aware that global mean sea level is not the same as the earth’s surface (i.e. land surface elevation). However, the SRTM-DEM data included in the database as land surface elevation data are also calculated relative to global mean sea level and can therefore be combined directly with UvA-BiTS movement data to calculate the bird’s altitude above land surface elevation. Different GPS tracker manufacturers may provide altitude measurements using different methods and/or reference systems, so use caution when combining GPS data collected with multiple tracking systems.

UvA-BiTS Tracking Data

2021-01-22T09:35:49Z

Stacy shinneman: /* Data processing and filtering */

== UvA-BiTS Tracking Data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_tracking_speed_limited ===

Shows the tracking data that your projects allow you to see.

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker, plus it adds speeds. See [[Reference systems for GPS coordinates and altitudes]] for further details.

It replaces old views gps.uva_tracking_limited, gps.uva_tracking_speed_3d_limited and gps.uva_tracking_speed. This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|longitude||float8||YES||Longitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS84 datum)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|vnorth||float8||YES||- vx * sin(lat) * cos(long) - vy * sin(lat) * sin (long) + vz * cos(lat), m/s
|-
|veast||float8||YES||- vx * sin(lon) + vy * cos(lon), m/s
|-
|vdown||float8||YES||- vx * cos(lat) * cos(lon) - vy * cos(lat) * sin(lon) - vz * sin(lat), m/s
|-
|speed||float8||YES||- vx * sin(lon) + vy * cos(lon)
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2), m/s
|-
|speed_2d||float8||YES|| Speed on 2D, m/s
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate based on Digital Elevation Model data used in the e-ecology database (for further information see: [[Elevation]]
|}

=== ''gps.''ee_acceleration_limited ===

Acceleration is measured in blocks of 1-200 measurements (records). The date_time mentioned in this block is the date and time of the first acceleration measurement. Sequential measurements within a block are numbered with an index 1... (max 200). Every row represents one measurement in 3 directions (X,Y,Z). The order is important within each block, so you should always sort the acceleration measurements by block and index. If you need the exact time for each row, you can calculate it from the index, the sample frequency (which is almost always 20 Hz) and the date_time of the first row in the block.

The date_time of the first block is the time of the GPS measurement preceding the accelerometer measurement. If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time in gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to (time in UTC, no time zone).
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_acc_start_limited ===

Each row represents the start of a block of acceleration measurements when a GPS fix is not available at that acceleration block's time.

If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

To know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

== Bird data ==
=== ''gps.''ee_individual_limited ===

Each row shows bird's details for all birds that your projects allow you to see.

This view replaces the old gps.uva_individual_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|ring_number||varchar||NO||Ring number.
|-
|species_latin_name||varchar||NO||Latin name of the species of the bird.
|-
|colour_ring||varchar||YES||String for ring colour.
|-
|sex||varchar||NO||Sex (M=male, F=female, X=unknown/unspecified)
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|start_date||timestamp||NO||Date when the bird was first tracked with an UvA-BiTS logger.
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant for the tracking project.
|-
|remarks||text||YES||Text for user remarks
|-
|individual_id||int8||NO||Bird unique id. Included just to enable joins.
|}

=== ''gps.''ee_species_limited ===

Each row represents a species that you can assign to your birds to include them in your project. No bird in the system can exist without being linked to one of the species from this list. If you are tagging a bird whose species does not exist in the list, please [http://www.uva-bits.nl/contact/ contact] us.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|abbr||varchar||NO||Unique abbreviation for this species.
|-
|english_name||varchar||YES||Name of the species in English.
|-
|latin_name||varchar||NO||Name of the species in Latin. It is always filled in.
|-
|species_id||int8||NO||Unique id. Added to allow joins.
|}

=== ''gps.''ee_nest_limited ===
This view shows nests you have access to. A nest is always for a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found, decimal degrees (WGS84 datum)
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found, decimal degrees (WGS84 datum)
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|nest_id||int8||NO||Unique id. Added to allow joins to nest_inhabitant_limited.
|}

=== ''gps.''ee_nest_inhabitant_limited ===
This view shows who lives on which nest, for birds you have access to.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string belonging to the nest that you can use to identify that nest. Because it is NOT enforced to be unique, DO NOT USE THIS FIELD FOR JOINING.
|-
|ring_number||varchar||NO||The ring number of a bird on a nest.
|-
|device_info_serial||int8||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the bird's trackers within that period are resolved so that they can be shown here. This is the device_info_serial of a tracker.
|-
|key_name||varchar||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the track sessions' projects within that period are resolved so that they can be shown here. This is the key_name of a project.
|-
|nest_id||int8||NO||Reference to the nest id. Added to allow joins.
|-
|individual_id||int8||NO||Reference to the individual id. Added to allow joins.
|-
|track_session_id||int8||NO||Reference to a resolved track session id. Added to allow joins.
|}
== Tracker and project data ==

=== ''gps.''ee_project_limited ===

Each row shows the details of each project, for the projects you are allowed to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Unique key name, with the form <species>_<place>
|-
|station_name||varchar||NO||Where the receiving station is
|-
|start_date||timestamp||NO||Project start date
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|project_id||int8||NO||Project unique id. Included just to enable joins.
|-
|parent_id||int8||YES||Reference to the parent project in the hierarchy.
|}

=== ''gps.''ee_tracker_limited ===

Each row shows tracker details for all trackers that your projects allow you to see.

This view replaces the old gps.uva_device_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (or surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (or surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (or sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (or heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or heave) axis
|-
|tracker_id||int8||NO||Tracker unique id. Included just to enable joins.
|}

=== ''gps.''ee_track_session_limited ===

A track session links a tracker and a bird for a given period of time. Therefore, it is '''mandatory for projects''' to supply this piece of '''metadata''' before they can see any tracking data related to a tracker they own.

In order to create a track session, the project must create first an entry in the system for the bird that they are attaching the tracker to, and this is the way for the UvA-BiTS community to make sense of the tracking data that we are receiving. This is why providing the metadata is so important.

By editing the start and end dates, administrators of the project that this track session belongs to, can delimit (and therefore, establish) when the coupling makes sense (or is to be understood as useful).

Whenever a tracker is attached to a bird, a track session must be started (start_date) to indicate that the tracking data that is received is already live data. If the tracker is separated from the bird, the tracking data ceases to make sense, so the track session's end date should be set to denote it has ended. New track sessions can be started if the tracker is re-attached to a bird (possibly the same as before, or maybe a different one) afterwards.

Each row shows track session's information. There exists a row for each track session that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the track session belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker this track session involves.
|-
|ring_number||varchar||NO||Ring number of the bird that this track session involves.
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|remarks||text||YES||Free text for comments
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged, decimal degrees (WGS84 datum)
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged, decimal degrees (WGS84 datum)
|-
|project_id||int8||NO||Reference to the project this track session belongs to. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker this track session involves. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual this track session involves. Included just to enable joins.
|-
|track_session_id||int8||NO||Unique identifier of the track session. Included just to enable joins.
|}

== Relation to projects ==

=== ''gps.''ee_user_limited ===

Each row shows a colleague in a project that you belong to, indicating her role and status there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Project name
|-
|email||varchar||NO||e-mail address
|-
|login_name||varchar||NO||Login name
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|role||varchar||NO||Role in the project
|-
|status||varchar||NO||Status of the membership
|-
|user_id||int8||NO||Unique identifier of the user. Included to enable joins.
|}

=== ''gps.''ee_tracker_ownership_limited ===

Each row that you see expresses that a tracker belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the tracker belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker. Included just to enable joins.
|}

=== ''gps.''ee_individual_ownership_limited ===

Each row that you see expresses that a bird belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the individual belongs to.
|-
|ring_number||varchar||NO||Ring number of the bird.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual. Included just to enable joins.
|}

== Engineering data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_comm_limited ===

Each row indicates status and result of communications with ground stations for each tracker that your project allows you to see.

Only for newer firmware (v.1.0.1.4 and higher).

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''ee_energy_limited ===

Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''ee_settings_limited ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

=== ''gps.''ee_trackingfile_limited ===

Each row shows the status of the last attempt to parse (process) the raw data of a file for a given tracker, for each tracker that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''ee_warning_limited ===

Since firmware version 1.0.2.7 this table was added for events that do not occur very often (C10, C20) or errors that occur during parsing.

For example all CRC failures can be selected with :

select * from gps.ee_warning_limited
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||File name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line contents in the file
|}

== Sharings ==

Projects can now share trackers with other projects during periods of time. This means that people in the receiving projects can access information that these shared trackers provide during those time periods.

The following views allow you to see this shared information:

* Tracking data
** ''gps.''ee_shared_tracking_speed_limited: tracking information (gps, speed), during the shared period, for trackers shared
** ''gps.''ee_shared_acc_start_limited: acceleration start information, during the shared period, for trackers shared
** ''gps.''ee_shared_acceleration_limited: acceleration information for trackers shared
* Metadata
** ''gps.''ee_shared_tracker_limited: information about each shared tracker
** ''gps.''ee_shared_individual_limited: information about birds with shared trackers
** ''gps.''ee_shared_track_session_limited: information about track sessions which involve shared trackers
* Relation to projects
** ''gps.''ee_shared_tracker_sharing_limited: view information for each sharing that your projects allow you to see

== Accelerometer calibration ==

The accelerometer measures the acceleration in 3D (20 times per second). Acceleration is caused by gravity (static component) and by a change in speed (dynamic component).
When the logger is static in horizontal position (solar cells up), gravity (green arrow) shows a heave (vertical acceleration, z) of +1 g (1 g = 9.81 m.s-2) while surge and heave are both 0. When the front of the logger is vertically pointing down (forward acceleration, x), surge is +1g and when it is rolled to the right (sidewards acceleration, y) sway is +1g. Due to gravity, the pitch of a logger is thus translated into the values of heave, surge and sway (or z, x, y).

However, the values are measured in mV instead of g. Before delivering the GPS-loggers, they have been calibrated at UvA. The acceleration (A) expressed in g is calculated from the measured value (M) and the calibration factors offset (''O'') and sensitivity (''S'').

Thus acceleration for heave (z), surge (x) and sway (y) is calculated as:
Az = (Mz-Oz)/Sz ; Ax = (Mx-Ox)/Sx; Ay = (My-Oy)/Sy ;

The values of ''O'' (in general ranging from 0-60, sometimes even 300) and ''S'' (ranging from 1300-1450) are available in the device table of the data base. If not available, use the factory defaults ''O'' = 0 and ''S'' = 1365.

{|
| [[File:accelero1.png|300px]]||[[File:accelero2.png|300px]]|
|}

In the figure above heave (z) is shown in green, surge(x) in red and sway (y) in blue, all values are expressed in g (on the y-axis).

== Data processing and filtering ==
The raw log files are processed before being entered into the database. Each firmware version has a dedicated parser for the task. In certain cases data that is found in the log files is not stored in the database. This can occur for two reasons:

1. The data did not pass the CRC8 checksum (cyclic redundancy check). This is a standard procedure designed to detect accidental changes to raw computer data.

2. If the data has an invalid signature. For example, if a certain data format (eg numeric) or number of values (eg 4 digits) is expected but not returned, this record is not stored in the database.

When a line is deleted, the log file name, line number and the type of error is stored in the table gps.uva_warning102.

== Information about GPS receivers ==
The first series of tags used the UBlox LEAS4S chip (~up to tag 541), newer tags use UBlox LEAS6S. The series developed in 2012 will use the Max6G receivers. From 2012 receiver versions will be stored in the database.
The UBX protocol is used for programming firmware, packaging and processing data.
For more information about the receivers themselves see the ublox documentation.
Ublox 4 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/ANTARIS_Protocol_Specification%28GPS.G3-X-03002%29.zip]
Ublox 6 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/u-blox6_ReceiverDescriptionProtocolSpec_%28GPS.G6-SW-10018%29.pdf]

Reference systems for GPS coordinates and altitudes

2021-01-22T09:33:51Z

Stacy shinneman:

All of our UvA-BiTS GPS trackers use the Ublox Max-6G GPS receiver to collect vertical and horizontal position data.

The GPS unit measures the distance from the satellites to the device and then it calculates the distance relative to the WGS84 ellipsoid (World Geodetic System WGS84). GPS coordinates in the database are calculated relative to the WGS84 datum.

The altitude measurements provided in the database are altitude above a geoid that approximates global mean sea level rather than altitudes relative to the WGS84 ellipsoid. (A good explanation about the difference between ellipsoids and geoid is available here: https://www.esri.com/news/arcuser/0703/geoid1of3.html)

Be aware that global mean sea level is not the same as the earth’s surface (i.e. land surface elevation). However, the SRTM-DEM data included in the database as land surface elevation data are also calculated relative to global mean sea level and can therefore be combined directly with UvA-BiTS movement data to calculate the bird’s altitude above land surface elevation. Different GPS tracker manufacturers may provide altitude measurements using different methods and/or reference systems, so use caution when combining GPS data collected with multiple tracking systems.

DB Tables 2015

2021-01-22T09:33:08Z

Stacy shinneman: /* gps.uva_tracking_data101 */

'''Note:''' What follows is a list of tables and explanations intended for the database maintainers or otherwise people actively involved in the development of the e-Ecology system itself. Most e-Ecology users should focus on the documentation on the [[UvA-BiTS_Tracking_Data|UvA-BiTS Tracking Data page]]. This page is intended as a reference only.

The database is split into two useful schemas: ''admin'' and ''gps''. The ''gps'' schema is meant for data that should be readily available for users, and that they may edit via the services (software tools) offered to that effect. The ''admin'' schema is meant to host data only accessible for e-Ecology administrators.

== ''Source'' data ==

=== ''admin.''ee_project_limited ===
Each row represents a project.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|key_name||varchar||NO||Unique key name, of the form species_place
|-
|lft||int8||NO||"Modified preorder tree traversal" field. Ignore it as a user.
|-
|rgt||int8||NO||"Modified preorder tree traversal" field. Ignore it as a user.
|-
|start_date||timestamp||NO||Project start date
|-
|station_name||varchar||NO||Where the receiving station is
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||NO||Reference to the user who last edited the project
|-
|parent||int8||YES||Reference to the parent project. Only the root project has no parent.
|}

=== ''admin.''ee_user ===

Each row represents a user.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
||id||int8||NO||Unique identifier. Primary key
|-
||address||varchar||YES||String for postal address
|-
|email||varchar||NO||e-mail address
|-
|encrypted_password||varchar||NO||Encrypted password
|-
|login_name||varchar||NO||Login name
|-
|mobile_phone||varchar||YES||String for mobile phone number
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||YES||Id of the user who last edited this user's data.
|}

=== ''gps.'' ee_tracker ===

Each row represents a tracker.

This table replaces the old gps.uva_device.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (heave) axis
|}

=== ''gps.''ee_individual ===

Each row represents a bird.

This table replaces gps.uva_individual from the old database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|colour_ring||varchar||YES||String for ring colour
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|remarks||text||YES||Text for user remarks
|-
|ring_number||varchar||NO||Ring number
|-
|sex||varchar||NO||Sex
|-
|start_date||timestamp||NO||Date before which the bird is not relevant
|-
|version||int4||YES||Optimistic locking field. Ignore it as user.
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|species||int8||NO||Reference to the species
|}

=== ''gps.''ee_track_session ===

Each row represents a track session.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|remarks||text||YES||Free text for comments
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|device||int8||NO||Reference to the tracker
|-
|individual||int8||NO||Reference to the individual
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this track session belongs to
|}

== Tracking data ==

=== ''gps.''uva_tracking_data101 ===

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker. See [[Reference systems for GPS coordinates and altitudes]] for further details.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix (WGS1984 datum)
|-
|longitude||float8||YES||Longitude of the GPS fix (WGS1984 datum)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS1984 datum)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2)
|-
|speed_2d||float8||YES|| Speed on 2D
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate
|}

=== ''gps.''uva_acc_start102 ===

Each row represents the start of a block of accelerations when a GPS fix is not available.

The logger tries to mark the start of blocks from the GPS fix that coincides with the begin of the acceleration block. If there's no GPS fix during that block, then it extrapolates the date and time and makes a special record which is interpreted by our accelerometer processing software and inserted into this specific table.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on table gps.ee_tracking_data101 for the given tracker, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

This table structure has not changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

=== ''gps.''uva_acceleration101 ===

Each row represents an acceleration measurement as it is interpreted from the data that comes via the accelerometer. Acceleration is looked at in batches for a period of time, so the 3 components (X,Y,Z). The blocks are delimited by a starting date and time, and then each observation is numbered consecutively.

The logger tries to mark the start of blocks from the GPS fix that coincides with the begin of the acceleration block. If there's no GPS fix during that block, then it extrapolates the date and time and makes a special record which is interpreted by our accelerometer processing software and inserted into a specific table called gps.ee_acc_start102. Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on table gps.ee_tracking_data101 for the given tracker, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to.
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_nest ===
This table holds nests' information. A nest is always for (at least) a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique id. Primary key.
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found (WGS1984 datum)
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found (WGS1984 datum)
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|operated_date||timestamp||NO||When was this row last edited.
|-
|operator_user||int8||NO||Reference to a user id. Last one who edited the row.
|}

=== ''gps.''ee_nest_inhabitant ===
This table is the materialisation of the many-to-many relation between nests and birds. The primary key of the table is the pair of both columns.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|nest_id||int8||NO||Reference to the nest id.
|-
|individual_id||int8||NO||Reference to the individual id.
|}

== Relation to projects ==

=== ''admin.''ee_membership ===

Each row represents the fact that a user has a given role in a given project at the moment.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|comments||text||YES||Free text for comments
|-
|label||text||YES||Labels that Willem wants to have for people in projects
|-
|role||varchar||NO||Role that the user holds during this period
|-
|status||varchar||NO||Status that the row holds during that period. Look at the attached file to know the allowed transitions.
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|member_user||int8||NO||Reference to the user that this membership applies to
|-
|operated_date||timestamp||NO||Date that the row was last modified
|-
|operator_user||int8||NO||Reference to the user who last edited this row
|-
|project||int8||NO||Reference to the project that this membership applies to
|}

=== ''admin.''ee_tracker_ownership ===

Each row represents the fact that a tracker belongs to a project within a specified time period.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the tracker ceases to belong to the project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the project begins to own the tracker
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|device||int8||NO||Reference to the tracker
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this tracker belongs to
|}

=== ''admin.''ee_individual_ownership ===

Each row represents the fact that an individual belongs to a project within a specified time period.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the individual ceases to belong to the project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the project begins to own the individual
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|individual||int8||NO||Reference to the individual
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this individual belongs to
|}

=== ''gps.''ee_tracker_sharing ===

Each row represents the fact that a tracker's information within a specified period, can be accessed by users belonging to another project.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the tracker information ceases to be available to the receiving project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the tracker information becomes available to the receiving project
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project that receives access to the tracker information
|-
|tracker_ownership||int8||NO||Reference to the tracker ownership by virtue of which, the tracker information is shared
|}

== Engineering data ==

=== ''gps.''uva_trackingfile_parsing ===

Each row on this table represents the status of the last attempt to parse (process) the raw data file that came from a tracker.

It is heavily used by the dashboard to determine how a given file is represented there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''uva_trackingfile_messages ===

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||serial||NO||Unique identifier of the row
|-
|tracking_bookkeeping_id||int4||YES||Reference to the gps.uva_trackingfile_parsing row for this file
|-
|message_type||int4||YES||Code for the type of information being told by the message.
|-
|message||text||YES||Textual message from the parsing process telling what it found.
|-
|message_code||int4||YES||
|-
|line_number||int4||YES||Line in the file where the message is reported
|}

=== ''gps.''uva_warning102 ===

Since firmware version 1.0.2.7 this table was added as catchall for events that do not occur very often (C10, C20) or errors that occur during parsing.
For example all CRC failures can be selected with :

select * from gps.uva_warning102
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||log file name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line in the file
|}

=== ''gps.''uva_comm101 ===

Communication table indicates status and result of communications with ground stations. Only for newer firmware (v.1.0.1.4 and higher).

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''uva_energy101 ===

Only for newer firmware.

This table structure has not changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''uva_settings101 ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

This table structure has ''not'' changed from the old database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

DB Tables 2015

2021-01-22T09:32:00Z

Stacy shinneman: /* gps.ee_nest */

'''Note:''' What follows is a list of tables and explanations intended for the database maintainers or otherwise people actively involved in the development of the e-Ecology system itself. Most e-Ecology users should focus on the documentation on the [[UvA-BiTS_Tracking_Data|UvA-BiTS Tracking Data page]]. This page is intended as a reference only.

The database is split into two useful schemas: ''admin'' and ''gps''. The ''gps'' schema is meant for data that should be readily available for users, and that they may edit via the services (software tools) offered to that effect. The ''admin'' schema is meant to host data only accessible for e-Ecology administrators.

== ''Source'' data ==

=== ''admin.''ee_project_limited ===
Each row represents a project.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|key_name||varchar||NO||Unique key name, of the form species_place
|-
|lft||int8||NO||"Modified preorder tree traversal" field. Ignore it as a user.
|-
|rgt||int8||NO||"Modified preorder tree traversal" field. Ignore it as a user.
|-
|start_date||timestamp||NO||Project start date
|-
|station_name||varchar||NO||Where the receiving station is
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||NO||Reference to the user who last edited the project
|-
|parent||int8||YES||Reference to the parent project. Only the root project has no parent.
|}

=== ''admin.''ee_user ===

Each row represents a user.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
||id||int8||NO||Unique identifier. Primary key
|-
||address||varchar||YES||String for postal address
|-
|email||varchar||NO||e-mail address
|-
|encrypted_password||varchar||NO||Encrypted password
|-
|login_name||varchar||NO||Login name
|-
|mobile_phone||varchar||YES||String for mobile phone number
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||YES||Id of the user who last edited this user's data.
|}

=== ''gps.'' ee_tracker ===

Each row represents a tracker.

This table replaces the old gps.uva_device.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (heave) axis
|}

=== ''gps.''ee_individual ===

Each row represents a bird.

This table replaces gps.uva_individual from the old database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|colour_ring||varchar||YES||String for ring colour
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|remarks||text||YES||Text for user remarks
|-
|ring_number||varchar||NO||Ring number
|-
|sex||varchar||NO||Sex
|-
|start_date||timestamp||NO||Date before which the bird is not relevant
|-
|version||int4||YES||Optimistic locking field. Ignore it as user.
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|species||int8||NO||Reference to the species
|}

=== ''gps.''ee_track_session ===

Each row represents a track session.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|remarks||text||YES||Free text for comments
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|device||int8||NO||Reference to the tracker
|-
|individual||int8||NO||Reference to the individual
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this track session belongs to
|}

== Tracking data ==

=== ''gps.''uva_tracking_data101 ===

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix (WGS1984 datum)
|-
|longitude||float8||YES||Longitude of the GPS fix (WGS1984 datum)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS1984 datum)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2)
|-
|speed_2d||float8||YES|| Speed on 2D
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate
|}

=== ''gps.''uva_acc_start102 ===

Each row represents the start of a block of accelerations when a GPS fix is not available.

The logger tries to mark the start of blocks from the GPS fix that coincides with the begin of the acceleration block. If there's no GPS fix during that block, then it extrapolates the date and time and makes a special record which is interpreted by our accelerometer processing software and inserted into this specific table.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on table gps.ee_tracking_data101 for the given tracker, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

This table structure has not changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

=== ''gps.''uva_acceleration101 ===

Each row represents an acceleration measurement as it is interpreted from the data that comes via the accelerometer. Acceleration is looked at in batches for a period of time, so the 3 components (X,Y,Z). The blocks are delimited by a starting date and time, and then each observation is numbered consecutively.

The logger tries to mark the start of blocks from the GPS fix that coincides with the begin of the acceleration block. If there's no GPS fix during that block, then it extrapolates the date and time and makes a special record which is interpreted by our accelerometer processing software and inserted into a specific table called gps.ee_acc_start102. Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on table gps.ee_tracking_data101 for the given tracker, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to.
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_nest ===
This table holds nests' information. A nest is always for (at least) a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique id. Primary key.
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found (WGS1984 datum)
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found (WGS1984 datum)
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|operated_date||timestamp||NO||When was this row last edited.
|-
|operator_user||int8||NO||Reference to a user id. Last one who edited the row.
|}

=== ''gps.''ee_nest_inhabitant ===
This table is the materialisation of the many-to-many relation between nests and birds. The primary key of the table is the pair of both columns.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|nest_id||int8||NO||Reference to the nest id.
|-
|individual_id||int8||NO||Reference to the individual id.
|}

== Relation to projects ==

=== ''admin.''ee_membership ===

Each row represents the fact that a user has a given role in a given project at the moment.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|comments||text||YES||Free text for comments
|-
|label||text||YES||Labels that Willem wants to have for people in projects
|-
|role||varchar||NO||Role that the user holds during this period
|-
|status||varchar||NO||Status that the row holds during that period. Look at the attached file to know the allowed transitions.
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|member_user||int8||NO||Reference to the user that this membership applies to
|-
|operated_date||timestamp||NO||Date that the row was last modified
|-
|operator_user||int8||NO||Reference to the user who last edited this row
|-
|project||int8||NO||Reference to the project that this membership applies to
|}

=== ''admin.''ee_tracker_ownership ===

Each row represents the fact that a tracker belongs to a project within a specified time period.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the tracker ceases to belong to the project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the project begins to own the tracker
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|device||int8||NO||Reference to the tracker
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this tracker belongs to
|}

=== ''admin.''ee_individual_ownership ===

Each row represents the fact that an individual belongs to a project within a specified time period.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the individual ceases to belong to the project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the project begins to own the individual
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|individual||int8||NO||Reference to the individual
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this individual belongs to
|}

=== ''gps.''ee_tracker_sharing ===

Each row represents the fact that a tracker's information within a specified period, can be accessed by users belonging to another project.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the tracker information ceases to be available to the receiving project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the tracker information becomes available to the receiving project
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project that receives access to the tracker information
|-
|tracker_ownership||int8||NO||Reference to the tracker ownership by virtue of which, the tracker information is shared
|}

== Engineering data ==

=== ''gps.''uva_trackingfile_parsing ===

Each row on this table represents the status of the last attempt to parse (process) the raw data file that came from a tracker.

It is heavily used by the dashboard to determine how a given file is represented there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''uva_trackingfile_messages ===

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||serial||NO||Unique identifier of the row
|-
|tracking_bookkeeping_id||int4||YES||Reference to the gps.uva_trackingfile_parsing row for this file
|-
|message_type||int4||YES||Code for the type of information being told by the message.
|-
|message||text||YES||Textual message from the parsing process telling what it found.
|-
|message_code||int4||YES||
|-
|line_number||int4||YES||Line in the file where the message is reported
|}

=== ''gps.''uva_warning102 ===

Since firmware version 1.0.2.7 this table was added as catchall for events that do not occur very often (C10, C20) or errors that occur during parsing.
For example all CRC failures can be selected with :

select * from gps.uva_warning102
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||log file name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line in the file
|}

=== ''gps.''uva_comm101 ===

Communication table indicates status and result of communications with ground stations. Only for newer firmware (v.1.0.1.4 and higher).

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''uva_energy101 ===

Only for newer firmware.

This table structure has not changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''uva_settings101 ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

This table structure has ''not'' changed from the old database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

DB Tables 2015

2021-01-22T09:31:41Z

Stacy shinneman: /* gps.uva_tracking_data101 */

'''Note:''' What follows is a list of tables and explanations intended for the database maintainers or otherwise people actively involved in the development of the e-Ecology system itself. Most e-Ecology users should focus on the documentation on the [[UvA-BiTS_Tracking_Data|UvA-BiTS Tracking Data page]]. This page is intended as a reference only.

The database is split into two useful schemas: ''admin'' and ''gps''. The ''gps'' schema is meant for data that should be readily available for users, and that they may edit via the services (software tools) offered to that effect. The ''admin'' schema is meant to host data only accessible for e-Ecology administrators.

== ''Source'' data ==

=== ''admin.''ee_project_limited ===
Each row represents a project.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|key_name||varchar||NO||Unique key name, of the form species_place
|-
|lft||int8||NO||"Modified preorder tree traversal" field. Ignore it as a user.
|-
|rgt||int8||NO||"Modified preorder tree traversal" field. Ignore it as a user.
|-
|start_date||timestamp||NO||Project start date
|-
|station_name||varchar||NO||Where the receiving station is
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||NO||Reference to the user who last edited the project
|-
|parent||int8||YES||Reference to the parent project. Only the root project has no parent.
|}

=== ''admin.''ee_user ===

Each row represents a user.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
||id||int8||NO||Unique identifier. Primary key
|-
||address||varchar||YES||String for postal address
|-
|email||varchar||NO||e-mail address
|-
|encrypted_password||varchar||NO||Encrypted password
|-
|login_name||varchar||NO||Login name
|-
|mobile_phone||varchar||YES||String for mobile phone number
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||YES||Id of the user who last edited this user's data.
|}

=== ''gps.'' ee_tracker ===

Each row represents a tracker.

This table replaces the old gps.uva_device.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (heave) axis
|}

=== ''gps.''ee_individual ===

Each row represents a bird.

This table replaces gps.uva_individual from the old database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|colour_ring||varchar||YES||String for ring colour
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|remarks||text||YES||Text for user remarks
|-
|ring_number||varchar||NO||Ring number
|-
|sex||varchar||NO||Sex
|-
|start_date||timestamp||NO||Date before which the bird is not relevant
|-
|version||int4||YES||Optimistic locking field. Ignore it as user.
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|species||int8||NO||Reference to the species
|}

=== ''gps.''ee_track_session ===

Each row represents a track session.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|remarks||text||YES||Free text for comments
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|device||int8||NO||Reference to the tracker
|-
|individual||int8||NO||Reference to the individual
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this track session belongs to
|}

== Tracking data ==

=== ''gps.''uva_tracking_data101 ===

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix (WGS1984 datum)
|-
|longitude||float8||YES||Longitude of the GPS fix (WGS1984 datum)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS1984 datum)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2)
|-
|speed_2d||float8||YES|| Speed on 2D
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate
|}

=== ''gps.''uva_acc_start102 ===

Each row represents the start of a block of accelerations when a GPS fix is not available.

The logger tries to mark the start of blocks from the GPS fix that coincides with the begin of the acceleration block. If there's no GPS fix during that block, then it extrapolates the date and time and makes a special record which is interpreted by our accelerometer processing software and inserted into this specific table.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on table gps.ee_tracking_data101 for the given tracker, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

This table structure has not changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

=== ''gps.''uva_acceleration101 ===

Each row represents an acceleration measurement as it is interpreted from the data that comes via the accelerometer. Acceleration is looked at in batches for a period of time, so the 3 components (X,Y,Z). The blocks are delimited by a starting date and time, and then each observation is numbered consecutively.

The logger tries to mark the start of blocks from the GPS fix that coincides with the begin of the acceleration block. If there's no GPS fix during that block, then it extrapolates the date and time and makes a special record which is interpreted by our accelerometer processing software and inserted into a specific table called gps.ee_acc_start102. Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on table gps.ee_tracking_data101 for the given tracker, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to.
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_nest ===
This table holds nests' information. A nest is always for (at least) a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique id. Primary key.
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found.
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found.
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|operated_date||timestamp||NO||When was this row last edited.
|-
|operator_user||int8||NO||Reference to a user id. Last one who edited the row.
|}

=== ''gps.''ee_nest_inhabitant ===
This table is the materialisation of the many-to-many relation between nests and birds. The primary key of the table is the pair of both columns.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|nest_id||int8||NO||Reference to the nest id.
|-
|individual_id||int8||NO||Reference to the individual id.
|}

== Relation to projects ==

=== ''admin.''ee_membership ===

Each row represents the fact that a user has a given role in a given project at the moment.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|comments||text||YES||Free text for comments
|-
|label||text||YES||Labels that Willem wants to have for people in projects
|-
|role||varchar||NO||Role that the user holds during this period
|-
|status||varchar||NO||Status that the row holds during that period. Look at the attached file to know the allowed transitions.
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|member_user||int8||NO||Reference to the user that this membership applies to
|-
|operated_date||timestamp||NO||Date that the row was last modified
|-
|operator_user||int8||NO||Reference to the user who last edited this row
|-
|project||int8||NO||Reference to the project that this membership applies to
|}

=== ''admin.''ee_tracker_ownership ===

Each row represents the fact that a tracker belongs to a project within a specified time period.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the tracker ceases to belong to the project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the project begins to own the tracker
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|device||int8||NO||Reference to the tracker
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this tracker belongs to
|}

=== ''admin.''ee_individual_ownership ===

Each row represents the fact that an individual belongs to a project within a specified time period.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the individual ceases to belong to the project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the project begins to own the individual
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|individual||int8||NO||Reference to the individual
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project this individual belongs to
|}

=== ''gps.''ee_tracker_sharing ===

Each row represents the fact that a tracker's information within a specified period, can be accessed by users belonging to another project.
{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier. Primary key.
|-
|end_date||timestamp||NO||Date at which the tracker information ceases to be available to the receiving project
|-
|operated_date||timestamp||NO||Recording of the last time the row was edited
|-
|start_date||timestamp||NO||Date at which the tracker information becomes available to the receiving project
|-
|version||int4||YES||Optimistic locking field. Ignore it as a user.
|-
|operator_user||int8||NO||Reference to the user who last edited the row
|-
|project||int8||NO||Reference to the project that receives access to the tracker information
|-
|tracker_ownership||int8||NO||Reference to the tracker ownership by virtue of which, the tracker information is shared
|}

== Engineering data ==

=== ''gps.''uva_trackingfile_parsing ===

Each row on this table represents the status of the last attempt to parse (process) the raw data file that came from a tracker.

It is heavily used by the dashboard to determine how a given file is represented there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''uva_trackingfile_messages ===

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||serial||NO||Unique identifier of the row
|-
|tracking_bookkeeping_id||int4||YES||Reference to the gps.uva_trackingfile_parsing row for this file
|-
|message_type||int4||YES||Code for the type of information being told by the message.
|-
|message||text||YES||Textual message from the parsing process telling what it found.
|-
|message_code||int4||YES||
|-
|line_number||int4||YES||Line in the file where the message is reported
|}

=== ''gps.''uva_warning102 ===

Since firmware version 1.0.2.7 this table was added as catchall for events that do not occur very often (C10, C20) or errors that occur during parsing.
For example all CRC failures can be selected with :

select * from gps.uva_warning102
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||log file name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line in the file
|}

=== ''gps.''uva_comm101 ===

Communication table indicates status and result of communications with ground stations. Only for newer firmware (v.1.0.1.4 and higher).

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''uva_energy101 ===

Only for newer firmware.

This table structure has not changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''uva_settings101 ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

This table structure has ''not'' changed from the old database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

UvA-BiTS Tracking Data

2021-01-22T09:06:49Z

Stacy shinneman: /* gps.ee_tracking_speed_limited */

== UvA-BiTS Tracking Data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_tracking_speed_limited ===

Shows the tracking data that your projects allow you to see.

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker, plus it adds speeds. See [[Reference systems for GPS coordinates and altitudes]] for further details.

It replaces old views gps.uva_tracking_limited, gps.uva_tracking_speed_3d_limited and gps.uva_tracking_speed. This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|longitude||float8||YES||Longitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS84 datum)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|vnorth||float8||YES||- vx * sin(lat) * cos(long) - vy * sin(lat) * sin (long) + vz * cos(lat), m/s
|-
|veast||float8||YES||- vx * sin(lon) + vy * cos(lon), m/s
|-
|vdown||float8||YES||- vx * cos(lat) * cos(lon) - vy * cos(lat) * sin(lon) - vz * sin(lat), m/s
|-
|speed||float8||YES||- vx * sin(lon) + vy * cos(lon)
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2), m/s
|-
|speed_2d||float8||YES|| Speed on 2D, m/s
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate based on Digital Elevation Model data used in the e-ecology database (for further information see: [[Elevation]]
|}

=== ''gps.''ee_acceleration_limited ===

Acceleration is measured in blocks of 1-200 measurements (records). The date_time mentioned in this block is the date and time of the first acceleration measurement. Sequential measurements within a block are numbered with an index 1... (max 200). Every row represents one measurement in 3 directions (X,Y,Z). The order is important within each block, so you should always sort the acceleration measurements by block and index. If you need the exact time for each row, you can calculate it from the index, the sample frequency (which is almost always 20 Hz) and the date_time of the first row in the block.

The date_time of the first block is the time of the GPS measurement preceding the accelerometer measurement. If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time in gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to (time in UTC, no time zone).
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_acc_start_limited ===

Each row represents the start of a block of acceleration measurements when a GPS fix is not available at that acceleration block's time.

If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

To know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

== Bird data ==
=== ''gps.''ee_individual_limited ===

Each row shows bird's details for all birds that your projects allow you to see.

This view replaces the old gps.uva_individual_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|ring_number||varchar||NO||Ring number.
|-
|species_latin_name||varchar||NO||Latin name of the species of the bird.
|-
|colour_ring||varchar||YES||String for ring colour.
|-
|sex||varchar||NO||Sex (M=male, F=female, X=unknown/unspecified)
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|start_date||timestamp||NO||Date when the bird was first tracked with an UvA-BiTS logger.
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant for the tracking project.
|-
|remarks||text||YES||Text for user remarks
|-
|individual_id||int8||NO||Bird unique id. Included just to enable joins.
|}

=== ''gps.''ee_species_limited ===

Each row represents a species that you can assign to your birds to include them in your project. No bird in the system can exist without being linked to one of the species from this list. If you are tagging a bird whose species does not exist in the list, please [http://www.uva-bits.nl/contact/ contact] us.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|abbr||varchar||NO||Unique abbreviation for this species.
|-
|english_name||varchar||YES||Name of the species in English.
|-
|latin_name||varchar||NO||Name of the species in Latin. It is always filled in.
|-
|species_id||int8||NO||Unique id. Added to allow joins.
|}

=== ''gps.''ee_nest_limited ===
This view shows nests you have access to. A nest is always for a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found, decimal degrees (WGS84 datum)
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found, decimal degrees (WGS84 datum)
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|nest_id||int8||NO||Unique id. Added to allow joins to nest_inhabitant_limited.
|}

=== ''gps.''ee_nest_inhabitant_limited ===
This view shows who lives on which nest, for birds you have access to.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string belonging to the nest that you can use to identify that nest. Because it is NOT enforced to be unique, DO NOT USE THIS FIELD FOR JOINING.
|-
|ring_number||varchar||NO||The ring number of a bird on a nest.
|-
|device_info_serial||int8||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the bird's trackers within that period are resolved so that they can be shown here. This is the device_info_serial of a tracker.
|-
|key_name||varchar||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the track sessions' projects within that period are resolved so that they can be shown here. This is the key_name of a project.
|-
|nest_id||int8||NO||Reference to the nest id. Added to allow joins.
|-
|individual_id||int8||NO||Reference to the individual id. Added to allow joins.
|-
|track_session_id||int8||NO||Reference to a resolved track session id. Added to allow joins.
|}
== Tracker and project data ==

=== ''gps.''ee_project_limited ===

Each row shows the details of each project, for the projects you are allowed to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Unique key name, with the form <species>_<place>
|-
|station_name||varchar||NO||Where the receiving station is
|-
|start_date||timestamp||NO||Project start date
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|project_id||int8||NO||Project unique id. Included just to enable joins.
|-
|parent_id||int8||YES||Reference to the parent project in the hierarchy.
|}

=== ''gps.''ee_tracker_limited ===

Each row shows tracker details for all trackers that your projects allow you to see.

This view replaces the old gps.uva_device_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (or surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (or surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (or sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (or heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or heave) axis
|-
|tracker_id||int8||NO||Tracker unique id. Included just to enable joins.
|}

=== ''gps.''ee_track_session_limited ===

A track session links a tracker and a bird for a given period of time. Therefore, it is '''mandatory for projects''' to supply this piece of '''metadata''' before they can see any tracking data related to a tracker they own.

In order to create a track session, the project must create first an entry in the system for the bird that they are attaching the tracker to, and this is the way for the UvA-BiTS community to make sense of the tracking data that we are receiving. This is why providing the metadata is so important.

By editing the start and end dates, administrators of the project that this track session belongs to, can delimit (and therefore, establish) when the coupling makes sense (or is to be understood as useful).

Whenever a tracker is attached to a bird, a track session must be started (start_date) to indicate that the tracking data that is received is already live data. If the tracker is separated from the bird, the tracking data ceases to make sense, so the track session's end date should be set to denote it has ended. New track sessions can be started if the tracker is re-attached to a bird (possibly the same as before, or maybe a different one) afterwards.

Each row shows track session's information. There exists a row for each track session that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the track session belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker this track session involves.
|-
|ring_number||varchar||NO||Ring number of the bird that this track session involves.
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|remarks||text||YES||Free text for comments
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged, decimal degrees (WGS84 datum)
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged, decimal degrees (WGS84 datum)
|-
|project_id||int8||NO||Reference to the project this track session belongs to. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker this track session involves. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual this track session involves. Included just to enable joins.
|-
|track_session_id||int8||NO||Unique identifier of the track session. Included just to enable joins.
|}

== Relation to projects ==

=== ''gps.''ee_user_limited ===

Each row shows a colleague in a project that you belong to, indicating her role and status there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Project name
|-
|email||varchar||NO||e-mail address
|-
|login_name||varchar||NO||Login name
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|role||varchar||NO||Role in the project
|-
|status||varchar||NO||Status of the membership
|-
|user_id||int8||NO||Unique identifier of the user. Included to enable joins.
|}

=== ''gps.''ee_tracker_ownership_limited ===

Each row that you see expresses that a tracker belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the tracker belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker. Included just to enable joins.
|}

=== ''gps.''ee_individual_ownership_limited ===

Each row that you see expresses that a bird belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the individual belongs to.
|-
|ring_number||varchar||NO||Ring number of the bird.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual. Included just to enable joins.
|}

== Engineering data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_comm_limited ===

Each row indicates status and result of communications with ground stations for each tracker that your project allows you to see.

Only for newer firmware (v.1.0.1.4 and higher).

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''ee_energy_limited ===

Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''ee_settings_limited ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

=== ''gps.''ee_trackingfile_limited ===

Each row shows the status of the last attempt to parse (process) the raw data of a file for a given tracker, for each tracker that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''ee_warning_limited ===

Since firmware version 1.0.2.7 this table was added for events that do not occur very often (C10, C20) or errors that occur during parsing.

For example all CRC failures can be selected with :

select * from gps.ee_warning_limited
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||File name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line contents in the file
|}

== Sharings ==

Projects can now share trackers with other projects during periods of time. This means that people in the receiving projects can access information that these shared trackers provide during those time periods.

The following views allow you to see this shared information:

* Tracking data
** ''gps.''ee_shared_tracking_speed_limited: tracking information (gps, speed), during the shared period, for trackers shared
** ''gps.''ee_shared_acc_start_limited: acceleration start information, during the shared period, for trackers shared
** ''gps.''ee_shared_acceleration_limited: acceleration information for trackers shared
* Metadata
** ''gps.''ee_shared_tracker_limited: information about each shared tracker
** ''gps.''ee_shared_individual_limited: information about birds with shared trackers
** ''gps.''ee_shared_track_session_limited: information about track sessions which involve shared trackers
* Relation to projects
** ''gps.''ee_shared_tracker_sharing_limited: view information for each sharing that your projects allow you to see

== Accelerometer calibration ==

The accelerometer measures the acceleration in 3D (20 times per second). Acceleration is caused by gravity (static component) and by a change in speed (dynamic component).
When the logger is static in horizontal position (solar cells up), gravity (green arrow) shows a heave (vertical acceleration, z) of +1 g (1 g = 9.81 m.s-2) while surge and heave are both 0. When the front of the logger is vertically pointing down (forward acceleration, x), surge is +1g and when it is rolled to the right (sidewards acceleration, y) sway is +1g. Due to gravity, the pitch of a logger is thus translated into the values of heave, surge and sway (or z, x, y).

However, the values are measured in mV instead of g. Before delivering the GPS-loggers, they have been calibrated at UvA. The acceleration (A) expressed in g is calculated from the measured value (M) and the calibration factors offset (''O'') and sensitivity (''S'').

Thus acceleration for heave (z), surge (x) and sway (y) is calculated as:
Az = (Mz-Oz)/Sz ; Ax = (Mx-Ox)/Sx; Ay = (My-Oy)/Sy ;

The values of ''O'' (in general ranging from 0-60, sometimes even 300) and ''S'' (ranging from 1300-1450) are available in the device table of the data base. If not available, use the factory defaults ''O'' = 0 and ''S'' = 1365.

{|
| [[File:accelero1.png|300px]]||[[File:accelero2.png|300px]]|
|}

In the figure above heave (z) is shown in green, surge(x) in red and sway (y) in blue, all values are expressed in g (on the y-axis).

== Data processing and filtering ==
The raw log files are processed before being entered into the database. Each firmware version has a dedicated parser for the task. In certain cases data that is found in the log files is not stored in the database. This can occur for two reasons:

1. The data did not pass the CRC8 checksum (cyclic redundancy check). This is a standard procedure designed to detect accidental changes to raw computer data.

2. If the data has an invalid signature. For example, if a certain data format (eg numeric) or number of values (eg 4 digits) is expected but not returned, this record is not stored in the database.

When a line is deleted, the log file name, line number and the type of error is stored in the table gps.uva_warning102

== Information about GPS receivers ==
The first series of tags used the UBlox LEAS4S chip (~up to tag 541), newer tags use UBlox LEAS6S. The series developed in 2012 will use the Max6G receivers. From 2012 receiver versions will be stored in the database.
The UBX protocol is used for programming firmware, packaging and processing data.
For more information about the receivers themselves see the ublox documentation.
Ublox 4 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/ANTARIS_Protocol_Specification%28GPS.G3-X-03002%29.zip]
Ublox 6 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/u-blox6_ReceiverDescriptionProtocolSpec_%28GPS.G6-SW-10018%29.pdf]

Reference systems for GPS coordinates and altitudes

2021-01-21T17:01:35Z

Stacy shinneman: /* Reference systems for GPS coordinates and altitudes */

All of our UvA-BiTS GPS trackers use the Ublox Max-6G GPS receiver to collect vertical and horizontal position data.

The GPS unit measures the distance from the satellites to the device and then it calculates the distance relative to the WGS84 ellipsoid (World Geodetic System WGS84). GPS coordinates in the database are calculated relative to the WGS84 datum.

The altitude measurements provided in the database are altitude above a geoid that approximates global mean sea level rather than altitudes relative to the WGS84 ellipsoid. (A good explanation about the difference between ellipsoids and geoid is available here: https://www.esri.com/news/arcuser/0703/geoid1of3.html)

Be aware that global mean sea level is not the same as the earth’s surface (=land surface elevation). However, the SRTM-DEM data included in the database as land surface elevation data are also calculated relative to global mean sea level and can therefore be combined directly with UvA-BiTS movement data to calculate the bird’s altitude above land surface elevation. Different GPS tracker manufacturers may provide altitude measurements using different methods and/or reference systems, so use caution when combining GPS data collected with multiple tracking systems.

Reference systems for GPS coordinates and altitudes

2021-01-21T17:00:47Z

Stacy shinneman: Created page with "== Reference systems for GPS coordinates and altitudes == All of our UvA-BiTS GPS trackers use the Ublox Max-6G GPS receiver to collect vertical and horizontal position data...."

== Reference systems for GPS coordinates and altitudes ==

All of our UvA-BiTS GPS trackers use the Ublox Max-6G GPS receiver to collect vertical and horizontal position data.

The GPS unit measures the distance from the satellites to the device and then it calculates the distance relative to the WGS84 ellipsoid (World Geodetic System WGS84). GPS coordinates in the database are calculated relative to the WGS84 datum.

The altitude measurements provided in the database are altitude above a geoid that approximates global mean sea level rather than altitudes relative to the WGS84 ellipsoid. (A good explanation about the difference between ellipsoids and geoid is available here: https://www.esri.com/news/arcuser/0703/geoid1of3.html)

Be aware that global mean sea level is not the same as the earth’s surface (=land surface elevation). However, the SRTM-DEM data included in the database as land surface elevation data are also calculated relative to global mean sea level and can therefore be combined directly with UvA-BiTS movement data to calculate the bird’s altitude above land surface elevation. Different GPS tracker manufacturers may provide altitude measurements using different methods and/or reference systems, so use caution when combining GPS data collected with multiple tracking systems.

E-Ecology Documentation and Guidance

2021-01-21T16:59:26Z

Stacy shinneman: /* UvA-BiTS Data */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[How to access the e-Ecology database]]
::* [[Reference systems for GPS coordinates and altitudes]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::'''''This service is currently not available.''''' Access meterological and radar data and overlay with GPS bird tracking data.
:: [[BirdSim]]: 
:::'''''This service is currently not available.''''' More dynamic version of BirdView that allows the user to scroll through weather data in time and space.

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users. '''''This service is currently not available.'''''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[How to access the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

E-Ecology Documentation and Guidance

2021-01-21T16:59:00Z

Stacy shinneman: /* UvA-BiTS Data */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[How to access the e-Ecology database]]
::* [[Reference systems for GPS coordinates and altitudes]]
::* [[Useful queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::'''''This service is currently not available.''''' Access meterological and radar data and overlay with GPS bird tracking data.
:: [[BirdSim]]: 
:::'''''This service is currently not available.''''' More dynamic version of BirdView that allows the user to scroll through weather data in time and space.

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users. '''''This service is currently not available.'''''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[How to access the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

UvA-BiTS Tracking Data

2021-01-21T16:57:21Z

Stacy shinneman: /* gps.ee_track_session_limited */

== UvA-BiTS Tracking Data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_tracking_speed_limited ===

Shows the tracking data that your projects allow you to see.

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker, plus it adds speeds.

It replaces old views gps.uva_tracking_limited, gps.uva_tracking_speed_3d_limited and gps.uva_tracking_speed. This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|longitude||float8||YES||Longitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS84 datum)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|vnorth||float8||YES||- vx * sin(lat) * cos(long) - vy * sin(lat) * sin (long) + vz * cos(lat), m/s
|-
|veast||float8||YES||- vx * sin(lon) + vy * cos(lon), m/s
|-
|vdown||float8||YES||- vx * cos(lat) * cos(lon) - vy * cos(lat) * sin(lon) - vz * sin(lat), m/s
|-
|speed||float8||YES||- vx * sin(lon) + vy * cos(lon)
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2), m/s
|-
|speed_2d||float8||YES|| Speed on 2D, m/s
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate based on Digital Elevation Model data used in the e-ecology database (for further information see: [[Elevation]]
|}

=== ''gps.''ee_acceleration_limited ===

Acceleration is measured in blocks of 1-200 measurements (records). The date_time mentioned in this block is the date and time of the first acceleration measurement. Sequential measurements within a block are numbered with an index 1... (max 200). Every row represents one measurement in 3 directions (X,Y,Z). The order is important within each block, so you should always sort the acceleration measurements by block and index. If you need the exact time for each row, you can calculate it from the index, the sample frequency (which is almost always 20 Hz) and the date_time of the first row in the block.

The date_time of the first block is the time of the GPS measurement preceding the accelerometer measurement. If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time in gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to (time in UTC, no time zone).
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_acc_start_limited ===

Each row represents the start of a block of acceleration measurements when a GPS fix is not available at that acceleration block's time.

If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

To know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

== Bird data ==
=== ''gps.''ee_individual_limited ===

Each row shows bird's details for all birds that your projects allow you to see.

This view replaces the old gps.uva_individual_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|ring_number||varchar||NO||Ring number.
|-
|species_latin_name||varchar||NO||Latin name of the species of the bird.
|-
|colour_ring||varchar||YES||String for ring colour.
|-
|sex||varchar||NO||Sex (M=male, F=female, X=unknown/unspecified)
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|start_date||timestamp||NO||Date when the bird was first tracked with an UvA-BiTS logger.
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant for the tracking project.
|-
|remarks||text||YES||Text for user remarks
|-
|individual_id||int8||NO||Bird unique id. Included just to enable joins.
|}

=== ''gps.''ee_species_limited ===

Each row represents a species that you can assign to your birds to include them in your project. No bird in the system can exist without being linked to one of the species from this list. If you are tagging a bird whose species does not exist in the list, please [http://www.uva-bits.nl/contact/ contact] us.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|abbr||varchar||NO||Unique abbreviation for this species.
|-
|english_name||varchar||YES||Name of the species in English.
|-
|latin_name||varchar||NO||Name of the species in Latin. It is always filled in.
|-
|species_id||int8||NO||Unique id. Added to allow joins.
|}

=== ''gps.''ee_nest_limited ===
This view shows nests you have access to. A nest is always for a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found, decimal degrees (WGS84 datum)
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found, decimal degrees (WGS84 datum)
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|nest_id||int8||NO||Unique id. Added to allow joins to nest_inhabitant_limited.
|}

=== ''gps.''ee_nest_inhabitant_limited ===
This view shows who lives on which nest, for birds you have access to.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string belonging to the nest that you can use to identify that nest. Because it is NOT enforced to be unique, DO NOT USE THIS FIELD FOR JOINING.
|-
|ring_number||varchar||NO||The ring number of a bird on a nest.
|-
|device_info_serial||int8||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the bird's trackers within that period are resolved so that they can be shown here. This is the device_info_serial of a tracker.
|-
|key_name||varchar||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the track sessions' projects within that period are resolved so that they can be shown here. This is the key_name of a project.
|-
|nest_id||int8||NO||Reference to the nest id. Added to allow joins.
|-
|individual_id||int8||NO||Reference to the individual id. Added to allow joins.
|-
|track_session_id||int8||NO||Reference to a resolved track session id. Added to allow joins.
|}
== Tracker and project data ==

=== ''gps.''ee_project_limited ===

Each row shows the details of each project, for the projects you are allowed to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Unique key name, with the form <species>_<place>
|-
|station_name||varchar||NO||Where the receiving station is
|-
|start_date||timestamp||NO||Project start date
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|project_id||int8||NO||Project unique id. Included just to enable joins.
|-
|parent_id||int8||YES||Reference to the parent project in the hierarchy.
|}

=== ''gps.''ee_tracker_limited ===

Each row shows tracker details for all trackers that your projects allow you to see.

This view replaces the old gps.uva_device_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (or surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (or surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (or sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (or heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or heave) axis
|-
|tracker_id||int8||NO||Tracker unique id. Included just to enable joins.
|}

=== ''gps.''ee_track_session_limited ===

A track session links a tracker and a bird for a given period of time. Therefore, it is '''mandatory for projects''' to supply this piece of '''metadata''' before they can see any tracking data related to a tracker they own.

In order to create a track session, the project must create first an entry in the system for the bird that they are attaching the tracker to, and this is the way for the UvA-BiTS community to make sense of the tracking data that we are receiving. This is why providing the metadata is so important.

By editing the start and end dates, administrators of the project that this track session belongs to, can delimit (and therefore, establish) when the coupling makes sense (or is to be understood as useful).

Whenever a tracker is attached to a bird, a track session must be started (start_date) to indicate that the tracking data that is received is already live data. If the tracker is separated from the bird, the tracking data ceases to make sense, so the track session's end date should be set to denote it has ended. New track sessions can be started if the tracker is re-attached to a bird (possibly the same as before, or maybe a different one) afterwards.

Each row shows track session's information. There exists a row for each track session that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the track session belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker this track session involves.
|-
|ring_number||varchar||NO||Ring number of the bird that this track session involves.
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|remarks||text||YES||Free text for comments
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged, decimal degrees (WGS84 datum)
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged, decimal degrees (WGS84 datum)
|-
|project_id||int8||NO||Reference to the project this track session belongs to. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker this track session involves. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual this track session involves. Included just to enable joins.
|-
|track_session_id||int8||NO||Unique identifier of the track session. Included just to enable joins.
|}

== Relation to projects ==

=== ''gps.''ee_user_limited ===

Each row shows a colleague in a project that you belong to, indicating her role and status there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Project name
|-
|email||varchar||NO||e-mail address
|-
|login_name||varchar||NO||Login name
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|role||varchar||NO||Role in the project
|-
|status||varchar||NO||Status of the membership
|-
|user_id||int8||NO||Unique identifier of the user. Included to enable joins.
|}

=== ''gps.''ee_tracker_ownership_limited ===

Each row that you see expresses that a tracker belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the tracker belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker. Included just to enable joins.
|}

=== ''gps.''ee_individual_ownership_limited ===

Each row that you see expresses that a bird belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the individual belongs to.
|-
|ring_number||varchar||NO||Ring number of the bird.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual. Included just to enable joins.
|}

== Engineering data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_comm_limited ===

Each row indicates status and result of communications with ground stations for each tracker that your project allows you to see.

Only for newer firmware (v.1.0.1.4 and higher).

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''ee_energy_limited ===

Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''ee_settings_limited ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

=== ''gps.''ee_trackingfile_limited ===

Each row shows the status of the last attempt to parse (process) the raw data of a file for a given tracker, for each tracker that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''ee_warning_limited ===

Since firmware version 1.0.2.7 this table was added for events that do not occur very often (C10, C20) or errors that occur during parsing.

For example all CRC failures can be selected with :

select * from gps.ee_warning_limited
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||File name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line contents in the file
|}

== Sharings ==

Projects can now share trackers with other projects during periods of time. This means that people in the receiving projects can access information that these shared trackers provide during those time periods.

The following views allow you to see this shared information:

* Tracking data
** ''gps.''ee_shared_tracking_speed_limited: tracking information (gps, speed), during the shared period, for trackers shared
** ''gps.''ee_shared_acc_start_limited: acceleration start information, during the shared period, for trackers shared
** ''gps.''ee_shared_acceleration_limited: acceleration information for trackers shared
* Metadata
** ''gps.''ee_shared_tracker_limited: information about each shared tracker
** ''gps.''ee_shared_individual_limited: information about birds with shared trackers
** ''gps.''ee_shared_track_session_limited: information about track sessions which involve shared trackers
* Relation to projects
** ''gps.''ee_shared_tracker_sharing_limited: view information for each sharing that your projects allow you to see

== Accelerometer calibration ==

The accelerometer measures the acceleration in 3D (20 times per second). Acceleration is caused by gravity (static component) and by a change in speed (dynamic component).
When the logger is static in horizontal position (solar cells up), gravity (green arrow) shows a heave (vertical acceleration, z) of +1 g (1 g = 9.81 m.s-2) while surge and heave are both 0. When the front of the logger is vertically pointing down (forward acceleration, x), surge is +1g and when it is rolled to the right (sidewards acceleration, y) sway is +1g. Due to gravity, the pitch of a logger is thus translated into the values of heave, surge and sway (or z, x, y).

However, the values are measured in mV instead of g. Before delivering the GPS-loggers, they have been calibrated at UvA. The acceleration (A) expressed in g is calculated from the measured value (M) and the calibration factors offset (''O'') and sensitivity (''S'').

Thus acceleration for heave (z), surge (x) and sway (y) is calculated as:
Az = (Mz-Oz)/Sz ; Ax = (Mx-Ox)/Sx; Ay = (My-Oy)/Sy ;

The values of ''O'' (in general ranging from 0-60, sometimes even 300) and ''S'' (ranging from 1300-1450) are available in the device table of the data base. If not available, use the factory defaults ''O'' = 0 and ''S'' = 1365.

{|
| [[File:accelero1.png|300px]]||[[File:accelero2.png|300px]]|
|}

In the figure above heave (z) is shown in green, surge(x) in red and sway (y) in blue, all values are expressed in g (on the y-axis).

== Data processing and filtering ==
The raw log files are processed before being entered into the database. Each firmware version has a dedicated parser for the task. In certain cases data that is found in the log files is not stored in the database. This can occur for two reasons:

1. The data did not pass the CRC8 checksum (cyclic redundancy check). This is a standard procedure designed to detect accidental changes to raw computer data.

2. If the data has an invalid signature. For example, if a certain data format (eg numeric) or number of values (eg 4 digits) is expected but not returned, this record is not stored in the database.

When a line is deleted, the log file name, line number and the type of error is stored in the table gps.uva_warning102

== Information about GPS receivers ==
The first series of tags used the UBlox LEAS4S chip (~up to tag 541), newer tags use UBlox LEAS6S. The series developed in 2012 will use the Max6G receivers. From 2012 receiver versions will be stored in the database.
The UBX protocol is used for programming firmware, packaging and processing data.
For more information about the receivers themselves see the ublox documentation.
Ublox 4 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/ANTARIS_Protocol_Specification%28GPS.G3-X-03002%29.zip]
Ublox 6 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/u-blox6_ReceiverDescriptionProtocolSpec_%28GPS.G6-SW-10018%29.pdf]

UvA-BiTS Tracking Data

2021-01-21T16:56:59Z

Stacy shinneman: /* gps.ee_nest_limited */

== UvA-BiTS Tracking Data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_tracking_speed_limited ===

Shows the tracking data that your projects allow you to see.

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker, plus it adds speeds.

It replaces old views gps.uva_tracking_limited, gps.uva_tracking_speed_3d_limited and gps.uva_tracking_speed. This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|longitude||float8||YES||Longitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS84 datum)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|vnorth||float8||YES||- vx * sin(lat) * cos(long) - vy * sin(lat) * sin (long) + vz * cos(lat), m/s
|-
|veast||float8||YES||- vx * sin(lon) + vy * cos(lon), m/s
|-
|vdown||float8||YES||- vx * cos(lat) * cos(lon) - vy * cos(lat) * sin(lon) - vz * sin(lat), m/s
|-
|speed||float8||YES||- vx * sin(lon) + vy * cos(lon)
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2), m/s
|-
|speed_2d||float8||YES|| Speed on 2D, m/s
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate based on Digital Elevation Model data used in the e-ecology database (for further information see: [[Elevation]]
|}

=== ''gps.''ee_acceleration_limited ===

Acceleration is measured in blocks of 1-200 measurements (records). The date_time mentioned in this block is the date and time of the first acceleration measurement. Sequential measurements within a block are numbered with an index 1... (max 200). Every row represents one measurement in 3 directions (X,Y,Z). The order is important within each block, so you should always sort the acceleration measurements by block and index. If you need the exact time for each row, you can calculate it from the index, the sample frequency (which is almost always 20 Hz) and the date_time of the first row in the block.

The date_time of the first block is the time of the GPS measurement preceding the accelerometer measurement. If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time in gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to (time in UTC, no time zone).
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_acc_start_limited ===

Each row represents the start of a block of acceleration measurements when a GPS fix is not available at that acceleration block's time.

If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

To know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

== Bird data ==
=== ''gps.''ee_individual_limited ===

Each row shows bird's details for all birds that your projects allow you to see.

This view replaces the old gps.uva_individual_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|ring_number||varchar||NO||Ring number.
|-
|species_latin_name||varchar||NO||Latin name of the species of the bird.
|-
|colour_ring||varchar||YES||String for ring colour.
|-
|sex||varchar||NO||Sex (M=male, F=female, X=unknown/unspecified)
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|start_date||timestamp||NO||Date when the bird was first tracked with an UvA-BiTS logger.
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant for the tracking project.
|-
|remarks||text||YES||Text for user remarks
|-
|individual_id||int8||NO||Bird unique id. Included just to enable joins.
|}

=== ''gps.''ee_species_limited ===

Each row represents a species that you can assign to your birds to include them in your project. No bird in the system can exist without being linked to one of the species from this list. If you are tagging a bird whose species does not exist in the list, please [http://www.uva-bits.nl/contact/ contact] us.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|abbr||varchar||NO||Unique abbreviation for this species.
|-
|english_name||varchar||YES||Name of the species in English.
|-
|latin_name||varchar||NO||Name of the species in Latin. It is always filled in.
|-
|species_id||int8||NO||Unique id. Added to allow joins.
|}

=== ''gps.''ee_nest_limited ===
This view shows nests you have access to. A nest is always for a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found, decimal degrees (WGS84 datum)
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found, decimal degrees (WGS84 datum)
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|nest_id||int8||NO||Unique id. Added to allow joins to nest_inhabitant_limited.
|}

=== ''gps.''ee_nest_inhabitant_limited ===
This view shows who lives on which nest, for birds you have access to.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string belonging to the nest that you can use to identify that nest. Because it is NOT enforced to be unique, DO NOT USE THIS FIELD FOR JOINING.
|-
|ring_number||varchar||NO||The ring number of a bird on a nest.
|-
|device_info_serial||int8||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the bird's trackers within that period are resolved so that they can be shown here. This is the device_info_serial of a tracker.
|-
|key_name||varchar||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the track sessions' projects within that period are resolved so that they can be shown here. This is the key_name of a project.
|-
|nest_id||int8||NO||Reference to the nest id. Added to allow joins.
|-
|individual_id||int8||NO||Reference to the individual id. Added to allow joins.
|-
|track_session_id||int8||NO||Reference to a resolved track session id. Added to allow joins.
|}
== Tracker and project data ==

=== ''gps.''ee_project_limited ===

Each row shows the details of each project, for the projects you are allowed to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Unique key name, with the form <species>_<place>
|-
|station_name||varchar||NO||Where the receiving station is
|-
|start_date||timestamp||NO||Project start date
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|project_id||int8||NO||Project unique id. Included just to enable joins.
|-
|parent_id||int8||YES||Reference to the parent project in the hierarchy.
|}

=== ''gps.''ee_tracker_limited ===

Each row shows tracker details for all trackers that your projects allow you to see.

This view replaces the old gps.uva_device_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (or surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (or surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (or sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (or heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or heave) axis
|-
|tracker_id||int8||NO||Tracker unique id. Included just to enable joins.
|}

=== ''gps.''ee_track_session_limited ===

A track session links a tracker and a bird for a given period of time. Therefore, it is '''mandatory for projects''' to supply this piece of '''metadata''' before they can see any tracking data related to a tracker they own.

In order to create a track session, the project must create first an entry in the system for the bird that they are attaching the tracker to, and this is the way for the UvA-BiTS community to make sense of the tracking data that we are receiving. This is why providing the metadata is so important.

By editing the start and end dates, administrators of the project that this track session belongs to, can delimit (and therefore, establish) when the coupling makes sense (or is to be understood as useful).

Whenever a tracker is attached to a bird, a track session must be started (start_date) to indicate that the tracking data that is received is already live data. If the tracker is separated from the bird, the tracking data ceases to make sense, so the track session's end date should be set to denote it has ended. New track sessions can be started if the tracker is re-attached to a bird (possibly the same as before, or maybe a different one) afterwards.

Each row shows track session's information. There exists a row for each track session that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the track session belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker this track session involves.
|-
|ring_number||varchar||NO||Ring number of the bird that this track session involves.
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|remarks||text||YES||Free text for comments
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged, decimal degrees (WGS1984 Ellipsoid)
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged, decimal degrees (WGS1984 Ellipsoid)
|-
|project_id||int8||NO||Reference to the project this track session belongs to. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker this track session involves. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual this track session involves. Included just to enable joins.
|-
|track_session_id||int8||NO||Unique identifier of the track session. Included just to enable joins.
|}

== Relation to projects ==

=== ''gps.''ee_user_limited ===

Each row shows a colleague in a project that you belong to, indicating her role and status there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Project name
|-
|email||varchar||NO||e-mail address
|-
|login_name||varchar||NO||Login name
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|role||varchar||NO||Role in the project
|-
|status||varchar||NO||Status of the membership
|-
|user_id||int8||NO||Unique identifier of the user. Included to enable joins.
|}

=== ''gps.''ee_tracker_ownership_limited ===

Each row that you see expresses that a tracker belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the tracker belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker. Included just to enable joins.
|}

=== ''gps.''ee_individual_ownership_limited ===

Each row that you see expresses that a bird belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the individual belongs to.
|-
|ring_number||varchar||NO||Ring number of the bird.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual. Included just to enable joins.
|}

== Engineering data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_comm_limited ===

Each row indicates status and result of communications with ground stations for each tracker that your project allows you to see.

Only for newer firmware (v.1.0.1.4 and higher).

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''ee_energy_limited ===

Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''ee_settings_limited ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

=== ''gps.''ee_trackingfile_limited ===

Each row shows the status of the last attempt to parse (process) the raw data of a file for a given tracker, for each tracker that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''ee_warning_limited ===

Since firmware version 1.0.2.7 this table was added for events that do not occur very often (C10, C20) or errors that occur during parsing.

For example all CRC failures can be selected with :

select * from gps.ee_warning_limited
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||File name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line contents in the file
|}

== Sharings ==

Projects can now share trackers with other projects during periods of time. This means that people in the receiving projects can access information that these shared trackers provide during those time periods.

The following views allow you to see this shared information:

* Tracking data
** ''gps.''ee_shared_tracking_speed_limited: tracking information (gps, speed), during the shared period, for trackers shared
** ''gps.''ee_shared_acc_start_limited: acceleration start information, during the shared period, for trackers shared
** ''gps.''ee_shared_acceleration_limited: acceleration information for trackers shared
* Metadata
** ''gps.''ee_shared_tracker_limited: information about each shared tracker
** ''gps.''ee_shared_individual_limited: information about birds with shared trackers
** ''gps.''ee_shared_track_session_limited: information about track sessions which involve shared trackers
* Relation to projects
** ''gps.''ee_shared_tracker_sharing_limited: view information for each sharing that your projects allow you to see

== Accelerometer calibration ==

The accelerometer measures the acceleration in 3D (20 times per second). Acceleration is caused by gravity (static component) and by a change in speed (dynamic component).
When the logger is static in horizontal position (solar cells up), gravity (green arrow) shows a heave (vertical acceleration, z) of +1 g (1 g = 9.81 m.s-2) while surge and heave are both 0. When the front of the logger is vertically pointing down (forward acceleration, x), surge is +1g and when it is rolled to the right (sidewards acceleration, y) sway is +1g. Due to gravity, the pitch of a logger is thus translated into the values of heave, surge and sway (or z, x, y).

However, the values are measured in mV instead of g. Before delivering the GPS-loggers, they have been calibrated at UvA. The acceleration (A) expressed in g is calculated from the measured value (M) and the calibration factors offset (''O'') and sensitivity (''S'').

Thus acceleration for heave (z), surge (x) and sway (y) is calculated as:
Az = (Mz-Oz)/Sz ; Ax = (Mx-Ox)/Sx; Ay = (My-Oy)/Sy ;

The values of ''O'' (in general ranging from 0-60, sometimes even 300) and ''S'' (ranging from 1300-1450) are available in the device table of the data base. If not available, use the factory defaults ''O'' = 0 and ''S'' = 1365.

{|
| [[File:accelero1.png|300px]]||[[File:accelero2.png|300px]]|
|}

In the figure above heave (z) is shown in green, surge(x) in red and sway (y) in blue, all values are expressed in g (on the y-axis).

== Data processing and filtering ==
The raw log files are processed before being entered into the database. Each firmware version has a dedicated parser for the task. In certain cases data that is found in the log files is not stored in the database. This can occur for two reasons:

1. The data did not pass the CRC8 checksum (cyclic redundancy check). This is a standard procedure designed to detect accidental changes to raw computer data.

2. If the data has an invalid signature. For example, if a certain data format (eg numeric) or number of values (eg 4 digits) is expected but not returned, this record is not stored in the database.

When a line is deleted, the log file name, line number and the type of error is stored in the table gps.uva_warning102

== Information about GPS receivers ==
The first series of tags used the UBlox LEAS4S chip (~up to tag 541), newer tags use UBlox LEAS6S. The series developed in 2012 will use the Max6G receivers. From 2012 receiver versions will be stored in the database.
The UBX protocol is used for programming firmware, packaging and processing data.
For more information about the receivers themselves see the ublox documentation.
Ublox 4 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/ANTARIS_Protocol_Specification%28GPS.G3-X-03002%29.zip]
Ublox 6 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/u-blox6_ReceiverDescriptionProtocolSpec_%28GPS.G6-SW-10018%29.pdf]

UvA-BiTS Tracking Data

2021-01-21T16:56:33Z

Stacy shinneman: /* gps.ee_tracking_speed_limited */

== UvA-BiTS Tracking Data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_tracking_speed_limited ===

Shows the tracking data that your projects allow you to see.

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker, plus it adds speeds.

It replaces old views gps.uva_tracking_limited, gps.uva_tracking_speed_3d_limited and gps.uva_tracking_speed. This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|longitude||float8||YES||Longitude of the GPS fix, decimal degrees (WGS84 datum)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS84 datum)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|vnorth||float8||YES||- vx * sin(lat) * cos(long) - vy * sin(lat) * sin (long) + vz * cos(lat), m/s
|-
|veast||float8||YES||- vx * sin(lon) + vy * cos(lon), m/s
|-
|vdown||float8||YES||- vx * cos(lat) * cos(lon) - vy * cos(lat) * sin(lon) - vz * sin(lat), m/s
|-
|speed||float8||YES||- vx * sin(lon) + vy * cos(lon)
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2), m/s
|-
|speed_2d||float8||YES|| Speed on 2D, m/s
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate based on Digital Elevation Model data used in the e-ecology database (for further information see: [[Elevation]]
|}

=== ''gps.''ee_acceleration_limited ===

Acceleration is measured in blocks of 1-200 measurements (records). The date_time mentioned in this block is the date and time of the first acceleration measurement. Sequential measurements within a block are numbered with an index 1... (max 200). Every row represents one measurement in 3 directions (X,Y,Z). The order is important within each block, so you should always sort the acceleration measurements by block and index. If you need the exact time for each row, you can calculate it from the index, the sample frequency (which is almost always 20 Hz) and the date_time of the first row in the block.

The date_time of the first block is the time of the GPS measurement preceding the accelerometer measurement. If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time in gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to (time in UTC, no time zone).
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_acc_start_limited ===

Each row represents the start of a block of acceleration measurements when a GPS fix is not available at that acceleration block's time.

If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

To know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

== Bird data ==
=== ''gps.''ee_individual_limited ===

Each row shows bird's details for all birds that your projects allow you to see.

This view replaces the old gps.uva_individual_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|ring_number||varchar||NO||Ring number.
|-
|species_latin_name||varchar||NO||Latin name of the species of the bird.
|-
|colour_ring||varchar||YES||String for ring colour.
|-
|sex||varchar||NO||Sex (M=male, F=female, X=unknown/unspecified)
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|start_date||timestamp||NO||Date when the bird was first tracked with an UvA-BiTS logger.
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant for the tracking project.
|-
|remarks||text||YES||Text for user remarks
|-
|individual_id||int8||NO||Bird unique id. Included just to enable joins.
|}

=== ''gps.''ee_species_limited ===

Each row represents a species that you can assign to your birds to include them in your project. No bird in the system can exist without being linked to one of the species from this list. If you are tagging a bird whose species does not exist in the list, please [http://www.uva-bits.nl/contact/ contact] us.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|abbr||varchar||NO||Unique abbreviation for this species.
|-
|english_name||varchar||YES||Name of the species in English.
|-
|latin_name||varchar||NO||Name of the species in Latin. It is always filled in.
|-
|species_id||int8||NO||Unique id. Added to allow joins.
|}

=== ''gps.''ee_nest_limited ===
This view shows nests you have access to. A nest is always for a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found, decimal degrees.
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found, decimal degrees.
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|nest_id||int8||NO||Unique id. Added to allow joins to nest_inhabitant_limited.
|}

=== ''gps.''ee_nest_inhabitant_limited ===
This view shows who lives on which nest, for birds you have access to.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string belonging to the nest that you can use to identify that nest. Because it is NOT enforced to be unique, DO NOT USE THIS FIELD FOR JOINING.
|-
|ring_number||varchar||NO||The ring number of a bird on a nest.
|-
|device_info_serial||int8||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the bird's trackers within that period are resolved so that they can be shown here. This is the device_info_serial of a tracker.
|-
|key_name||varchar||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the track sessions' projects within that period are resolved so that they can be shown here. This is the key_name of a project.
|-
|nest_id||int8||NO||Reference to the nest id. Added to allow joins.
|-
|individual_id||int8||NO||Reference to the individual id. Added to allow joins.
|-
|track_session_id||int8||NO||Reference to a resolved track session id. Added to allow joins.
|}
== Tracker and project data ==

=== ''gps.''ee_project_limited ===

Each row shows the details of each project, for the projects you are allowed to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Unique key name, with the form <species>_<place>
|-
|station_name||varchar||NO||Where the receiving station is
|-
|start_date||timestamp||NO||Project start date
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|project_id||int8||NO||Project unique id. Included just to enable joins.
|-
|parent_id||int8||YES||Reference to the parent project in the hierarchy.
|}

=== ''gps.''ee_tracker_limited ===

Each row shows tracker details for all trackers that your projects allow you to see.

This view replaces the old gps.uva_device_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (or surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (or surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (or sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (or heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or heave) axis
|-
|tracker_id||int8||NO||Tracker unique id. Included just to enable joins.
|}

=== ''gps.''ee_track_session_limited ===

A track session links a tracker and a bird for a given period of time. Therefore, it is '''mandatory for projects''' to supply this piece of '''metadata''' before they can see any tracking data related to a tracker they own.

In order to create a track session, the project must create first an entry in the system for the bird that they are attaching the tracker to, and this is the way for the UvA-BiTS community to make sense of the tracking data that we are receiving. This is why providing the metadata is so important.

By editing the start and end dates, administrators of the project that this track session belongs to, can delimit (and therefore, establish) when the coupling makes sense (or is to be understood as useful).

Whenever a tracker is attached to a bird, a track session must be started (start_date) to indicate that the tracking data that is received is already live data. If the tracker is separated from the bird, the tracking data ceases to make sense, so the track session's end date should be set to denote it has ended. New track sessions can be started if the tracker is re-attached to a bird (possibly the same as before, or maybe a different one) afterwards.

Each row shows track session's information. There exists a row for each track session that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the track session belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker this track session involves.
|-
|ring_number||varchar||NO||Ring number of the bird that this track session involves.
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|remarks||text||YES||Free text for comments
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged, decimal degrees (WGS1984 Ellipsoid)
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged, decimal degrees (WGS1984 Ellipsoid)
|-
|project_id||int8||NO||Reference to the project this track session belongs to. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker this track session involves. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual this track session involves. Included just to enable joins.
|-
|track_session_id||int8||NO||Unique identifier of the track session. Included just to enable joins.
|}

== Relation to projects ==

=== ''gps.''ee_user_limited ===

Each row shows a colleague in a project that you belong to, indicating her role and status there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Project name
|-
|email||varchar||NO||e-mail address
|-
|login_name||varchar||NO||Login name
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|role||varchar||NO||Role in the project
|-
|status||varchar||NO||Status of the membership
|-
|user_id||int8||NO||Unique identifier of the user. Included to enable joins.
|}

=== ''gps.''ee_tracker_ownership_limited ===

Each row that you see expresses that a tracker belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the tracker belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker. Included just to enable joins.
|}

=== ''gps.''ee_individual_ownership_limited ===

Each row that you see expresses that a bird belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the individual belongs to.
|-
|ring_number||varchar||NO||Ring number of the bird.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual. Included just to enable joins.
|}

== Engineering data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_comm_limited ===

Each row indicates status and result of communications with ground stations for each tracker that your project allows you to see.

Only for newer firmware (v.1.0.1.4 and higher).

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''ee_energy_limited ===

Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''ee_settings_limited ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

=== ''gps.''ee_trackingfile_limited ===

Each row shows the status of the last attempt to parse (process) the raw data of a file for a given tracker, for each tracker that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''ee_warning_limited ===

Since firmware version 1.0.2.7 this table was added for events that do not occur very often (C10, C20) or errors that occur during parsing.

For example all CRC failures can be selected with :

select * from gps.ee_warning_limited
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||File name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line contents in the file
|}

== Sharings ==

Projects can now share trackers with other projects during periods of time. This means that people in the receiving projects can access information that these shared trackers provide during those time periods.

The following views allow you to see this shared information:

* Tracking data
** ''gps.''ee_shared_tracking_speed_limited: tracking information (gps, speed), during the shared period, for trackers shared
** ''gps.''ee_shared_acc_start_limited: acceleration start information, during the shared period, for trackers shared
** ''gps.''ee_shared_acceleration_limited: acceleration information for trackers shared
* Metadata
** ''gps.''ee_shared_tracker_limited: information about each shared tracker
** ''gps.''ee_shared_individual_limited: information about birds with shared trackers
** ''gps.''ee_shared_track_session_limited: information about track sessions which involve shared trackers
* Relation to projects
** ''gps.''ee_shared_tracker_sharing_limited: view information for each sharing that your projects allow you to see

== Accelerometer calibration ==

The accelerometer measures the acceleration in 3D (20 times per second). Acceleration is caused by gravity (static component) and by a change in speed (dynamic component).
When the logger is static in horizontal position (solar cells up), gravity (green arrow) shows a heave (vertical acceleration, z) of +1 g (1 g = 9.81 m.s-2) while surge and heave are both 0. When the front of the logger is vertically pointing down (forward acceleration, x), surge is +1g and when it is rolled to the right (sidewards acceleration, y) sway is +1g. Due to gravity, the pitch of a logger is thus translated into the values of heave, surge and sway (or z, x, y).

However, the values are measured in mV instead of g. Before delivering the GPS-loggers, they have been calibrated at UvA. The acceleration (A) expressed in g is calculated from the measured value (M) and the calibration factors offset (''O'') and sensitivity (''S'').

Thus acceleration for heave (z), surge (x) and sway (y) is calculated as:
Az = (Mz-Oz)/Sz ; Ax = (Mx-Ox)/Sx; Ay = (My-Oy)/Sy ;

The values of ''O'' (in general ranging from 0-60, sometimes even 300) and ''S'' (ranging from 1300-1450) are available in the device table of the data base. If not available, use the factory defaults ''O'' = 0 and ''S'' = 1365.

{|
| [[File:accelero1.png|300px]]||[[File:accelero2.png|300px]]|
|}

In the figure above heave (z) is shown in green, surge(x) in red and sway (y) in blue, all values are expressed in g (on the y-axis).

== Data processing and filtering ==
The raw log files are processed before being entered into the database. Each firmware version has a dedicated parser for the task. In certain cases data that is found in the log files is not stored in the database. This can occur for two reasons:

1. The data did not pass the CRC8 checksum (cyclic redundancy check). This is a standard procedure designed to detect accidental changes to raw computer data.

2. If the data has an invalid signature. For example, if a certain data format (eg numeric) or number of values (eg 4 digits) is expected but not returned, this record is not stored in the database.

When a line is deleted, the log file name, line number and the type of error is stored in the table gps.uva_warning102

== Information about GPS receivers ==
The first series of tags used the UBlox LEAS4S chip (~up to tag 541), newer tags use UBlox LEAS6S. The series developed in 2012 will use the Max6G receivers. From 2012 receiver versions will be stored in the database.
The UBX protocol is used for programming firmware, packaging and processing data.
For more information about the receivers themselves see the ublox documentation.
Ublox 4 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/ANTARIS_Protocol_Specification%28GPS.G3-X-03002%29.zip]
Ublox 6 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/u-blox6_ReceiverDescriptionProtocolSpec_%28GPS.G6-SW-10018%29.pdf]

UvA-BiTS Tracking Data

2021-01-21T15:55:03Z

Stacy shinneman: /* gps.ee_track_session_limited */

== UvA-BiTS Tracking Data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_tracking_speed_limited ===

Shows the tracking data that your projects allow you to see.

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker, plus it adds speeds.

It replaces old views gps.uva_tracking_limited, gps.uva_tracking_speed_3d_limited and gps.uva_tracking_speed. This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix, decimal degrees (WGS1984 Ellipsoid)
|-
|longitude||float8||YES||Longitude of the GPS fix, decimal degrees (WGS1984 Ellipsoid)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS1984 Ellipsoid)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|vnorth||float8||YES||- vx * sin(lat) * cos(long) - vy * sin(lat) * sin (long) + vz * cos(lat), m/s
|-
|veast||float8||YES||- vx * sin(lon) + vy * cos(lon), m/s
|-
|vdown||float8||YES||- vx * cos(lat) * cos(lon) - vy * cos(lat) * sin(lon) - vz * sin(lat), m/s
|-
|speed||float8||YES||- vx * sin(lon) + vy * cos(lon)
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2), m/s
|-
|speed_2d||float8||YES|| Speed on 2D, m/s
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate based on Digital Elevation Model data used in the e-ecology database (for further information see: [[Elevation]]
|}

=== ''gps.''ee_acceleration_limited ===

Acceleration is measured in blocks of 1-200 measurements (records). The date_time mentioned in this block is the date and time of the first acceleration measurement. Sequential measurements within a block are numbered with an index 1... (max 200). Every row represents one measurement in 3 directions (X,Y,Z). The order is important within each block, so you should always sort the acceleration measurements by block and index. If you need the exact time for each row, you can calculate it from the index, the sample frequency (which is almost always 20 Hz) and the date_time of the first row in the block.

The date_time of the first block is the time of the GPS measurement preceding the accelerometer measurement. If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time in gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to (time in UTC, no time zone).
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_acc_start_limited ===

Each row represents the start of a block of acceleration measurements when a GPS fix is not available at that acceleration block's time.

If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

To know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

== Bird data ==
=== ''gps.''ee_individual_limited ===

Each row shows bird's details for all birds that your projects allow you to see.

This view replaces the old gps.uva_individual_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|ring_number||varchar||NO||Ring number.
|-
|species_latin_name||varchar||NO||Latin name of the species of the bird.
|-
|colour_ring||varchar||YES||String for ring colour.
|-
|sex||varchar||NO||Sex (M=male, F=female, X=unknown/unspecified)
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|start_date||timestamp||NO||Date when the bird was first tracked with an UvA-BiTS logger.
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant for the tracking project.
|-
|remarks||text||YES||Text for user remarks
|-
|individual_id||int8||NO||Bird unique id. Included just to enable joins.
|}

=== ''gps.''ee_species_limited ===

Each row represents a species that you can assign to your birds to include them in your project. No bird in the system can exist without being linked to one of the species from this list. If you are tagging a bird whose species does not exist in the list, please [http://www.uva-bits.nl/contact/ contact] us.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|abbr||varchar||NO||Unique abbreviation for this species.
|-
|english_name||varchar||YES||Name of the species in English.
|-
|latin_name||varchar||NO||Name of the species in Latin. It is always filled in.
|-
|species_id||int8||NO||Unique id. Added to allow joins.
|}

=== ''gps.''ee_nest_limited ===
This view shows nests you have access to. A nest is always for a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found, decimal degrees.
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found, decimal degrees.
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|nest_id||int8||NO||Unique id. Added to allow joins to nest_inhabitant_limited.
|}

=== ''gps.''ee_nest_inhabitant_limited ===
This view shows who lives on which nest, for birds you have access to.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string belonging to the nest that you can use to identify that nest. Because it is NOT enforced to be unique, DO NOT USE THIS FIELD FOR JOINING.
|-
|ring_number||varchar||NO||The ring number of a bird on a nest.
|-
|device_info_serial||int8||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the bird's trackers within that period are resolved so that they can be shown here. This is the device_info_serial of a tracker.
|-
|key_name||varchar||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the track sessions' projects within that period are resolved so that they can be shown here. This is the key_name of a project.
|-
|nest_id||int8||NO||Reference to the nest id. Added to allow joins.
|-
|individual_id||int8||NO||Reference to the individual id. Added to allow joins.
|-
|track_session_id||int8||NO||Reference to a resolved track session id. Added to allow joins.
|}
== Tracker and project data ==

=== ''gps.''ee_project_limited ===

Each row shows the details of each project, for the projects you are allowed to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Unique key name, with the form <species>_<place>
|-
|station_name||varchar||NO||Where the receiving station is
|-
|start_date||timestamp||NO||Project start date
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|project_id||int8||NO||Project unique id. Included just to enable joins.
|-
|parent_id||int8||YES||Reference to the parent project in the hierarchy.
|}

=== ''gps.''ee_tracker_limited ===

Each row shows tracker details for all trackers that your projects allow you to see.

This view replaces the old gps.uva_device_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (or surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (or surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (or sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (or heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or heave) axis
|-
|tracker_id||int8||NO||Tracker unique id. Included just to enable joins.
|}

=== ''gps.''ee_track_session_limited ===

A track session links a tracker and a bird for a given period of time. Therefore, it is '''mandatory for projects''' to supply this piece of '''metadata''' before they can see any tracking data related to a tracker they own.

In order to create a track session, the project must create first an entry in the system for the bird that they are attaching the tracker to, and this is the way for the UvA-BiTS community to make sense of the tracking data that we are receiving. This is why providing the metadata is so important.

By editing the start and end dates, administrators of the project that this track session belongs to, can delimit (and therefore, establish) when the coupling makes sense (or is to be understood as useful).

Whenever a tracker is attached to a bird, a track session must be started (start_date) to indicate that the tracking data that is received is already live data. If the tracker is separated from the bird, the tracking data ceases to make sense, so the track session's end date should be set to denote it has ended. New track sessions can be started if the tracker is re-attached to a bird (possibly the same as before, or maybe a different one) afterwards.

Each row shows track session's information. There exists a row for each track session that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the track session belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker this track session involves.
|-
|ring_number||varchar||NO||Ring number of the bird that this track session involves.
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|remarks||text||YES||Free text for comments
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged, decimal degrees (WGS1984 Ellipsoid)
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged, decimal degrees (WGS1984 Ellipsoid)
|-
|project_id||int8||NO||Reference to the project this track session belongs to. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker this track session involves. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual this track session involves. Included just to enable joins.
|-
|track_session_id||int8||NO||Unique identifier of the track session. Included just to enable joins.
|}

== Relation to projects ==

=== ''gps.''ee_user_limited ===

Each row shows a colleague in a project that you belong to, indicating her role and status there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Project name
|-
|email||varchar||NO||e-mail address
|-
|login_name||varchar||NO||Login name
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|role||varchar||NO||Role in the project
|-
|status||varchar||NO||Status of the membership
|-
|user_id||int8||NO||Unique identifier of the user. Included to enable joins.
|}

=== ''gps.''ee_tracker_ownership_limited ===

Each row that you see expresses that a tracker belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the tracker belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker. Included just to enable joins.
|}

=== ''gps.''ee_individual_ownership_limited ===

Each row that you see expresses that a bird belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the individual belongs to.
|-
|ring_number||varchar||NO||Ring number of the bird.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual. Included just to enable joins.
|}

== Engineering data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_comm_limited ===

Each row indicates status and result of communications with ground stations for each tracker that your project allows you to see.

Only for newer firmware (v.1.0.1.4 and higher).

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''ee_energy_limited ===

Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''ee_settings_limited ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

=== ''gps.''ee_trackingfile_limited ===

Each row shows the status of the last attempt to parse (process) the raw data of a file for a given tracker, for each tracker that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''ee_warning_limited ===

Since firmware version 1.0.2.7 this table was added for events that do not occur very often (C10, C20) or errors that occur during parsing.

For example all CRC failures can be selected with :

select * from gps.ee_warning_limited
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||File name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line contents in the file
|}

== Sharings ==

Projects can now share trackers with other projects during periods of time. This means that people in the receiving projects can access information that these shared trackers provide during those time periods.

The following views allow you to see this shared information:

* Tracking data
** ''gps.''ee_shared_tracking_speed_limited: tracking information (gps, speed), during the shared period, for trackers shared
** ''gps.''ee_shared_acc_start_limited: acceleration start information, during the shared period, for trackers shared
** ''gps.''ee_shared_acceleration_limited: acceleration information for trackers shared
* Metadata
** ''gps.''ee_shared_tracker_limited: information about each shared tracker
** ''gps.''ee_shared_individual_limited: information about birds with shared trackers
** ''gps.''ee_shared_track_session_limited: information about track sessions which involve shared trackers
* Relation to projects
** ''gps.''ee_shared_tracker_sharing_limited: view information for each sharing that your projects allow you to see

== Accelerometer calibration ==

The accelerometer measures the acceleration in 3D (20 times per second). Acceleration is caused by gravity (static component) and by a change in speed (dynamic component).
When the logger is static in horizontal position (solar cells up), gravity (green arrow) shows a heave (vertical acceleration, z) of +1 g (1 g = 9.81 m.s-2) while surge and heave are both 0. When the front of the logger is vertically pointing down (forward acceleration, x), surge is +1g and when it is rolled to the right (sidewards acceleration, y) sway is +1g. Due to gravity, the pitch of a logger is thus translated into the values of heave, surge and sway (or z, x, y).

However, the values are measured in mV instead of g. Before delivering the GPS-loggers, they have been calibrated at UvA. The acceleration (A) expressed in g is calculated from the measured value (M) and the calibration factors offset (''O'') and sensitivity (''S'').

Thus acceleration for heave (z), surge (x) and sway (y) is calculated as:
Az = (Mz-Oz)/Sz ; Ax = (Mx-Ox)/Sx; Ay = (My-Oy)/Sy ;

The values of ''O'' (in general ranging from 0-60, sometimes even 300) and ''S'' (ranging from 1300-1450) are available in the device table of the data base. If not available, use the factory defaults ''O'' = 0 and ''S'' = 1365.

{|
| [[File:accelero1.png|300px]]||[[File:accelero2.png|300px]]|
|}

In the figure above heave (z) is shown in green, surge(x) in red and sway (y) in blue, all values are expressed in g (on the y-axis).

== Data processing and filtering ==
The raw log files are processed before being entered into the database. Each firmware version has a dedicated parser for the task. In certain cases data that is found in the log files is not stored in the database. This can occur for two reasons:

1. The data did not pass the CRC8 checksum (cyclic redundancy check). This is a standard procedure designed to detect accidental changes to raw computer data.

2. If the data has an invalid signature. For example, if a certain data format (eg numeric) or number of values (eg 4 digits) is expected but not returned, this record is not stored in the database.

When a line is deleted, the log file name, line number and the type of error is stored in the table gps.uva_warning102

== Information about GPS receivers ==
The first series of tags used the UBlox LEAS4S chip (~up to tag 541), newer tags use UBlox LEAS6S. The series developed in 2012 will use the Max6G receivers. From 2012 receiver versions will be stored in the database.
The UBX protocol is used for programming firmware, packaging and processing data.
For more information about the receivers themselves see the ublox documentation.
Ublox 4 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/ANTARIS_Protocol_Specification%28GPS.G3-X-03002%29.zip]
Ublox 6 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/u-blox6_ReceiverDescriptionProtocolSpec_%28GPS.G6-SW-10018%29.pdf]

UvA-BiTS Tracking Data

2021-01-21T15:53:41Z

Stacy shinneman: /* gps.ee_tracking_speed_limited */

== UvA-BiTS Tracking Data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_tracking_speed_limited ===

Shows the tracking data that your projects allow you to see.

Each row represents a GPS fix as it is interpreted from the data that comes via the GPS tracker, plus it adds speeds.

It replaces old views gps.uva_tracking_limited, gps.uva_tracking_speed_3d_limited and gps.uva_tracking_speed. This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time when the fix was taken. UTC (i.e.: no timezone). Part of the unique identifier of the GPS fix.
|-
|latitude||float8||YES||Latitude of the GPS fix, decimal degrees (WGS1984 Ellipsoid)
|-
|longitude||float8||YES||Longitude of the GPS fix, decimal degrees (WGS1984 Ellipsoid)
|-
|altitude||int4||YES||Altitude above sea level measured by GPS tag in meters (WGS1984 Ellipsoid)
|-
|pressure||int4||YES||Pressure measured by GPS tag sensor in Pascals
|-
|temperature||float8||YES||Temperature measured by GPS tag sensor in Celsius
|-
|h_accuracy||float8||YES||Horizontal accuracy (meters)
|-
|v_accuracy||float8||YES||vertical accuracy (meters)
|-
|x_speed||float8||YES||meters/second
|-
|y_speed||float8||YES||meters/second
|-
|z_speed||float8||YES||meters/second
|-
|gps_fixtime||float8||YES||GPS fix time (also known as Time to Fix) in seconds
|-
|userflag||int4||YES||Acceptable = 0; Data flagged as unacceptable by user if not equal to 0.
|-
|satellites_used||int2||YES||Number of satellites used for fix the GPS measurement
|-
|positiondop||float8||YES||Positional DOP (Dilution of Precision), unitless
|-
|speed_accuracy||float8||YES||
|-
|location||geometry||YES||postgreSQL geometry; to allow GIS operations
|-
|vnorth||float8||YES||- vx * sin(lat) * cos(long) - vy * sin(lat) * sin (long) + vz * cos(lat), m/s
|-
|veast||float8||YES||- vx * sin(lon) + vy * cos(lon), m/s
|-
|vdown||float8||YES||- vx * cos(lat) * cos(lon) - vy * cos(lat) * sin(lon) - vz * sin(lat), m/s
|-
|speed||float8||YES||- vx * sin(lon) + vy * cos(lon)
|-
|speed_3d||float8||YES|| sqrt(d.x_speed^2 + d.y_speed^2 + d.z_speed^2), m/s
|-
|speed_2d||float8||YES|| Speed on 2D, m/s
|-
|direction||numeric||YES|| Heading in degrees; 0 is north, 90 is east
|-
|altitude_agl||float8||YES|| Reported GPS altitude MINUS terrain elevation on the coordinate based on Digital Elevation Model data used in the e-ecology database (for further information see: [[Elevation]]
|}

=== ''gps.''ee_acceleration_limited ===

Acceleration is measured in blocks of 1-200 measurements (records). The date_time mentioned in this block is the date and time of the first acceleration measurement. Sequential measurements within a block are numbered with an index 1... (max 200). Every row represents one measurement in 3 directions (X,Y,Z). The order is important within each block, so you should always sort the acceleration measurements by block and index. If you need the exact time for each row, you can calculate it from the index, the sample frequency (which is almost always 20 Hz) and the date_time of the first row in the block.

The date_time of the first block is the time of the GPS measurement preceding the accelerometer measurement. If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

Therefore, to know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time in gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker.

This table structure has ''not'' changed with respect to our previous database.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the fix. Part of the unique identifier of the GPS fix.
|-
|date_time||timestamp||NO||Time of the acceleration block that this measurement belongs to (time in UTC, no time zone).
|-
|index||int2||NO||Sorting order of the measurement within the acceleration block (each block starts at 0).
|-
|x_acceleration||int2||YES||Acceleration measured on the x (or surge) axis (range between -2666 and 2666).
|-
|y_acceleration||int2||YES||Acceleration measured on the y (or sway) axis (range between -2666 and 2666).
|-
|z_acceleration||int2||YES||Acceleration measured on the z (or heave) axis (range between -2666 and 2666).
|}

=== ''gps.''ee_acc_start_limited ===

Each row represents the start of a block of acceleration measurements when a GPS fix is not available at that acceleration block's time.

If there is no valid GPS measurement, this time is calculated from the last GPS measurement and the time of the internal clock of the tracker.

To know all valid dates and times that mark a valid block for a tracker, you need to consider all of the values in column date_time on gps.ee_tracking_speed_limited for the given tracker, column date_time on gps.ee_acc_start_limited, and all the values on column date_time for that tracker. And the order is important within each block, so you should sort the acceleration measurements by block and index.

Added in firmware 1.0.2.7.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker that reported the acceleration measurement.
|-
|date_time||timestamp||NO||Time to mark the start of the acceleration block.
|-
|line_counter||int4||NO||Number of lines associated with this set of accelerations.
|-
|timesynced||int2||YES||The source of the time that marks the acceleration block (0=not synced, 1=user set, 2=Gps Synced).
|-
|ascii||int4||YES||Acceleration measurement interval.
|-
|accsn||int4||YES||Number of samples (0-65536).
|-
|f||int2||YES||Acceleration sampling frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz).
|}

== Bird data ==
=== ''gps.''ee_individual_limited ===

Each row shows bird's details for all birds that your projects allow you to see.

This view replaces the old gps.uva_individual_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|ring_number||varchar||NO||Ring number.
|-
|species_latin_name||varchar||NO||Latin name of the species of the bird.
|-
|colour_ring||varchar||YES||String for ring colour.
|-
|sex||varchar||NO||Sex (M=male, F=female, X=unknown/unspecified)
|-
|mass||numeric(5,0)||YES||Mass (grams)
|-
|start_date||timestamp||NO||Date when the bird was first tracked with an UvA-BiTS logger.
|-
|end_date||timestamp||NO||Date at which the bird is no longer relevant for the tracking project.
|-
|remarks||text||YES||Text for user remarks
|-
|individual_id||int8||NO||Bird unique id. Included just to enable joins.
|}

=== ''gps.''ee_species_limited ===

Each row represents a species that you can assign to your birds to include them in your project. No bird in the system can exist without being linked to one of the species from this list. If you are tagging a bird whose species does not exist in the list, please [http://www.uva-bits.nl/contact/ contact] us.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|abbr||varchar||NO||Unique abbreviation for this species.
|-
|english_name||varchar||YES||Name of the species in English.
|-
|latin_name||varchar||NO||Name of the species in Latin. It is always filled in.
|-
|species_id||int8||NO||Unique id. Added to allow joins.
|}

=== ''gps.''ee_nest_limited ===
This view shows nests you have access to. A nest is always for a bird with a tracker (so a track session must exist for that bird), and must contain a short name (a reference name for you to use) and a latitude and longitude pair.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
|-
|latitude||numeric||YES||Latitude of the coordinates where the nest was found, decimal degrees.
|-
|longitude||numeric||YES||Longitude of the coordinates where the nest was found, decimal degrees.
|-
|start_date_time||timestamp||NO||When the nest was spotted.
|-
|end_date_time||timestamp||NO||When the nest ceases to be relevant.
|-
|found_by_whom||varchar||YES||Place to credit people involved in finding the nest.
|-
|remarks||text||YES||Free text.
|-
|location||bytearray||YES||Calculated Geometry of the coordinates where the nest was found.
|-
|nest_id||int8||NO||Unique id. Added to allow joins to nest_inhabitant_limited.
|}

=== ''gps.''ee_nest_inhabitant_limited ===
This view shows who lives on which nest, for birds you have access to.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|reference_name||varchar||NO||A string belonging to the nest that you can use to identify that nest. Because it is NOT enforced to be unique, DO NOT USE THIS FIELD FOR JOINING.
|-
|ring_number||varchar||NO||The ring number of a bird on a nest.
|-
|device_info_serial||int8||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the bird's trackers within that period are resolved so that they can be shown here. This is the device_info_serial of a tracker.
|-
|key_name||varchar||NO||Following the link from a bird to her track sessions, and considering the nest's start and end time, the track sessions' projects within that period are resolved so that they can be shown here. This is the key_name of a project.
|-
|nest_id||int8||NO||Reference to the nest id. Added to allow joins.
|-
|individual_id||int8||NO||Reference to the individual id. Added to allow joins.
|-
|track_session_id||int8||NO||Reference to a resolved track session id. Added to allow joins.
|}
== Tracker and project data ==

=== ''gps.''ee_project_limited ===

Each row shows the details of each project, for the projects you are allowed to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Unique key name, with the form <species>_<place>
|-
|station_name||varchar||NO||Where the receiving station is
|-
|start_date||timestamp||NO||Project start date
|-
|end_date||timestamp||NO||Date at which the project is not relevant any more
|-
|description||text||YES||String for free description. Updatable by the user.
|-
|project_id||int8||NO||Project unique id. Included just to enable joins.
|-
|parent_id||int8||YES||Reference to the parent project in the hierarchy.
|}

=== ''gps.''ee_tracker_limited ===

Each row shows tracker details for all trackers that your projects allow you to see.

This view replaces the old gps.uva_device_limited.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int8||NO||Serial number of the tracker
|-
|firmware_version||varchar||YES||String for firmware version
|-
|mass||numeric(4,2)||YES||Mass (in grams)
|-
|start_date||timestamp||NO||Date before which the device is not relevant
|-
|end_date||timestamp||NO||Date at which the device is not relevant any more
|-
|x_o||numeric(30,6)||YES||Accelerometer calibration: offset on the x (or surge) axis
|-
|x_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the x (or surge) axis
|-
|y_o||numeric(30,6)||YES||Accelerometer calibration: offset on the y (or sway) axis
|-
|y_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or sway) axis
|-
|z_o||numeric(30,6)||YES||Accelerometer calibration: offset on the z (or heave) axis
|-
|z_s||numeric(30,6)||YES||Accelerometer calibration: sensitivity on the y (or heave) axis
|-
|tracker_id||int8||NO||Tracker unique id. Included just to enable joins.
|}

=== ''gps.''ee_track_session_limited ===

A track session links a tracker and a bird for a given period of time. Therefore, it is '''mandatory for projects''' to supply this piece of '''metadata''' before they can see any tracking data related to a tracker they own.

In order to create a track session, the project must create first an entry in the system for the bird that they are attaching the tracker to, and this is the way for the UvA-BiTS community to make sense of the tracking data that we are receiving. This is why providing the metadata is so important.

By editing the start and end dates, administrators of the project that this track session belongs to, can delimit (and therefore, establish) when the coupling makes sense (or is to be understood as useful).

Whenever a tracker is attached to a bird, a track session must be started (start_date) to indicate that the tracking data that is received is already live data. If the tracker is separated from the bird, the tracking data ceases to make sense, so the track session's end date should be set to denote it has ended. New track sessions can be started if the tracker is re-attached to a bird (possibly the same as before, or maybe a different one) afterwards.

Each row shows track session's information. There exists a row for each track session that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the track session belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker this track session involves.
|-
|ring_number||varchar||NO||Ring number of the bird that this track session involves.
|-
|start_date||timestamp||NO||Date before which the track session is not relevant
|-
|end_date||timestamp||NO||Date at which the track session is not relevant any more
|-
|remarks||text||YES||Free text for comments
|-
|start_latitude||numeric(11,8)||NO||Latitude where the bird was tagged, decimal degrees
|-
|start_longitude||numeric(11,8)||NO||Longitude where the bird was tagged, decimal degrees
|-
|project_id||int8||NO||Reference to the project this track session belongs to. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker this track session involves. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual this track session involves. Included just to enable joins.
|-
|track_session_id||int8||NO||Unique identifier of the track session. Included just to enable joins.
|}

== Relation to projects ==

=== ''gps.''ee_user_limited ===

Each row shows a colleague in a project that you belong to, indicating her role and status there.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||Project name
|-
|email||varchar||NO||e-mail address
|-
|login_name||varchar||NO||Login name
|-
|name||varchar||NO||(first) name
|-
|surname||varchar||NO||Surname
|-
|role||varchar||NO||Role in the project
|-
|status||varchar||NO||Status of the membership
|-
|user_id||int8||NO||Unique identifier of the user. Included to enable joins.
|}

=== ''gps.''ee_tracker_ownership_limited ===

Each row that you see expresses that a tracker belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the tracker belongs to.
|-
|device_info_serial||int8||NO||Serial number of the tracker.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|tracker_id||int8||NO||Reference to the tracker. Included just to enable joins.
|}

=== ''gps.''ee_individual_ownership_limited ===

Each row that you see expresses that a bird belongs to one of the projects you are involved with.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|key_name||varchar||NO||The project that the individual belongs to.
|-
|ring_number||varchar||NO||Ring number of the bird.
|-
|project_id||int8||NO||Reference to the project. Included just to enable joins.
|-
|individual_id||int8||NO||Reference to the individual. Included just to enable joins.
|}

== Engineering data ==

'''Note:''' These views will show no rows for trackers not yet attached to birds.

=== ''gps.''ee_comm_limited ===

Each row indicates status and result of communications with ground stations for each tracker that your project allows you to see.

Only for newer firmware (v.1.0.1.4 and higher).

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
| device_info_serial || int4 || NO || Serial number of the tracker.
|-
| date_time || timestamp || NO || timestamp of start communication
|-
| date_time_end || timestamp ||YES|| timestamp of end communication
|-
| date_time_utc || timestamp ||YES|| date_time in utc (if available)
|-
| date_time_end_utc || timestamp ||YES|| date_time_end in utc (if available)
|-
| action || text||YES|| JOINED / RESCHEDULE ??
|-
| bytes_communicated || int4 ||YES|| Number of bytes communicated (total of all bytes per DATA line)
|-
|status || text ||YES|| READY, TIMEOUT, LOWPOWER, MISSING
|-
| network_join_time || int4||YES||milliseconds
|-
| number_identical_lines || int4 ||YES|| Number identical lines in data block. Gives indication
|-
| non_ascii_char || int4 ||YES|| Number of non ascii characters (indicated by question-mark ?)
|-
| sts_time || timestamp ||YES|| derived from S0 hh,mi
|-
| timesynced || int2||YES|| TimeSynced (0=not sy:wnced, 1=user set, 2=Gps Synced)
|-
|devicemode || int2||YES||DeviceMode (0-99). This describes whether the device is in a special operational mode. 0 = normal operational mode; 80 hex = Memory save mode (memory is almost full and all activity is drastically reduced so the circular memory probably won't be overwritten or, if it does, it will be kept to a minimum)
|-
| vbat || float8 ||YES|| Battery voltage in V (0-~4.200V)
|-
|logb || int4 ||YES|| logged bytes, waiting for download (0-4Mbyte)
|-
| logp || int2 ||YES|| Logged percentage (units 0.1%) of memory waiting to be downloaded (0-1000)
|-
| gpsii || int4 ||YES|| GpsIval, current (0-65535)
|-
|comii || int4||YES|| ComIval, current (0-65535)
|-
|accii || int4||YES|| AcceleroIval, current (0-65535)
|-
|swddd || int4||YES|| User switches, current (0-65535)
|-
|last_gps_fix ||timestamp||YES||timestamp of last fix
|-
|latitude||float8||YES||Latitude of last fix. Decimal degrees
|-
|longitude||float8||YES||Longitude of last fix. Decimal degrees
|}

=== ''gps.''ee_energy_limited ===

Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|date_time||timestamp||NO||Date and time of the reported data
|-
|timesynced ||int2||YES||TimeSynced (0=not synced, 1=user set, 2=Gps synced)
|-
|devicemode ||int2||YES||(0-99)
|-
|vsll ||float8||YES||Solar Voltage during charging in mV (0-~2500)
|-
|vbat ||float8||YES||Battery Voltage in V (0-~4.200)
|-
|ssw ||int2||YES|| Solar Cell Switches ... (0-255) fw1022 and up
|-
| ||||YES||extended fields if special user switch is on
|-
|vsoo ||float8||YES|| Solar voltage open in V (0-~2.500)
|-
|vdrf ||float8||YES||reference voltage Dac in V (0-~2.435)
|-
|vchr ||float8||YES||charge current voltage
|-
|g ||int2||YES|| Gps battery threshold OK
|-
|c || int2||YES||Com battery threshold OK
|-
|m || int2||YES||Charge Mode (0,1,2 Com battery threshold OK
|-
|pressure || int4||YES|| in pascal (1mBar = 100Pa, 1pressuremeter = ~12Pa)
|-
|temperature || float8||YES|| in celcius degrees
|}

=== ''gps.''ee_settings_limited ===

Settings are either from C0-C1-C9 lines or parsed from uplink communication lines.
If C0-C1 is parsed its values are compared with the previous value and if the settings are the same, then no new records need to be added.
If uplink settings are parsed, then wait until the last setting has been seen (DL command or EOF); then store all the new settings in one record. Only for newer firmware.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||
|-
|date_time||timestamp||NO||Derived timestamp from Hh:mi fields in C0 combined with computer timestamp in JOINED line
|-
|timesynced || int2 ||YES||
|-
|timezone_time || int2 ||YES||
|-
|swddd || int4||YES|| SpecialUserSwitches
|-
|gpsnavmode || int2||YES|| GpsNavMode 0-8. This is the Kalman filter used for GPS navigation mode often called the "Dynamic platform model". The following codes are provided by the GPS chip manufacturer for the UBlox GPS chip LEAS4S: 1 = stationary, 2 = pedestrian, 3 = automotive, 4 = sea, 5 = airborne 1G, 6 = airborne 2G, 7 = airborne 3G. The UBlox GPS chip LEAS6S uses the following codes: 0 = portable, 2 = stationary, 3 = pedestrian, 4 = automotive, 5 = sea, 6 = airborne with < 1g acceleration, 7 = airborne with < 2g acceleration, 8 = airborne with < 4g acceleration
|-
|gpsreceivermode || int2 ||YES|| GpsReceiverNode 0-3
|-
|gpsfixtime || int2||YES|| GpsFixTime 0-255
|-
|gps_ivaln_on_h || int2||YES|| hours Gps IvalN on
|-
|gps_ivaln_on_m || int2||YES|| minutes Gps IvalN on
|-
|gps_ivaln || int4||YES|| GpsIvalN (0-65530)
|-
|gps_ivale_on_h || int2||YES|| hours Gps IvalE on
|-
|gps_ivale_on_m || int2||YES|| minutes GpS IvalE on
|-
|gps_ivale || int4||YES|| GpsIvalE
|-
|com_ivaln_on_h || int2||YES|| hours Com IvalN on
|-
|com_ivaln_on_m || int2||YES|| minutes Com IvalN on
|-
|com_ivaln || int4||YES|| ComIvalN (0-65530)
|-
|com_ivale_on_h || int2||YES|| hours Com IvalE on
|-
|com_ivale_on_m || int2||YES|| minutes Com IvalE on
|-
|com_ivale || int4||YES|| ComIvalE
|-
|accii || int4||YES|| Accelero measurement interval
|-
|accbn || int4||YES|| Accelero number of blocks
|-
|accsn || int4||YES|| Accelero number of samples (0-65536)
|-
|f || int2||YES||Accelero sample frequency (0=20Hz, 1=10Hz, 2=5Hz, 3=1Hz)
|-
|pdp || int2||YES|| PdopMsk (3-250)
|-
|pac||int2||YES|| PaccMsk (10-100)
|-
|tdp ||int2||YES|| TdopMsk (3-250)
|-
|tac ||int2||YES|| TaccMsk (30-300)
|-
|ga || int2||YES|| Additional Fixes GpsIval mode (3-25) before interval fix is logged
|-
|aol || int2||YES||AccMemOffLev (100 - 950) units 0.1%; if Memfull above this level all Accelero activity will be overruled and switched off untill MemFull < level - 5
|-
|eventnum || text||YES|| 12 eventnumbers (0-0xff) only printed if not zero, therfore converted to text
|}

=== ''gps.''ee_trackingfile_limited ===

Each row shows the status of the last attempt to parse (process) the raw data of a file for a given tracker, for each tracker that your projects allow you to see.

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|id||int8||NO||Unique identifier of the row
|-
|date_reported||date||NO||Date that the file arrived
|-
|filename||varchar||NO||Name of the file
|-
|last_modified_date||timestamp||NO||Date that the file was last modified on
|-
|device_info_serial||int4||YES||Serial number of the tracker this file comes from
|-
|filesize||int8||NO||Size of the file in KB
|-
|parsed_date||timestamp||YES||Date and time that the file was last attempted to be parsed on
|-
|version||int4||NO||Optimistic locking field. Ignore it as a user.
|-
|last_found_date||timestamp||YES||Last time that a process that runs to look for new files saw the file
|-
|n_lines||int4||YES||Count of lines found in the file last time it was processed
|-
|n_data_lines||int4||YES||Count of actual data lines, out of the total lines
|-
|n_tracking_records||int4||YES||Count of GPS fixes found during the parsing process
|-
|n_communication_records||int4||YES||Count of communication records found during the parsing process
|-
|n_energy_records||int4||YES||Count of energy records found during the parsing process
|-
|n_acceleration_records||int4||YES||Count of acceleration records found during the parsing process
|-
|n_acceleration_start_records||int4||YES||Count of acceleration start records found during the parsing process
|-
|n_errors||int4||YES||Count of errors encountered during the parsing process
|-
|n_settings_records||int4||YES||Count of settings records found during the parsing process
|-
|filepath||text||NO||Directory path of the file on the server
|}

=== ''gps.''ee_warning_limited ===

Since firmware version 1.0.2.7 this table was added for events that do not occur very often (C10, C20) or errors that occur during parsing.

For example all CRC failures can be selected with :

select * from gps.ee_warning_limited
where warning = 'CRC8 failure'
order by device_info_serial;

{| class="wikitable"
! COLUMN_NAME
! TYPE_NAME
! IS_NULLABLE
! DESCRIPTION
|-
|device_info_serial||int4||NO||Serial number of the tracker
|-
|filename||text||NO||File name
|-
|linenumber||integer||YES||line number in file
|-
|warning||text||YES||The warning message
|-
|line||text||YES||The actual line contents in the file
|}

== Sharings ==

Projects can now share trackers with other projects during periods of time. This means that people in the receiving projects can access information that these shared trackers provide during those time periods.

The following views allow you to see this shared information:

* Tracking data
** ''gps.''ee_shared_tracking_speed_limited: tracking information (gps, speed), during the shared period, for trackers shared
** ''gps.''ee_shared_acc_start_limited: acceleration start information, during the shared period, for trackers shared
** ''gps.''ee_shared_acceleration_limited: acceleration information for trackers shared
* Metadata
** ''gps.''ee_shared_tracker_limited: information about each shared tracker
** ''gps.''ee_shared_individual_limited: information about birds with shared trackers
** ''gps.''ee_shared_track_session_limited: information about track sessions which involve shared trackers
* Relation to projects
** ''gps.''ee_shared_tracker_sharing_limited: view information for each sharing that your projects allow you to see

== Accelerometer calibration ==

The accelerometer measures the acceleration in 3D (20 times per second). Acceleration is caused by gravity (static component) and by a change in speed (dynamic component).
When the logger is static in horizontal position (solar cells up), gravity (green arrow) shows a heave (vertical acceleration, z) of +1 g (1 g = 9.81 m.s-2) while surge and heave are both 0. When the front of the logger is vertically pointing down (forward acceleration, x), surge is +1g and when it is rolled to the right (sidewards acceleration, y) sway is +1g. Due to gravity, the pitch of a logger is thus translated into the values of heave, surge and sway (or z, x, y).

However, the values are measured in mV instead of g. Before delivering the GPS-loggers, they have been calibrated at UvA. The acceleration (A) expressed in g is calculated from the measured value (M) and the calibration factors offset (''O'') and sensitivity (''S'').

Thus acceleration for heave (z), surge (x) and sway (y) is calculated as:
Az = (Mz-Oz)/Sz ; Ax = (Mx-Ox)/Sx; Ay = (My-Oy)/Sy ;

The values of ''O'' (in general ranging from 0-60, sometimes even 300) and ''S'' (ranging from 1300-1450) are available in the device table of the data base. If not available, use the factory defaults ''O'' = 0 and ''S'' = 1365.

{|
| [[File:accelero1.png|300px]]||[[File:accelero2.png|300px]]|
|}

In the figure above heave (z) is shown in green, surge(x) in red and sway (y) in blue, all values are expressed in g (on the y-axis).

== Data processing and filtering ==
The raw log files are processed before being entered into the database. Each firmware version has a dedicated parser for the task. In certain cases data that is found in the log files is not stored in the database. This can occur for two reasons:

1. The data did not pass the CRC8 checksum (cyclic redundancy check). This is a standard procedure designed to detect accidental changes to raw computer data.

2. If the data has an invalid signature. For example, if a certain data format (eg numeric) or number of values (eg 4 digits) is expected but not returned, this record is not stored in the database.

When a line is deleted, the log file name, line number and the type of error is stored in the table gps.uva_warning102

== Information about GPS receivers ==
The first series of tags used the UBlox LEAS4S chip (~up to tag 541), newer tags use UBlox LEAS6S. The series developed in 2012 will use the Max6G receivers. From 2012 receiver versions will be stored in the database.
The UBX protocol is used for programming firmware, packaging and processing data.
For more information about the receivers themselves see the ublox documentation.
Ublox 4 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/ANTARIS_Protocol_Specification%28GPS.G3-X-03002%29.zip]
Ublox 6 Protocol specification [http://www.u-blox.com/images/downloads/Product_Docs/u-blox6_ReceiverDescriptionProtocolSpec_%28GPS.G6-SW-10018%29.pdf]

BirdSim

2021-01-21T11:21:44Z

Stacy shinneman: /* Introduction */

== Introduction ==

BirdSim is an interactive web-browser embedded application for interactively looking at GPS tracks, from both top-down view and from Birds-eye view. It is based on the [[BirdView]] application, and as such has some similarities. However, it contains some adjustments and additions, allowing for new features such as Touring and location/time dependent (meteorological) data.

BirdSim has two main goals:
* To enable looking at GPS tracks from a birds perspective.
* To dynamically link (meteorological) data to GPS-data.

'''''This service is currently not available.'''''

===Requirements===

To be able to use BirdSim, you need to fulfill a couple of requirements. Most of these requirements are usually already fulfilled, or in case of installing Google Earth browser plugin, very easy.
* You need to run either Microsoft Windows (XP or later) or Maxc OS X 10.4+ as operating system.
* You need to have a compatible web browser. Internet explorer, Google Chrome and Firefox are known to be compatible, but Firefox is recommended since it provides additional functionality such as opening local kmz/kml files.
* You need to have the Google Earth plugin installed. This can be done very simply provided you have administrator rights. It will query you to install the first time you open
* To retrieve data, you need to be logged in with your UvA-Bit password. See [[#Security]] for more details.

We recommend Firefox as browser: While other browser will work too, only Firefox support loading of local KMZ/KML files.

== Control Panel ==

The Google Earth map is controlled by standard mouse interactions, for functions such as zooming, panning and rotation.

The interface for querying and controlling GPS data in BirdSim is located in the Control Panel on the left side of the screen.

===BirdSim tab===
[[image:birdsimControl1.png|thumb|right|260px|Screenshot of the BirdSim Control interface]]

In the BirdSim tab you can find the controls for querying data from the GPS e-Ecology database.

Here we will go over each field, and explain what this does:
* Device Id: This is the serial number of the bird you want to look at. Note that only the device that you have access to are visible.
* Start date/time and End date/time: Here your set the timespan of the data you want to see. The time is in UTC
* Tour Threshold: Measurement points with an average speed (in m/s) under the threshold are grouped together. This option is to clean up the bird tracks and aggregate spots where they were not travelling.
* Tour Speed Multiplier: The default tour speed is the same as the speed of the measurements. Since this can be extremely slow on migration flights, the default speed can be modified. Note that the touring interface also has buttons to speed up the tour, this option is just an alternative.
* Points checkbox: Adds the measurements to the GPS track/tour as icons.
* Lines checkbox: Add straigt, shortest distance lines between the measurement points (green line)
* Spline checkbox: Add a spline (blue line) through the measurement points. It is a Catmull-Rom spline, which goes trough every measurement point. Note that this the route that the tour follows.
* Terrain Exaggeration checkbox: If this checkbox is checked, the height of the terrain and bird are multiplied with a factor of 3, in order to see more details in the terrain. This is interesting for tour trough hilly/mountainous terrain.

* Download Button: This downloads a KML file containing both the tour and the GPS track.
* GPS button: This displays the requested GPS track in the Earth Panel
* Tour Button: This will show the requested GPS tarck as Tour in the Earth Panel. You have to use the Touring interface in the Earth Panel to start the tour.

Finally, there is a TimeRate (in seconds per second) together with a play/pause button.
The reason for these buttons is to have a finer control over the timeslider of Google Earth. The default speed of the Time Slider makes it play very fast, making a migration of a month finish in 10 seconds. By setting the Time rate to for example 3600, the timeslider will advance an hour each second. To update the speed after filling in a new speed, press the play button again.

=== Environment tab ===
[[image:birdsimControl2.png|thumb|right|260px|Screenshot of the BirdSim Environment Control interface]]

''For the Environment tab to be visible, you do need ECMWF access on the UvA-Bits database. If you do not have this, environment tab won't be visible.''

The meteorological data in BirdSim is dynamically queried, based on two factor:
* The viewpoint (both zoom level and position)
* The current time in the Google Earth Timeslider

If the timeslider is giving a single timestamp, this will be used for querying the weather data. If the Timeslider is expanded to a timerange, the '''end''' is used to query the data.
The position and zoom level are used to determine the extend of the queried data. An area is queried that fills the view in the Google Earth panel, depending on the center of the screen, and the zoom level.

The pressure level dropdown is used for both the temperature and the wind barbs, it has no effect on the precipitation.
There are 3 checkboxes for showing and hiding the Wind, Temperature and Precipitation data. A 4th checkbox will show/hide the legend for these visualizations.

Finally, this dynamic querying functionality places a strain on both local processing and server processing, by constantly updating data depending on time and location. Showing a large area by zooming out will load noticeably slower. In order to limit the strain, you can set the update rate to a different value. Furthermore, you have to start the Weather update process yourself by pressing the "start update" button, and vice versa, you can stop the update again by using the "stop update" button.

==== Meteorological data ====
[[Image:ScreenShotMeteo.png|thumb|right|500px|Wind, temperature and precipitation data in BirdView]]
The e-Ecology database contains the weather prediction data acquired from ECMWF. The meteorological data goes back as far as 01-01-2005, although only wind vector is available for that particular period.

ECMWF data properties:
* The prediction is 72 hours into the future
* The data is updated every 12 hours, overwriting previous data.
* The last prediction for any 12 hour period is kept in the database
* The data interval is 3 hours, e.g 00:00, 03:00, etc.
* The resolution is 0.25 degrees latitude and longitude (currently, was 0.5 in the past)
* The current boundaries are Longitude -17 to 45 degrees, Latitude 5 to 62 degrees
* Some variables (temperature, wind speed) have multiple altitude levels. Currently, these include ''ground, ''1000'', ''925'', ''850'', and ''700'', given in milliBar.

===== Temperature =====
By selecting the temperature option, you get the temperature as a slightly transparent filled contour plot, in an area bounded by the bounding box. The temperature range displayed is limited to a -40 to 50 degrees Celsius range, with a small jump at the freezing point, making it stand out slightly. The legend of the temperature can be found by using the ''legend'' option.

Slightly visible white lines have been added to the contour plot to make the different levels of the contour plot easier to see. The levels are 0.33 degrees each, so three different bands will represent one degree in temperature shift.

===== Wind Vector (Wind Barb)=====
Wind speed is an important factor in bird behavior. Flying against a strong wind is very inefficient, while flying along with the wind can save a lot of energy. It can often be seen that the general direction of birds is strongly dependent on the wind speed.

If the wind Vector option can be used to visualize the wind speed. It will display the wind in yellow arrows (vectors) where the direction is the wind direction, and the size of the arrow gives an indication of the wind speed.

When the ''legend'' option is checked, the wind vectors will be extended with very small icons on the base of the arrow. These icons can be clicked on, and a popup table will tell you the exact properties of the wind vector in that location, with speed in both U/V component, and in absolute speed/heading.

===== Precipitation =====
By selecting the precipitation option, you get the accumulated precipitation for the last full 3 hour period (e.g. if you do it for example at 04:00, you get the precipitation from 00:00-03:00). This value is calculated by subtracting the previous accumulated value from the current accumulated value, giving us the accumulated value of that 3 hour period.

The precipitation is visualized as a slightly transparent filled contour plot, in an area bounded by the bounding box. Everything under 1mm of precipitation is ignored, the rest is shown in a scale of dark blue to light blue. The range goes to 15+ mm, everything higher gets the same color as the 15mm level.

The precision of the plot is 0.5 mm per color level.

==== Wadden data ====

There is a Tide data of the Wadden Sea available between 01-01-2010 and 01-06-2010. This data includes sea surface height and flow speeds with a temporal resolution of an hour.
This data is the result of a model run by Deltares, based on meteorological and boundary conditions, not measurements.

===== Tide-Flow =====

The tide flow shows arrows (vectors) representing the current speed and direction of the tide.

===== Tide-Level =====

The tide level shows a detailed contour of the sea surface height in the Wadden sea

'''''Note:''''' Due to an excessive demand on the server, this option is currently disabled and does not work as of this time. See [[#Known problems/Troubleshooting]]

===KML Documents ===

The KML document panel contains the interface to add custom data to BirdView, in the form of KML/KMZ files. These files can be located either on your local machine, in which case you upload them, or they can be retrieved directly form an online location.

Note that while all browsers support loading of online KML/KMZ files, only firefox supports opening local KML and KMZ files.

==== Add online KML/KMZ files ====

Here you can add the url of an online KML or KMZ file, and so load it directly in BirdView.
Either press "enter" or the little magnifying glass icon.

==== Add Local KML/KMZ ====
'''''Note:''''' These are only supported in Firefox

Both the KML and the KMZ options work similar: Press the "browse" button and select the KML or KMZ file you want to upload.
The file is then uploaded and opened in BirdView. Depending on the size of the file, this might take a short while.

=== Other Tabs ===

There are a few other tabs, such as "Find Location", "Google Earth Layers" and "Options". These control some of the Google Earth plugin basic behaviors and properties, and are not documented here.

== Security ==

The BirdSim application itself runs on the https protocol, and requires a UvA-bits user account to access.

The access to different birds is limited per account. You can only look at data that you have access to.
The same goes for the ECMWF weather prediction data. This functionality is limited to accounts with access to the ECMWF data.

== Known problems/Troubleshooting ==

Updated 6-12-2013

Known issues:
* Tide Level option in Wadden data is disabled due to server strain
* Wadden data server is offline

BirdView

2021-01-21T11:21:18Z

Stacy shinneman: /* Introduction */

'''UNDER CONSTRUCTION'''

== Introduction ==

BirdView is an interactive web-browser embedded application that allows users to look at gps tracks from the database, superimposing them of different data sources, such as satellite images and meteorological data.
It is built using the Google Earth browser plugin, which enables it to make use of all the Google earth GIS functionality (pan, zoom, high quality satellite images, etc), and combine it with the ease of a web browser.

'''''This service is currently not available.'''''

===Requirements===

To be able to use BirdView, there are a couple of requirements. Most of these requirements are usually already fulfilled, or in case of installing Google Earth browser plugin, very easy.
* You need to run either Microsoft Windows (XP or later) or Maxc OS X 10.4+ as operating system.
* You need to have a compatible web browser. Internet explorer, Google Chrome and Firefox are known to be compatible, but Firefox is recommended since it provides additional functionality such as opening local kmz/kml files.
* You need to have the Google Earth plugin installed. This can be done very simply provided you have administrator rights. It will query you to install the first time you open
* To retrieve data, you need to be logged in with your UvA-Bit password. See [[#Security]] for more details.

We recommend Firefox as browser: While other browser will work too, only Firefox support loading of local KMZ/KML files.

== Control Panel ==

===BirdView tab===
[[image:interfaceBirdView.png|thumb|right|260px|Screenshot of the BirdView Control interface]]

In the BirdView tab you can find most of the interface controls for selecting the data in the database. It allows you to specify the date and time your interested in, the radar beams you want to include in your queries, the bounding box for the meteorological data, and allows you to switch part of the visualization off and on as desired. Below we go into more detail on different parts of the interface.

==== Time Control ====
The topmost 3 fields, named "Date/Time", "Minutes" and "Animation", and their related buttons, are used to navigate trough time, and enable the user to quickly and easily find periods of interest, such as heavy migration.

When BirdView is started, the date and time default to UTC, rounded down to the hour. UTC is used, because the data in the database uses UTC timestamps, and by using UTC we avoid confusion as to timezones and daylight savings.

To navigate trough time, there are two options:
* To go to a specified date and/or time, you can simply fill in a specific date/time in the box, and press the "go" button. The data in BirdView will automatically update for the new period.
* To move backward and forward trough time, you can use the back and forward arrow icons respectively. The amount of minutes in the box is the size of the time-steps. The data in BirdView will automatically update for the new time-step. The default time-step is 30 minutes, equal to the time between consecutive radar images.
* The animation controls can be used to let BirdView Automatically play trough consecutive time-steps. It uses the same size of time-steps as the "back/forward" controls. The interval in seconds between steps of the animation is equal to the number in the box. Default is 3 seconds, but if large data selections are made, it is recommended to take a larger interval for the animation. The animation can be started and stopped with the play and pause button respectively.

==== Radar Selection ====
'''''note:''''' Radar is only enabled for certain users.

Currently there are three different MPR radars (Glons, Wier and Millingen) in the database, that each have both a high beam and a low beam, making a total of 6 different radar images each update.

Often when you are looking at the MPR data you only want a few of the radar images available. Therefore each radar (both low and high beam) can be selected individually.
This limits the amount of information on your screen, giving a better overview. Furthermore, it reduces the loading strain, since only the data of the selected radars is retrieved from the database.

The radar selection is used for the following visualization options:
* Bird tracks
* Bird Summary Plot
* Radar Images
* Land Mask
* Rain Mask

==== Bounding Box ====
'''''note:''''' Meteorological data is only enabled for certain users.
See also [[#Meteorological data]]

The database contains a large collection of meteorological data, including wind speed, temperature and precipitation, for an area covering Scandinavia to North Africa.
For looking at continental migration, this data can be an invaluable asset, but it also puts a strain on the database, since loading large areas can result in longer loading times.
If you are only interested in the weather in the Benelux, the long loading time is unnecessary. Therefore we provide an interface to specify the area of interest. In this interface the user can set a bounding box by providing the minimum and maximum latitude and longitude coordinates. This way, only the meteorological data inside this area is queried and displayed.

In some cases the meteorological data also has different values at different pressure levels in the atmosphere. For example, on lower pressure levels (i.e. higher in the air) the wind direction and speed can be significantly different compared to closer to the ground. Since birds can fly on different heights, this is can be an important factor when looking at bird migration.

These settings are used for :
* Wind Barbs
* Precipitation
* Temperature

==== MPR Radar data ====
[[image:ScreenShotBirdView1.jpg|thumb|right|400px|Screenshot of BirdView with summary plots of 6 selected radars]]

There is no full temporal coverage for the MPR data in the FlySafe database. Out of every half hour, 10 consecutive radar revolutions are taken.
Therefor the default time step in BirdView is set to 30 minutes.

The following options can be used to display different aspects of the MPR radar data

=====Bird Tracks =====

The MPR radar data contains a large amount of echoes, of which only a smalls set are bird echoes. In order to extract the echoes that are considered to be birds, an algorithm (ROBIN) is used. The algorithm extracts bird echoes, and then connects the echoes that it considers the same bird, resulting on small bird tracks of up to 10 different points.

By selecting the Bird tracks option, the bird tracks for all selected radars are shown. The low beam birds are shown in red, and the high beam birds are shown in green. There currently is no mechanism to identify which bird belongs to which radar, only the high and low beam distinction is made.

===== Bird summary Plot =====

The bird tracks give an indication of the amount of birds detected by the MPR radar, but not of other characteristics, such as average speed and heading for each track.
In order to give an indication of these factors, a dynamically generated bird summary plot can be shown. In this plot, the average speed and direction of each bird track (taken from the entire radar area for a given beam) are plotted on a circular graph. The angular component is the average flight direction, the radial component is the average speed, and the color indicates the density of tracks in the plot exhibiting a particular combination of speed and direction.

The plot itself is dynamically generated by a separate R (http://www.r-project.org) process, which can run preloaded R scripts. In this case, a bird summary plot is generated that gives a general indication of the direction and speed of bird migration. It is also possible to write other R scripts that generate different kinds of plots. This still requires some rewriting of code, but is a relatively simple operation, and could be something for future extensions of BirdView.

If multiple radars or beams are selected, a unique plot for each radar and beam combination is generated. BirdView displays all the generated plots along the left of the Google Earth window (see figure on the right), stacked under one other, in the same order as the radars appear in the radar selection. If the total height of the plots is bigger than the height of the window, the images are automatically resized so that they fit.

Note: If data for the selected radar has not been uploaded for the selected time period, there are no bird tracks with which to produce a plot. In that case a default image (a large red "X") is shown in place of the plot.

===== MPR Images =====
'''''Note:''''' Viewing MPR images requires access
Every half hour, 10 radar revolutions are recorded, and the resulting echo densities are added together and stored as an image. In this image, we can distinguish different types of echoes, such as permanent landmarks, rainclouds, and birds. Since we are mainly interested in the birds echoes, we want to be able to distinguish between different type of echoes. This is done by apply certain filters on the echo density image, forming seperate images. There are currently two type of filters in the FlySafe database, in addition to the unfiltered radar image, namely the ''Land mask'' and ''Rain mask''. However it must be noted that echoes filtered by either the land or rain masks are not per definition land or rain, but only echoes fulfilling certain filtering criteria.

Each of the different type of images can be added in BirdView as an overlay, centered on the location of the radar. These layers can be turned on and off individually, in order to look at different details of the radar image. Furthermore, each of the overlays is partly transparent where there are no echoes, in order to be able to see how the radar image relates to the real-world satellite images.

When adding the overlays, they are always added in the following order, bottom to top:
* bottom is the full echo density image, and this is colored in a range from bright green to red, where red is the most dense echo.
* The middle level is the Rain Mask, which is shown as a Blue layer.
* The top level is the Land Mask, which is shown as a Green layer.

Note:
*The images are mapped to approximately fit their real location in BirdView. The aligning of the radar image and the real-world coordinates was done by hand, by comparing certain landmarks (church towers) and their echoes. Additionally, certain image and coordinate conversion steps produce deviation errors, that can result in errors in both radial and angular position.

----

==== Meteorological data ====
[[Image:ScreenShotMeteo.png|thumb|right|500px|Wind, temperature and precipitation data in BirdView]]
The FlySafe database contains the weather prediction data acquired from ECMWF. The meteorological data goes back as far as 01-01-2005, although only wind vector is available for that particular period.

ECMWF data properties:
* The prediction is 72 hours into the future
* The data is updated every 12 hours, overwriting previous data.
* The last prediction for any 12 hour period is kept in the database
* The data interval is 3 hours, e.g 00:00, 03:00, etc.
* The resolution is 0.25 degrees latitude and longitude (currently, was 0.5 in the past)
* The current boundaries are Longitude -17 to 45 degrees, Latitude 5 to 62 degrees
* Some variables (temperature, wind speed) have multiple altitude levels. Currently, these include ''ground, ''1000'', ''925'', ''850'', and ''700'', given in milliBar.

===== Temperature =====
By selecting the temperature option, you get the temperature as a slightly transparent filled contour plot, in an area bounded by the bounding box. The temperature range displayed is limited to a -40 to 50 degrees Celsius range, with a small jump at the freezing point, making it stand out slightly. The legend of the temperature can be found by using the ''legend'' option.

Slightly visible white lines have been added to the contour plot to make the different levels of the contour plot easier to see. The levels are 0.33 degrees each, so three different bands will represent one degree in temperature shift.

===== Wind Vector (Wind Barb)=====
Wind speed is an important factor in bird behavior. Flying against a strong wind is very inefficient, while flying along with the wind can save a lot of energy. It can often be seen that the general direction of birds is strongly dependent on the wind speed.

If the wind Vector option can be used to visualize the wind speed. It will display the wind in yellow arrows (vectors) where the direction is the wind direction, and the size of the arrow gives an indication of the wind speed.

When the ''legend'' option is checked, the wind vectors will be extended with very small icons on the base of the arrow. These icons can be clicked on, and a popup table will tell you the exact properties of the wind vector in that location, with speed in both U/V component, and in absolute speed/heading.

===== Precipitation =====

By selecting the precipitation option, you get the accumulated precipitation for the last full 3 hour period (e.g. if you do it for example at 04:00, you get the precipitation from 00:00-03:00). This value is calculated by subtracting the previous accumulated value from the current accumulated value, giving us the accumulated value of that 3 hour period.

The precipitation is visualized as a slightly transparent filled contour plot, in an area bounded by the bounding box. Everything under 1mm of precipitation is ignored, the rest is shown in a scale of dark blue to light blue. The range goes to 15+ mm, everything higher gets the same color as the 15mm level.

The precision of the plot is 0.5 mm per color level.

----

===KML Documents ===

The KML document panel contains the interface to add custom data to BirdView, in the form of KML/KMZ files. These files can be located either on your local machine, in which case you upload them, or they can be retrieved directly form an online location.

Note that while all browsers support loading of online KML/KMZ files, only firefox supports opening local KML and KMZ files.

==== Add online KML/KMZ files ====

Here you can add the url of an online KML or KMZ file, and so load it directly in BirdView.
Either press "enter" or the little magnifying glass icon.

==== Add Local KML/KMZ ====
'''''Note:''''' These are only supported in Firefox

Both the KML and the KMZ options work similar: Press the "browse" button and select the KML or KMZ file you want to upload.
The file is then uploaded and opened in BirdView. Depending on the size of the file, this might take a short while.

=== Miscellaneous tabs ===

The tabs "Find location", "Google Earth Layers" and "Options" provide some of additional functionality that is usually found in Google earth.
The functions and options function similar as a normal Google earth/Google maps client.

==== Credits ====

In this overview tab the main partners in the FlySafe Project are listed as images.
Each image is a link to the organizations website, and a mouse-over shows the name of the partner.

Furthermore, there is a checkbox that switches on the logo-icons in the Google earth plugin. These icons show the location of the the partners on the map.
The logo-icons are visible when you start BirdView, but are turned off automatically after 10 seconds.

== Security ==

The security of the UvA-Bits database is an important issue, and as such there are a few security measures in place to protect both the data in the database, and the system itself.

The BirdView application itself is open to the public, and can be accessed by anyone. However, to access any kind of data from the database, a user is required to log in using her UvA Bits username and password.

Some functionality, such as the ECMWF data and the radar images, require additional access rights, and might be disabled for some users.

== Known problems/Troubleshooting ==
''updated 6-12-2013''

* Loading local KML/KMZ only works from Firefox
* Loading local KML/KMZ does not work correctly when embedded in the FlySafe services environment (https://services.e-ecology.sara.nl)

Flight Generator

2021-01-21T11:15:56Z

Stacy shinneman: /* Save selection */

== Flight generator ==

Doarama (http://http://www.doarama.com/) is a GPS track visualization tool. To add a track to Doarama a GPX file is required.
A GPX file of a UvA-BiTS track can be generated using this service.

The service itself can be found via the Virtual Lab [http://www.uva-bits.nl/virtual-lab/] by selecting "Flight Generator". This requires special access: You need to have credentials and your IP address should be registered. Please contact [mailto://UvA-BiTS@uva.nl UvA-BiTS@uva.nl] if you have problems getting access.

To make nice flights you need high resolution data. http://www.doarama.com/view/135589 is an example of a foraging flight of our famous Lesser Black backed Gull that was breeding on Texel and went to Amsterdam for some time (http://www.uva-bits.nl/kml/visiting-amsterdam-for-a-day/) . This example shows nice high resolution data (16 sec interval; 9:00 -14:25), lower resolution data suggesting angulated flight (5 min interval; up to 8:20), and data that are difficult to interpret without landscape reference above the sea (11:00-15:00).

=== System Requirements ===

==== Software Requirements ====
The webservice itself is standard javascript website, and as such is supported by all modern browsers and operating systems.

==== Hardware requirements ====
The Doarama (http://http://www.doarama.com/) visualization is a visual application, and therefore does require a bit of graphical power. Fortunately, most modern computers have decent hardware for graphical acceleration, so this should not be a problem. But for example, if the graphical drivers are not installed correctly, or if remote desktop tools are used, performance can be limited.

Additionally, Doarama require a decent internet connection. They use satellite images as background, and these have to be retrieved from the internet. Furthermore, if large amounts of data are selected in the flight generator, performance issues may arise.

==== The interface ====

=== The Submit Interface ===
[[Image:GPXSubmit.png|thumb|right|400px|abc|]]

==== Select start date and end date ====

For the begin and end date, there are two options: You can either type a date directly, or you can use the calender button. When you use the calender button, a small window pops up where a desired date can be selected. Here you can either use the arrows, or you can use ctrl up/down and ctrl left/right to change the years and months respectively. If you have selected a year and month, you can click on a day to finalize you date selection. For more detailed time tuning, there are also fields for the hours and minutes. Again, you can either type it, or you can use the drop down to select the desired hour.

==== Select tracker ====

To select tracker click on the row. The checkbox should be checked.

==== Save selection ====

Users often are interested in the same tracker. It would be tedious to select the same tracker each time. Therefore we added functionality to save and load specific selections.
To save the current selection, press the ''Save selection'' button, choose a name, by default the current time will be used.
To restore a saved selection, press the ''Restore saved selection'' button, double click on the name or press the icon with the arrow.

As a bonus the last submitted selection is always saved as ''Last used''. This can be used when you want to make the same gpx file again or make small changes without configuring everything again.

Please note that the saved selections are browser-specific, so they will not show up when using a different computer.

=== Limitations ===

The Flight generator will give a error when you try to make a GPXfile with more than 50.000 datapoints.
The error will tell you how many data point are currently selected.

=== Visualization ===

To make a Doarama visualization:

# Download the GPX file from the result page
# Go to http://www.doarama.com and Login
# Click 'Upload' in topleft corner of page
# Select the downloaded GPX file and upload it
# Edit description
# Set 'Activity Type' to 'Fly Bird / Aves'
# Create a visualization
# Press save button
# The url that now appears in your browser is one you can share with others

Flight Generator

2021-01-21T11:15:17Z

Stacy shinneman: /* Hardware requirements */

== Flight generator ==

Doarama (http://http://www.doarama.com/) is a GPS track visualization tool. To add a track to Doarama a GPX file is required.
A GPX file of a UvA-BiTS track can be generated using this service.

The service itself can be found via the Virtual Lab [http://www.uva-bits.nl/virtual-lab/] by selecting "Flight Generator". This requires special access: You need to have credentials and your IP address should be registered. Please contact [mailto://UvA-BiTS@uva.nl UvA-BiTS@uva.nl] if you have problems getting access.

To make nice flights you need high resolution data. http://www.doarama.com/view/135589 is an example of a foraging flight of our famous Lesser Black backed Gull that was breeding on Texel and went to Amsterdam for some time (http://www.uva-bits.nl/kml/visiting-amsterdam-for-a-day/) . This example shows nice high resolution data (16 sec interval; 9:00 -14:25), lower resolution data suggesting angulated flight (5 min interval; up to 8:20), and data that are difficult to interpret without landscape reference above the sea (11:00-15:00).

=== System Requirements ===

==== Software Requirements ====
The webservice itself is standard javascript website, and as such is supported by all modern browsers and operating systems.

==== Hardware requirements ====
The Doarama (http://http://www.doarama.com/) visualization is a visual application, and therefore does require a bit of graphical power. Fortunately, most modern computers have decent hardware for graphical acceleration, so this should not be a problem. But for example, if the graphical drivers are not installed correctly, or if remote desktop tools are used, performance can be limited.

Additionally, Doarama require a decent internet connection. They use satellite images as background, and these have to be retrieved from the internet. Furthermore, if large amounts of data are selected in the flight generator, performance issues may arise.

==== The interface ====

=== The Submit Interface ===
[[Image:GPXSubmit.png|thumb|right|400px|abc|]]

==== Select start date and end date ====

For the begin and end date, there are two options: You can either type a date directly, or you can use the calender button. When you use the calender button, a small window pops up where a desired date can be selected. Here you can either use the arrows, or you can use ctrl up/down and ctrl left/right to change the years and months respectively. If you have selected a year and month, you can click on a day to finalize you date selection. For more detailed time tuning, there are also fields for the hours and minutes. Again, you can either type it, or you can use the drop down to select the desired hour.

==== Select tracker ====

To select tracker click on the row. The checkbox should be checked.

==== Save selection ====

Users often are interested in the same tracker. It would be tedious to select the same tracker each time. Therefore we added functionality to save and load specific selections.
To save the current selection, press the ''Save selection'' button, choose a name, by default the current time will be used.
To restore a saved selection, press the ''Restored saved selection'' button, double click on the name or press the icon with the arrow.

As a bonus the last submitted selection is always saved as ''Last used''. This can be used when you want to make the same gpx file again or make small changes without configuring everything again.

Please note that the saved selections are browser-specific, so they will not show up when using a different computer.

=== Limitations ===

The Flight generator will give a error when you try to make a GPXfile with more than 50.000 datapoints.
The error will tell you how many data point are currently selected.

=== Visualization ===

To make a Doarama visualization:

# Download the GPX file from the result page
# Go to http://www.doarama.com and Login
# Click 'Upload' in topleft corner of page
# Select the downloaded GPX file and upload it
# Edit description
# Set 'Activity Type' to 'Fly Bird / Aves'
# Create a visualization
# Press save button
# The url that now appears in your browser is one you can share with others

Dashboard

2021-01-21T11:10:14Z

Stacy shinneman: /* Monitoring your trackers: Dashboard tab */

The dashboard is the tool to view the details of your projects. If you have administrative rights in a project you can also use it to administer them (e.g.: see trackers, create track sessions, add users...)

= Monitoring your trackers: Dashboard tab =

The direct link: https://dashboard.e-ecology.nl

The Dashboard tab is the facility that helps you to get an overview of which trackers recently made contact and how much data they uploaded to the database.

On the Dashboard there are two tabs, “Overview” and “Files Dashboard”:
* Overview: gives an overview of the tracker activity for all your projects for the last three days.
* Files Dashboard: allows you to examine tracker activity for specific projects for adjustable time periods.

A tracker is highlighted in colour if it uploaded data. The small horizontal bar indicates how much data the tracker uploaded to the database. See the legend on the “Files Dashboard” for an explanation.

''Which trackers are shown on the dashboard?''
; Overview: only active trackers are shown, that is trackers with an active track session
; Files Dashboard: By default only active trackers are shown. Click on “show all trackers” to show all trackers of the project, that is also those without an active track session (e.g. to show trackers that you have turned on, but that have not been put on a bird yet, and therefore do not have an active track session).

= Administering your projects: Projects tab =

If you have administrative rights in a project, then you can see an "Project admin" tab on the dashboard. The direct link: https://services.e-ecology.sara.nl/dashboard/projects/

== Users in your project ==
# Users should register themselves with Uva-Bits on https://services.e-ecology.sara.nl/dashboard/open/signup. Only after users have successfully registered themselves can you make them a member of your project, by following the next steps:
# Click Project Admin on the top of your screen
# On the Overview tab, select the project you want to add a member to
# Click on the Memberships tab
# Click on Add new to add a user membership to the selected project
# You will now see a screen “Create Membership” where you need to fill out:
#* User: the login name of the user (which has already registered himself on UvaBits) that you want to add (when you start typing the user’s name, a list will pop up with users’ names that match what you have just typed in so far)
#* Role: the membership role, i.e. the administrative rights you want to give to the user. Pick one of four options:
#** Enrolled: the user can only see the project exists, but cannot access data or perform administrative rights.
#** Base: assign this role to standard users. They can view and use the project data, but have no administrative rights
#** Admin: assign this role to project administrators. They can add/change user memberships, birds and track sessions
#** Leader: the role with highest administrative rights, typically for the principal investigator. Admin cannot change the administrative rights of a leader, but leaders can change the rights of Admins.
#* Status: select Approved to add the user to your project. If you want to put the membership temporarily on hold, select Pending_User (this will trigger the user to submit it back to you for re-approval).
#* Comments: any comments you like to add to the new membership

== Trackers in your project ==
# Click Project Admin on the top of your screen
# On the Overview tab, select the project for which you want to see the trackers
# Click on the Trackers tab
# This will give you an overview of all the trackers part of your project. Most important are the numbers listed in the column “Tracker”, these correspond to the number on the physical tracker device (this number is called device_info_serial in the database).
# In case you have several projects or collaborate with other projects, you may want to share trackers of your current project with other projects. To do so click “Add new share” and enter the following:
#* Share with project: enter the project name of the project you want to share a tracker with
#* Tracker: the tracker number of the tracker you want to share
#* Tracker ownership: usually, leave it blank (it is meaningful only if you have had a tracker in your project within two or more time periods)
#* Start date / End date: the start and end dates during which you want to share the tracker with another project. This gives you the flexibility to share data for a restricted time period only (e.g.: to prevent disclosing a nest location during mating season).

== Birds in your project ==
# Click Project Admin on the top of your screen
# On the Overview tab, select the project you want to add a bird to
# Click on the Birds tab
# Click on Add new to add a bird to the selected project
# Fill out the following information (if available)
#* ring number: the metal ring number issued by your national ringing centre
#* Colour ring: colour ring code of your bird (if applicable)
#* Species: the latin name of the species (must be selected out the list we recognise; if you miss yours, please contact the system administrators)
#* Sex
#* Mass in grams
#* Start Date: the date at which the bird was either born or seen in the study area for the first time (start of the bird’s lifespan)
#* End Date: the date at which the bird died. Put into the far future if unknown or still alive (end of the bird’s lifespan)
#* Any remarks (e.g. the circumstances under which the bird was found dead)
# Click on Submit to save the bird to the selected project

== Track sessions: assigning a tracker to a bird ==
Trackers have to be assigned to birds through a so-called Track Session, which establishes the link between a tracker and a bird during a period of time. You can create a Track Session like this:

# Click Project Admin on the top of your screen
# On the Overview tab, select the project you want to add a bird to
# Click on the Track sessions tab
# Click on Add new to add a bird to the selected project
# Fill out the following fields:
#* Tracker: the number of the tracker (this number is called device_info_serial in the database)
#* Individual: the ring number of the bird
#* Tagging date: this is the start date of the Track Session
#* End date: this is the date at which you want to end the Track Session (e.g.:this can be either the date at which the bird died, the date at which you removed the tracker during a recapture, or the date at which the tracker was lost for other reasons).
#* Place of capture: select on the map as accurately as possible, or enter a coordinate and click “Bring map to given lat/lon”
#* Remarks: put any remarks here, in particular the reason why the track session was ended.

Note that on the Dashboard overview only trackers with active track sessions in the period you queried for are shown by default, which are trackers for which at least a day within the period you queried for lies between the track session start and end date.

IMPORTANT: Adding track sessions may be the most involving step of the project administration, and several conditions need to be taken into account before a track session can be successfully defined:
# The tracker you want to put on a bird must be owned by your project (i.e. it must be listed under the tab “Trackers”)
# You must have added the bird in the system
# A tracker can be assigned only to one bird at the time. This means you must close any existing track session of the tracker, before you can assign it to a new bird. So if you have removed the tracker from a bird and want to put it on a different bird, you must first close the existing track session of the tracker before you can assign it to a new bird. Closing the track session is simply done by entering the end date of the track session, which should lie before the date at which you want to start the new track session.
# A bird can have assigned only one active track session at the same time, because a bird cannot carry more than one tracker simultaneously. So if you have removed a tracker from a bird, and want to put a new tracker on that same bird, you first have to close the track session of the first tracker, before you can start the track session of the new tracker. Closing a track session is simply done by entering the end date of the previous track session, which should lie before the date at which you want to start the new track session.

To avoid problems we urge users to properly administer their track sessions and bird metadata at all times. So if you replace trackers, or find birds dead with a tracker, always update your track sessions immediately by providing the updated start and end dates of the existing track sessions.

== Including a bird in a nest ==
You can keep track of nests for a bird. You just need to create the nest and tell which bird is living there (and eventually, partners or offspring). The bird must exist in the system before you can create a nest for it.

To create a nest:
# Click Project Admin on the top of your screen
# On the Overview tab, select the project you want to add a nest to
# Click on the Nests tab
# Click on Add new to add a nest for a bird from the selected project
# Fill in the following fields:
#* Ref. name: A string with a useful reference for you. We recommend its being unique for each nest in a project, but the system does NOT enforce it.
#* Birds (max. 2): a maximum of 2 existing birds that you have identified living on that nest
#* Nest location: the important fields are Latitude and Longitude; these will be stored in the database. The map is there as a visual cue and you can double click on the map to fill the Latitude and Longitude. You can search for a location on the textbox above the map to bring the area of the map close to a known point.
#* Start date: when the nest was spotted
#* End date: When the nest ceases to be relevant to you.
#* Found by: place to credit people involved in finding the nest
#* Remarks: free text. Maybe you want to include an address or use it as a log of subsequent observations.
# Finally, click on Submit to store the nest.

E-Ecology Documentation and Guidance

2021-01-21T09:53:21Z

Stacy shinneman: /* UvA-BiTS Services */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[How to access the e-Ecology database]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::'''''This service is currently not available.''''' Access meterological and radar data and overlay with GPS bird tracking data.
:: [[BirdSim]]: 
:::'''''This service is currently not available.''''' More dynamic version of BirdView that allows the user to scroll through weather data in time and space.

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users. '''''This service is currently not available.'''''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[How to access the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

E-Ecology Documentation and Guidance

2021-01-21T09:52:59Z

Stacy shinneman: /* UvA-BiTS Services */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[How to access the e-Ecology database]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::'''''This service is currently not available.''''' Access meterological and radar data and overlay with GPS bird tracking data.
:: [[BirdSim]]: 
:::'''''This service is currently not available.''''' More dynamic version of BirdView that allows the user to scroll through weather data in time and space.

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* '''''This service is currently not available.''''' [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users.
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[How to access the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

How to access the e-Ecology database

2018-04-10T14:41:51Z

Stacy shinneman: /* Setting up an ODBC connection (Windows) */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Via the UvA-BiTS Virtual Lab''' : http://www.uva-bits.nl/virtual-lab/
The database can be accessed from any computer with an internet connection by clicking on the database icon on the Virtual Lab webpage. Access to the postgreSQL database is provided via a phpPgAdmin web service. Login to the database. To see the tables and views available in the database, expand the 'eEcology' folder by clicking directly on it or on the + next to it, then click the + next to the 'eecology' folder, next expand 'Schemas' and finally 'gps'. Now you should be able to expand the database tables and views. Most users will only have access to the data included in tables or views that end with “_limited”; these are subsets of the entire database and include only the data to which you officially have access.

'''Via PgAdmin client software:'''
PgAdmin must installed on your computer: http://www.pgadmin.org/download/
Once you have installed the pgAdmin software on your computer, go to File - Add Server. In the 'New Server Registration' window, enter the following:

Properties tab:
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Enter Username and Password as provided to you by SURF.
Leave all other fields blank. Generally it is not recommended to store the password, especially on shared computer.
SSL tab: SSL = require
Advanced tab: Check "Connect now" and "Restore env?" boxes.

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

'''UvA students and employees working on the UvA network:'''
The database can be directly accessed from computers connected to the university network through PgAdmin software. The computer you are working on must first have the pgAdmin software installed. You can also access the database from other programs (Access, R, etc.) by installing an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.) In pgAdmin go to File - Add Server. In the 'New Server Registration' window, enter the following:

Properties tab:
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Enter Username and Password as provided to you by SURF.
Leave all other fields blank. Do not store your user name and password on shared computers.
SSL tab: SSL = require
Advanced tab: Check "Connect now" and "Restore env?" boxes.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pubserv.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pubserv.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pubserv.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2018-04-10T14:41:31Z

Stacy shinneman: /* Setting up an ODBC connection (Windows) */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Via the UvA-BiTS Virtual Lab''' : http://www.uva-bits.nl/virtual-lab/
The database can be accessed from any computer with an internet connection by clicking on the database icon on the Virtual Lab webpage. Access to the postgreSQL database is provided via a phpPgAdmin web service. Login to the database. To see the tables and views available in the database, expand the 'eEcology' folder by clicking directly on it or on the + next to it, then click the + next to the 'eecology' folder, next expand 'Schemas' and finally 'gps'. Now you should be able to expand the database tables and views. Most users will only have access to the data included in tables or views that end with “_limited”; these are subsets of the entire database and include only the data to which you officially have access.

'''Via PgAdmin client software:'''
PgAdmin must installed on your computer: http://www.pgadmin.org/download/
Once you have installed the pgAdmin software on your computer, go to File - Add Server. In the 'New Server Registration' window, enter the following:

Properties tab:
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Enter Username and Password as provided to you by SURF.
Leave all other fields blank. Generally it is not recommended to store the password, especially on shared computer.
SSL tab: SSL = require
Advanced tab: Check "Connect now" and "Restore env?" boxes.

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

'''UvA students and employees working on the UvA network:'''
The database can be directly accessed from computers connected to the university network through PgAdmin software. The computer you are working on must first have the pgAdmin software installed. You can also access the database from other programs (Access, R, etc.) by installing an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.) In pgAdmin go to File - Add Server. In the 'New Server Registration' window, enter the following:

Properties tab:
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Enter Username and Password as provided to you by SURF.
Leave all other fields blank. Do not store your user name and password on shared computers.
SSL tab: SSL = require
Advanced tab: Check "Connect now" and "Restore env?" boxes.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pubserv.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.
If you are having problems setting up the connection, go back to Step 1 above and make sure that you have installed the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pubserv.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pubserv.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

How to access the e-Ecology database

2018-04-10T14:41:09Z

Stacy shinneman: /* Setting up an ODBC connection (Windows) */

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Via the UvA-BiTS Virtual Lab''' : http://www.uva-bits.nl/virtual-lab/
The database can be accessed from any computer with an internet connection by clicking on the database icon on the Virtual Lab webpage. Access to the postgreSQL database is provided via a phpPgAdmin web service. Login to the database. To see the tables and views available in the database, expand the 'eEcology' folder by clicking directly on it or on the + next to it, then click the + next to the 'eecology' folder, next expand 'Schemas' and finally 'gps'. Now you should be able to expand the database tables and views. Most users will only have access to the data included in tables or views that end with “_limited”; these are subsets of the entire database and include only the data to which you officially have access.

'''Via PgAdmin client software:'''
PgAdmin must installed on your computer: http://www.pgadmin.org/download/
Once you have installed the pgAdmin software on your computer, go to File - Add Server. In the 'New Server Registration' window, enter the following:

Properties tab:
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Enter Username and Password as provided to you by SURF.
Leave all other fields blank. Generally it is not recommended to store the password, especially on shared computer.
SSL tab: SSL = require
Advanced tab: Check "Connect now" and "Restore env?" boxes.

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

'''UvA students and employees working on the UvA network:'''
The database can be directly accessed from computers connected to the university network through PgAdmin software. The computer you are working on must first have the pgAdmin software installed. You can also access the database from other programs (Access, R, etc.) by installing an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.) In pgAdmin go to File - Add Server. In the 'New Server Registration' window, enter the following:

Properties tab:
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Enter Username and Password as provided to you by SURF.
Leave all other fields blank. Do not store your user name and password on shared computers.
SSL tab: SSL = require
Advanced tab: Check "Connect now" and "Restore env?" boxes.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pubserv.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it. If you are having problems setting up the connection, go back to Step 1 above and make sure that you have the most recent version of the PostgreSQL ODBC driver.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pubserv.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pubserv.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

E-Ecology Documentation and Guidance

2018-04-05T08:45:00Z

Stacy shinneman: /* UvA-BiTS Services */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[How to access the e-Ecology database]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::Access meterological and radar data and overlay with GPS bird tracking data. '''''This service is currently not available.'''''
:: [[BirdSim]]: 
:::More dynamic version of BirdView that allows the user to scroll through weather data in time and space. '''''This service is currently not available.'''''

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users. '''''This service is currently not available.'''''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[How to access the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

Setting up a connection to the e-Ecology database

2018-02-22T11:42:52Z

Stacy shinneman: Replaced content with "Disregard: I created this page by mistake and now cannot delete - Stacy Feb 2018"

Disregard: I created this page by mistake and now cannot delete - Stacy Feb 2018

Connecting to the database

2018-02-22T11:41:43Z

Stacy shinneman:

Disregard: I created this page by mistake and now cannot delete - Stacy Feb 2018

E-Ecology Documentation and Guidance

2018-02-22T10:43:47Z

Stacy shinneman: /* Other e-Ecology Datasets */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[How to access the e-Ecology database]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::Access meterological and radar data and overlay with GPS bird tracking data. ''This service is currently not available.''
:: [[BirdSim]]: 
:::More dynamic version of BirdView that allows the user to scroll through weather data in time and space. ''This service is currently not available.''

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users. ''This service is currently not available.''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[How to access the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

E-Ecology Documentation and Guidance

2018-02-22T10:43:34Z

Stacy shinneman: /* UvA-BiTS Data */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[How to access the e-Ecology database]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::Access meterological and radar data and overlay with GPS bird tracking data. ''This service is currently not available.''
:: [[BirdSim]]: 
:::More dynamic version of BirdView that allows the user to scroll through weather data in time and space. ''This service is currently not available.''

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users. ''This service is currently not available.''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[Setting up a connection to the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

How to access the e-Ecology database

2018-02-22T10:43:14Z

Stacy shinneman: moved Connecting to the UvA-BiTS database to How to access the e-Ecology database

=General Information=

The UvA-BiTS database is a spatial postgreSQL database([http://www.postgresql.org/]). In order to access the database you must first have a user name and password provided to you by SURF.

=How to access the database=
'''Via the UvA-BiTS Virtual Lab''' : http://www.uva-bits.nl/virtual-lab/
The database can be accessed from any computer with an internet connection by clicking on the database icon on the Virtual Lab webpage. Access to the postgreSQL database is provided via a phpPgAdmin web service. Login to the database. To see the tables and views available in the database, expand the 'eEcology' folder by clicking directly on it or on the + next to it, then click the + next to the 'eecology' folder, next expand 'Schemas' and finally 'gps'. Now you should be able to expand the database tables and views. Most users will only have access to the data included in tables or views that end with “_limited”; these are subsets of the entire database and include only the data to which you officially have access.

'''Via PgAdmin client software:'''
PgAdmin must installed on your computer: http://www.pgadmin.org/download/
Once you have installed the pgAdmin software on your computer, go to File - Add Server. In the 'New Server Registration' window, enter the following:

Properties tab:
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Enter Username and Password as provided to you by SURF.
Leave all other fields blank. Generally it is not recommended to store the password, especially on shared computer.
SSL tab: SSL = require
Advanced tab: Check "Connect now" and "Restore env?" boxes.

'''Connecting via other programs:'''
You can also connect to the database via other programs (Access, R, etc.) by setting up an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.)

'''Now that I have connected, what do all these tables and views really mean?''' See the UvA-BiTS Data Model page of this wiki.

'''UvA students and employees working on the UvA network:'''
The database can be directly accessed from computers connected to the university network through PgAdmin software. The computer you are working on must first have the pgAdmin software installed. You can also access the database from other programs (Access, R, etc.) by installing an ODBC driver for postgreSQL and establishing a "User Data Source". (See "Setting Up an ODBC Connection" below.) In pgAdmin go to File - Add Server. In the 'New Server Registration' window, enter the following:

Properties tab:
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Enter Username and Password as provided to you by SURF.
Leave all other fields blank. Do not store your user name and password on shared computers.
SSL tab: SSL = require
Advanced tab: Check "Connect now" and "Restore env?" boxes.

=Setting up an ODBC connection (Windows) =

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.

Step 1: Download and install the most recent ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/]

Step 2: Set up an ODBC connection

Do the following (variations may exist between windows operating systems):
# Go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''. If you are having trouble finding it, type "ODBC" into the Search box at the top of the Control Panel window.
# Go to '''User DSN''' (first tab in Data sources).
# Click '''ADD'''.
# In the window '''Create''' '''New Data Source''' select '''PostgreSQL ANSI''' and click '''FINISH'''.
# The '''ODBC Driver Setup''' window opens. '''Data source name''': give your data source a name (e.g. GPS). Keep it simple and easily identifiable since this is the name you will use to identify your database from other applications; this will be the '''ODBC driver name'''.
# In the same window, fill in the following information: '''database''' ''eecology'', '''Server''' ''pubserv.e-ecology.nl'', '''User Name''' ''your user name'', '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' ''your password''. Click '''TEST''' to test your connection. When the window is completed click '''SAVE'''.
# Back in the '''User DSN''' window you should now see the ODBC driver you just named. Click '''OK'''.
After completing these steps many other programs will now recognize your database and be able to communicate with it.

=Setting up an ODBC connection (MacOSX 10.8.5 ) =

The PostgreSQL ODBC driver provided here is a 64-bit driver. It will probably only work if the other components involved in accessing the UvA-BiTS database are also 64-bit. This includes R, iODBC, and libpq. It has been tested on MacOSX 10.8.5 (Mountain Lion) with 64-bit R 3.0.2 (Frisbee Sailing). If you have an older version of MacOSX, and/or a 32-bit version of R installed, this driver may or may not work for you. There's no harm in trying. If it does not work, and you have a 32-bit version of R installed, try using the 64-bit version instead (this will allow R to use more memory as well, possibly speeding up your calculations).

1. Download the ODBC Administrator Tool for Mac OS X v1.0 using the following URL http://support.apple.com/kb/DL895

2. Download the zip file found here [http://horizon.science.uva.nl/public/pgodbc_macosx/]. Unpack the .zip file into your home directory by double clicking it. Your home directory will usually have your name and is located in Macintosh HD > Users next to the little symbol of a house. This will create a folder called "pgodbc".

3. Open the iODBC Administator.app from within Applications.

4. Select the tab "ODBC Drivers" and click on "Add a driver". A new window will open.

5. The first field is "Description of the driver". Type in "PostgreSQL".

6. Click on "Browse" next to the second field to locate and select the driver file you unpacked from the .zip file you downloaded.

7. Go to folder "pgodbc" in your home directory, and select the driver file "psqlodbcw.so"

8. Make sure the driver is defined as "System", i.e., that the "System" option is selected and not the "User" option.

9. Click on "Add" at the bottom of this window, and "OK"

You should now be able to access the database through R using the R script provided.

=Setting up an ODBC connection (Linux) =

1. Install unixODBC and its -dev file
$ sudo apt-get install unixODBC unixODBC-dev

2. Install the postgresql package

$ sudo apt-get install odbc-postgresql

3. Configure the PostgreSQL ODBC driver in the /etc/odbcinst.ini file

$ gksudo gedit /etc/odbcinst.ini

4. On Ubuntu, the /etc/odbcinst.ini file will be blank, so you’ll need to add the data to that configuration file. Add the following to the odbcinst.ini file:

[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/odbc/psqlodbca.so
Setup = /usr/lib/odbc/libodbcpsqlS.so
FileUsage = 1
'''Be careful,''' on x64, the path for the driver and setup should look
something like that /usr/lib/x86_64-linux-gnu/odbc/ (use the command $
mlocate psql to find the appropriate path...)

5. Verify that the system is able to see the driver by running the
following command. It should return the label name [PostgreSQL] if all is
well:

$ odbcinst -q -d

6. Next, configure the /etc/odbc.ini

$ gksudo gedit /etc/odbc.ini

Here is what the text file should look like

[GPStracker]
Driver = PostgreSQL
Description = PostgreSQL connection to the UvA BiTS DataBase
Servername = pubserv.e-ecology.nl
Port = 5432
UserName = user_name
Password = yourpassword
Database = eecology
SSLMode = prefer
OPTION = 3
SOCKET =

You should now be able to connect to the database, for example via RODBC.

= Connecting other programs to the UvA-BiTS database (e.g. R, Access, ArcGIS) =

This section provides information on how to set up an Open Database Connectivity (ODBC) which will enable many software packages to extract data directly from the database.

==Connecting to the database in R==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Here you provide the name of the database file or the user DSN name for a given ODBC data source (as described above).

db.file <-"GPS"

Load the RODBC library. Visit the CRAN repository <http://cran.r-project.org/web/packages/> for a list of contributed packages and instructions for installation. Packages may be installed using the GUI interface or using ?install.packages.

library(RODBC)

Establish a connection to the database, the following code is used if reading from an ODBC data source.

db <- odbcConnect(db.file)

View the table names in the database.

sqlTables(db)

View the field (column) names of a specific table.

sqlColumns(db, table.name)

Query the database, returning data from table gps.uva_tracking_speed_limited, for device_info_serial 1 as dataframe "test".

==Connecting to the database in R (mac) ==

Here is a very short example that will give an idea of how to connect to the database via the ODBC driver you have just established.

Connect to the database in R with the package RODBC, using your username and password.
Install the RODBC package and any dependencies
install.packages("RODBC", dep=T)
require(RODBC)

Replace "user" and "password" with your own username and password provided by SURF:
my_username <- "user"
my_password <- "password"

Define the connection and connect from within R:
l.connstring <- 'DRIVER=PostgreSQL;'
l.connstring <- paste(l.connstring, 'SERVER=pubserv.e-ecology.nl;', sep = '')
l.connstring <- paste(l.connstring, 'PORT=5432;', sep = '')
l.connstring <- paste(l.connstring, 'SSLMODE=require;', sep = '')
l.connstring <- paste(l.connstring, 'DATABASE=eecology;', sep = '')
l.connstring <- paste(l.connstring, paste("USERNAME=", my_username, ";", sep=''), sep = '')
l.connstring <- paste(l.connstring, paste("PASSWORD=", my_password, ";", sep='') , sep = '')

GPS <- odbcDriverConnect(l.connstring, case = "tolower")

From here you can write SQL queries and retrieve the data through R. Here is an example.

To extract GPS data, remember to replace <insert bird id> with the device serial number of one of your tracked birds:
gps_test <- sqlQuery(GPS, query = "SELECT *
FROM gps.ee_tracking_speed_limited s
WHERE s.device_info_serial = <insert bird ID>", as.is=c(2))

==Connecting to the database in Access==
After completing the steps for setting up an ODBC connection, you can connect to the database via Access. You have two options. One is to import the data; once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database. (You must be connected to the network.) In this case you are always viewing updated data that is in the actual SURF database. The first set of instructions is for Access 2007-2010, while the second is for the older version of Access.

# Open a new database in '''Access 2007-2010'''.
# Go to '''External data | ODBC Database'''.
# Select '''Link to the data source by creating a linked table'''. Click OK.
# In '''Select Data Source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

# Open a new database in '''Access 2003'''.
# Go to '''File | Get external data | Link tables'''.
# In '''Files of type''', select '''ODBC databases'''.
# In '''Select data source''' click on the tab '''Machine Data Source'''.
# Select the data source name you created when setting up the ODBC connection (e.g. GPS), then click OK.
# Select the tables you want to import and click OK.

You now have access to your selected tables in the database. Have fun.

Connecting to the UvA-BiTS database

2018-02-22T10:43:14Z

Stacy shinneman: moved Connecting to the UvA-BiTS database to How to access the e-Ecology database

#REDIRECT [[How to access the e-Ecology database]]

E-Ecology Documentation and Guidance

2018-02-22T10:33:14Z

Stacy shinneman: /* UvA-BiTS Services */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[Setting up a connection to the e-Ecology database]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::Access meterological and radar data and overlay with GPS bird tracking data. ''This service is currently not available.''
:: [[BirdSim]]: 
:::More dynamic version of BirdView that allows the user to scroll through weather data in time and space. ''This service is currently not available.''

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema and sharing your tables with other users. ''This service is currently not available.''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[Setting up a connection to the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

ECMWF

2018-02-22T10:26:17Z

Stacy shinneman: /* Historical data */

== ECMWF ==

Datamodel for the tables in the ECMWF schema.

Data is created by the ECMWF and retrieved from the KNMI ftp site as GRIB files. The original GRIB files can also be retrieved from the Flysafe server from :

[https://flysafe-pps.grid.sara.nl/archive/knmi/ecmwf/ ECMWF Data Archive]

More information about the parameters in these GRIB files can be retrieved from the [http://www.ecmwf.int/services/dissemination/3.1/Products_in_GRIB_code.html ECMWF website].

== INVARIABLE ==

The INVARIABLE table contains data that is invariant over time.

{|border=1
|'''LATITUDE''' || degrees * 100 to make this a integer
|-
|'''LONGITUDE''' || degrees * 100 to make this a integer
|-
|Z || 129 Z Geopotential surface/model orography
|-
|LSM || 172 LSM Land sea mask
|-
|LOCATION || Latitude/longitude as GIS location
|}

The GIS LOCATION is only added to this table. This is because always the same values for latitude and longitude are used and then if the GIS location is required for the ANALYSIS or FORECAST tables it can be retrieved by joining to this table on the LATITUDE, LONGUTUDE fields.

== FORECAST and UVA_FORECAST ==

The FORECAST table contains predicted values in steps of 3 hours till 72 hours ahead. The MODELTS is the timestamp for which this prediction applies. Two times a day new Forecast GRIB files will arrive. The newer FORECAST data that arrives for the same timestamp will overwrite the older data.

The FORECAST (and UVA_FORECAST) table itself is a master table that does not contain any data. The inherited tables
FORECAST<YEAR><MONTH> contain the data, but the master FORECAST or UVA_FORECAST table can be used to SELECT the data.

Data till 2009-09 is owned by the UvA and can be accessed by anybody with ECMWF_READ rights by using the FORECAST_UVA table. The FORECAST table itself also references data after 2009-09 that is owned by RNLAF for which special access rights ECMWF_RNLAF_READ are required.

{|border=1
|'''MODELTS'''||Time for which this forecast applies = model run + step hours
|-
|'''LATITUDE'''||degrees * 100 to make this a integer
|-
|'''LONGITUDE'''||degrees * 100 to make this a integer
|-
|STEP||model step (smaller is more recent forecast)
|-
|SSRD||169 SSRD surface solar radiation downwards accumulated
|-
|TP||228 TP total precipitation accumulated
|-
|SP||134 SP surface pressure
|-
|BLH||159 BLH boundary layer height
|-
|TCC||164 TCC total cloud coverage
|-
|U10||165 10U 10 meters wind U-component m/s
|-
|V10||166 10V 10 meters wind V-component m/s
|-
|T2||167 2T 2 metres air temperature
|-
|D2||168 2D 2 metres dew point temperature
|-
|LCC||186 LCC low cloud coverage
|-
|''Pressure level 1000''||
|-
|T_1000||130 T temperature
|-
|U_1000||131 U U-wind component
|-
|V_1000||132 V V-wind component
|-
|GH_1000||156 GH geopotential height
|-
|R_1000||157 R relative humidity
|-
|''Pressure level 925''||
|-
|T_925 real||130 T temperature
|-
|U_925 real||131 U U-wind component
|-
|V_925 real||132 V V-wind component
|-
|GH_925 real||156 GH geopotential height
|-
|R_925 real||157 R relative humidity
|-
|''Pressure level 850''||
|-
|T_850||130 T temperature
|-
|U_850 real||131 U U-wind component
|-
|V_850 real||132 V V-wind component
|-
|GH_850 real||156 GH geopotential height
|-
|R_850 real||157 R relative humidity
|-
|''Pressure level 700''||
|-
|T_700 real||130 T temperature
|-
|U_700 real||131 U U-wind component
|-
|V_700 real||132 V V-wind component
|-
|GH_700 real||156 GH geopotential height
|-
|R_700 real||157 R relative humidity
|}

== ANALYSIS and UVA_ANALYSIS ==

The ANALYSIS and UVA_ANALYSIS table contains values from the Analysis GRIB files at 18, 00, 06 and 12 UTC

Data till 2009-09 is owned by the UvA and can be accessed by anybody with ECMWF_READ rights by using the ANALYSIS_UVA table. The ANALYSIS table itself also references data after 2009-09 that is owned by RNLAF for which special access rights ECMWF_RNLAF_READ are required.

{|border=1
|'''MODELTS'''||Time for which this forecast applies = model run
|-
|'''LATITUDE'''||degrees * 100 to make this a integer
|-
|'''LONGITUDE'''||degrees * 100 to make this a integer
|-
|SP||134 SP surface pressure
|-
|TCC||164 TCC total cloud coverage
|-
|U10||165 10U 10 meters wind U-component
|-
|V10||166 10V 10 meters wind V-component
|-
|T2||167 2T 2 metres air temperature
|-
|D2||168 2D 2 metres dew point temperature
|-
|LCC||186 LCC low cloud coverage
|-
|''Pressure level 1000''||
|-
|T_1000||130 T temperature
|-
|U_1000||131 U U-wind component
|-
|V_1000||132 V V-wind component
|-
|GH_1000||156 GH geopotential height
|-
|R_1000||157 R relative humidity
|-
|Pressure level 925||
|-
|T_925||130 T temperature
|-
|U_925||131 U U-wind component
|-
|V_925||132 V V-wind component
|-
|GH_925||156 GH geopotential height
|-
|R_925||157 R relative humidity
|-
|Pressure level 850||
|-
|T_850||130 T temperature
|-
|U_850||131 U U-wind component
|-
|V_850||132 V V-wind component
|-
|GH_850||156 GH geopotential height
|-
|R_850||157 R relative humidity
|-
|Pressure level 700||
|-
|T_700||130 T temperature
|-
|U_700||131 U U-wind component
|-
|V_700||132 V V-wind component
|-
|GH_700||156 GH geopotential height
|-
|R_700||157 R relative humidity
|}

== COMBINED ==

If you want to use Analysis data if available for a timestimp but otherwise Forecast, the COMBINED VIEW can be used. If a ANALYSIS value is available for a timestamp and latitude/longitude the ANALYSIS value will be used, otherwise the FORECAST value is given.

== Historical data ==

In addition ongoing process of adding ECMWF data since November 11 2007, a historical data set has been loaded. That historical dataset is stored in additional inherited tables FORECASTHISTORY and
ANALYSISHISTORY which inherit from FORECAST and ANALYSIS respectively.

The FORECASTHISTORY does not always contain TP (228 TP total precipitation accumulated). Instead, it contains the following values:

{|border=1
|LSP||142 LSP Large scale precipitation
|-
|CP||143 CP Convective precipitation
|}

The historical dat sets span the following periods and locations

{|border=1
|'''Start'''||'''End'''||'''Area'''||'''Grid'''
|-
|2007-01-01||2007-11-15||62degN 10degW 335degN 20degE||0.25x0.25deg
|-
|2003-02-15||2003-06-15||62degN 10degW 335degN 20degE||0.5x0.5deg
|-
|2005-01-01||2006-12-31||62degN 10degW 335degN 20degE||0.5x0.5deg
|}

== Helpful Functions & Queries ==

The following functions have been defined in Postgres to select the nearest gridpoint in spacetime for the ECMWF analysis and forecast tables.

These are '''ecmwf.forecast_nearest''' and '''ecmwf.analysis_nearest''' and they take a latitude, longitude and timestamp as parameter.

They can be used in the following way, which will return all the fields in the relevant ECMWF data:

SELECT * FROM ecmwf.forecast_nearest( 52.35983, 5.615, timestamp '2008-05-19 13:34:00');

SELECT * FROM ecmwf.analysis_nearest( 52.35983, 5.615, timestamp '2008-05-19 13:34:00');

Note that the model latitude and model longitude are returned as integers and multiples of 100. To return latititude and longitude as a decimal (for example to compare to the latitude and longitude of your GPS data) you need to first convert these fields to decimals:

SELECT f.latitude/100::decimal, f.longitude/100::decimal
FROM ecmwf.forecast_nearest( 52.35983, 5.615, timestamp '2008-05-19 13:34:00') f;

These function can for example be used in a join as follows :

SELECT g.device_info_serial
, g.date_time
, g.latitude
, g.longitude
, g.speed_2d
, g.altitude
, (ecmwf.forecast_nearest( g.latitude, g.longitude,g.date_time)).*
FROM gps.ee_tracking_speed_limited g
WHERE g.device_info_serial = 1
AND g.latitude <> 0
AND g.userflag = 0
ORDER BY g.date_time;

Although these functions were converted to PLPgSQL to speed them up, a faster ways to retrieve the same data is with (thanks to Floris Sluiters):

SELECT device_info_serial, date_time, latitude, longitude, speed_2d, altitude, (x::ecmwf.forecast).*
FROM (SELECT
g.*
,((SELECT f
FROM ecmwf.forecast f
WHERE f.latitude = round(g.latitude * 100 / 25)::integer * 25
AND f.longitude = round(g.longitude * 100 / 25)::integer * 25
AND f.modelts = TIMESTAMP 'epoch' + round(EXTRACT(EPOCH FROM g.date_time)
/ (60*60*3))::integer * INTERVAL '3 hours' limit 1)::ecmwf.forecast) as x
FROM gps.ee_tracking_speed_limited g
WHERE g.device_info_serial = 1
AND g.latitude <> 0
AND g.userflag = 0
ORDER BY g.date_time) t

Below is example query that retrieves data for a model grid point closest to user provided coordinates and for one model time stamp (eg. 12:00) for a series of consecutive days

SELECT f.modelts, f.latitude, f.longitude, f.blh
FROM ecmwf.forecast as f
WHERE f.latitude = round(42.718104*100/25)::integer*25 AND f.longitude = round(11.517727*100/25)::integer*25 and
f.modelts >= '2007-08-01 12:00' and f.modelts <= '2008-08-01 12:00' and date_part('hour', f.modelts) = 12
ORDER BY f.modelts

=== Interpolation ===

The following functions are added so that simple linear interpolation for a specific point in space and time can be done in Postgres.

These are '''ecmwf.forecast_interpolate''' and '''ecmwf.analysis_interpolate''' and they take a latitude, longitude and timestamp as parameter.

They can be used in the same way:

select * from ecmwf.forecast_interpolate( 52.35983, 5.615, timestamp '2008-05-19 13:34:00');

select * from ecmwf.analysis_interpolate( 52.35983, 5.615, timestamp '2008-05-19 13:34:00');

And they can also be used in joins as above. These functions select the 8 surrounding points in latitude, longitude and date_time and interpolate linear in latitude, longitude and date_time successively. The '''interpolate''' functions are much slower then the '''nearest''' functions.

E-Ecology Documentation and Guidance

2018-02-22T10:25:14Z

Stacy shinneman: /* Changes to the UvA-BiTS data model in 2014 */

Last updated: February 2018, Stacy Shinneman

The e-Ecology Virtual Laboratories represent a collaborative effort between the University of Amsterdam Institute for Biodiversity and Ecosystem Dynamics and SURFsara Computing to manage bird tracking data and develop tools for data exploration, analysis and visualization. This wiki is designed to facilitate access to the data and tools within the Virtual Laboratories and to provide a forum for collaboration.

The e-Ecology database contains many different datasets, each of which is contained within its own schema. Prior to 2018, the e-Ecology database was also referred to as the FlySafe database.

== UvA-BiTS Data ==

Bird data collected with the UvA-BiTS Bird Tracking System is managed within the gps schema of the e-Ecology database.

Note: We are continually working on improving UvA-BiTS services, including data processing and Virtual Lab functionality. In some cases, this has enabled us to extract more information from old GPS log files which is then incorporated into the database. These changes may result in slight differences when you reprocess historic data.

::* [[ DB_Views_2015 | e-Ecology UvA-BiTS database for general use (views) ]]
::* [[ DB_Tables_2015 | e-Ecology UvA-BiTS tables behind the views (not accessible for normal use) ]]
::* [[Setting up a connection to the e-Ecology database]]
::* [[Useful Queries]]

==UvA-BiTS Services==

* Visualization Tools

:: [[KMZ Generator|UvA-BiTS KMZ Generator]]: 
:::Produces kmz files of UvA-BiTS bird tracking data to be viewed in Google Earth.
:: [[Tracker Calendar]]: 
:::Calendar overview with daily statistics of GPS-tracker
:: [[Flight Generator]]: 
:::This tool produces GPX files from UvA-BiTS tracking data for dynamic [http://www.doarama.com Doarama] visualizations.
:: [[BirdView]]: 
:::Access meterological and radar data and overlay with GPS bird tracking data. ''This service is currently not available.''
:: [[BirdSim]]: 
:::More dynamic version of BirdView that allows the user to scroll through weather data in time and space. ''This service is currently not available.''

* [[Dashboard]]: These web pages allow UvA-BiTS users to get an overview of the raw data files of GPS bird tracking devices they have access to and to access the dashboard for administering projects.

* [[ Annotation tables ]] for uploading GPS annotation data into your own schema. ''This service is currently not available.''
* [[ SMS tables ]] for storage of SMS messages send by trackers.
* [[ CartoDB ]] usage and limitation of CartoDB server

== Changes to the UvA-BiTS data model in 2014 ==
'''Important'''
In the summer of 2014 we have '''re-arranged the administrative''' part of UvA-BiTS (and e-Ecology) in the database. End users may have to adapt their programs the following way:
:* The '''uva_*_limited''' views that used to exist in the gps schema are gone. Most of them should have been replaced with an equivalent '''ee_*_limited counterpart'''.
:* The new ee_*_limited views show a superset of the same information as before, but enhanced with new project information.
:* Some '''column names''' in the ee_*_limited views may differ from the previous views.
:* There are new limited views:
:** gps.ee_'''project'''_limited: shows your projects' information
:** gps.ee_'''shared'''_*_limited: set of data you are able to see related to trackers that a project is sharing with a project you belong to
:* Special attention: gps.uva_device_limited is now replaced by '''gps.ee_tracker_limited'''. For each tracker, you now see which project provides you access to that tracker.
:* Also, '''gps.uva_tracking_limited''' is gone. Instead of that we have gps.ee_tracker_speed_limited, which adds speed columns to the original data. This amounts to a merger of gps.ee_tracking_limited, gps.ee_tracking_speed_3d_limited and gps.ee_tracking_speed_limited.
:* [[ UvAGPS|UvA-BiTS Data Model]] - OLD

==Other e-Ecology Datasets==

'''ENVIRONMENTAL DATA'''

::* [[ Elevation|Elevation Data ]]
::* [[ ECMWF|Weather Forecast Data (ECMWF)]] Restricted to FlySafe users.

'''FLYSAFE'''

:Data related to European Space Agency FlySafe project. Financed by the Royal Netherlands Air Force.

::* [[Argos Data Model|Argos GPS Data Model]]
::* [[ RADAR|Radar Data Model]]
::* [[ Wind profile|Wind Profile Data Model]]
::* [[Setting up a connection to the e-Ecology database]]

==Other e-Ecology Services==
[[e-Ecology GPS Bird Track Generator]]: Produce kmz files of ARGOS/GPS bird tracking data from the FlySafe project to be viewed in Google Earth.

[[ Administration]]: Information about user rights and data processing tasks.

[[ Bookkeeping]]: Description of tables related to e-Ecology data processing and archiving.

[[ ToDo ]]

RADAR

2018-02-22T10:23:58Z

Stacy shinneman: /* Introduction */

=== Description of tables in RADAR schema in e-Ecology database ===

Data model for Motion Analyses images collected with ROBIN-system for the Medium Power Radar

Originally Hans van Gasteren, May 2007

=Introduction=
Radar data are one of the most important measurements in Bird Avoidance System (BAS) and FlySafe project. Until now data are archived in structured files instead of in a database. In this document data model for Motion Analysis images of the ROBIN4 system is designed, based on file structure of ROBIN4-MA images as defined in High Level Design Document of ROBIN4 (HLDD, TNO 2006).
MA (Motion Analysis) images are summated images of ten radar antenna rotations and are recorded twice per hour, two radar beams per radar, 3 radars in total (equalling 12 images per hour).
The data comprise all recorded radar echoes with summated intensities, plus rain clutter masks, land clutter masks and all separate objects of all recognised tracks. The huge amount of data per image (360°, 150km range, 10Mb) require a flexible database. Numbers of tracks can be as high as 15.000 tracks per image, with a mean about 1000 tracks. Per day we collect 48 images * 2 beams * 10Mb * 3 radars equipped with Robin 3Gb data.
Because the database must be readily and speedily accessible, only aggregated data can be recorded in the database. Original data will have to be recorded in a file system. The level of aggregation is still under debate, as the database should also allow a certain level of reanalysis. It is not desirable to store all original recorded echoes and raw radar data, which are the bulk of the data, in the database. Because the number of tracks is rather low (1000) for most of the images, it may be sufficient to store summaries of each track. This will lead to an estimated data reduction of 90%.
In the near future, already during precursor phase of FlySafe project, bird echo tracks will be recorded continuously. This means that if we want to store individual tracks, also positions of each antenna rotation is stored and database could be increasing much faster than it does now.

=Data model for MA images collected by ROBIN4 system .=

== RADAR ==

The radar table contains details for the different radars.

{|border=1
|'''RADAR_ID'''||unique radar id
|-
|RADAR_NAME||logical name
|-
|LATITUDE||degrees
|-
|LONGITUDE||degrees
|-
|X_POSITION||in meters, rijksdriehoekmeting
|-
|Y_POSITION||in meters, rijksdriehoekmeting
|-
|Z_POSITION||In meters, rijksdriehoekmeting
|-
|ALTITUDE_ANTENNA||In meters with respect to sea level
|-
|Radar type:||
|-
|MIN_RANGE||In meters
|-
|MAX_RANGE||In meters
|-
|RANGE_RESOLUTION||In meters, normal MPR 30m
|-
|AZIMUTH_RESOLUTION|| In radials azimuth if horizontal radar else elevation
|-
|REVOLUTION_TIME||In seconds, normal MPR 10s
|-
|Transmitter:||
|-
|MEAN_CARRIER_FREQUENCY||In GigaHz
|-
|PEAK_POWER||In dB
|-
|PULSE_LENGTH||In μS
|-
|PULSE_REPETITION_FREQUENCY||In Hz
|-
|Antenna:||
|-
|TYPE||circ, rectangular, omni
|-
|VERTICAL_ILLUMINATION||uniform, parabolic, parabolic², cosec²
|-
|POLARISATION||(horizontal, vertical, circular)
|-
|TRANSMITTER_GAIN||In dB
|}

==COVERAGE_DIAGRAM==

(for each azimuth horizontal elevation is given).

{|border=1
|'''RADAR_ID'''||Reference to RADAR_ID of table RADAR
|-
|'''COVDIAG_ID'''||
|-
|AZIMUTH||Radians (0-2PI)
|-
|ELEVATION||Radians
|}

==BEAM==

{|border=1
|'''RADAR_ID'''||Reference to RADAR_ID of table RADAR
|-
|'''BEAM_ID'''||
|-
|BEAM_NAME||
|-
|BEAM_ELEVATION||Elevation of beam (radians)
|-
|EL_BEAMWIDTH||Beam width (radians) in elevation
|-
|AZ_BEAMWIDTH||Beam width (radians) in azimuth
|}

==IMAGE_REQUEST==

{|border=1
|'''RADAR_ID'''||Reference to RADAR_ID of table RADAR
|-
|'''BEAM_ID'''||Reference to BEAM_ID of table BEAM
|-
|'''IMAGE_REQUEST_ID'''||
|-
|WINDOW_MIN_RANGE||Measurement range (m) of window
|-
|WINDOW_MAX_RANGE||meters
|-
|WINDOW_MIN_AZIMUTH||radials
|-
|WINDOW_MAX_AZIMUTH||radials
|-
|Motion_analysis_parameters||Note: for CMA different
|-
|RAIN_CLUTTER_MODE||Automatic, off, on
|-
|LAND_CLUTTER_MODE||Automatic, off, on
|-
|DETECTION_METHOD||Peak detection, threshold
|-
|MIN_TRACK_MEMBERS||2 ..10
|-
|ALLOWED_INTERVAL||0..9
|-
|MIN_MEMBER_SIZE||1..?
|-
|MIN_SPEED||5..50
|-
|MAX_SPEED||5..50
|-
|ALLOWED_SPEED_DEFLECTION||5..50
|-
|TANGENTIA_SPEED_DEVIATION||… part of score function
|-
|RADIAL_SPEED_DEVIATION||… part of score function
|-
|MASS_DEVIATION||… part of score function
|-
|BONUS_SCORE_PROBABILITY||… part of score function
|-
|PATH_FRACTION_THRESHOLD||… part of score function
|-
|RELAXATION_FACTOR||0.0..1.0 … part of score function
|-
|RANGE_RESOLUTION||Correction with respect to default range resolution
|-
|AZIMUTH_RESOLUTION||Correction with respect to default azimuth resolution
|}

==SUBWINDOW==

The subwindow table specifies size and location of subwindows

{|border=1
|'''RADAR_ID'''||Reference to RADAR_ID of table RADAR
|-
|'''BEAM_ID'''||Reference to BEAM_ID of table BEAM
|-
|'''IMAGE_REQUEST_ID'''||Reference to IMAGE_REQUEST_ID of table IMAGE_REQUEST
|-
|'''SUBWINDOW_ID'''||
|-
|SUBWINDOW_MIN_RANGE||meters
|-
|SUBWINDOW_MAX_RANGE||meters
|-
|SUBWINDOW_MIN_AZIMUTH||radials
|-
|SUBWINDOW_MAX_AZIMUTH||radials
|}

==IMAGE_RESULT==

The IMAGE_RESULT table contains references to actual images that have been recorded with the radar
at different timestamps. For the Medium Power Radar normally every half hour such an image is generated. So for one RADAR/ BEAM we can expect 17520 such images a year.

{|border=1
|'''RADAR_ID'''||Reference to RADAR_ID of table RADAR
|-
|'''BEAM_ID'''||Reference to BEAM_ID of table BEAM
|-
|'''IMAGE_REQUEST_ID'''||Reference to IMAGE_REQUEST_ID of table IMAGE_REQUEST
|-
|'''IMAGE_RESULT_ID'''||
|-
|ACQUISITION_TIME||Date and time of request
|-
|ERROR_STATUS||Status given back by Robin-system
|-
|CORRECTION_LEVEL||False alarm rate, FAR correction factor, important value, should have own graphical representation through out the year and correlate with the weather conditions
|-
|NR_OF_TRACKS||Number of tracks in image
|-
|RAIN_MASK||Rain clutter mask. Area were bird detection is switched off
|-
|LAND_MASK||Land clutter mask. No bird detection has been achieved in this area
|-
|IMAGE||Jpeg image of data (high resolution ) or georef TIFF
|}

==SUBWINDOW_RESULT==

The SUBWINDOW_RESULT table contains values for all the subwindows and for each image_result and contains some computed values where the most notable is the ROBIN_CORRECTED_DENSITY which is a measure of the bird density that can be used for warnings such as the [https://flysafe-pps.grid.sara.nl/birdtam.html BirdTam]

{|border=1
|'''RADAR_ID'''||Reference to RADAR_ID of table RADAR
|-
|'''BEAM_ID'''||Reference to BEAM_ID of table BEAM
|-
|'''IMAGE_REQUEST_ID'''||Reference to IMAGE_REQUEST_ID of table IMAGE_REQUEST
|-
|'''IMAGE_RESULT_ID'''||Reference to IMAGE_RESULT_ID of table IMAGE_RESULT
|-
|'''SUBWINDOW_ID'''||Reference to SUBWINDOW_ID of table SUBWINDOW
|-
|AREA||(km²)
|-
|LAND_CL_PERC||% land clutter in image
|-
|RAIN_CL_PERC||% rain clutter in image
|-
|CLUTTER_PERC||% land or rain clutter in image
|-
|TOTAL_MASS_DENSITY ||Total mass density per km², before MA
|-
|LAND_MASS_DENSITY ||Mass density per km², after MA in land clutter mask
|-
|RAIN_MASS_DENSITY ||Mass density per km², after MA in rain clutter mask
|-
|CLUTTER_MASS_DENSITY ||Mass density per km², after MA in rain or land clutter mask
|-
|BIRD_MASS_DENSITY||Mass density per km² of bird tracks
|-
|BIRD_ECHO_DENSITY||Mean bird echo density per km²
|-
|BIRD_MEAN_DIRECTION||Mean direction of all bird tracks in sub-window (radians)
|-
|BIRD_MEAN_SPEED||Mean speed of all bird tracks in sub-window (m/s)
|-
|MEAN_SPEED_VECTOR_RHO||Mean speed value added as vector
|-
|MEAN_SPEED_VECTOR_PHI||Mean speed direction added as vector
|-
|MEAN_BIRD_VECTOR|| ( sum(cos(alpha[i]) )^2 + sum(sin(alpha[i]) )^2 ) / n
|-
|STANDARD_DEVIATION_BIRD_SPEED||standard deviation BIRD_MEAN_SPEED
|-
|N||Number of echos in subwindow
|-
|ROBIN_CORRECTED_DENSITY||Corrected density
|-
|ROBIN_CORRECTED_QUALITY||Quality parameter associated with the ROBIN_CORRECTED_DENSITY
|}

The BIRD_MEAN_DIRECTION is the directional mean of the direction of all bird tracks in the subwindow. ( atan2(sum(sin(direction)) / N, sum(sin(direction)) / N) )

The BIRD_MEAN_SPEED is just the average speed of the birds in the subwindow.
The STANDARD_DEVIATION_BIRD_SPEED is the standard deviation for this BIRD_MEAN_SPEED.

The MEAN_BIRD_VECTOR is a measure of whether the birds fly in the same direction.
MEAN_BIRD_VECTOR = SQRT(sum(sin(direction)) /N + sum(cos(direction)) /N)

If this value is 1 they all fly in the same direction. If it is zero they all fly in opposite directions.

In addition the mean speed is also calculated as vector with the results in rho and phi in
MEAN_SPEED_VECTOR_RHO and MEAN_SPEED_VECTOR_PHI.

Details for the calculations of the ROBIN_CORRECTED_DENSITY and ROBIN_CORRECTED_QUALITY are in [[Robin subwindow density/quality_calculations]].

==TRACK_RESULT==

The TRACK_RESULT table contains tracks of birds that have been detected by the ROBIN4 system. It also contains as arrays all the different points in a track data. That data used to be in the TRACK_OBJECT table. The TRACK_RESULT table is a master table and the real data is contained in the inherited tables TRACK_RESULT<YEAR><MONTH>. Otherwise the TRACK_RESULT table would become too big.

{|border=1
|'''RADAR_ID'''||Reference to RADAR_ID of table RADAR
|-
|'''BEAM_ID'''||Reference to BEAM_ID of table BEAM
|-
|'''IMAGE_REQUEST_ID'''||Reference to IMAGE_REQUEST_ID of table IMAGE_REQUEST
|-
|'''IMAGE_RESULT_ID'''||Reference to IMAGE_RESULT_ID of table IMAGE_RESULT
|-
|'''TRACK_RESULT_ID'''||
|-
|''DATE_TIME''||Starttime (date + time) of track (for MA equal to ACQUISITION TIME, for CMA different)
|-
|TRACK_RANGE||Start position of track (m)
|-
|TRACK_AZIMUTH||Start position of track (radians)
|-
|TRACK_MASS||Mean mass of bird track
|-
|TRACK_SPEED||Mean speed of bird track (m/s)
|-
|TRACK_DIRECTION||Mean direction of bird track (radians)
|-
|TRACK_OBJECTS||Number of objects in track (0 – 10)
|-
|TRACK_SOURCE||MA / CAM
|-
|OBJECT_LATITUDE||Array of latitude position of object
|-
|OBJECT_LONGITUDE||Array of longitude position of object
|-
|OBJECT_MASS||Array of mass (reflection) of object
|-
|OBJECT_SIZE||Array of size of object
|-
|OBJECT_RHO||Array of rho coordinate of object
|-
|OBJECT_PHI||Array of phi coordinate of object
|}

[[ Image : Radar_datamodel.png ]]

Setting up a connection to the e-Ecology database

2018-02-20T15:54:38Z

Stacy shinneman: /* General Information */

= Connecting to the e-Ecology database via Access or other relational database programs=
Stacy Shinneman, Updated: February 2018; Judy Shamoun-Baranes / Bart Heupers, Updated: July 28, 2014

=General Information=

The e-Ecology database is a spatial postgreSQL ([http://www.postgresql.org/-http://www.postgresql.org/]). Before 2018, it was also referred to as the FlySafe database.

PgAdmin III or IV can be used to browse and query data. If you use pgAdmin you do not need to set up an ODBC connection.

The database can also be called from other programs by installing an ODBC driver for postgreSQL and establishing a "User Data Source" [http://www.postgresql.org/ftp/odbc/versions/msi/-http://www.postgresql.org/ftp/odbc/versions/msi/].

In order to access the database, you should send your IP address to uva-bits@uva.nl who will provide a user name and password. For organizations an IP range can also be given.

The e-Ecology server which is accessible to the UvA-BiTS community is at:

Server: pubserv.e-ecology.nl
Port: 5432
SSL: required

To set up a new connection to the e-Ecology database via '''pgAdmin III''', go to create New Server Registration and fill in the information below. To set up a connection in '''PgAdmin version 4''', click Add New Server and then enter the information on the General, Connection and SSL tabs.

'''Under the Properties tab'''
Name: e-Ecology
Host: pubserv.e-ecology.nl
Port: 5432
Maintenance DB: eecology
Username: <your user name>
Password: <your password>

'''Under the SSL tab'''
SSL: required

Restricted e-Ecology server information:

Server : db.e-ecology.sara.nl
Port : 5432
SSL: required

The tracking data can be found in the schema '''gps'''. However, these instructions can be used to access any of the data included in the e-Ecology database.

=Setting up an ODBC connection=

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.
Step 1: Download and install an ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/-http://www.postgresql.org/ftp/odbc/versions/msi/]
Step 2: Set up ODBC connection

Do the following:
# go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''
# go to '''User DSN''' (first tab in Data sources)
# click '''ADD'''
# in window '''create''' '''new data source''' select '''PostgreSQL ANSI''' click '''FINISH'''
# The '''ODBC Driver setup''' window opens. '''Data source name''': give your data source a name (eg GPS). Keep it simple, easily identifiable, this is the name you will use to identify your database from other applications, this will be the '''ODBC driver name'''.
# In the same window fill in the following information: '''database''' ''eecology'', '''Server''' ''public.e-ecology.sara.nl'', '''User Name''' your user name, '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' your password. When the window is completed click '''SAVE'''
# Back in the '''User DSN''' window you should now see your ODBC driver you just named, click '''OK'''
After completing these steps many other programs will now recognize your database and be able to communicate with it

=Connecting to Access=
You can now connect to the database via access. You have two options. One is to import the data, once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database (you must be connected to the network) in this case you are always viewing data that is in the actual SARA database.

# Open a new database in Access
# go to '''file | Get external data | Link tables'''
# in '''files of type''' select '''ODBC databases'''
# in '''select data source''' click on tab '''Machine Data Source'''
# select the data source name you created when setting up the ODBC connection (eg GPS) click '''OK'''
# Select the tables you want to import click OK

You now have access to your selected tables in the database. Have fun.

Original document [[ Media:Connect_flysafe.doc ]]

Setting up a connection to the e-Ecology database

2018-02-20T14:44:37Z

Stacy shinneman: /* General Information */

= Connecting to the e-Ecology database via Access or other relational database programs=
Stacy Shinneman, Updated: February 2018; Judy Shamoun-Baranes / Bart Heupers, Updated: July 28, 2014

=General Information=

The e-Ecology database is a spatial postgreSQL ([http://www.postgresql.org/-http://www.postgresql.org/]). Before 2018, it was also referred to as the FlySafe database.

PgAdmin III or IV can be used to browse and query data. If you use pgAdmin you do not need to set up an ODBC connection.

The database can also be called from other programs by installing an ODBC driver for postgreSQL and establishing a "User Data Source" [http://www.postgresql.org/ftp/odbc/versions/msi/-http://www.postgresql.org/ftp/odbc/versions/msi/].

In order to access the database, you should send your IP address to uva-bits@uva.nl who will provide a user name and password. For organizations an IP range can also be given.

The e-Ecology server which is accessible to the UvA-BiTS community is at:

Server: pubserv.e-ecology.nl
Port: 5432
SSL: required

To set up a new connection to the e-Ecology database via '''pgAdmin III''', go to create New Server Registration and fill in the information below. To set up a connection in '''PgAdmin version 4''', click Add New Server and then enter the information on the General, Connection and SSL tabs.

'''Under the Properties tab'''
Name: e-Ecology
Host: pubserv.e-ecology.sara.nl
Port: 5432
Maintenance DB: eecology
Username: <your user name>
Password: <your password>

'''Under the SSL tab'''
SSL: required

Restricted e-Ecology server information:

Server : db.e-ecology.sara.nl
Port : 5432
SSL: required

The tracking data can be found in the schema '''gps'''. However, these instructions can be used to access any of the data included in the e-Ecology database.

=Setting up an ODBC connection=

ODBC (Open Database connectivity) is a standard method of sharing data between databases and other programs. ODBC drivers use SQL to gain access to data from sources outside the program. Therefore, in order to connect between your database and other programs you must set up an ODBC connection.
Step 1: Download and install an ODBC driver for postgreSQL.
[http://www.postgresql.org/ftp/odbc/versions/msi/-http://www.postgresql.org/ftp/odbc/versions/msi/]
Step 2: Set up ODBC connection

Do the following:
# go to '''start | Control panel | Administrative tools | Data sources (ODBC)'''
# go to '''User DSN''' (first tab in Data sources)
# click '''ADD'''
# in window '''create''' '''new data source''' select '''PostgreSQL ANSI''' click '''FINISH'''
# The '''ODBC Driver setup''' window opens. '''Data source name''': give your data source a name (eg GPS). Keep it simple, easily identifiable, this is the name you will use to identify your database from other applications, this will be the '''ODBC driver name'''.
# In the same window fill in the following information: '''database''' ''eecology'', '''Server''' ''public.e-ecology.sara.nl'', '''User Name''' your user name, '''SSL Mode''' ''require'', '''Port''' ''5432'', '''Password''' your password. When the window is completed click '''SAVE'''
# Back in the '''User DSN''' window you should now see your ODBC driver you just named, click '''OK'''
After completing these steps many other programs will now recognize your database and be able to communicate with it

=Connecting to Access=
You can now connect to the database via access. You have two options. One is to import the data, once this is done your database is not automatically updated but does not require a link to the database.

The second option (described below) is to create a dynamic link to the database (you must be connected to the network) in this case you are always viewing data that is in the actual SARA database.

# Open a new database in Access
# go to '''file | Get external data | Link tables'''
# in '''files of type''' select '''ODBC databases'''
# in '''select data source''' click on tab '''Machine Data Source'''
# select the data source name you created when setting up the ODBC connection (eg GPS) click '''OK'''
# Select the tables you want to import click OK

You now have access to your selected tables in the database. Have fun.

Original document [[ Media:Connect_flysafe.doc ]]