Difference between revisions of "Files Dashboard"

From ecology
Jump to: navigation, search
(File processing lifecycle)
(User manual)
 
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
The files dashboard is a web page to get an overview of the raw data files of trackers. For a chosen project and a date interval, it displays a table with a row for every day in the date interval and a column for every tracker that belongs to the project.
+
The files dashboard is a web page to get an overview of the raw data files of trackers. For a chosen project and a date interval, it displays a table with a row for every day within the date interval and a column for every tracker that belongs to the project.
  
 
Each cell from the table will then contain information for the files of that tracker for that day; in principle, one box for each file. And usually, no more than one file per day. The purpose is to have information at a glance, so if there are no files for a tracker for one day, an empty box is to be shown.
 
Each cell from the table will then contain information for the files of that tracker for that day; in principle, one box for each file. And usually, no more than one file per day. The purpose is to have information at a glance, so if there are no files for a tracker for one day, an empty box is to be shown.
Line 7: Line 7:
 
* How big the file is
 
* How big the file is
 
* Whether errors happened during parsing of its contents
 
* Whether errors happened during parsing of its contents
 +
 +
== User manual ==
 +
If you can access the 'services' machine, you should be able to use your login/password and access the url:
 +
https://services.e-ecology.sara.nl/dashboard/
 +
 +
=== Interaction ===
 +
To customise the reported dashboard, the web page shows a bar at the top, with a couple of shortcuts:
 +
* for months (namely, the current month as the last in the list, and then 11 months in advance), which set the first and the last days to be included in the report
 +
* for projects, there is a dropdown list of all the project names; the request to change the project is sent by clicking on the submit button to the right of the list
 +
 +
=== URL Parameters ===
 +
You have more options if you look at the url of the page directly. The url takes 3 parameters: projectId, startDay and endDay. In principle, you can use the web interface to select the values that you want (click on a month, for instance, to set the dates), or select from a drop-down list the project you want to view the report for, as explained above.
 +
 +
However, if the interface does not provide the month you want to see (e.g.: you want to see something else than a specific month), then you can feed the parameters by hand in your browser: <code>...?projectId=2&startDay=20120401&endDay=20120430</code>.
 +
 +
If a projectId fails to appear on the url or the id provided does not make sense, then no project will be selected and therefore a message stating that no data has been found will be displayed.
 +
 +
If the dates fail to make sense, then the current date will be selected by default.
 +
 +
=== A bit more on the URL parameters ===
 +
; projectId
 +
: is the Identifier of the Project that you want to query (a number that is given to the project when it is created)
 +
; startDay
 +
: is the first day you want to include in the report, in format yyyyMMdd, where
 +
:* yyyy is the 4-digit year (e.g. 2012)
 +
:* MM is the 2-digit month within the year (e.g. 05 for May)
 +
:* dd is the 2-digit day of the month (e.g. 01 for the first day of the month)
 +
; endDay
 +
: is the last day you want included in the report, with the same format as the startDay
 +
 +
'''Note:''' The parameters are separated from the resource url by the standard HTTP symbol (a question mark, so '?'), and they are separated from each other by the standard HTTP parameter separator (the ampersand, so '&').
 +
 +
This has a number of advantages. Among others, you can make a bookmark of a specific report.
  
 
== Files processing ==
 
== Files processing ==
 
 
Raw tracking data comes into the system sent as plain text files through a [https://www.dropbox.com/ Dropbox] account. Dropbox makes the files available in the server file system.
 
Raw tracking data comes into the system sent as plain text files through a [https://www.dropbox.com/ Dropbox] account. Dropbox makes the files available in the server file system.
  
Line 27: Line 59:
 
; Size
 
; Size
 
: how big the file contents are
 
: how big the file contents are
 
=== File processing lifecycle ===
 
The goal being to show the three pieces of information described at the beginning, and considering the information that is available in the file name and properties, the system needs to keep track of these. Here is what it is looked at and how.
 
 
The file name (alone, without the path) uniquely identifies it in the system. If this is a new file, a new entry will be added to the system. When a file is discovered in the file system, first its properties are analyzed, as described above, and stored next to the file name.
 
 
If there already existed an entry for the file name, then the ''last modification date'' must be checked. If the one from the file differs from that in the entry in the system, then the information found in the newly found file must be taken into consideration; so the new ''last modification date'' overrides the one in the system entry (which, in turn, gets discarded) and along with the new ''file size''.
 
 
During all this process, no peek is taken inside the the file's contents.
 
 
'''Note:''' At this stage in the development there's no tracking done when parsing the files contents. Therefore, no errors are reported in the dashboard yet.
 

Latest revision as of 13:56, 24 July 2014

The files dashboard is a web page to get an overview of the raw data files of trackers. For a chosen project and a date interval, it displays a table with a row for every day within the date interval and a column for every tracker that belongs to the project.

Each cell from the table will then contain information for the files of that tracker for that day; in principle, one box for each file. And usually, no more than one file per day. The purpose is to have information at a glance, so if there are no files for a tracker for one day, an empty box is to be shown.

Three pieces of information are to be shown for each file:

  • Whether a file exists for a given day and tracker
  • How big the file is
  • Whether errors happened during parsing of its contents

User manual

If you can access the 'services' machine, you should be able to use your login/password and access the url: https://services.e-ecology.sara.nl/dashboard/

Interaction

To customise the reported dashboard, the web page shows a bar at the top, with a couple of shortcuts:

  • for months (namely, the current month as the last in the list, and then 11 months in advance), which set the first and the last days to be included in the report
  • for projects, there is a dropdown list of all the project names; the request to change the project is sent by clicking on the submit button to the right of the list

URL Parameters

You have more options if you look at the url of the page directly. The url takes 3 parameters: projectId, startDay and endDay. In principle, you can use the web interface to select the values that you want (click on a month, for instance, to set the dates), or select from a drop-down list the project you want to view the report for, as explained above.

However, if the interface does not provide the month you want to see (e.g.: you want to see something else than a specific month), then you can feed the parameters by hand in your browser: ...?projectId=2&startDay=20120401&endDay=20120430.

If a projectId fails to appear on the url or the id provided does not make sense, then no project will be selected and therefore a message stating that no data has been found will be displayed.

If the dates fail to make sense, then the current date will be selected by default.

A bit more on the URL parameters

projectId
is the Identifier of the Project that you want to query (a number that is given to the project when it is created)
startDay
is the first day you want to include in the report, in format yyyyMMdd, where
  • yyyy is the 4-digit year (e.g. 2012)
  • MM is the 2-digit month within the year (e.g. 05 for May)
  • dd is the 2-digit day of the month (e.g. 01 for the first day of the month)
endDay
is the last day you want included in the report, with the same format as the startDay

Note: The parameters are separated from the resource url by the standard HTTP symbol (a question mark, so '?'), and they are separated from each other by the standard HTTP parameter separator (the ampersand, so '&').

This has a number of advantages. Among others, you can make a bookmark of a specific report.

Files processing

Raw tracking data comes into the system sent as plain text files through a Dropbox account. Dropbox makes the files available in the server file system.

File properties

Once accessible through the local file system, they can be queried, read and parsed to extract the tracking information they contain.

The file name is expected to have the form Log_0533_13042012_xx.txt. This provides:

The tracker number
(in the example, 533)
The reported date
(in the example, April 13th 2012)


Apart from the name, the file has other attributes that the file system provides. Namely:

last modification date
tells when something was modified in the file for the last time.
Size
how big the file contents are