Difference between revisions of "Bookkeeping"
(New page: === Description of tables in BOOKKEEPING schema in the Flysafe database === == FILES == The FILES table describes the status of all data-files which have been or need to be processed. T...) |
(→Flysafe Grid-Store Catalog) |
||
| (11 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | === Description of tables in BOOKKEEPING schema in the | + | === Description of tables in BOOKKEEPING schema in the e-Ecology database === |
| + | |||
| + | [[Image : Bookkeeping_datamodel.png]] | ||
== FILES == | == FILES == | ||
The FILES table describes the status of all data-files which have been or need to be processed. | The FILES table describes the status of all data-files which have been or need to be processed. | ||
| − | This information is used by the bookkeeping service on the | + | This information is used by the bookkeeping service on the e-Ecology website. |
| − | |||
{|border=1 | {|border=1 | ||
|'''ID'''||unique file id | |'''ID'''||unique file id | ||
|- | |- | ||
| − | |''filename | + | |''filename''||unique file name |
|- | |- | ||
|file_type|| Type of file ie MPR, ECMWF, | |file_type|| Type of file ie MPR, ECMWF, | ||
| Line 18: | Line 19: | ||
|status|| Status of file ie ''new'', ''processed'', ''archived'' | |status|| Status of file ie ''new'', ''processed'', ''archived'' | ||
|- | |- | ||
| − | |filename_time||Time associated with filename as timestamp | + | |filename_time||Time associated with filename as timestamp. |
|- | |- | ||
|time_in||Timestamp when file was first registered | |time_in||Timestamp when file was first registered | ||
|- | |- | ||
| − | |time_changed|| | + | |time_changed||Timestamp of last status change |
| + | |} | ||
| + | |||
| + | = e-Ecology Grid-Store Catalog = | ||
| + | |||
| + | The following three tables are used to implement the e-Ecology Grid-Store catalog. If files are being archived or copied the Grid-Storage, this is administratred in the following tables. | ||
| + | |||
| + | For Grid-Storage it is preferable to have a limited number of big files instead of many small files. Therefore files are first collected into bigger archives, which are then stored as | ||
| + | archive to the Grid-Storage. This is also maintained in the tables below. | ||
| + | |||
| + | == FILE_CATALOG == | ||
| + | |||
| + | Every file to be stored gets an entry in the FILE_CATALOG table. | ||
| + | |||
| + | {|border=1 | ||
| + | |'''id'''||ID for file | ||
| + | |- | ||
| + | |''filename''||Unique filename | ||
| + | |- | ||
| + | |filetype||Type of file (MPR, tar, etc) | ||
| + | |- | ||
| + | |path||Original path of file on Flysafe server | ||
| + | |} | ||
| + | |||
| + | |||
| + | == ARCHIVE_FILE == | ||
| + | |||
| + | Files can be collected together in archive (usually tar files). These archive files are also registered as files in the FILE_CATALOG table. The ARCHIVE_FILE table contains the information about which files are in what archive file. | ||
| + | |||
| + | {|border=1 | ||
| + | |'''archive_id'''||Id of archive file in FILE_CATALOG table | ||
| + | |- | ||
| + | |'''file_id'''||Id of file in archive in FILE_CATALOG table | ||
| + | |- | ||
| + | |archive_path||Path of file in archive | ||
| + | |} | ||
| + | |||
| + | == FILE_REPLICA == | ||
| + | |||
| + | The FILE_REPLICA table contains the actual storage location of the stored files. It is possible to have more replicas of the same file. | ||
| + | |||
| + | {|border=1 | ||
| + | |'''file_id'''||References in file_catalog | ||
| + | |- | ||
| + | |'''replica_id'''||replicanumber, ie 1 for first replica, 2 for second etc | ||
| + | |- | ||
| + | |surl||SURL is the storage location. | ||
| + | |- | ||
| + | |stored||timestamp with time of storage | ||
| + | |- | ||
| + | |adler32|| adler32 checksum of file | ||
| + | |- | ||
| + | |verified||boolean indicator | ||
| + | |} | ||
| + | |||
| + | = Auxiliary tables = | ||
| + | |||
| + | == SRMREQUEST == | ||
| + | |||
| + | The SRMREQUEST table is used when it is requested to have a file or group of files restored from SRM. In this table the status of the request is maintained so that it can be shown on a website. | ||
| + | It is also possible to check all requests for a specific user. | ||
| + | |||
| + | |||
| + | {|border=1 | ||
| + | |'''id'''||Unique ID | ||
| + | |- | ||
| + | |username||Name of user who requested the restore | ||
| + | |- | ||
| + | |status||Status of the request | ||
| + | |- | ||
| + | |time_start||Timestamp when request was started | ||
| + | |- | ||
| + | |time_changed||Timestamp when request was changed | ||
|} | |} | ||
Latest revision as of 14:35, 20 February 2018
Contents
Description of tables in BOOKKEEPING schema in the e-Ecology database
FILES
The FILES table describes the status of all data-files which have been or need to be processed. This information is used by the bookkeeping service on the e-Ecology website.
| ID | unique file id |
| filename | unique file name |
| file_type | Type of file ie MPR, ECMWF, |
| file_size | Size of file in bytes |
| status | Status of file ie new, processed, archived |
| filename_time | Time associated with filename as timestamp. |
| time_in | Timestamp when file was first registered |
| time_changed | Timestamp of last status change |
e-Ecology Grid-Store Catalog
The following three tables are used to implement the e-Ecology Grid-Store catalog. If files are being archived or copied the Grid-Storage, this is administratred in the following tables.
For Grid-Storage it is preferable to have a limited number of big files instead of many small files. Therefore files are first collected into bigger archives, which are then stored as archive to the Grid-Storage. This is also maintained in the tables below.
FILE_CATALOG
Every file to be stored gets an entry in the FILE_CATALOG table.
| id | ID for file |
| filename | Unique filename |
| filetype | Type of file (MPR, tar, etc) |
| path | Original path of file on Flysafe server |
ARCHIVE_FILE
Files can be collected together in archive (usually tar files). These archive files are also registered as files in the FILE_CATALOG table. The ARCHIVE_FILE table contains the information about which files are in what archive file.
| archive_id | Id of archive file in FILE_CATALOG table |
| file_id | Id of file in archive in FILE_CATALOG table |
| archive_path | Path of file in archive |
FILE_REPLICA
The FILE_REPLICA table contains the actual storage location of the stored files. It is possible to have more replicas of the same file.
| file_id | References in file_catalog |
| replica_id | replicanumber, ie 1 for first replica, 2 for second etc |
| surl | SURL is the storage location. |
| stored | timestamp with time of storage |
| adler32 | adler32 checksum of file |
| verified | boolean indicator |
Auxiliary tables
SRMREQUEST
The SRMREQUEST table is used when it is requested to have a file or group of files restored from SRM. In this table the status of the request is maintained so that it can be shown on a website. It is also possible to check all requests for a specific user.
| id | Unique ID |
| username | Name of user who requested the restore |
| status | Status of the request |
| time_start | Timestamp when request was started |
| time_changed | Timestamp when request was changed |
