Difference between revisions of "Bookkeeping"

From ecology
Jump to: navigation, search
(FILES)
(Flysafe Grid-Store Catalog)
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
=== Description of tables in BOOKKEEPING schema in the Flysafe database ===
+
=== Description of tables in BOOKKEEPING schema in the e-Ecology database ===
 +
 
 +
[[Image : Bookkeeping_datamodel.png]]
  
 
== FILES ==  
 
== FILES ==  
  
 
The FILES table describes the status of all data-files which have been or need to be processed.
 
The FILES table describes the status of all data-files which have been or need to be processed.
This information is used by the bookkeeping service on the flysafe website
+
This information is used by the bookkeeping service on the e-Ecology website.
  
 
{|border=1
 
{|border=1
Line 21: Line 23:
 
|time_in||Timestamp when  file was first registered  
 
|time_in||Timestamp when  file was first registered  
 
|-
 
|-
|time_changed||Timastamp of last status change
+
|time_changed||Timestamp of last status change
 
|}
 
|}
  
=== Flysafe Grid-Store Catalog ===
+
= e-Ecology Grid-Store Catalog =
  
The following three tables are used to implement the Flysafe Grid-Store catalog. If files are being archived or copied the Grid-Storage, this is administratred in the following tables.  
+
The following three tables are used to implement the e-Ecology Grid-Store catalog. If files are being archived or copied the Grid-Storage, this is administratred in the following tables.  
  
 
For Grid-Storage it is preferable to have a limited number of  big files instead of many small files. Therefore files are first collected into bigger archives, which are then stored as  
 
For Grid-Storage it is preferable to have a limited number of  big files instead of many small files. Therefore files are first collected into bigger archives, which are then stored as  
archive to the Grid-Storage. This is also maintained in the tables below.  
+
archive to the Grid-Storage. This is also maintained in the tables below.
  
 
== FILE_CATALOG ==  
 
== FILE_CATALOG ==  
Line 46: Line 48:
  
  
== ARCHIVE_FILE ==  
+
== ARCHIVE_FILE ==
  
 +
Files can be collected together in archive (usually tar files). These archive files are also registered as files in the FILE_CATALOG table.  The ARCHIVE_FILE table contains the information about which files are in what archive file.
 +
 +
{|border=1
 +
|'''archive_id'''||Id of archive file in FILE_CATALOG table
 +
|-
 +
|'''file_id'''||Id of file in archive in FILE_CATALOG table
 +
|-
 +
|archive_path||Path of file in archive
 +
|}
  
 
== FILE_REPLICA ==
 
== FILE_REPLICA ==
 +
 +
The FILE_REPLICA table contains the actual storage location of the stored files. It is possible to have more replicas of the same file.
 +
 +
{|border=1
 +
|'''file_id'''||References in file_catalog
 +
|-
 +
|'''replica_id'''||replicanumber, ie 1 for first replica, 2 for second etc
 +
|-
 +
|surl||SURL is the storage location.
 +
|-
 +
|stored||timestamp with time of storage
 +
|-
 +
|adler32|| adler32 checksum of file
 +
|-
 +
|verified||boolean indicator
 +
|}
 +
 +
= Auxiliary tables =
 +
 +
== SRMREQUEST ==
 +
 +
The SRMREQUEST table is used when it is requested to have a file or group of files restored from SRM. In this table the status of the request is maintained so that it can be shown on a website.
 +
It is also possible to check all requests for a specific user.
 +
 +
 +
{|border=1
 +
|'''id'''||Unique ID
 +
|-
 +
|username||Name of user who requested the restore
 +
|-
 +
|status||Status of the request
 +
|-
 +
|time_start||Timestamp when request was started
 +
|-
 +
|time_changed||Timestamp when request was changed
 +
|}

Latest revision as of 14:35, 20 February 2018

Description of tables in BOOKKEEPING schema in the e-Ecology database

Bookkeeping datamodel.png

FILES

The FILES table describes the status of all data-files which have been or need to be processed. This information is used by the bookkeeping service on the e-Ecology website.

ID unique file id
filename unique file name
file_type Type of file ie MPR, ECMWF,
file_size Size of file in bytes
status Status of file ie new, processed, archived
filename_time Time associated with filename as timestamp.
time_in Timestamp when file was first registered
time_changed Timestamp of last status change

e-Ecology Grid-Store Catalog

The following three tables are used to implement the e-Ecology Grid-Store catalog. If files are being archived or copied the Grid-Storage, this is administratred in the following tables.

For Grid-Storage it is preferable to have a limited number of big files instead of many small files. Therefore files are first collected into bigger archives, which are then stored as archive to the Grid-Storage. This is also maintained in the tables below.

FILE_CATALOG

Every file to be stored gets an entry in the FILE_CATALOG table.

id ID for file
filename Unique filename
filetype Type of file (MPR, tar, etc)
path Original path of file on Flysafe server


ARCHIVE_FILE

Files can be collected together in archive (usually tar files). These archive files are also registered as files in the FILE_CATALOG table. The ARCHIVE_FILE table contains the information about which files are in what archive file.

archive_id Id of archive file in FILE_CATALOG table
file_id Id of file in archive in FILE_CATALOG table
archive_path Path of file in archive

FILE_REPLICA

The FILE_REPLICA table contains the actual storage location of the stored files. It is possible to have more replicas of the same file.

file_id References in file_catalog
replica_id replicanumber, ie 1 for first replica, 2 for second etc
surl SURL is the storage location.
stored timestamp with time of storage
adler32 adler32 checksum of file
verified boolean indicator

Auxiliary tables

SRMREQUEST

The SRMREQUEST table is used when it is requested to have a file or group of files restored from SRM. In this table the status of the request is maintained so that it can be shown on a website. It is also possible to check all requests for a specific user.


id Unique ID
username Name of user who requested the restore
status Status of the request
time_start Timestamp when request was started
time_changed Timestamp when request was changed