geolba/tethys.backend
2
0
Fork 1
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Update 'DataArchitectureDiagram'

Kaimbacher 2023-04-25 16:19:10 +00:00
parent 3ccae8e731
commit f1a278123d
1 changed files with 154 additions and 153 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

307
DataArchitectureDiagram.md

@ -1,153 +1,154 @@
<center> <h1>Data Architecture Diagram</h1> </center> <center> <h1>Data Architecture Diagram</h1> </center>
![Image](./data_diagram.drawio.svg) ![Image](./data_diagram.drawio.svg)
https://gitea.geologie.ac.at/geolba/tethys.backend/wiki/raw/data_diagram.drawio.svg https://gitea.geologie.ac.at/geolba/tethys.backend/wiki/raw/data_diagram.drawio.svg
# 1. Data Sources # 1. Data Sources
TETHYS collects and manages scanned geological maps, spatial data sets, and other types of research outputs. TETHYS collects and manages scanned geological maps, spatial data sets, and other types of research outputs.
The general guideline for publishing research data <a href="https://tethys.at/docs/HandbuchTethys.pdf#page=12" target="_blank">(manual, p. 11-2)</a> is providing information on how to prepare data sources for deposit (helpful for depositors and repository staff): The general guideline for publishing research data <a href="https://tethys.at/docs/HandbuchTethys.pdf#page=12" target="_blank">(manual, p. 11-2)</a> is providing information on how to prepare data sources for deposit (helpful for depositors and repository staff):
- the types of file formats that are acceptable. All **allowed file fomats** must be enabled by the administrator in the database table **'mime_types'**. Also see <a href="https://gitea.geologie.ac.at/geolba/tethys.backend/wiki/Database#tethys-lookup-tables" target="_blank">(db lookup tables)</a> - the types of file formats that are acceptable. All **allowed file fomats** must be enabled by the administrator in the database table **'mime_types'**. Also see <a href="https://gitea.geologie.ac.at/geolba/tethys.backend/wiki/Database#tethys-lookup-tables" target="_blank">(db lookup tables)</a>
- what kind of data can be included in the research data repository, - what kind of data can be included in the research data repository,
- the file upload limits **(512MB)**, - the file upload limits **(512MB)**,
- the strategy for deleting datasets, - the strategy for deleting datasets,
- how to handle the use of external research data, - how to handle the use of external research data,
- the required use of keywords - the required use of keywords
- Note that the uploaded data will be reviewed - Note that the uploaded data will be reviewed
- what CC licenses can be used: - what CC licenses can be used:
<table style="border: 1px solid black; border-collapse: collapse;"> <table style="border: 1px solid black; border-collapse: collapse;">
<tr> <tr>
<th>CC Code</th> <th>CC Code</th>
<th>Link to the definition</th> <th>Link to the definition</th>
<th>Description</th> <th>Description</th>
</tr> </tr>
<tr> <tr>
<td> <td>
CC-BY CC-BY
</td> </td>
<td> <td>
<a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a> <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a>
</td> </td>
<td>Nur Namensnennung</td> <td>Nur Namensnennung</td>
</tr> </tr>
<tr> <tr>
<td> <td>
CC-BY-SA CC-BY-SA
</td> </td>
<td> <td>
<a href="https://creativecommons.org/licenses/by-sa/4.0/" target="_blank">https://creativecommons.org/licenses/by-sa/4.0/</a> <a href="https://creativecommons.org/licenses/by-sa/4.0/" target="_blank">https://creativecommons.org/licenses/by-sa/4.0/</a>
</td> </td>
<td>Namensnennung; Weitergabe unter gleicher Bedingung</td> <td>Namensnennung; Weitergabe unter gleicher Bedingung</td>
</tr> </tr>
<tr> <tr>
<td> <td>
CC BY-NC CC BY-NC
</td> </td>
<td> <td>
<a href="http://creativecommons.org/licenses/by-nc/4.0" target="_blank">http://creativecommons.org/licenses/by-nc/4.0</a> <a href="http://creativecommons.org/licenses/by-nc/4.0" target="_blank">http://creativecommons.org/licenses/by-nc/4.0</a>
</td> </td>
<td>Namensnennung; keine kommerzielle Nutzung</td> <td>Namensnennung; keine kommerzielle Nutzung</td>
</tr> </tr>
<tr> <tr>
<td> <td>
CC-BY-ND CC-BY-ND
</td> </td>
<td> <td>
<a href="https://creativecommons.org/licenses/by-nd/4.0/" target="_blank">https://creativecommons.org/licenses/by-nd/4.0/</a> <a href="https://creativecommons.org/licenses/by-nd/4.0/" target="_blank">https://creativecommons.org/licenses/by-nd/4.0/</a>
</td> </td>
<td>Namensnennung; keine Veränderung erlaubt</td> <td>Namensnennung; keine Veränderung erlaubt</td>
</tr> </tr>
<tr> <tr>
<td> <td>
CC-BY-NC-SA CC-BY-NC-SA
</td> </td>
<td> <td>
<a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">https://creativecommons.org/licenses/by-nc-sa/4.0/</a> <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">https://creativecommons.org/licenses/by-nc-sa/4.0/</a>
</td> </td>
<td>Namensnennung; keine kommerzielle Nutzung; Weitergabe unter gleicher Bedingung</td> <td>Namensnennung; keine kommerzielle Nutzung; Weitergabe unter gleicher Bedingung</td>
</tr> </tr>
<tr> <tr>
<td> <td>
CC-BY-NC-ND CC-BY-NC-ND
</td> </td>
<td> <td>
<a href="https://creativecommons.org/licenses/by-nc-nd/4.0/" target="_blank">https://creativecommons.org/licenses/by-nc-nd/4.0/</a> <a href="https://creativecommons.org/licenses/by-nc-nd/4.0/" target="_blank">https://creativecommons.org/licenses/by-nc-nd/4.0/</a>
</td> </td>
<td>Namensnennung; keine kommerzielle Nutzung; keine Veränderung erlaubt</td> <td>Namensnennung; keine kommerzielle Nutzung; keine Veränderung erlaubt</td>
</tr> </tr>
</table> </table>
<!-- ![valid mime types per 25.04.2023](./allowed_mime_types.png) <!-- ![valid mime types per 25.04.2023](./allowed_mime_types.png)
*allowed mime_types and file extensions in TETHYS RDR 2023* --> *allowed mime_types and file extensions in TETHYS RDR 2023* -->
<p> <p>
<img src="./allowed_mime_types.png" alt="Mime Types"> <img src="./allowed_mime_types.png" alt="Mime Types">
<figcaption>Fig.1 - valid mime types per 25.04.2023.</figcaption> <figcaption>Fig.1 - valid mime types per 25.04.2023.</figcaption>
</p> </p>
# 2. Metadata Schema # 2. Metadata Schema
The **Metadata format guidelines** <a href="https://tethys.at/docs/HandbuchTethys.pdf#page=9" target="_blank">(manual, p. 8-10)</a> on the other hand, provides information on the specific types of metadata that should be included with the data. This may include information on the format and structure of the metadata, as well as the types of information that should be included. Tethys RDR is using three metadata standards (**Dublin Core, DataCite and ISO10130**) as well as one descriptive metadata element (dataset type). The dataset type is one of the following metadata elements: Analysis, Measurements, Monitoring, Remote Sensing, Geographic Information System (GIS), Models, and Mixed Type. The **Metadata format guidelines** <a href="https://tethys.at/docs/HandbuchTethys.pdf#page=9" target="_blank">(manual, p. 8-10)</a> on the other hand, provides information on the specific types of metadata that should be included with the data. This may include information on the format and structure of the metadata, as well as the types of information that should be included. Tethys RDR is using three metadata standards (**Dublin Core, DataCite and ISO10130**) as well as one descriptive metadata element (dataset type). The dataset type is one of the following metadata elements: Analysis, Measurements, Monitoring, Remote Sensing, Geographic Information System (GIS), Models, and Mixed Type.
# 3. Storage Infrastructure # 3. Storage Infrastructure
This is the physical or virtual infrastructure used to store the data and metadata in the repository. By using **PostgreSQL**, TETHYS is able to manage large volumes of metadata and provide fast and secure access to this information. The data files are stored on an **Ubuntu 22.04 file server** with **ext4**partition. This is the physical or virtual infrastructure used to store the data and metadata in the repository. By using **PostgreSQL**, TETHYS is able to manage large volumes of metadata and provide fast and secure access to this information. The data files are stored on an **Ubuntu 22.04 file server** with **ext4**partition.
# 4. Data processing and curation workflows # 4. Data processing and curation workflows
... ...
# 5. Data Discovery # 5. Data Discovery
TETHYS supports data discovery in various ways TETHYS supports data discovery in various ways
## TETHYS REST API ## TETHYS REST API
Tethys Metadata and File downloads can be queried by a REST API (Representational State Transfer Application Programming Interface), Tethys Metadata and File downloads can be queried by a REST API (Representational State Transfer Application Programming Interface),
which allows repository staff to interact with the Tethys system and retrieve metadata and data files programmatically. which allows repository staff to interact with the Tethys system and retrieve metadata and data files programmatically.
- **metadata API**: e.g.: [metadata for dataset 209](https://tethys.at/api/dataset/200) - **metadata API**: e.g.: [metadata for dataset 209](https://tethys.at/api/dataset/200)
- **file download**: e.g. [file for dataset 209: api/file/download/1009](https://www.tethys.at/api/file/download/1009) - **file download**: e.g. [file for dataset 209: api/file/download/1009](https://www.tethys.at/api/file/download/1009)
## Tethys Frontend ## Tethys Frontend
The datasets can always be found through the Data Frontend, https://tethys.at/search, browsing by subject, author, language or year, or by searching the metadata fields title, author or keyword. To enhance the discoverability of the datasets, TETHYS RDR stimulates adding as many metadata and other relevant documentation as possible. The datasets can always be found through the Data Frontend, https://tethys.at/search, browsing by subject, author, language or year, or by searching the metadata fields title, author or keyword. To enhance the discoverability of the datasets, TETHYS RDR stimulates adding as many metadata and other relevant documentation as possible.
## Solr Search Index ## Solr Search Index
All visible metadata and some metadata extracted from data files are indexed and searchable by Solr. All visible metadata and some metadata extracted from data files are indexed and searchable by Solr.
[SOLR wildcard search](https://www.tethys.at/solr/rdr_data/select?q=*%3A*)
## DataCite DOI Fabrica
## DataCite DOI Fabrica
Tethys Repository assigns a DOI for each published dataset.
Therefore the datasts are also accessible via DOI Fabrica: e.g. https://doi.datacite.org/dois/10.24341%2Ftethys.200 Tethys Repository assigns a DOI for each published dataset.
Therefore the datasts are also accessible via DOI Fabrica: e.g. https://doi.datacite.org/dois/10.24341%2Ftethys.200
# 6. Authentication and authorization systems:
# 6. Authentication and authorization systems:
TETHYS provides an Access Control List (ACL) system that is used to manage users, user roles and permissions.
![title](./tethys_acl.png) TETHYS provides an Access Control List (ACL) system that is used to manage users, user roles and permissions.
![title](./tethys_acl.png)
- accounts: This table stores information about the users of the system. Each user is assigned a unique identifier, and the table includes fields for the user's name, email address, and timestamps.
- accounts: This table stores information about the users of the system. Each user is assigned a unique identifier, and the table includes fields for the user's name, email address, and timestamps.
- roles: This table stores information about the roles that are available in your application. Each role has a unique identifier and a name that describes the role. TETHYS is using the following rules:
- roles: This table stores information about the roles that are available in your application. Each role has a unique identifier and a name that describes the role. TETHYS is using the following rules:
- admin role: for editing user accounts, roles and permissions
- submitter role: Submitters submit original research datasets. Submitters are responsible for ensuring that their submissions meet the repository's guidelines and standards. - admin role: for editing user accounts, roles and permissions
- editor role: TETHYS editors are responsible for overseeing the content and quality of the research repository. They review submissions to ensure they meet the repository's standards and may provide feedback to submitters. They may also work with reviewers to ensure that submissions are evaluated fairly. - submitter role: Submitters submit original research datasets. Submitters are responsible for ensuring that their submissions meet the repository's guidelines and standards.
- reviewer role: Reviewers are responsible for evaluating submissions to the research repository. They may be assigned specific submissions to review, and they provide feedback on the quality and relevance of the research. Reviewers typically have expertise in the field related to the research and may be asked to provide a written report or score to help the editor make a decision about whether to accept or reject the submission. - editor role: TETHYS editors are responsible for overseeing the content and quality of the research repository. They review submissions to ensure they meet the repository's standards and may provide feedback to submitters. They may also work with reviewers to ensure that submissions are evaluated fairly.
- reviewer role: Reviewers are responsible for evaluating submissions to the research repository. They may be assigned specific submissions to review, and they provide feedback on the quality and relevance of the research. Reviewers typically have expertise in the field related to the research and may be asked to provide a written report or score to help the editor make a decision about whether to accept or reject the submission.
- permissions: This table stores information about the permissions that are available for the roles. Each permission has a unique identifier, a name that describes the permission, and a short code that represents the permission.
- permissions: This table stores information about the permissions that are available for the roles. Each permission has a unique identifier, a name that describes the permission, and a short code that represents the permission.
- role_has_permissions: This table stores the relationship between roles and permissions. Each row in this table represents a role that has been granted a specific permission.
- role_has_permissions: This table stores the relationship between roles and permissions. Each row in this table represents a role that has been granted a specific permission.
- link_account_roles: This table stores the relationship between users (Accounts) and roles. Each row in this table represents a user that has been assigned a specific role.
- link_account_roles: This table stores the relationship between users (Accounts) and roles. Each row in this table represents a user that has been assigned a specific role.
# 7. System Integration
# 7. System Integration
System Integration with other systems involves integrating the research repository with other systems, such as project management tools, data platforms and reporting tools.
System Integration with other systems involves integrating the research repository with other systems, such as project management tools, data platforms and reporting tools.
By providing the Open Archives Initiative Protocol for Metadata Harvesting (**OAI-PMH**) any other data provider can harvest TETHYS metadata. An example would be the **Bielefeld Academic Search Engine BASE**: https://www.base-search.net/Search/Results?q=coll:fttethysrdr&refid=dctablede
By providing the Open Archives Initiative Protocol for Metadata Harvesting (**OAI-PMH**) any other data provider can harvest TETHYS metadata. An example would be the **Bielefeld Academic Search Engine BASE**: https://www.base-search.net/Search/Results?q=coll:fttethysrdr&refid=dctablede
**Matomo** is used to track statistics data for TETHYS. Matomo is an open-source web analytics platform that can be used to track user behavior on a website.
**Matomo** is used to track statistics data for TETHYS. Matomo is an open-source web analytics platform that can be used to track user behavior on a website.