---
title: Interacting with the gypsum REST API
author:
- name: Aaron Lun
  email: infinite.monkeys.with.keyboards@gmail.com
package: gypsum
date: "Revised: April 2, 2024"
output:
  BiocStyle::html_document
vignette: >
  %\VignetteIndexEntry{Hitting the gypsum API}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, echo=FALSE}
library(BiocStyle)
self <- Biocpkg("gypsum")
knitr::opts_chunk$set(error=FALSE, warning=FALSE, message=FALSE)
```

# Introduction

The `r self` package implements an R client for the [API of the same name](https://gypsum-test.aaron-lun.workers.dev).
This allows Bioconductor packages to easily store and retrieve data from the **gypsum** backend.
It also provides mechanisms to allow package maintainers to easily manage upload authorizations and third-party contributions.
Readers are referred to [API's documentation](https://github.com/ArtifactDB/gypsum-worker) for a description of the concepts;
this guide will strictly focus on the usage of the `r self` package.

# Reading files

`r self` provides several convenience methods for reading from the **gypsum** bucket:

```{r}
library(gypsum)
listAssets("test-R")
listVersions("test-R", "basic")
listFiles("test-R", "basic", "v1")

out <- saveFile("test-R", "basic", "v1", "blah.txt")
readLines(out)

dir <- saveVersion("test-R", "basic", "v1")
list.files(dir, all.files=TRUE, recursive=TRUE)
```

We can fetch the summaries and manifests for each version of a project's assets.

```{r}
fetchManifest("test-R", "basic", "v1")
fetchSummary("test-R", "basic", "v1")
```

We can get the latest version of an asset:

```{r}
fetchLatest("test-R", "basic")
```

All read operations involve a publicly accessible bucket so no authentication is required.

# Uploading files

## Basic usage

To demonstrate, let's say we have a directory of files that we wish to upload to the backend.

```{r}
tmp <- tempfile()
dir.create(tmp)
write(file=file.path(tmp, "foo"), letters)
write(file=file.path(tmp, "bar"), LETTERS)
write(file=file.path(tmp, "whee"), 1:10)
```

We run the upload sequence of `startUpload()`, `uploadFiles()` and `completeUpload()`.
This requires authentication via GitHub, which is usually prompted but can also be set beforehand via `setAccessToken()` (e.g., for batch jobs).

```r
init <- startUpload(
    project=project_name,
    asset=asset_name,
    version=version_name,
    files=list.files(tmp, recursive=TRUE),
    directory=tmp
)

tryCatch({
    uploadFiles(init, directory=tmp)
    completeUpload(init)
}, error=function(e) {
    abortUpload(init) # clean up if the upload fails.
    stop(e)
})
```

We can also set `concurrent=` to parallelize the uploads in `uploadFiles()`.

## Link generation

More advanced developers can use `links=` in `startUpload()` to improve efficiency by deduplicating redundant files on the **gypsum** backend.
For example, if we wanted to link to some files in our `test-R` project, we might do:

```r
# Creates links from lun/aaron.txt ==> test-R/basic/v1/foo/bar.txt
# and kancherla/jayaram ==> test-R/basic/v1/blah.txt
init <- startUpload(
    project=project_name,
    asset=asset_name,
    version=version_name,
    files=character(0),
    links=data.frame(
        from.path=c("lun/aaron.txt", "kancherla/jayaram.txt"),
        to.project=c("test-R", "test-R"),
        to.asset=c("basic", "basic"),
        to.version=c("v1", "v1"),
        to.path=c("foo/bar.txt", "blah.txt")
    ),
    directory=tmp
)
```

This functionality is particularly useful when creating new versions of existing assets.
Only the modified files need to be uploaded, while the rest of the files can be linked to their counterparts in the previous version.
In fact, this pattern is so common that it can be expedited via `cloneVersion()` and `prepareDirectoryUpload()`:

```{r}
dest <- tempfile()
cloneVersion("test-R", "basic", "v1", destination=dest)

# Do some modifications in 'dest' to create a new version, e.g., add a file.
# However, users should treat symlinks as read-only - so if you want to modify
# a file, instead delete the symlink and replace it with a new file.
write(file=file.path(dest, "BFFs"), c("Aaron", "Jayaram"))

to.upload <- prepareDirectoryUpload(dest)
to.upload
```

Then we can just pass these values along to `startUpload()` to take advantage of the upload links:

```r
init <- startUpload(
    project=project_name,
    asset=asset_name,
    version=version_name,
    files=to.upload$files,
    links=to.upload$links,
    directory=dest
)
```

# Changing permissions

Upload authorization is determined by each project's permissions, which are controlled by project owners.
Both uploaders and owners are identified based on their GitHub logins:

```{r}
fetchPermissions("test-R")
```

Owners can add more uploaders (or owners) via the `setPermissions()` function.
Uploaders can be scoped to individual assets or versions, and an expiry date may be attached to each authorization:

```r
setPermissions("test-R", 
    uploaders=list(
        list(
            id="jkanche", 
            until=Sys.time() + 24 * 60 * 60,
            asset="jays-happy-fun-time",
            version="1"
        )
    )
)
```

Organizations may also be added in the permissions, in which case the authorization extends to all members of that organization.

# Probational uploads

Unless specified otherwise, all uploaders are considered to be "untrusted".
Any uploads from such untrusted users are considered "probational" and must be approved by the project owners before they are considered complete.
Alternatively, an owner may reject an upload, which deletes all the uploaded files from the backend.

```r
approveProbation("test-R", "third-party-upload", "good")
rejectProbation("test-R", "third-party-upload", "bad")
```

An uploader can be trusted by setting `trusted=TRUE` in `setPermissions()`.
Owners and trusted uploaders may still perform probational uploads (e.g., for testing) by setting `probation=TRUE` in `startUpload()`.

# Inspecting the quota

Each project has a quota that specifies how much storage space is available for uploaders.
The quota is computed as a linear function of `baseline + growth_rate * (NOW - year)`,
which provides some baseline storage that grows over time.

```{r}
fetchQuota("test-R")
```

Once the project's contents exceed this quota, all uploads are blocked.
The current usage of the project can be easily inspected:

```{r}
fetchUsage("test-R")
```

Changes to the quota must be performed by administrators, see [below](#administration).

# Validating metadata

Databases can operate downstream of the **gypsum** backend to create performant search indices, usually based on special metadata files.
`r self` provides some utilities to check that metadata follows the JSON schema of some known downstream databases.

```{r}
schema <- fetchMetadataSchema()
cat(head(readLines(schema)), sep="\n")
```

Uploaders can verify that their metadata respects this schema via the `validateMetadata()` function.
This ensures that the uploaded files can be successfully indexed by the database, given that the **gypsum** backend itself applies no such checks.

```{r}
metadata <- list(                           
    title="Fatherhood",
    description="Luke ich bin dein Vater.",
    sources=list(
       list(provider="GEO", id="GSE12345")
    ),
    taxonomy_id=list("9606"),
    genome=list("GRCm38"),
    maintainer_name="Darth Vader",
    maintainer_email="vader@empire.gov",
    bioconductor_version="3.10"
)

validateMetadata(metadata, schema)
```

# Administration

Administrators of the **gypsum** instance can create projects with new permissions:

```r
createProject("my-new-project", 
    owners="jkanche", 
    uploaders=list(
        list(
            id="LTLA", 
            asset="aarons-stuff"
        )
    )
)
```

They can alter the quota parameters for a project:

```r
setQuota("my-new-project",
    baseline=10 * 2^30,
    growth=5 * 2^30,
    year=2019
)
```

They can manually refresh the latest version for an asset and the quota usage for a project.
This is only required on very rare occasions where there are simultaneous uploads to the same project.

```r
refreshLatest("test-R", "basic")
refreshUsage("test-R")
```

Administrators may also delete projects, assets or versions, though this should be done sparingly as it violates **gypsum**'s expectations of immutability.

```r
removeProject("my-new_project")
```

# Session information {-}

```{r}
sessionInfo()
```