SciLo: Long Term Data Archiving

SciLo is a long term archiving service at ACCRE based on the Spectra Logic BlackPearl converged storage solution. With SciLo you can archive data at a very low cost and minimal system administrator intervention. Movement of data from ACCRE (or anywhere) to SciLo and back is accomplished with command line client or, if you are using portal or doing X11 forwarding through SSH, an available GUI (dsb-gui). ACCRE provides accre_scilo CLI tool to help the user list, download and upload files to the bucket. The tool encapsulates the use of ds3_java_cli command inside, and the users don’t need to get down to the details of how to feed the correct file path information to the ds3_java_cli command.

Getting Started

Initial sign up to the SciLo requires creation of an account on our BlackPearl and issuance of an id and key. Open a helpdesk ticket with ACCRE requesting access. You will receive confirmation of account setup and the ID and secret key.

Scilo Archive CLI

A consolidated Python CLI tool for interacting with the Spectra Logic BlackPearl archive. It provides a simple, unified interface for listing, uploading, downloading, and deleting objects stored on the archive.

Prerequisites

• A .s3keys file in the project root with your BlackPearl credentials:

export DS3_ENDPOINT=archive1.accre.vanderbilt.edu
export DS3_ACCESS_KEY=your_access_key
export DS3_SECRET_KEY=your_secret_key

• Load accre_scilo module from the software stack.

module load accre_scilo

Usage

All commands follow the pattern:

accre_scilo <command> [arguments] [options]

The available commands are: ls, list-all, get, put, delete, and volume-quota.

---

1. List Buckets or Objects (ls)

Use the ls command to list all available buckets, or to browse the contents of a specific bucket. Optionally filter by a folder prefix and output results as JSON.

Arguments

Argument	Required	Description
bucket	No	Name of the bucket to list. If omitted, all buckets are listed.
prefix	No	A folder path prefix to filter results within the bucket. If omitted, all objects in the bucket are shown.

Options

Option	Description
--json	Output the listing in JSON format instead of plain text.

Examples

List all available buckets:

accre_scilo ls

List all objects in a specific bucket:

accre_scilo ls my-bucket

List objects under a specific folder prefix:

accre_scilo ls my-bucket data/2024/

List objects as JSON:

accre_scilo ls my-bucket --json

---

2. Comprehensive Listing (list-all)

Recursively list all objects across all buckets (or a specific one) and save the results to a JSON file. This command calculates progress by scanning folder hierarchies up to a configurable depth, making it suitable for large-scale inventory operations.

Arguments

Argument	Required	Description
bucket	No	Name of a specific bucket to list. If omitted, all buckets are scanned.

Options

Option	Default	Description
--output_file	scilo_list_all.json	Path to the file where the output will be saved.
--level	3	Hierarchy depth used for progress tracking. Higher values provide more granular progress reporting but may increase scan time.
--result_type	flat	Output structure. Accepts flat or nested.

Result Types

Type	Description
flat (default)	A simple list of all object keys in the bucket. Best for programmatic consumption and piping into other tools.
nested	A hierarchical JSON structure representing the folder and file tree. Useful for visualising the archive structure.

Examples

List everything and save to the default scilo_list_all.json:

accre_scilo list-all

List a specific bucket with custom scanning depth and output file:

accre_scilo list-all my-bucket --level 4 --output_file results.json

Get nested hierarchical output:

accre_scilo list-all my-bucket --result_type nested

---

3. Retrieve Objects or Folders (get)

Download one or more objects (or entire folders) from a bucket to a local destination directory. By default, all objects in the bucket are retrieved. Use the --objects flag to selectively download specific items.

Arguments

Argument	Required	Description
bucket	Yes	Name of the bucket to download from.
destination	Yes	Local directory where the downloaded files will be saved.

Options

Option	Description
--objects	One or more specific object keys or folder prefixes to retrieve. If omitted, all objects in the bucket are downloaded.
--recursive, -r	Recursively fetch all objects within a folder prefix. Required when downloading folders.

Examples

Download all objects from a bucket:

accre_scilo get my-bucket ./local_destination

Download specific objects:

accre_scilo get my-bucket ./local_destination --objects file1.txt path/to/file2.jpg

Recursively download an entire folder:

accre_scilo get my-bucket ./local_destination --objects my_folder/ --recursive

---

4. Upload Files or Directories (put)

Upload a local file or an entire directory to the archive. When uploading a directory, all files within it are uploaded recursively, preserving the folder structure. Use the --name option to control the remote object name or folder prefix.

Arguments

Argument	Required	Description
bucket	Yes	Name of the destination bucket.
file	Yes	Path to the local file or directory to upload.

Options

Option	Description
--name	Override the remote object name (for files) or folder prefix (for directories). If omitted, the original file or directory name is used.

Examples

Upload a single file:

accre_scilo put my-bucket ./local_file.dat

Upload an entire directory recursively:

accre_scilo put my-bucket ./my_folder/

Upload with a specific remote name:

accre_scilo put my-bucket ./local_file.dat --name remote_name.dat

---

5. Delete Objects or Folders (delete)

Remove an object or an entire folder from a bucket. Deleting a single object is straightforward. To delete a non-empty folder and all of its contents, the --force flag is required, and you will be prompted for secondary confirmation by typing the full path.

Arguments

Argument	Required	Description
bucket	Yes	Name of the bucket containing the object or folder.
object	Yes	The object key or folder path to delete.

Options

Option	Description
--force	Force recursive deletion of a non-empty folder. Without this flag, the tool will refuse to delete folders that contain objects.

Examples

Delete a single object:

accre_scilo delete my-bucket object_name

Delete a folder and all its contents (requires confirmation):

accre_scilo delete my-bucket folder_path/ --force

---

6. Volume Quota (volume-quota)

Display the total amount of storage space used within a specific bucket. This is useful for monitoring archive usage and capacity planning.

Arguments

Argument	Required	Description
bucket	Yes	Name of the bucket to check.

Examples

Check the used space for a bucket:

accre_scilo volume-quota my-bucket

Scilo GUI

The Scilo GUI (dsb-gui) is a graphical interface for interacting with the archive. It provides the same core functionality as the CLI — browsing buckets, uploading, downloading, and deleting objects — through a point-and-click interface.

Display Requirements

Because dsb-gui is a graphical application, you must have a way to forward its display to your local machine. There are two supported methods:

Method	Description
VizPortal	Connect to ACCRE through the VizPortal web interface, which provides a full remote desktop session.
X11 Forwarding	Connect to ACCRE over SSH with X11 forwarding enabled. Use the -X (or -Y on macOS) flag when connecting: ssh -X vunetid@accre.vu

Launching the GUI

Load the module and start the application:

module load scilo-gui
dsb-gui

Creating a BlackPearl session

To connect to the BlackPearl archive, create a new session in the GUI with the following settings:

Field	Value
Data Path Address	archive1.accre.vanderbilt.edu
Port	80
Access ID	Your access key (provided by us)
Secret Key	Your secret key (provided by us)