Warning

This documentation is actively being updated as the project evolves and may not be complete in all areas.

OpenDAL driver¶

jumpstarter-driver-opendal provides functionality for interacting with storages attached to the exporter.

Installation¶

$ pip3 install --extra-index-url https://pkg.jumpstarter.dev/simple jumpstarter-driver-opendal

Configuration¶

Example configuration:

type: "jumpstarter_driver_opendal.driver.Opendal"
config:
  # See https://docs.rs/opendal/latest/opendal/services/index.html
  # for list of supported services and their configuration parameters
  scheme: "fs"
  kwargs:
    root: "/tmp/jumpstarter"
  # Optional: automatically remove created files/directories when driver closes
  # Note: all tracked paths are normalized by stripping leading/trailing slashes
  remove_created_on_close: false

Configuration Parameters¶

scheme (required): The storage service type (e.g., “fs”, “s3”, “gcs”). See OpenDAL services for supported options.
kwargs (required): Service-specific configuration parameters passed to the OpenDAL operator.
remove_created_on_close (optional, default: false): When enabled, automatically removes all files and directories created during the session when the driver is closed.

File/Directory Tracking and Cleanup¶

The OpenDAL driver tracks all files and directories created during a session:

File Creation: Files opened in write modes ("wb", "w", "ab", "a")
Directory Creation: Directories created via create_dir()
Copy Operations: Target files/directories from copy() operations
Rename Operations: Target files/directories from rename() operations (source is removed from tracking)

Automatic Cleanup: The tracking is automatically updated when resources are removed:

Delete Operations: delete() removes the path from tracking
Remove Operations: remove_all() removes the path from tracking

Cleanup Behavior: When remove_created_on_close: true, all tracked files and directories are automatically removed when the driver closes (filesystem only)

Tracking API¶

# Get all created resources (files and directories)
created_resources = await driver.get_created_resources()  # Returns set[str]

# Example usage
for path in created_resources:
    print(f"Created: {path}")

Use Cases¶

Temporary File Management:

# Enable cleanup for temporary storage
remove_created_on_close: true

Persistent Storage:

# Disable cleanup to preserve files (default)
remove_created_on_close: false

Note: Pre-existing files that are written to are treated as “created” since they may be remnants from failed cleanup operations.

API Reference¶

Examples¶

>>> from tempfile import NamedTemporaryFile
>>> opendal.create_dir("test/directory/")
>>> opendal.write_bytes("test/directory/file", b"hello")
>>> assert opendal.hash("test/directory/file", "md5") == "5d41402abc4b2a76b9719d911017c592"
>>> opendal.remove_all("test/")

Client API¶

class jumpstarter_driver_opendal.client.OpendalClient¶

capability() → Capability¶

Get capabilities of the underlying storage

>>> cap = opendal.capability()
>>> cap.copy
True
>>> cap.presign_read
False

copy(source: str | PathLike, target: str | PathLike)¶

Copy source to target

>>> opendal.write_bytes("file.txt", b"content")
>>> opendal.copy("file.txt", "copy.txt")
>>> opendal.exists("copy.txt")
True

create_dir(path: str | PathLike)¶

Create a dir at given path

To indicate that a path is a directory, it is compulsory to include a trailing / in the path.

Create on existing dir will succeed. Create dir is always recursive, works like mkdir -p.

>>> opendal.create_dir("a/b/c/")
>>> opendal.exists("a/b/c/")
True

delete(path: str | PathLike)¶

Delete given path

Delete not existing error won’t return errors

>>> opendal.write_bytes("file.txt", b"content")
>>> opendal.exists("file.txt")
True
>>> opendal.delete("file.txt")
>>> opendal.exists("file.txt")
False

exists(path: str | PathLike) → bool¶

Check if given path exists

>>> opendal.exists("file.txt")
False
>>> opendal.write_bytes("file.txt", b"content")
>>> opendal.exists("file.txt")
True

get_created_resources() → set[str]¶

Get set of all paths that have been created during this session.

Returns:: set[str]: Set of all paths (files and directories) that were created

hash(path: str | PathLike, algo: Literal['md5', 'sha256'] = 'sha256') → str¶

Get current path’s hash

>>> opendal.write_bytes("file.txt", b"content")
>>> opendal.hash("file.txt")
'ed7002b439e9ac845f22357d822bac1444730fbdb6016d3ec9432297b9ec9f73'

list(path: str | PathLike) → Generator[str, None, None]¶

List files and directories under given path

>>> opendal.write_bytes("dir/file.txt", b"content")
>>> opendal.write_bytes("dir/another.txt", b"content")
>>> sorted(opendal.list("dir/"))
['dir/', 'dir/another.txt', 'dir/file.txt']

open(path: str | PathLike, mode: Literal['rb', 'wb']) → OpendalFile¶

Open a file-like reader for the given path

>>> file = opendal.open("file.txt", "wb")
>>> file.write_bytes(b"content")
>>> file.close()

presign_read(path: str | PathLike, expire_second: int) → PresignedRequest¶: Presign an operation for read (GET) which expires after expire_second seconds

presign_stat(path: str | PathLike, expire_second: int) → PresignedRequest¶: Presign an operation for stat (HEAD) which expires after expire_second seconds

presign_write(path: str | PathLike, expire_second: int) → PresignedRequest¶: Presign an operation for write (PUT) which expires after expire_second seconds

read_bytes(path: str | PathLike) → bytes¶

Read data from path

>>> opendal.write_bytes("file.txt", b"content")
>>> opendal.read_bytes("file.txt")
b'content'

read_into_path(src: str | PathLike, dst: str | PathLike, operator: Operator | None = None) → None¶

Read data into dst from src

>>> opendal.write_bytes("file.txt", b"content")
>>> opendal.read_into_path("file.txt", tmp / "dst")
>>> (tmp / "dst").read_bytes()
b'content'

remove_all(path: str | PathLike)¶

Remove all file under path

>>> opendal.write_bytes("dir/file.txt", b"content")
>>> opendal.remove_all("dir/")
>>> opendal.exists("dir/file.txt")
False

rename(source: str | PathLike, target: str | PathLike)¶

Rename source to target

>>> opendal.write_bytes("file.txt", b"content")
>>> opendal.rename("file.txt", "rename.txt")
>>> opendal.exists("file.txt")
False
>>> opendal.exists("rename.txt")
True

scan(path: str | PathLike) → Generator[str, None, None]¶

List files and directories under given path recursively

>>> opendal.write_bytes("dir/a/file.txt", b"content")
>>> opendal.write_bytes("dir/b/another.txt", b"content")
>>> sorted(opendal.scan("dir/"))
['dir/', 'dir/a/', 'dir/a/file.txt', 'dir/b/', 'dir/b/another.txt']

stat(path: str | PathLike) → Metadata¶

Get current path’s metadata

>>> opendal.write_bytes("file.txt", b"content")
>>> opendal.stat("file.txt").mode.is_file()
True

write_bytes(path: str | PathLike, data: bytes) → None¶

Write data into path

>>> opendal.write_bytes("file.txt", b"content")

write_from_path(dst: str | PathLike, src: str | PathLike, operator: Operator | None = None) → None¶

Write data from src into dst

>>> _ = (tmp / "src").write_bytes(b"content")
>>> opendal.write_from_path("file.txt", tmp / "src")
>>> opendal.read_bytes("file.txt")
b'content'

class jumpstarter_driver_opendal.client.OpendalFile¶

A file-like object representing a remote file

close() → None¶: Close the file

property closed: bool¶: Check if the file is closed

read_into_path(path: str | PathLike, operator: Operator | None = None)¶: Read content from remote file into local file

readable() → bool¶: Check if the file is readable

seek(pos: int, whence: int = 0) → int¶

Change the cursor position to the given byte offset. Offset is interpreted relative to the position indicated by whence. The default value for whence is SEEK_SET. Values for whence are:

SEEK_SET or 0 – start of the file (the default); offset should be zero or positive

SEEK_CUR or 1 – current cursor position; offset may be negative

SEEK_END or 2 – end of the file; offset is usually negative

Return the new cursor position

seekable() → bool¶: Check if the file is seekable

tell() → int¶: Return the current cursor position

writable() → bool¶: Check if the file is writable

write_from_path(path: str | PathLike, operator: Operator | None = None)¶: Write into remote file with content from local file

class jumpstarter_driver_opendal.common.Metadata¶

content_disposition: str | None¶

content_length: int¶

content_md5: str | None¶

content_type: str | None¶

etag: str | None¶

mode: EntryMode¶

class jumpstarter_driver_opendal.common.EntryMode¶

entry_is_dir: bool¶

entry_is_file: bool¶

is_dir() → bool¶

is_file() → bool¶

class jumpstarter_driver_opendal.common.PresignedRequest¶

Presigned HTTP request

Allows you to delegate access to a specific file in your storage backend without sharing access credentials

headers: dict[str, str]¶: Additional HTTP headers to send with the request

method: str¶: HTTP method

GET: download file

PUT: upload file

DELETE: delete file

url: str¶: HTTP request URL

class jumpstarter_driver_opendal.common.Capability¶

blocking: bool¶

create_dir: bool¶

delete: bool¶

list: bool¶

list_with_limit: bool¶

list_with_recursive: bool¶

list_with_start_after: bool¶

presign: bool¶

presign_read: bool¶

presign_stat: bool¶

presign_write: bool¶

read: bool¶

read_with_if_match: bool¶

read_with_if_none_match: bool¶

read_with_override_cache_control: bool¶

read_with_override_content_disposition: bool¶

read_with_override_content_type: bool¶

rename: bool¶

shared: bool¶

stat: bool¶

stat_with_if_match: bool¶

stat_with_if_none_match: bool¶

write: bool¶

write_can_append: bool¶

write_can_empty: bool¶

write_can_multi: bool¶

write_multi_max_size: int | None¶

write_multi_min_size: int | None¶

write_total_max_size: int | None¶

write_with_cache_control: bool¶

write_with_content_disposition: bool¶

write_with_content_type: bool¶