palimpsest-storage

Content-addressable blob storage with three backends: in-memory, filesystem, and object store (S3/GCS/Azure). Deduplication is structural — same content is stored once.

BlobStore Trait

#![allow(unused)]
fn main() {
pub trait BlobStore: Send + Sync {
    async fn put(&self, data: Bytes) -> Result<ContentHash, StorageError>;
    async fn get(&self, hash: &ContentHash) -> Result<Bytes, StorageError>;
    async fn exists(&self, hash: &ContentHash) -> Result<bool, StorageError>;
    async fn delete(&self, hash: &ContentHash) -> Result<(), StorageError>;
    async fn metadata(&self, hash: &ContentHash) -> Result<BlobMetadata, StorageError>;
}
}

BlobMetadata

#![allow(unused)]
fn main() {
pub struct BlobMetadata { pub size: u64, pub stored_at: DateTime<Utc> }
}

StorageError

#![allow(unused)]
fn main() {
pub enum StorageError {
    Backend(String),
    NotFound(ContentHash),
    IntegrityError { expected: ContentHash, actual: ContentHash },
}
}

InMemoryBlobStore

#![allow(unused)]
fn main() {
impl InMemoryBlobStore {
    pub fn new() -> Self;
    pub fn len(&self) -> usize;
    pub fn is_empty(&self) -> bool;
    pub fn total_bytes(&self) -> u64;
}
}

Uses BTreeMap for deterministic ordering.

FileSystemBlobStore

#![allow(unused)]
fn main() {
impl FileSystemBlobStore {
    pub async fn new(root: impl Into<PathBuf>) -> Result<Self, StorageError>;
    pub fn root(&self) -> &Path;
}
}

Git-style layout: {root}/{hash[0..2]}/{hash[2..]}. Atomic writes via temp file + rename. Integrity verification on every read.

ObjectStoreBlobStore

S3, GCS, and Azure support via the object_store crate. Same BlobStore trait interface.

Key Invariant

Every put computes ContentHash::of(data). Every get verifies the hash of retrieved data. Tampering is always detectable.