Contribution Guide

Development Environment Setup

Strata is a multi-language monorepo consisting of a high-performance Rust core, Python bindings (via PyO3), and a React-based dashboard.

Prerequisites

• Rust: 1.75 or higher
• Python: 3.9 or higher (with venv recommended)
• Node.js: 18.x or higher (for the Dashboard)
• Docker & Docker Compose: For containerized testing and dependency management

Initializing the Workspace

1. Clone the repository:

   git clone https://github.com/syrilj/Strata.git
   cd Strata
   

2. Setup Rust dependencies: The workspace uses Cargo to manage multiple crates.

   cargo build
   

3. Setup Python environment:

   python -m venv venv
   source venv/bin/activate
   pip install -e ./crates/python-bindings
   

4. Setup Dashboard:

   cd dashboard
   npm install
   

Workspace Structure

The project is divided into several specialized crates within the crates/ directory:

| Crate | Responsibility | |:---|:---| | runtime-core | Shared types (CheckpointMetadata, ShardAssignment), error handling, and worker state definitions. | | coordinator | The central gRPC server and Axum-based HTTP API for dashboard integration. | | data-shard | Core logic for consistent hashing and deterministic shard distribution. | | storage | Pluggable backends for local disk and AWS S3 persistence. | | checkpoint | High-level management for async checkpoint writing and versioning. | | python-bindings | PyO3-based wrappers for exposing Rust logic to ML training scripts. |

Coding Standards

To ensure consistency across the distributed runtime, please adhere to the following standards:

Rust

• Style: All code must be formatted using cargo fmt.
• Linting: Ensure cargo clippy returns no warnings.
• Documentation: Public functions and structs in the core crates must have doc comments. Use cargo doc --open to verify.
• Error Handling: Use the custom Error and Result types defined in runtime-core::error. Avoid expect() or unwrap() in the coordinator or storage crates.

Dashboard (React/TypeScript)

• Typing: All API interactions should use the interfaces defined in dashboard/src/types.ts to maintain parity with the Rust http_api definitions.
• Styling: Use Tailwind CSS for UI components.

Testing Framework

Contributions must include relevant tests to prevent regressions in high-concurrency scenarios.

Unit & Integration Tests

Run the full Rust suite from the root:

cargo test --workspace

To test specific logic, such as consistent hashing:

cargo test -p data-shard

Protocol Verification

The coordinator relies on Protobuf definitions. If you modify proto/coordinator.proto, ensure you regenerate and verify the gRPC service implementation in crates/coordinator/src/service.rs.

Local Development Workflow

1. Running the Coordinator

Start the coordinator in development mode (defaults to port 50051 for gRPC and 3000 for the dashboard API):

cargo run -p coordinator

2. Running the Dashboard

In a separate terminal, launch the React development server:

cd dashboard
npm run dev

3. Simulating Workers

You can use the provided Docker Compose configuration to spin up a local cluster for integration testing:

docker-compose up --build

Submission Process

1. Create a Feature Branch: Use descriptive names like feature/consistent-hashing-optimization or fix/s3-multipart-upload.
2. Atomic Commits: Group related changes. Follow Conventional Commits (e.g., feat:, fix:, docs:, perf:).
3. PR Requirements:
   • Verified passing of cargo test.
   • Updated documentation for any changes to the public API in runtime-core or coordinator.
   • If adding a new storage backend, ensure it implements the StorageBackend trait in crates/storage/src/backend.rs.
4. Review: A maintainer will review the PR for performance implications, specifically regarding synchronization latency and async I/O efficiency.