Apiome Bulk
Big-data bulk loader for object stores and data lakes — Parquet to apiome-db at cluster scale.
Apiome Bulk ingests Parquet, Avro, ORC, Delta Lake, CSV, and JSON Lines from S3, GCS, Azure Blob, HDFS, and local uploads directly into apiome-db. Chunked resumable uploads, partition-aware fanout, and a choice of DataFusion for single-node or Spark for cluster-scale, all over Arrow Flight as the high-throughput transport.
What ships with Bulk
Every Bulk surface is wired into the rest of the Apiome platform — no glue code, no separate identity, no bolt-on integrations.
Multi-format lake ingestion
Load Parquet, Avro, ORC, Delta Lake, CSV, and JSON Lines, inferring schema from footers or headers before the run starts.
Object store & HDFS connectors
Manage S3, GCS, Azure Blob, HDFS, and local upload connections with connectivity tests and credential rotation.
Visual job planner
A three-pane configurator pairs the source selector with a live fanout execution-plan preview and job parameters.
Partition strategy control
Choose hash, range, or time-based partitioning and preview shard distribution across the partition count before committing.
Chunked resumable loads
Per-chunk checkpointing shows rows loaded versus target and resumes failed partitions without restarting the whole job.
Pluggable execution engines
Run DataFusion for single-node loads or Spark for cluster scale, moving data over Arrow Flight for throughput.
A look inside Apiome Bulk
Live design previews from the Bulk mockup pack — 9 surfaces in total.
Real-time view of every bulk load with row-count progress, engine badge, ETA, source, and target class.
Map source file columns to apiome-db class properties with type coercion, null handling, and required-field enforcement.
Full manifest for a completed run: per-file chunk outcomes, loaded row counts, error log, and coercion summary.
Use cases
Bulk is designed around the way real teams actually work — not the way a tool wants them to work.
Backfill billions of rows from a Parquet lake into apiome-db with resumable, partition-aware Spark jobs.
Preview inferred schemas and map columns to class properties before a load instead of discovering type errors mid-run.
Watch active loads and drill into per-chunk checkpoint state to resume the exact partitions that failed.
- Cluster-scale Spark execution with parallelism and checkpoint tuning
- Credential rotation and connectivity tests across every object store
- Schema drift detection against target class versions
- Full run audit trail with throughput, error counts, and coercion logs
- Arrow Flight transport for high-throughput, secure data movement
- Auto-map source columns to class properties with coercion rules
- Recommend hash, range, or time partitioning from data shape
- Size engine and parallelism from file volume and cluster capacity
- Explain and triage per-partition load failures from error logs
AI load planner
Bulk inspects your source files and target classes to propose the fastest, safest load plan — mapping columns, picking a partition strategy, and sizing engine parallelism automatically.
Every BulkAI feature is grounded in your tenant's data, runs under your data-residency policy, and respects every role and ACL the platform enforces.
Every surface in Apiome Bulk
A look at the 9 screens designed for this suite — covering everything from day-1 onboarding to day-100 operations.
