Enterprises' thirst for real-time data is driving profound changes in data architecture. However, building modern, efficient data streaming platforms generally faces two major bottlenecks: First, Apache Kafka, as the central hub for real-time data, exposes severe cost and operational challenges in cloud environments due to its traditional architecture. High holding costs, near-zero elasticity, and complex partition migrations force enterprises to compromise between cost and performance.

Second, the selection of object storage—the cornerstone of data persistence—is equally challenging. Traditional distributed storage solutions, while powerful, are complex in architecture and have extremely high deployment and operational barriers; while lightweight solutions use the AGPL license, posing potential compliance risks and restrictions for enterprise commercial use. Enterprises need a new data architecture that balances high performance, low cost, ease of maintenance, and license friendliness.

To address these challenges, AutoMQ and RustFS have announced a strategic partnership. The two parties will deeply integrate AutoMQ's cloud-native stream processing capabilities, which are 100% compatible with the Apache Kafka protocol, with RustFS, a high-performance distributed object storage built on Rust, licensed under Apache 2.0, and compatible with S3. This collaboration aims to provide global enterprises with a next-generation Diskless Kafka platform that offers a superior architecture, lower costs, higher performance, and complete avoidance of licensing risks, thereby fundamentally solving the cost and efficiency challenges of real-time data stream processing in the cloud era.

About AutoMQ

AutoMQ is a next-generation cloud-native Kafka-compatible stream processing platform dedicated to solving the core pain points of traditional Kafka in cloud environments: high cost, limited elasticity, and complex operation and maintenance. AutoMQ employs an advanced compute-storage separation architecture, directly persisting streaming data to S3-compatible object storage, while the compute layer (Broker) is completely stateless. This revolutionary design ensures 100% compatibility with the Apache Kafka protocol, guaranteeing seamless migration to existing ecosystems such as Flink and Spark, while delivering a hundredfold increase in elasticity efficiency and second-level partition migration capabilities, helping enterprises run real-time streaming services with up to 90% savings in Total Cost of Ownership (TCO).

💡 Core Advantages

• Extreme Cost Efficiency : Achieves unlimited storage and pay-as-you-go pricing based on object storage, and completely eliminates expensive cross-Availability Zone (AZ) data replication traffic through a multi-point write architecture, achieving up to 90% TCO savings.

• Hundredfold Elasticity Efficiency : Stateless Broker supports second-level auto-scaling of compute resources. Partition migration time has been reduced from several hours to 1.5 seconds, enabling truly seamless cluster expansion for business operations.

• 100% Kafka Compatibility : Fully compatible with the Apache Kafka protocol and ecosystem, supporting zero-downtime migration from existing clusters without any code modifications.

• Fully Managed and Zero-Maintenance : Built-in automatic data rebalancing and fault self-healing capabilities, providing a BYOC (Bring Your Own Cloud) deployment mode. Data remains 100% within the customer's VPC, ensuring data privacy and security.

Figure 1: AutoMQ Advanced Cloud-Native Architecture

About RustFS

RustFS is a high-performance distributed object storage system developed in Rust and compliant with the S3 protocol. It aims to provide a robust data foundation for AI/ML, big data, and cloud-native applications. Unlike heavyweight architectures like Ceph, RustFS adopts a lightweight "metadata-free" design, where all nodes are equal, greatly simplifying deployment, operation, and scaling, and avoiding single points of failure in metadata. Leveraging the memory safety, high concurrency, and high performance advantages of the Rust language, RustFS achieves extremely high read/write performance and memory stability while providing exabyte-scale scalability.

💡 Core Advantages

• Apache 2.0 Friendly License : Utilizing the Apache-2.0 open-source license, it is completely friendly to enterprise commercial use, avoiding the intellectual property and compliance risks associated with licenses such as AGPL.

• High Performance and High Stability : Developed in Rust, it naturally possesses advantages in memory safety and high concurrency; under equivalent configurations, its read/write performance far surpasses Ceph, and its memory usage is stable with no high-concurrency jitter.

• Lightweight, metadata-free architecture : Deployment and maintenance are extremely simple, requiring no dedicated metadata server. Scaling is easy, requiring only a single command to start, significantly lowering the operational threshold.

• 100% S3 Compatibility : Fully compatible with the S3 API, supporting existing toolchains and SDKs. It also supports enterprise-level features such as version control, object read-only, and cross-region replication, seamlessly replacing existing S3 storage solutions.

Figure 2: RustFS Architecture (Source: RustFS Official Website)

AutoMQ × RustFS: Building a secure, scalable, cross-cloud Diskless Kafka architecture

The deep integration of AutoMQ's compute-storage separation architecture and RustFS's high-performance distributed storage achieves end-to-end technical synergy across three dimensions: cross-cloud support, security and reliability, and unlimited scalability, jointly forming a next-generation Diskless Kafka platform.

1. Building a Unified, Highly Available Data Flow

Both architectures are designed specifically for multi-cloud environments. AutoMQ provides a flexible BYOC (Bring Your Own Cloud) deployment model, supporting compute instances deployed in multi-cloud environments such as Amazon Web Services, Google Cloud, and Azure, as well as private data centers (IDCs). RustFS provides "true multi-cloud storage" capabilities at the storage layer, supporting bucket-level proactive-proactive cross-region replication. The combination of AutoMQ's BYOC compute layer and RustFS's cross-cloud storage layer enables enterprises to build unified, vendor-locked real-time data services, truly achieving highly available data flow across clouds and regions.

2. Achieving End-to-End Data Security and Privacy

Both technologies jointly construct end-to-end security and reliability. AutoMQ provides TLS/mTLS encryption at the access layer and deploys the data plane within the user's VPC using BYOC mode, ensuring "data stays within the VPC" and "superior data privacy." RustFS provides high-performance object storage security encryption at the storage layer and ensures data integrity and high availability through version control, object read-only (WORM), and proactive replication mechanisms. AutoMQ is responsible for access security and data privacy at the computation layer, while RustFS is responsible for encryption and data redundancy at the persistence layer. Together, they provide enterprises with end-to-end security from access and computation to storage, along with a strict read-after-write consistency model.

3. Independent flexibility in computing and storage

The architectures of both solutions completely resolve the pain point of traditional Kafka's difficulty in scaling. RustFS adopts a lightweight architecture with a "centralized metadata server," supporting EB-level storage capacity and "unlimited scaling." AutoMQ, with its storage-compute separation and stateless broker design, achieves auto-scaling at the compute layer and second-level partition migration. The perfect combination of AutoMQ's compute layer elasticity (second-level scaling) and RustFS's storage layer elasticity (EB-level capacity) finally allows enterprises to independently and on-demand expand compute resources (to cope with traffic surges) or storage resources (to cope with data growth), achieving true unlimited scaling.

Figure 3: AutoMQ & RustFS Collaborative Architecture

What It Brings

The deep technical synergy between AutoMQ and RustFS brings enterprises a high-performance, low-cost, secure, and compliant real-time data infrastructure. Its core advantages are:

• True unlimited elasticity and high performance : Both architectures support independent scaling. AutoMQ's stateless broker supports second-level elasticity, reducing partition migration time from hours to seconds, while maintaining a low latency of less than 10ms for P99. RustFS provides EB-level storage capacity and "unlimited scaling" capabilities. Enterprises can independently scale compute or storage as needed to cope with traffic surges and data growth.

• End-to-end security, reliability, and compliance : AutoMQ's BYOC deployment model ensures "data never leaves the VPC," guaranteeing excellent data privacy and providing TLS/mTLS encryption. RustFS provides high-performance encryption, version control, and WORM (read-only object) functionality at the storage layer. More importantly, RustFS's use of the Apache 2.0 license allows enterprises to completely avoid "license traps" like AGPL, achieving 100% commercial friendliness and compliance.

• A Fully Open-Source Joint Solution : At the heart of this joint solution lies our shared belief in the open-source ecosystem. Both parties have built on the Apache License, collaborating to create a 100% full-stack Apache-licensed solution for enterprises, from the compute layer to the storage layer. This not only provides enterprises with a secure and compliant commercial option with a high degree of autonomy and control, but this open practice has also won widespread recognition from the community.

  AutoMQ open source project address

RustFS open-source project address

• Extreme Cost Optimization : AutoMQ, with its compute-storage separation architecture, can save up to 90% on Kafka TCO and, through multi-point read/write architecture optimization, completely avoids expensive cross-Availability Zone (AZ) traffic. RustFS's lightweight architecture has low hardware requirements, and combined with the Apache 2.0 friendly license, further reduces storage and software licensing costs.

• Cross-Cloud Support and Greatly Simplified Operations : Both companies provide "true multi-cloud storage" and deployment capabilities, helping enterprises prevent single-cloud vendor lock-in. AutoMQ's fully managed and automatic data rebalancing features, combined with RustFS's lightweight architecture with "no metadata center" and low difficulty of use, greatly reduce the operational complexity of the entire system.

Looking Ahead

AutoMQ and RustFS will continue to deepen their technological integration, jointly driving the development of cloud-native real-time data infrastructure. Together, they will provide global enterprises with lower-cost, higher-performance, easier-to-maintain, and compliance-risk-free data streaming solutions, accelerating the transformation of enterprises from data to intelligence and jointly building an open and efficient cloud-native data ecosystem.

Visit the AutoMQ website now to learn about the ultimate performance and cost advantages of the next-generation cloud-native Kafka: AutoMQ

Also visit the RustFS website to explore high-performance S3 storage solutions based on the Apache-2.0 license: RustFS