A Fluvio client communicates with a Fluvio cluster to manage streams and to emit or receive events. The client uses a purpose-built communication protocol that is optimized for maximum performance, scalability, and low latency. Websocket is currently in the works, and future versions will provide adaptors to other protocols, such as: HTTP, gRPC, protobuf and more.

All communication between the clients and the servers is encrypted in TLS for maximum privacy and security.

The Fluvio client library is written in Rust and can be natively embedded into other programming languages.

Check out our APIs page for more information

The Fluvio CLI an application written on top of the client library. The CLI can manage a Fluvio cluster as well as produce and consume data using the terminal.

For additional information, checkout:

• Fluvio CLI

Future versions of Fluvio will provide additional programming language bindings, such as:

• Objective-C
• C/C++

Fluvio client library has three core APIs: Producer, Consumer, and Admin.

The Producer API is responsible for sending records to data streams. A data record is a key/value object, where key is an optional field. Key and value fields can be of arbitrary format.

For example, a producer with a key mapped to countries would use the following API:

let timber_resources: HashMap<&str, i32> =
    [("Norway", 100), ("Denmark", 50), ("Iceland", 10)]
    .into_iter().collect();

producer.send(timber_resources).await;

Producers can send records one at a time or in batches. The producer API is multi-threaded, which enables applications to stream data in parallel.

The Consumer API is responsible for receiving records from data streams. Records can be retrieved one at a time or continuously from any position in the data stream.

For example, a consumer reading key/value records (one at a time) from offset 100, would use the following API:

let records = consumer.fetch(100).await;

for record in records {
    for (k, v) in record {
        println!("{}, {}", k, v);
    }
}

Records are transmitted in binary format and it is up to the Application developer to provide a conversion into their custom type.

The Admin API is the management interface for the Fluvio cluster. The API can perform the following operations:

• create/delete/update objects such as: create topic.
• inspect configuration, such as: list spus.
• inspect status, such as: partitions status - Online, Offline, or LeaderOffline.

Configuration object models follow a similar paradigm. Each object has the following components:

• Name - unique identifier of the object
• Spec - the configuration specification (aka. desired state)
• Status - the actual provisioning status (aka. actual state)
• Owner - provides a parent/child relationship (for resource removal)

A Fluvio administrator configures the object Spec, and the cluster updates the object Status to match. The Status is a read-only element from the administrator’s perspective.

Fluvio has the following configuration objects:

• SPUs - streaming processing unit (custom or managed)
• SPGs - groups of managed SPUs
• topics - data streaming configuration element
• partitions - provisioned data streaming element of a topic
  • partitions are children of topics

Each configuration object goes through its own lifecycle. Object status tracks the state as it progresses through various lifecycle stages.

For detailed schema definition and object life cycles, checkout the Architecture Overview.

Each configuration object can converted to different data formats, such as json, or yaml. Additional data formats are available and can be exposed if required.

Configuration objects may be fetched using filters such as object name.

Consumers are also multi-threaded which allows each consumer to read records from multiple data streams simultaneously. Each connection can specify different retrieval properties:

• Consistency Model - retrieve records based on their committed state across replicas:
  • COMMITTED: fetch only the records that have been replicated n times (where n defined by min-live-replicas)
  • UNCOMMITTED: fetch records that have been stored by replica leader. When using UNCOMMITTED read strategy, it is possible to lose records that have already been seen by the consumers. Hence, it should only be used when sporadic message loss is acceptable.
• Max Bytes - the maximum number of bytes sent in single message.
  • When a consumer fetches multiple records, the SPU batches the result into buffers up to the maximum number of bytes.
  • Default batch size is 1Mb.

The Fluvio client can survive SPU failures. All data streams are replicated across multiple SPUs to prevent data loss.

When a data stream is created, one of the SPUs is elected as leader and the others become followers. Fluvio clients look-up the SPU leaders to produce or consume records.

If the SPU leader becomes unreachable, an election is triggered and one of the SPU followers becomes the leader. The client detects the SPU leader failure and automatically switches over to the new leader.

For additional information on the election algorithm, checkout Election Design.

The client library utilizes profiles to hide the complexity associated with the connection configuration. Furthermore, profiles allows the client library to manage multiple Fluvio clusters from the same client instance. Simply switch the profile and all subsequent operations are applied to a different cluster.

For additional information on Profile management, checkout Fluvio Profiles section.

All client operations follow a similar pattern.

1. Create a profile (one time operation).
2. Connect to the cluster, using the profile created above.
3. Use the Admin API to configure or retrieve objects (optional).
4. Produce or Consume records:
   • Use the Producer API to send records to a data stream.
   • Use the Consumer API to retrieve records from a data stream.

The Fluvio Client library is multi-threaded, and it can simultaneously connect to multiple clusters, and concurrently produce and consume one or more data streams.