Apache Kafka is a distributed event-streaming platform designed for high throughput and fault tolerance. Among its many configurations, min.insync.replicas plays an important role in ensuring durability and consistency of messages. This configuration defines the minimum number of replicas that must acknowledge a write before a producer can consider the write successful.
This article explains the purpose of min.insync.replicas, its interaction with producer acknowledgments, and how it impacts message reliability. Understanding this configuration is essential for configuring Kafka brokers and topics effectively to meet your application’s delivery guarantees.
What is min.insync.replicas?
The min.insync.replicas configuration specifies the minimum number of replicas that must be in sync and acknowledge a write for it to be considered successful. This parameter works in conjunction with the producer’s acks configuration to enforce strong durability guarantees.
- Default Value: 1
- Scope: Topic-level or broker-level.
Relationship with In-Sync Replicas (ISR)
In Kafka, In-Sync Replicas (ISR) are replicas that have fully caught up with the leader’s log. The min.insync.replicas configuration ensures that a minimum number of replicas in the ISR must acknowledge a write before it is committed.
How min.insync.replicas Works
The behavior of min.insync.replicas is influenced by the producer’s acknowledgment (acks) settings:
acks=all:- The producer waits for acknowledgment from all replicas specified by
min.insync.replicas. - If the number of in-sync replicas falls below
min.insync.replicas, the broker will reject the write, and the producer receives an error. - Ensures strong durability guarantees.
- The producer waits for acknowledgment from all replicas specified by
acks=1oracks=0:min.insync.replicashas no effect as the producer does not wait for acknowledgments from replicas.
Configuring min.insync.replicas
Broker-Level Configuration
Set min.insync.replicas in the server.properties file to apply it globally across all topics.
Example:
min.insync.replicas=2 This ensures that a minimum of two replicas must acknowledge a write for every topic in the cluster.
Topic-Level Configuration
To apply the configuration to a specific topic, use the kafka-topics.sh tool:
Example:
kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics \
--entity-name my-topic \
--alter \
--add-config min.insync.replicas=2 Verify the setting:
kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type topics \
--entity-name my-topic \
--describe Output:
Configs for topic 'my-topic':
min.insync.replicas=2 Impact of min.insync.replicas on Durability
The min.insync.replicas setting affects the trade-off between durability and availability:
- Durability:
With a highermin.insync.replicas, Kafka ensures stronger delivery guarantees. However, if the number of in-sync replicas drops below the configured value, producers withacks=allwill fail to write. - Availability:
Lowering the value increases availability at the cost of weaker durability. Writes are accepted even if fewer replicas acknowledge the message.
Example Scenarios
1: High Durability Requirements
- Configuration:
min.insync.replicas=3- Producer
acks=all
- Outcome:
- Writes are successful only when at least three replicas in the ISR acknowledge the message.
- Ensures that data is durably written across multiple nodes.
2: Balancing Durability and Availability
- Configuration:
min.insync.replicas=2- Producer
acks=all
- Outcome:
- Writes are allowed as long as two replicas in the ISR are available.
- Provides moderate durability without compromising on availability significantly.
3: Low Durability Requirements
- Configuration:
min.insync.replicas=1- Producer
acks=1
- Outcome:
- Writes succeed even if only one replica acknowledges the message.
- Data loss is possible if the leader fails before replication completes.
Common Issues and Troubleshooting min.insync.replicas
Producer Errors with acks=all:
If the number of in-sync replicas falls below min.insync.replicas, producers with acks=all will receive errors.
- Solution: Troubleshoot any issues with brokers that may be preventing the replicas being in sync, or temporarily reduce
min.insync.replicasor increase replication factor to restore availability.
ISR Shrinking:
ISR may shrink due to slow replicas or network issues, impacting min.insync.replicas.
- Solution: Investigate and resolve replica lag using metrics like
UnderReplicatedPartitions.
Balancing Availability and Durability:
- Overly strict
min.insync.replicasvalues can reduce availability during maintenance or failures.- Solution: Adjust the value based on your system’s fault tolerance and workload requirements.
Best Practices for min.insync.replicas
Align with Replication Factor:
Set min.insync.replicas slightly lower than the replication factor. For example, with a replication factor of 3, use min.insync.replicas=2 to tolerate one replica failure.
Monitor ISR:
Regularly monitor ISR metrics to ensure that replicas are staying in sync.
Use with acks=all:
Combine min.insync.replicas with acks=all to enforce stronger durability guarantees.
Test Before Deployment:
Validate your configuration in a staging environment to ensure it meets application requirements.
Reference Links
- Apache Kafka Documentation: min.insync.replicas
- Apache Kafka: Replication
- Confluent Documentation: Durability in Kafka
- Kafka GitHub Repository
