kafka.producer package

Submodules

kafka.producer.base module

class kafka.producer.base.Producer(client, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20)[source]

Bases: object

Base class to be used by producers

Parameters:
  • client – The Kafka client instance to use
  • async – If set to true, the messages are sent asynchronously via another thread (process). We will not wait for a response to these WARNING!!! current implementation of async producer does not guarantee message delivery. Use at your own risk! Or help us improve with a PR!
  • req_acks – A value indicating the acknowledgements that the server must receive before responding to the request
  • ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
  • batch_send – If True, messages are send in batches
  • batch_send_every_n – If set, messages are send in batches of this size
  • batch_send_every_t – If set, messages are send after this timeout
ACK_AFTER_CLUSTER_COMMIT = -1
ACK_AFTER_LOCAL_WRITE = 1
ACK_NOT_REQUIRED = 0
DEFAULT_ACK_TIMEOUT = 1000
send_messages(topic, partition, *msg)[source]

Helper method to send produce requests @param: topic, name of topic for produce request – type str @param: partition, partition number for produce request – type int @param: *msg, one or more message payloads – type bytes @returns: ResponseRequest returned by server raises on error

Note that msg type must be encoded to bytes by user. Passing unicode message will not work, for example you should encode before calling send_messages via something like unicode_message.encode(‘utf-8’)

All messages produced via this method will set the message ‘key’ to Null

stop(timeout=1)[source]

Stop the producer. Optionally wait for the specified timeout before forcefully cleaning up.

kafka.producer.keyed module

class kafka.producer.keyed.KeyedProducer(client, partitioner=None, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20)[source]

Bases: kafka.producer.base.Producer

A producer which distributes messages to partitions based on the key

Parameters:

client – The kafka client instance

Keyword Arguments:
 
  • partitioner – A partitioner class that will be used to get the partition to send the message to. Must be derived from Partitioner
  • async – If True, the messages are sent asynchronously via another thread (process). We will not wait for a response to these
  • ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
  • batch_send – If True, messages are send in batches
  • batch_send_every_n – If set, messages are send in batches of this size
  • batch_send_every_t – If set, messages are send after this timeout
send(topic, key, msg)[source]
send_messages(topic, key, *msg)[source]

kafka.producer.simple module

class kafka.producer.simple.SimpleProducer(client, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20, random_start=True)[source]

Bases: kafka.producer.base.Producer

A simple, round-robin producer. Each message goes to exactly one partition

Parameters:

client – The Kafka client instance to use

Keyword Arguments:
 
  • async – If True, the messages are sent asynchronously via another thread (process). We will not wait for a response to these
  • req_acks – A value indicating the acknowledgements that the server must receive before responding to the request
  • ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
  • batch_send – If True, messages are send in batches
  • batch_send_every_n – If set, messages are send in batches of this size
  • batch_send_every_t – If set, messages are send after this timeout
  • random_start – If true, randomize the initial partition which the the first message block will be published to, otherwise if false, the first message block will always publish to partition 0 before cycling through each partition
send_messages(topic, *msg)[source]

Module contents

class kafka.producer.SimpleProducer(client, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20, random_start=True)[source]

Bases: kafka.producer.base.Producer

A simple, round-robin producer. Each message goes to exactly one partition

Parameters:

client – The Kafka client instance to use

Keyword Arguments:
 
  • async – If True, the messages are sent asynchronously via another thread (process). We will not wait for a response to these
  • req_acks – A value indicating the acknowledgements that the server must receive before responding to the request
  • ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
  • batch_send – If True, messages are send in batches
  • batch_send_every_n – If set, messages are send in batches of this size
  • batch_send_every_t – If set, messages are send after this timeout
  • random_start – If true, randomize the initial partition which the the first message block will be published to, otherwise if false, the first message block will always publish to partition 0 before cycling through each partition
send_messages(topic, *msg)[source]
class kafka.producer.KeyedProducer(client, partitioner=None, async=False, req_acks=1, ack_timeout=1000, codec=None, batch_send=False, batch_send_every_n=20, batch_send_every_t=20)[source]

Bases: kafka.producer.base.Producer

A producer which distributes messages to partitions based on the key

Parameters:

client – The kafka client instance

Keyword Arguments:
 
  • partitioner – A partitioner class that will be used to get the partition to send the message to. Must be derived from Partitioner
  • async – If True, the messages are sent asynchronously via another thread (process). We will not wait for a response to these
  • ack_timeout – Value (in milliseconds) indicating a timeout for waiting for an acknowledgement
  • batch_send – If True, messages are send in batches
  • batch_send_every_n – If set, messages are send in batches of this size
  • batch_send_every_t – If set, messages are send after this timeout
send(topic, key, msg)[source]
send_messages(topic, key, *msg)[source]