re: streaming - kafka partition transition time from (stage change logger)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

re: streaming - kafka partition transition time from (stage change logger)

Peter Liu
Hi there,

Working on the streaming processing latency time based on timestamps from Kafka, I have two quick general questions triggered by looking at the kafka stage change log file:

(a) the partition state change from OfflineReplica state to OnlinePartition state seems to take more than 20 sec. Would it mean that an incoming message/event into kafka would need to go thru all these state transitions (see below) to become ready for consumer (in this case, after 20+ sec)? 

(b) For which stage would be the kafka time stampe for?

Any help/clarification would be very much appreciated!

Thanks ...

Peter

[2018-06-08 15:34:36,518] TRACE Controller 0 epoch 53 changed state of replica 0 for partition [events,79] from ReplicaDeletionIneligible to OfflineReplica (state.change.logger)
[2018-06-08 15:34:36,945] TRACE Controller 0 epoch 53 changed state of replica 0 for partition [events,79] from OfflineReplica to OnlineReplica (state.change.logger)
[2018-06-08 15:34:37,025] TRACE Controller 0 epoch 53 sending become-follower LeaderAndIsr request (Leader:-1,ISR:,LeaderEpoch:1,ControllerEpoch:53) to broker 0 for partition [events,79] (state.change.logger)
[2018-06-08 15:34:37,079] TRACE Broker 0 received LeaderAndIsr request PartitionState(controllerEpoch=53, leader=-1, leaderEpoch=1, isr=[], zkVersion=1, replicas=[0]) correlation id 1 from controller 0 epoch 53 for partition [events,79] (state.change.logger)
[2018-06-08 15:34:38,481] TRACE Controller 0 epoch 53 changed state of replica 0 for partition [events,79] from OnlineReplica to OfflineReplica (state.change.logger)
[2018-06-08 15:34:38,588] TRACE Controller 0 epoch 53 changed state of replica 0 for partition [events,79] from OfflineReplica to ReplicaDeletionStarted (state.change.logger)
[2018-06-08 15:34:39,427] TRACE Controller 0 epoch 53 changed state of replica 0 for partition [events,79] from ReplicaDeletionStarted to ReplicaDeletionSuccessful (state.change.logger)
[2018-06-08 15:34:39,560] TRACE Controller 0 epoch 53 changed state of replica 0 for partition [events,79] from ReplicaDeletionSuccessful to NonExistentReplica (state.change.logger)
[2018-06-08 15:34:39,564] TRACE Controller 0 epoch 53 changed partition [events,79] state from OfflinePartition to OfflinePartition (state.change.logger)
[2018-06-08 15:34:39,571] TRACE Controller 0 epoch 53 changed partition [events,79] state from OfflinePartition to NonExistentPartition (state.change.logger)
[2018-06-08 15:35:01,893] TRACE Controller 0 epoch 53 changed partition [events,79] state from NonExistentPartition to NewPartition with assigned replicas 0 (state.change.logger)
[2018-06-08 15:35:01,960] TRACE Controller 0 epoch 53 changed state of replica 0 for partition [events,79] from NonExistentReplica to NewReplica (state.change.logger)
[2018-06-08 15:35:02,026] TRACE Controller 0 epoch 53 changed partition [events,79] from NewPartition to OnlinePartition with leader 0 (state.change.logger)
[2018-06-08 15:35:02,207] TRACE Controller 0 epoch 53 sending become-leader LeaderAndIsr request (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:53) to broker 0 for partition [events,79] (state.change.logger)
[2018-06-08 15:35:02,219] TRACE Broker 0 received LeaderAndIsr request PartitionState(controllerEpoch=53, leader=0, leaderEpoch=0, isr=[0], zkVersion=0, replicas=[0]) correlation id 88 from controller 0 epoch 53 for partition [events,79] (state.change.logger)