Skip to content

Instantly share code, notes, and snippets.

@okoye

okoye/kafka logs

Created January 16, 2019 00:25
Show Gist options
  • Save okoye/14b63fa3ff88804794a004ce4505bb0b to your computer and use it in GitHub Desktop.
Save okoye/14b63fa3ff88804794a004ce4505bb0b to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. okoye created this gist Jan 16, 2019.
    127 changes: 127 additions & 0 deletions kafka logs
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,127 @@
    WARNING: Could not determine Java version
    [2019-01-15 16:23:59,933] INFO Verbose logging enabled (io.confluent.cloud.Main$BaseCommand)
    [2019-01-15 16:23:59,946] INFO Config values:
    admin.client.timeout.ms = 5000
    bootstrap.servers = [pkc-43xj4.us-east-1.aws.confluent.cloud:9092]
    (io.confluent.cloud.Config)
    [2019-01-15 16:23:59,958] INFO AdminClientConfig values:
    bootstrap.servers = [pkc-43xj4.us-east-1.aws.confluent.cloud:9092]
    client.id =
    connections.max.idle.ms = 300000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 20000
    retries = 5
    retry.backoff.ms = 500
    sasl.jaas.config = [hidden]
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.mechanism = PLAIN
    security.protocol = SASL_SSL
    send.buffer.bytes = 131072
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    (org.apache.kafka.clients.admin.AdminClientConfig)
    [2019-01-15 16:24:00,262] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin)
    [2019-01-15 16:24:00,438] DEBUG Added sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,445] DEBUG Added sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,445] DEBUG Added sensor with name successful-authentication: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,445] DEBUG Added sensor with name failed-authentication: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,446] DEBUG Added sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,446] DEBUG Added sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,447] DEBUG Added sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,448] DEBUG Added sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,449] DEBUG Added sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:00,638] DEBUG Updated cluster metadata version 1 to Cluster(id = null, nodes = [pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: -1 rack: null)], partitions = []) (org.apache.kafka.clients.Metadata)
    [2019-01-15 16:24:00,644] INFO Kafka version : 1.0.0 (org.apache.kafka.common.utils.AppInfoParser)
    [2019-01-15 16:24:00,644] INFO Kafka commitId : aaa7af6d4a11b29d (org.apache.kafka.common.utils.AppInfoParser)
    [2019-01-15 16:24:00,647] DEBUG [AdminClient clientId=adminclient-1] Kafka admin client initialized (org.apache.kafka.clients.admin.KafkaAdminClient)
    [2019-01-15 16:24:00,659] DEBUG [AdminClient clientId=adminclient-1] Queueing Call(callName=listTopics, deadlineMs=1547598260658) with a timeout 20000 ms from now. (org.apache.kafka.clients.admin.KafkaAdminClient)
    [2019-01-15 16:24:01,140] DEBUG [AdminClient clientId=adminclient-1] Initialize connection to node pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:01,140] DEBUG [AdminClient clientId=adminclient-1] Initiating connection to node pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:01,209] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:01,213] DEBUG Creating SaslClient: client=null;service=kafka;serviceHostname=pkc-43xj4.us-east-1.aws.confluent.cloud;mechs=[PLAIN] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:01,247] DEBUG Added sensor with name node--1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:01,248] DEBUG Added sensor with name node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:01,249] DEBUG Added sensor with name node--1.latency (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:01,252] DEBUG [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 66608, SO_SNDBUF = 131768, SO_TIMEOUT = 0 to node -1 (org.apache.kafka.common.network.Selector)
    [2019-01-15 16:24:01,352] DEBUG [AdminClient clientId=adminclient-1] Completed connection to node -1. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:01,584] DEBUG SSL handshake completed successfully with peerHost 'pkc-43xj4.us-east-1.aws.confluent.cloud' peerPort 9092 peerPrincipal 'CN=*.us-east-1.aws.confluent.cloud' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer)
    [2019-01-15 16:24:01,587] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:01,678] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:01,679] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:01,762] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:01,764] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:01,967] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:01,967] DEBUG [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -1. (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:02,073] DEBUG [AdminClient clientId=adminclient-1] Recorded API versions for node -1: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0 to 7 [usable: 6], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0 to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3 [usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10): 0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12): 0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14): 0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1], ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2 [usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21): 0 [usable: 0], InitProducerId(22): 0 [usable: 0], OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0 [usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0 [usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0 [usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to 1 [usable: 0], AlterConfigs(33): 0 [usable: 0], AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0 [usable: 0], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 [usable: 0], UNKNOWN(38): 0, UNKNOWN(39): 0, UNKNOWN(40): 0, UNKNOWN(41): 0, UNKNOWN(42): 0) (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:02,074] DEBUG [AdminClient clientId=adminclient-1] Sending metadata request (type=MetadataRequest, topics=) to node pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:02,193] DEBUG Updated cluster metadata version 2 to Cluster(id = lkc-4j83w, nodes = [b4-pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: 4 rack: 1), b0-pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: 0 rack: 0), b1-pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: 1 rack: 1), b5-pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: 5 rack: 2), b3-pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: 3 rack: 0), b2-pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: 2 rack: 2)], partitions = []) (org.apache.kafka.clients.Metadata)
    [2019-01-15 16:24:02,193] DEBUG [AdminClient clientId=adminclient-1] Initiating connection to node b0-pkc-43xj4.us-east-1.aws.confluent.cloud:9092 (id: 0 rack: 0) (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:02,352] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:02,352] DEBUG Creating SaslClient: client=null;service=kafka;serviceHostname=b0-pkc-43xj4.us-east-1.aws.confluent.cloud;mechs=[PLAIN] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:02,426] DEBUG Added sensor with name node-0.bytes-sent (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:02,428] DEBUG Added sensor with name node-0.bytes-received (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:02,430] DEBUG Added sensor with name node-0.latency (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:02,431] DEBUG [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 66608, SO_SNDBUF = 131768, SO_TIMEOUT = 0 to node 0 (org.apache.kafka.common.network.Selector)
    [2019-01-15 16:24:02,431] DEBUG [AdminClient clientId=adminclient-1] Completed connection to node 0. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:02,649] DEBUG SSL handshake completed successfully with peerHost 'b0-pkc-43xj4.us-east-1.aws.confluent.cloud' peerPort 9092 peerPrincipal 'CN=*.us-east-1.aws.confluent.cloud' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer)
    [2019-01-15 16:24:02,649] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:02,731] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:02,732] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:03,344] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:03,345] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:03,613] DEBUG Set SASL client state to CLIENT_COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:03,614] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
    [2019-01-15 16:24:03,614] DEBUG [AdminClient clientId=adminclient-1] Initiating API versions fetch from node 0. (org.apache.kafka.clients.NetworkClient)
    [2019-01-15 16:24:03,744] DEBUG [AdminClient clientId=adminclient-1] Recorded API versions for node 0: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0 to 7 [usable: 6], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0 to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3 [usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10): 0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12): 0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14): 0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1], ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2 [usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21): 0 [usable: 0], InitProducerId(22): 0 [usable: 0], OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0 [usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0 [usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0 [usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to 1 [usable: 0], AlterConfigs(33): 0 [usable: 0], AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0 [usable: 0], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 [usable: 0], UNKNOWN(38): 0, UNKNOWN(39): 0, UNKNOWN(40): 0, UNKNOWN(41): 0, UNKNOWN(42): 0) (org.apache.kafka.clients.NetworkClient)
    production.av.backend.derived_data
    av-metrics-mission-ingest-production
    av-lyftbag-document-production
    production.av.lyftbag_doc
    production.av.backend.metrics_mission_ingest
    production.av.backend.lyftbag_doc
    production.av.camera_doc
    production.av.backend.camera_doc
    production.av.backend.yolo_label
    [2019-01-15 16:24:04,170] DEBUG [AdminClient clientId=adminclient-1] Initiating close operation. (org.apache.kafka.clients.admin.KafkaAdminClient)
    [2019-01-15 16:24:04,170] DEBUG [AdminClient clientId=adminclient-1] Waiting for the I/O thread to exit. Hard shutdown in 31536000000 ms. (org.apache.kafka.clients.admin.KafkaAdminClient)
    [2019-01-15 16:24:04,172] DEBUG Removed sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,172] DEBUG Removed sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,173] DEBUG Removed sensor with name successful-authentication: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,173] DEBUG Removed sensor with name failed-authentication: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,173] DEBUG Removed sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,173] DEBUG Removed sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,174] DEBUG Removed sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,174] DEBUG Removed sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,174] DEBUG Removed sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,174] DEBUG Removed sensor with name node--1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,175] DEBUG Removed sensor with name node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,175] DEBUG Removed sensor with name node--1.latency (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,175] DEBUG Removed sensor with name node-0.bytes-sent (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,175] DEBUG Removed sensor with name node-0.bytes-received (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,175] DEBUG Removed sensor with name node-0.latency (org.apache.kafka.common.metrics.Metrics)
    [2019-01-15 16:24:04,176] DEBUG [AdminClient clientId=adminclient-1] Exiting AdminClientRunnable thread. (org.apache.kafka.clients.admin.KafkaAdminClient)
    [2019-01-15 16:24:04,176] DEBUG [AdminClient clientId=adminclient-1] Kafka admin client closed. (org.apache.kafka.clients.admin.KafkaAdminClient)