Skip to content

Instantly share code, notes, and snippets.

@bwann

bwann/gist:9fce7b27d94e70918e85e39cd7a4fcf8

Created October 15, 2018 23:08
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save bwann/9fce7b27d94e70918e85e39cd7a4fcf8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save bwann/9fce7b27d94e70918e85e39cd7a4fcf8 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. bwann created this gist Oct 15, 2018.
    65 changes: 65 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,65 @@
    2018-10-15 16:06:48,287 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,288 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,288 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,288 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,288 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,289 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,289 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,289 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,289 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,290 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,290 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,290 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,290 INFO [AsyncHBase I/O Worker #3] HBaseClient: Added client for region RegionInfo(table="tsdb", region_name="tsdb,\x00\x01<X&?`\x00\x00\x
    01\x00\x01\xC4\x00\x00\x08\x00\x00U\x00\x00\x0E\x00\x00S,1522508611724.27a44ad052647662fd0cc8509fe35a04.", stop_key=[0, 1, 62, 87, 112, 108, -128, 0, 0, 1, 0, 0
    , -100, 0, 0, 8, 0, 0, 87, 0, 0, 14, 0, 0, 83]), which was updated in the regions cache. Now we know that RegionClient@1183367206(chan=[id: 0xbc11eb49, /192.16
    8.130.10:46660 => /192.168.135.23:16020], #pending_rpcs=0, #batched=119, #rpcs_inflight=1) is hosting 36 regions.
    2018-10-15 16:06:48,290 INFO [AsyncHBase I/O Worker #3] HBaseClient: Resolved IP address for host: basic04.wann.net is: 192.168.135.23
    2018-10-15 16:06:48,290 INFO [AsyncHBase I/O Worker #3] HBaseClient: Added client for region RegionInfo(table="tsdb", region_name="tsdb,\x00\x01>Wpl\x80\x00\x0
    0\x01\x00\x00\x9C\x00\x00\x08\x00\x00W\x00\x00\x0E\x00\x00S,1522508611724.8b473b89ebc63a1a3f36b74b395854fb.", stop_key=[0, 1, 64, 85, -110, 34, 80, 0, 0, 1, 0,
    0, -108, 0, 0, 15, 0, 0, 88, 0, 0, 16, 0, 0, 89]), which was added to the regions cache. Now we know that RegionClient@1183367206(chan=[id: 0xbc11eb49, /192.16
    8.130.10:46660 => /192.168.135.23:16020], #pending_rpcs=0, #batched=120, #rpcs_inflight=0) is hosting 37 regions.
    2018-10-15 16:06:48,644 ERROR [AsyncHBase I/O Worker #3] RegionClient: Uncaught error during de-serialization of MultiAction(batch=[PutRequest(table="tsdb", key=[0, 0, 106, 91, -59, 27, -16, 0, 0, 1, 0, 3, -78], family="t", qualifiers=["\x19#"], values=[[24, -68, 106, -33]], timestamp=1539644802000, lockid=-1, durable=true, bufferable=true, attempt=3, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00iP&\xB9\xC0\x00\x00\x01\x00\x00],1367455284500.15e5527bcf780c299075698c19c6e592.", stop_key=[0, 0, 107, 80, 84, 81, -96, 0, 0, 1, 0, 0, -121])), PutRequest(table="tsdb", key=[0, 0, 107, 91, -59, 27, -16, 0, 0, 1, 0, 3, -78], family="t", qualifiers=["\x19#"], values=[[0, 37, -123, -8]], timestamp=1539644802000, lockid=-1, durable=true, bufferable=true, attempt=3, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00kPTQ\xA0\x00\x00\x01\x00\x00\x87,1364772770201.9007d89d8f0eb0edd2c0396dab45b68e.", stop_key=[0, 0, 108, 80, 63, -30, 96, 0, 0, 1, 0, 0, -78, 0, 0, 3, 0, 0, 62, 0, 0, 9, 0, 0, 61])), PutRequest(table="tsdb", key=[0, 0, 108, 91, -59, 27, -16, 0, 0, 1, 0, 3, -78, 0, 0, 6, 0, 0, 22], family="t", qualifiers=["\x19#"], values=[[0, 98, 25, -83]], timestamp=1539644802000, lockid=-1, durable=true, bufferable=true, attempt=3, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x00lX+\x14\xE0\x00\x00\x01\x00\x00\xF5\x00\x00\x06\x00\x00\x1B,1513409342560.6ee06ee4c60cff2b3b8e6c2c2e577c88.", stop_key=[0, 0, 109, 80, 62, 116, -64, 0, 0, 1, 0, 0, -78, 0, 0, 9, 0, 0, 61])), ... 117 RPCs not shown ..., PutRequest(table="tsdb", key=[0, 11, 12, 91, -59, 27, -16, 0, 0, 1, 0, 1, -60, 0, 0, 8, 0, 7, -75, 0, 0, 52, 0, 0, -117], family="t", qualifiers=["\x193"], values=[[19, -36, -117, -112]], timestamp=1539644803000, lockid=-1, durable=true, bufferable=true, attempt=2, region=RegionInfo(table="tsdb", region_name="tsdb,\x00\x0B\x0CZ\xEF\xEB`\x00\x00\x01\x00\x00\x9C\x00\x00\x08\x00\x07\xB2\x00\x004\x00\x00\x85,1532542801561.2d35c174dcbaed5dc62d5f2718ae6382.", stop_key=[0, 11, 13, 86, -17, -101, -16, 0, 0, 1, 0, 0, 116, 0, 0, 8, 0, 7, -73]))]), rpcid=94
    2018-10-15 16:06:48,646 ERROR [AsyncHBase I/O Worker #3] RegionClient: Unexpected exception from downstream on [id: 0xbc11eb49, /192.168.130.10:46660 => /192.168.135.23:16020]

    2018-10-15 16:06:48,646 ERROR [AsyncHBase I/O Worker #3] RegionClient: Unexpected exception from downstream on [id: 0xbc11eb49, /192.168.130.10:46660 => /192.168.135.23:16020]
    java.lang.AssertionError: Must not be used.
    at org.hbase.async.HBaseException.make(HBaseException.java:62) ~[asynchbase-1.8.0.jar:na]
    at org.hbase.async.RegionClient$1MultiActionCallback.handleException(RegionClient.java:903) ~[asynchbase-1.8.0.jar:na]
    at org.hbase.async.RegionClient$1MultiActionCallback.call(RegionClient.java:867) ~[asynchbase-1.8.0.jar:na]
    at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[async-1.4.0.jar:na]
    at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[async-1.4.0.jar:na]
    at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) ~[async-1.4.0.jar:na]
    at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712) ~[asynchbase-1.8.0.jar:na]
    at org.hbase.async.RegionClient.decode(RegionClient.java:1512) ~[asynchbase-1.8.0.jar:na]
    at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.8.0.jar:na]
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]
    at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1226) ~[asynchbase-1.8.0.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]
    at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3678) [asynchbase-1.8.0.jar:na]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
    2018-10-15 16:06:48,647 INFO [AsyncHBase I/O Worker #3] HBaseClient: Channel [id: 0xbc11eb49, /192.168.130.10:46660 => /192.168.135.23:16020] is disconnecting: [id: 0xbc11eb49, /192.168.130.10:46660 => /192.168.135.23:16020] CLOSE
    2018-10-15 16:06:48,647 INFO [AsyncHBase I/O Worker #3] HBaseClient: Lost connection with the .META. region