-
-
Save fuyou001/1232136 to your computer and use it in GitHub Desktop.
Revisions
-
rednaxelafx revised this gist
Jul 14, 2011 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,4 +1,4 @@ A HiveServer running on JDK 6 update 18 early access, 32-bit Linux, is suffering from sudden raise of CPU usage. The process' `pid` is `400`. ``` $ java -version @@ -11,7 +11,7 @@ Linux xxxx.xxx 2.6.9-89.ELxenU #1 SMP Mon Apr 20 10:56:05 EDT 2009 i686 i686 i38 The steps taken to analyze the problem: 1. Use `top -H` to find out which threads are suffering from high CPU usage problem. See log1. We can target a specific process by using `top -p <pid> -H`, in this case that'd be `top -p 400 -H`. + Use `jstack <pid>` to find out what the threads are doing in this program. See log2. + The `nid`s in `jstack <pid>`'s output are hexadecimal representations of the `pid`s from `top -H`'s output (where as `jstack -F <pid>` and `jstack -m <pid>` show decimal representation of `pid`). + From the logs collected above, it's clear that the threads in question are: -
Jul 14, 2011 . 1 changed file with 2 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -26,10 +26,12 @@ The steps taken to analyze the problem: java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) ... "pool-1-thread-50" prio=10 tid=0x9a156000 nid=0x6920 runnable [0x9b35c000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) ... "pool-1-thread-51" prio=10 tid=0x9a134400 nid=0x15e7 runnable [0x9c10b000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) -
Jul 14, 2011 . 1 changed file with 1 addition and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -10,6 +10,7 @@ Linux xxxx.xxx 2.6.9-89.ELxenU #1 SMP Mon Apr 20 10:56:05 EDT 2009 i686 i686 i38 ``` The steps taken to analyze the problem: 1. Use `top -H` to find out which threads are suffering from high CPU usage problem. See log1. + Use `jstack <pid>` to find out what the threads are doing in this program. See log2. + The `nid`s in `jstack <pid>`'s output are hexadecimal representations of the `pid`s from `top -H`'s output (where as `jstack -F <pid>` and `jstack -m <pid>` show decimal representation of `pid`). -
Jul 14, 2011 . 1 changed file with 3 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,4 +1,5 @@ A HiveServer running on JDK 6 update 18 early access, 32-bit Linux, is suffering from sudden raise of CPU usage. ``` $ java -version java version "1.6.0_18-ea" @@ -13,6 +14,7 @@ The steps taken to analyze the problem: + Use `jstack <pid>` to find out what the threads are doing in this program. See log2. + The `nid`s in `jstack <pid>`'s output are hexadecimal representations of the `pid`s from `top -H`'s output (where as `jstack -F <pid>` and `jstack -m <pid>` show decimal representation of `pid`). + From the logs collected above, it's clear that the threads in question are: ``` "pool-1-thread-25" prio=10 tid=0x99389c00 nid=0x4aab runnable [0x9c1ac000] java.lang.Thread.State: RUNNABLE @@ -35,6 +37,7 @@ The steps taken to analyze the problem: ``` So all 4 threads in question are stuck in an endless loop in `HashMap.get()`: ```java public V get(Object key) { if (key == null) -
Jul 14, 2011 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,7 @@ $ top -H ... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19115 root 25 0 576m 217m 9432 R 99.7 5.3 1357:12 java 28347 root 25 0 576m 217m 9432 R 99.7 5.3 1191:05 java 26912 root 25 0 576m 217m 9432 R 99.3 5.3 114:58.57 java 5607 root 25 0 576m 217m 9432 R 99.0 5.3 54:49.02 java This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,302 @@ $ sudo -u root jstack 400 2011-07-14 10:27:41 Full thread dump Java HotSpot(TM) Server VM (16.0-b10 mixed mode): "Attach Listener" daemon prio=10 tid=0x09966400 nid=0x3597 runnable [0x00000000] java.lang.Thread.State: RUNNABLE "pool-1-thread-53" prio=10 tid=0x971f7800 nid=0x29c0 waiting on condition [0x9b7fe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xa175ed10> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) "pool-1-thread-51" prio=10 tid=0x9a134400 nid=0x15e7 runnable [0x9c10b000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415) at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159) at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(RDBMSMapEntrySetStore.java:327) at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java:221) - locked <0xa26c2110> (a org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore) at org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987) at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258) at org.datanucleus.sco.backed.Map.load(Map.java:229) at org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1609) at org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(PersistentClean.java:189) at org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470) at org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131) at org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534) at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551) - locked <0xa2672c58> (a org.datanucleus.jdo.JDOPersistenceManager) at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560) - locked <0xa2672c58> (a org.datanucleus.jdo.JDOPersistenceManager) at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:618) at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:551) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:924) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:921) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:921) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:558) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:515) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:475) at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:1838) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:176) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384) at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115) at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378) at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) "pool-1-thread-50" prio=10 tid=0x9a156000 nid=0x6920 runnable [0x9b35c000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415) at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159) at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(RDBMSMapEntrySetStore.java:322) at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java:221) - locked <0xa2636548> (a org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore) at org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987) at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258) at org.datanucleus.sco.backed.Map.load(Map.java:229) at org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1609) at org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(PersistentClean.java:189) at org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470) at org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131) at org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534) at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551) - locked <0xa2503a60> (a org.datanucleus.jdo.JDOPersistenceManager) at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560) - locked <0xa2503a60> (a org.datanucleus.jdo.JDOPersistenceManager) at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:618) at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:551) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:924) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:921) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:921) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:558) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:515) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:475) at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:1838) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:176) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384) at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115) at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378) at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) "LeaseChecker" daemon prio=10 tid=0x09af4c00 nid=0x29fd waiting on condition [0x9b3ad000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:978) at java.lang.Thread.run(Thread.java:619) "pool-1-thread-43" prio=10 tid=0x9b6e1000 nid=0x6ebb runnable [0x9bead000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415) at org.datanucleus.store.mapped.MappedTypeManager.isSupportedMappedType(MappedTypeManager.java:84) at org.datanucleus.store.rdbms.query.legacy.AbstractIteratorStatement.<init>(AbstractIteratorStatement.java:83) at org.datanucleus.store.rdbms.query.legacy.UnionIteratorStatement.<init>(UnionIteratorStatement.java:147) at org.datanucleus.store.rdbms.query.legacy.ClassTableExtent.newQueryStatement(ClassTableExtent.java:204) at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:323) at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225) at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175) at org.datanucleus.store.query.Query.executeQuery(Query.java:1628) at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245) at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499) at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:290) at org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:826) at org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:797) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(HiveMetaStore.java:1282) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(HiveMetaStore.java:1279) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:1279) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:542) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:919) at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:282) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:191) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384) at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115) at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378) at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) "pool-1-thread-25" prio=10 tid=0x99389c00 nid=0x4aab runnable [0x9c1ac000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185) at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415) at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159) at org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getSQLStatementForGet(RDBMSJoinMapStore.java:469) at org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getValue(RDBMSJoinMapStore.java:320) - locked <0xa1fed690> (a org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore) at org.datanucleus.store.mapped.scostore.JoinMapStore.putAll(JoinMapStore.java:167) at org.datanucleus.store.mapped.mapping.MapMapping.postInsert(MapMapping.java:135) at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:515) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121) at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275) at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249) at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428) at org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(PersistableMapping.java:664) at org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(PersistableMapping.java:423) at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:197) at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1023) at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideField(MStorageDescriptor.java) at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideFields(MStorageDescriptor.java) at org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(JDOStateManagerImpl.java:2803) - locked <0xa1fedaa8> (a java.lang.Object) at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:294) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121) at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275) at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249) at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428) at org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(PersistableMapping.java:664) at org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(PersistableMapping.java:423) at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:197) at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1023) at org.apache.hadoop.hive.metastore.model.MTable.jdoProvideField(MTable.java) at org.apache.hadoop.hive.metastore.model.MTable.jdoProvideFields(MTable.java) at org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(JDOStateManagerImpl.java:2803) - locked <0xa1fedc30> (a java.lang.Object) at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:294) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146) at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121) at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275) at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249) at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428) at org.datanucleus.ObjectManagerImpl.persistObject(ObjectManagerImpl.java:1241) at org.datanucleus.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:655) at org.datanucleus.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:680) - locked <0xa1fedc88> (a org.datanucleus.jdo.JDOPersistenceManager) at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:515) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:793) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$600(HiveMetaStore.java:79) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(HiveMetaStore.java:815) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(HiveMetaStore.java:812) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:812) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:279) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:394) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:2126) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:166) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384) at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115) at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378) at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) "MySQL Statement Cancellation Timer" daemon prio=10 tid=0x082ab800 nid=0x2b0a in Object.wait() [0x9ad9c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0xa1a09490> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0xa1a09490> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "Low Memory Detector" daemon prio=10 tid=0x08119000 nid=0x1ef runnable [0x00000000] java.lang.Thread.State: RUNNABLE "CompilerThread1" daemon prio=10 tid=0x08117800 nid=0x1ee waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "CompilerThread0" daemon prio=10 tid=0x08114400 nid=0x1ed waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x08112c00 nid=0x1ec runnable [0x00000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x080ff000 nid=0x1eb in Object.wait() [0x9d0f2000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0xa1717c28> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=0x080fd800 nid=0x1ea in Object.wait() [0x9d143000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0xa1749a58> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x0805d400 nid=0x1e4 runnable [0xb7471000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) - locked <0xa175ebe8> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118) at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34) at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183) at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:390) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:165) at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) "VM Thread" prio=10 tid=0x080f9c00 nid=0x1e9 runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x08064400 nid=0x1e5 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x08065c00 nid=0x1e6 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x08067400 nid=0x1e7 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x08068800 nid=0x1e8 runnable "VM Periodic Task Thread" prio=10 tid=0x0811b000 nid=0x1f0 waiting on condition JNI global references: 760 This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,264 @@ $ sudo -u root jstack -F 400 Attaching to process ID 400, please wait... Debugger attached successfully. Server compiler detected. JVM version is 16.0-b10 Deadlock Detection: No deadlocks found. Thread 13719: (state = BLOCKED) Thread 10688: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) - java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=158, line=422 (Compiled frame) - java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=323 (Interpreted frame) - java.util.concurrent.SynchronousQueue.take() @bci=7, line=857 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 5607: (state = IN_JAVA) - org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise) - org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame) - org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame) - org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Compiled frame) - org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(org.datanucleus.StateManager) @bci=137, line=327 (Compiled frame) - org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(org.datanucleus.StateManager) @bci=20, line=221 (Compiled frame) - org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(java.util.Map, org.datanucleus.store.scostore.MapStore, org.datanucleus.StateManager) @bci=164, line=987 (Compiled frame) - org.datanucleus.sco.backed.Map.loadFromStore() @bci=74, line=258 (Compiled frame) - org.datanucleus.sco.backed.Map.load() @bci=8, line=229 (Compiled frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields() @bci=160, line=1609 (Compiled frame) - org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(org.datanucleus.StateManager, boolean) @bci=14, line=189 (Interpreted frame) - org.datanucleus.state.AbstractStateManager.retrieve(boolean) @bci=11, line=470 (Interpreted frame) - org.datanucleus.ObjectManagerImpl.retrieveObject(java.lang.Object, boolean) @bci=72, line=1131 (Interpreted frame) - org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(java.lang.Object, boolean) @bci=6, line=534 (Interpreted frame) - org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object, boolean) @bci=7, line=551 (Interpreted frame) - org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object) @bci=3, line=560 (Interpreted frame) - org.apache.hadoop.hive.metastore.ObjectStore.getMTable(java.lang.String, java.lang.String) @bci=77, line=618 (Interpreted frame) - org.apache.hadoop.hive.metastore.ObjectStore.getTable(java.lang.String, java.lang.String) @bci=14, line=551 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=924 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=921 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(java.lang.String, java.lang.String) @bci=30, line=921 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(java.lang.String, java.lang.String) @bci=7, line=558 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String, java.lang.String, boolean) @bci=32, line=515 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String) @bci=7, line=475 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.DropTableDesc) @bci=7, line=1838 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=152, line=176 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Compiled frame) - org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Compiled frame) - org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame) - org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame) - org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame) - org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Compiled frame) - org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 26912: (state = IN_JAVA) - org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise) - org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame) - org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame) - org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Compiled frame) - org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(org.datanucleus.StateManager) @bci=94, line=322 (Compiled frame) - org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(org.datanucleus.StateManager) @bci=20, line=221 (Compiled frame) - org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(java.util.Map, org.datanucleus.store.scostore.MapStore, org.datanucleus.StateManager) @bci=164, line=987 (Compiled frame) - org.datanucleus.sco.backed.Map.loadFromStore() @bci=74, line=258 (Compiled frame) - org.datanucleus.sco.backed.Map.load() @bci=8, line=229 (Compiled frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields() @bci=160, line=1609 (Compiled frame) - org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(org.datanucleus.StateManager, boolean) @bci=14, line=189 (Interpreted frame) - org.datanucleus.state.AbstractStateManager.retrieve(boolean) @bci=11, line=470 (Interpreted frame) - org.datanucleus.ObjectManagerImpl.retrieveObject(java.lang.Object, boolean) @bci=72, line=1131 (Interpreted frame) - org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(java.lang.Object, boolean) @bci=6, line=534 (Interpreted frame) - org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object, boolean) @bci=7, line=551 (Interpreted frame) - org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object) @bci=3, line=560 (Interpreted frame) - org.apache.hadoop.hive.metastore.ObjectStore.getMTable(java.lang.String, java.lang.String) @bci=77, line=618 (Interpreted frame) - org.apache.hadoop.hive.metastore.ObjectStore.getTable(java.lang.String, java.lang.String) @bci=14, line=551 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=924 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=921 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(java.lang.String, java.lang.String) @bci=30, line=921 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(java.lang.String, java.lang.String) @bci=7, line=558 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String, java.lang.String, boolean) @bci=32, line=515 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String) @bci=7, line=475 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.DropTableDesc) @bci=7, line=1838 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=152, line=176 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Compiled frame) - org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Compiled frame) - org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame) - org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame) - org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame) - org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Compiled frame) - org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 10749: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) - org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run() @bci=73, line=978 (Compiled frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 28347: (state = IN_JAVA) - org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise) - org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame) - org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame) - org.datanucleus.store.mapped.MappedTypeManager.isSupportedMappedType(java.lang.String) @bci=21, line=84 (Interpreted frame) - org.datanucleus.store.rdbms.query.legacy.AbstractIteratorStatement.<init>(java.lang.Class, org.datanucleus.ClassLoaderResolver, boolean, org.datanucleus.store.StoreManager) @bci=88, line=83 (Interpreted frame) - org.datanucleus.store.rdbms.query.legacy.UnionIteratorStatement.<init>(org.datanucleus.ClassLoaderResolver, java.lang.Class, boolean, org.datanucleus.store.StoreManager, java.lang.Class, org.datanucleus.store.mapped.mapping.JavaTypeMapping, org.datanucleus.store.mapped.DatastoreContainerObject, boolean, java.lang.Boolean, boolean, boolean) @bci=6, line=147 (Interpreted frame) - org.datanucleus.store.rdbms.query.legacy.ClassTableExtent.newQueryStatement(java.lang.Class, org.datanucleus.store.mapped.DatastoreIdentifier) @bci=343, line=204 (Interpreted frame) - org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile() @bci=77, line=323 (Interpreted frame) - org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(int) @bci=567, line=225 (Interpreted frame) - org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(boolean, java.util.Map) @bci=60, line=175 (Interpreted frame) - org.datanucleus.store.query.Query.executeQuery(java.util.Map) @bci=51, line=1628 (Interpreted frame) - org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(java.util.Map) @bci=6, line=245 (Interpreted frame) - org.datanucleus.store.query.Query.executeWithArray(java.lang.Object[]) @bci=102, line=1499 (Interpreted frame) - org.datanucleus.jdo.JDOQuery.execute(java.lang.Object, java.lang.Object, java.lang.Object) @bci=20, line=290 (Interpreted frame) - org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(java.lang.String, java.lang.String, java.util.List) @bci=117, line=826 (Interpreted frame) - org.apache.hadoop.hive.metastore.ObjectStore.getPartition(java.lang.String, java.lang.String, java.util.List) @bci=10, line=797 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(org.apache.hadoop.hive.metastore.RawStore) @bci=13, line=1282 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=1279 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(java.lang.String, java.lang.String, java.util.List) @bci=33, line=1279 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(java.lang.String, java.lang.String, java.util.List) @bci=8, line=542 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.Hive.getPartition(org.apache.hadoop.hive.ql.metadata.Table, java.util.Map, boolean) @bci=190, line=919 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.AddPartitionDesc) @bci=28, line=282 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=227, line=191 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame) - org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame) - org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame) - org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Interpreted frame) - org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 19115: (state = IN_JAVA) - java.util.HashMap.get(java.lang.Object) @bci=74, line=303 (Compiled frame; information may be imprecise) - org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=9, line=186 (Compiled frame) - org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame) - org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame) - org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Interpreted frame) - org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getSQLStatementForGet(org.datanucleus.StateManager) @bci=220, line=469 (Interpreted frame) - org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getValue(org.datanucleus.StateManager, java.lang.Object) @bci=32, line=320 (Interpreted frame) - org.datanucleus.store.mapped.scostore.JoinMapStore.putAll(org.datanucleus.StateManager, java.util.Map) @bci=102, line=167 (Interpreted frame) - org.datanucleus.store.mapped.mapping.MapMapping.postInsert(org.datanucleus.StateManager) @bci=387, line=135 (Interpreted frame) - org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=1914, line=515 (Interpreted frame) - org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame) - org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame) - org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame) - org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=689, line=664 (Interpreted frame) - org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=25, line=423 (Interpreted frame) - org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(int, java.lang.Object) @bci=160, line=197 (Interpreted frame) - org.datanucleus.state.AbstractStateManager.providedObjectField(javax.jdo.spi.PersistenceCapable, int, java.lang.Object) @bci=6, line=1023 (Compiled frame) - org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideField(int) @bci=226 (Interpreted frame) - org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideFields(int[]) @bci=28 (Interpreted frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(int[], org.datanucleus.store.fieldmanager.FieldManager) @bci=23, line=2803 (Compiled frame) - org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=562, line=294 (Interpreted frame) - org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame) - org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame) - org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame) - org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=689, line=664 (Interpreted frame) - org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=25, line=423 (Interpreted frame) - org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(int, java.lang.Object) @bci=160, line=197 (Interpreted frame) - org.datanucleus.state.AbstractStateManager.providedObjectField(javax.jdo.spi.PersistenceCapable, int, java.lang.Object) @bci=6, line=1023 (Compiled frame) - org.apache.hadoop.hive.metastore.model.MTable.jdoProvideField(int) @bci=216 (Interpreted frame) - org.apache.hadoop.hive.metastore.model.MTable.jdoProvideFields(int[]) @bci=28 (Interpreted frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(int[], org.datanucleus.store.fieldmanager.FieldManager) @bci=23, line=2803 (Compiled frame) - org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=562, line=294 (Interpreted frame) - org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame) - org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame) - org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame) - org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame) - org.datanucleus.ObjectManagerImpl.persistObject(java.lang.Object) @bci=97, line=1241 (Interpreted frame) - org.datanucleus.jdo.JDOPersistenceManager.jdoMakePersistent(java.lang.Object) @bci=5, line=655 (Interpreted frame) - org.datanucleus.jdo.JDOPersistenceManager.makePersistent(java.lang.Object) @bci=16, line=680 (Interpreted frame) - org.apache.hadoop.hive.metastore.ObjectStore.createTable(org.apache.hadoop.hive.metastore.api.Table) @bci=18, line=515 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(org.apache.hadoop.hive.metastore.RawStore, org.apache.hadoop.hive.metastore.api.Table) @bci=371, line=793 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$600(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler, org.apache.hadoop.hive.metastore.RawStore, org.apache.hadoop.hive.metastore.api.Table) @bci=3, line=79 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=815 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=812 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(org.apache.hadoop.hive.metastore.api.Table) @bci=55, line=812 (Interpreted frame) - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(org.apache.hadoop.hive.metastore.api.Table) @bci=24, line=279 (Interpreted frame) - org.apache.hadoop.hive.ql.metadata.Hive.createTable(org.apache.hadoop.hive.ql.metadata.Table, boolean) @bci=69, line=394 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.createTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.CreateTableDesc) @bci=771, line=2126 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=102, line=166 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame) - org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Interpreted frame) - org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame) - org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame) - org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame) - org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Interpreted frame) - org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 11018: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=28, line=483 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 492: (state = BLOCKED) Thread 491: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame) - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame) Thread 490: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise) - java.lang.Object.wait() @bci=2, line=485 (Compiled frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame) Thread 484: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=390 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame) - org.apache.thrift.transport.TServerSocket.acceptImpl() @bci=22, line=118 (Interpreted frame) - org.apache.thrift.transport.TServerSocket.acceptImpl() @bci=1, line=34 (Interpreted frame) - org.apache.thrift.transport.TServerTransport.accept() @bci=1, line=31 (Interpreted frame) - org.apache.thrift.server.TThreadPoolServer.serve() @bci=39, line=183 (Interpreted frame) - org.apache.hadoop.hive.service.HiveServer.main(java.lang.String[]) @bci=114, line=390 (Interpreted frame) - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame) - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame) - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame) - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Interpreted frame) - org.apache.hadoop.util.RunJar.main(java.lang.String[]) @bci=595, line=165 (Interpreted frame) - org.apache.hadoop.mapred.JobShell.run(java.lang.String[]) @bci=12, line=54 (Interpreted frame) - org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.conf.Configuration, org.apache.hadoop.util.Tool, java.lang.String[]) @bci=38, line=65 (Interpreted frame) - org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.util.Tool, java.lang.String[]) @bci=8, line=79 (Interpreted frame) - org.apache.hadoop.mapred.JobShell.main(java.lang.String[]) @bci=10, line=68 (Interpreted frame) This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,391 @@ $ sudo -u root jstack -m 400 | c++filt Attaching to process ID 400, please wait... Debugger attached successfully. Server compiler detected. JVM version is 16.0-b10 Deadlock Detection: No deadlocks found. ----------------- 484 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0x9cb93e19 Java_java_net_PlainSocketImpl_socketAccept + 0x129 0xb43c25aa * java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) bci:0 (Interpreted frame) 0xb43bb08d * java.net.PlainSocketImpl.accept(java.net.SocketImpl) bci:7 line:390 (Interpreted frame) 0xb43bb08d * java.net.ServerSocket.implAccept(java.net.Socket) bci:60 line:453 (Interpreted frame) 0xb43bb08d * java.net.ServerSocket.accept() bci:48 line:421 (Interpreted frame) 0xb43baf27 * org.apache.thrift.transport.TServerSocket.acceptImpl() bci:22 line:118 (Interpreted frame) 0xb43baf27 * org.apache.thrift.transport.TServerSocket.acceptImpl() bci:1 line:34 (Interpreted frame) 0xb43baf27 * org.apache.thrift.transport.TServerTransport.accept() bci:1 line:31 (Interpreted frame) 0xb43baf27 * org.apache.thrift.server.TThreadPoolServer.serve() bci:39 line:183 (Interpreted frame) 0xb43bb08d * org.apache.hadoop.hive.service.HiveServer.main(java.lang.String[]) bci:114 line:390 (Interpreted frame) 0xb43b834c <StubRoutines> 0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190 0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18 0xb77f66cf JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) + 0x2f 0xb7a1d92b Reflection::invoke(instanceKlassHandle, methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*) + 0x44b 0xb7a20891 Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*) + 0x111 0xb787267f JVM_InvokeMethod + 0x1bf 0xb7406314 Java_sun_reflect_NativeMethodAccessorImpl_invoke0 + 0x34 0xb43c25aa * sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) bci:0 (Interpreted frame) 0xb43baf27 * sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:87 line:39 (Interpreted frame) 0xb43baf27 * sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:6 line:25 (Interpreted frame) 0xb43bb403 * java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) bci:161 line:597 (Interpreted frame) 0xb43baf27 * org.apache.hadoop.util.RunJar.main(java.lang.String[]) bci:595 line:165 (Interpreted frame) 0xb43bb08d * org.apache.hadoop.mapred.JobShell.run(java.lang.String[]) bci:12 line:54 (Interpreted frame) 0xb43bb445 * org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.conf.Configuration, org.apache.hadoop.util.Tool, java.lang.String[]) bci:38 line:65 (Interpreted frame) 0xb43baf69 * org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.util.Tool, java.lang.String[]) bci:8 line:79 (Interpreted frame) 0xb43baf69 * org.apache.hadoop.mapred.JobShell.main(java.lang.String[]) bci:10 line:68 (Interpreted frame) 0xb43b834c <StubRoutines> 0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190 0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18 0xb77f66cf JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) + 0x2f 0xb7828c93 jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) + 0x133 0xb7818eec jni_CallStaticVoidMethod + 0x13c 0x08049b98 JavaMain + 0x2c8 0x00623413 start_thread + 0xb3 ----------------- 485 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb79aebce Monitor::IWait(Thread*, long long) + 0xee 0xb79af405 Monitor::wait(bool, long, bool) + 0x255 0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66 0xb7796c9b GCTaskThread::run() + 0xdb 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 486 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb79aebce Monitor::IWait(Thread*, long long) + 0xee 0xb79af405 Monitor::wait(bool, long, bool) + 0x255 0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66 0xb7796c9b GCTaskThread::run() + 0xdb 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 487 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb79aebce Monitor::IWait(Thread*, long long) + 0xee 0xb79af405 Monitor::wait(bool, long, bool) + 0x255 0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66 0xb7796c9b GCTaskThread::run() + 0xdb 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 488 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb79aebce Monitor::IWait(Thread*, long long) + 0xee 0xb79af405 Monitor::wait(bool, long, bool) + 0x255 0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66 0xb7796c9b GCTaskThread::run() + 0xdb 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 489 -----------------sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4 0x004a57a2 _dl_sysinfo_int80 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) + at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) 0x2 0x020cd378 ? at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)? ?????? at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) ----------------- 490 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb 0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53 0xb7855a1c JVM_MonitorWait + 0x20c 0xb4522f12 <Unknown compiled code> 0xb488d994 <Unknown compiled code> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x096de19e with alignment: 4 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) 0x9d77d6a8 ???????? ----------------- 491 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb 0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53 0xb7855a1c JVM_MonitorWait + 0x20c 0xb4522f12 <Unknown compiled code> 0xb43bb08d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:118 (Interpreted frame) 0xb483ef14 * java.lang.ref.ReferenceQueue.remove() bci:2 line:134 (Compiled frame) * java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159 (Compiled frame) sun.jvm.hotspot.debugger.UnmappedAddressException at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208) at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63) at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) ----------------- 492 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79cb56d os::signal_wait() + 0xd 0xb79c7b2a signal_thread_entry(JavaThread*, Thread*) + 0x7a 0xb7ab2c3e JavaThread::run() + 0x11e 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 493 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb79aebce Monitor::IWait(Thread*, long long) + 0xee 0xb79af29e Monitor::wait(bool, long, bool) + 0xee 0xb76e2ebf CompileQueue::get() + 0x6f 0xb76e4aaa CompileBroker::compiler_thread_loop() + 0x1ea 0xb7ab9586 compiler_thread_entry(JavaThread*, Thread*) + 0x16 0xb7ab2c3e JavaThread::run() + 0x11e 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 494 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb79aebce Monitor::IWait(Thread*, long long) + 0xee 0xb79af29e Monitor::wait(bool, long, bool) + 0xee 0xb76e2ebf CompileQueue::get() + 0x6f 0xb76e4aaa CompileBroker::compiler_thread_loop() + 0x1ea 0xb7ab9586 compiler_thread_entry(JavaThread*, Thread*) + 0x16 0xb7ab2c3e JavaThread::run() + 0x11e 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 495 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb79aebce Monitor::IWait(Thread*, long long) + 0xee 0xb79af405 Monitor::wait(bool, long, bool) + 0x255 0xb79652bf LowMemoryDetector::low_memory_detector_thread_entry(JavaThread*, Thread*) + 0x9f 0xb7ab2c3e JavaThread::run() + 0x11e 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 496 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) 0x00dd7750 ???????? ----------------- 11018 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79ce7f5 os::PlatformEvent::park() + 0xe5 0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb 0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53 0xb7855a1c JVM_MonitorWait + 0x20c 0xb43c25aa * java.lang.Object.wait(long) bci:0 (Interpreted frame) 0xb43bb08d * java.lang.Object.wait() bci:2 line:485 (Interpreted frame) 0xb43bb08d * java.util.TimerThread.mainLoop() bci:28 line:483 (Interpreted frame) 0xb43bb08d * java.util.TimerThread.run() bci:1 line:462 (Interpreted frame) 0xb43b834c <StubRoutines> 0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190 0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18 0xb77f6077 JavaCalls::call_virtual(JavaValue*, KlassHandle, symbolHandle, symbolHandle, JavaCallArguments*, Thread*) + 0xb7 0xb77f612a JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, symbolHandle, symbolHandle, Thread*) + 0x6a 0xb7876b95 thread_entry(JavaThread*, Thread*) + 0xa5 0xb7ab2c3e JavaThread::run() + 0x11e 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 19115 ----------------- 0xb45706dd <Unknown compiled code> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) 0x9d777780 ???????? ----------------- 28347 ----------------- 0xb45706e0 <Unknown compiled code> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) 0x9d777780 ???????? ----------------- 10749 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) 0x02fe3088 ???????? ----------------- 26912 ----------------- 0xb45707bf <Unknown compiled code> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) 0x9d777780 ???????? ----------------- 5607 ----------------- 0xb45706e0 <Unknown compiled code> sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) 0x9d777780 ???????? ----------------- 10688 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb79cee18 Parker::park(bool, long long) + 0x2d8 0xb7adb660 Unsafe_Park + 0x1d0 0xb43c25aa * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame) 0xb43bb08d * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:158 (Interpreted frame) 0xb4994050 * java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:158 line:422 (Compiled frame) sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74) at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60) at sun.jvm.hotspot.tools.PStack.run(PStack.java:161) at sun.jvm.hotspot.tools.PStack.run(PStack.java:53) at sun.jvm.hotspot.tools.PStack.run(PStack.java:48) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) 0x9dbb1848 ???????? ----------------- 13719 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0xb75ce9fb AttachListener::dequeue() + 0x7b 0xb75cdb97 attach_listener_thread_entry(JavaThread*, Thread*) + 0x47 0xb7ab2c3e JavaThread::run() + 0x11e 0xb79cf15e java_start(Thread*) + 0x14e 0x00623413 start_thread + 0xb3 ----------------- 400 ----------------- 0x004a57a2 _dl_sysinfo_int80 + 0x2 0x0804dd78 ContinueInNewThread + 0x88 0x080497f6 main + 0x276 0x004d3df3 __libc_start_main + 0xd3 This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,55 @@ A HiveServer running on JDK 6 update 18 early access, 32-bit Linux, is suffering from sudden raise of CPU usage. ``` $ java -version java version "1.6.0_18-ea" Java(TM) SE Runtime Environment (build 1.6.0_18-ea-b03) Java HotSpot(TM) Server VM (build 16.0-b10, mixed mode) $ uname -a Linux xxxx.xxx 2.6.9-89.ELxenU #1 SMP Mon Apr 20 10:56:05 EDT 2009 i686 i686 i386 GNU/Linux ``` The steps taken to analyze the problem: 1. Use `top -H` to find out which threads are suffering from high CPU usage problem. See log1. + Use `jstack <pid>` to find out what the threads are doing in this program. See log2. + The `nid`s in `jstack <pid>`'s output are hexadecimal representations of the `pid`s from `top -H`'s output (where as `jstack -F <pid>` and `jstack -m <pid>` show decimal representation of `pid`). + From the logs collected above, it's clear that the threads in question are: ``` "pool-1-thread-25" prio=10 tid=0x99389c00 nid=0x4aab runnable [0x9c1ac000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) ... "pool-1-thread-43" prio=10 tid=0x9b6e1000 nid=0x6ebb runnable [0x9bead000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) "pool-1-thread-50" prio=10 tid=0x9a156000 nid=0x6920 runnable [0x9b35c000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) "pool-1-thread-51" prio=10 tid=0x9a134400 nid=0x15e7 runnable [0x9c10b000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186) ... ``` So all 4 threads in question are stuck in an endless loop in `HashMap.get()`: ```java public V get(Object key) { if (key == null) return getForNullKey(); int hash = hash(key.hashCode()); for (Entry<K,V> e = table[indexFor(hash, table.length)]; e != null; e = e.next) { Object k; if (e.hash == hash && ((k = e.key) == key || key.equals(k))) return e.value; } return null; } ``` where line 303 is `e = e.next`. Now we can come to the conclusion that the program has hit a known issue of misusing `HashMap`: unprotected concurrent access to `HashMap` may cause data to be corrupted, which in turn may cause `HashMap.get()` to fall into an endless loop.