SlideShare a Scribd company logo
bluedavy
2012-03
   Java Runtime Data Area

                 -Xss

              Local Vars     Native method Stack
               Operator                         -XX:PermSize –
       PC                   Method Area        XX:MaxPermSize
              Stack Frame

            Method Stack       Heap           -Xms -Xmx
   Oracle JDK Java Heap

      -Xmn   New Generation

         Eden               S0      S1   Old Generation
                -XX:SurvivorRatio
   Oracle JDK startup options for memory manage
     -Xms –Xmx –Xmn –XX:PermSize –XX:MaxPermSize
     -XX:SurvivorRatio
     -XX:+UseParallelGC || -XX:+UseParallelOldGC
      ▪ -XX:MaxTenuringThreshold -XX:ParallelGCThreads
      ▪ -XX:-UseAdaptiveSizePolicy
     -XX:+UseConcMarkSweepGC
      ▪ -XX:CMSInitiatingOccupancyFraction -
        XX:+UseCMSInitiatingOccupancyOnly
     -XX:+DisableExplicitGC –
     XX:+ExplicitGCInvokesConcurrent
   Startup options or tools for memory problem solution
       -XX:+PrintGCDateStamps –XX:+PrintGCDetails -Xloggc
       -XX:+HeapDumpOnOutOfMemoryError
       -XX:+HeapDumpBeforeFullGC
       -XX:+PrintFlagsFinal(JDK 6u21+)
       for CMS GC
        ▪ -XX:+PrintPromotionFailure -XX:+CMSDumpAtPromotionFailure
       jinfo
       jstat
       jmap
       MAT
       btrace
       google perf-tools
   OOM
     java.lang.OutOfMemoryError: {$reason}
   Full GC Frequently
   CMS GC
     promotion failed
     concurrent mode failure
Java memory problem cases solutions
Java memory problem cases solutions
-Xms4g –Xmx4g –Xmn2560m
Java memory problem cases solutions
Java memory problem cases solutions
Then to find who use Deflater.init
   java.lang.OutOfMemoryError: {$reason}
     GC overhead limit exceeded
     Java Heap Space
     Unable to create new native thread
     PermGen Space
     Direct buffer memory
     request {} bytes for {}. Out of swap space?
   GC overhead limit exceeded
    Java Heap Space
     if code didn’t catch OutOfMemoryError,thread
      will exit;
     slow response or no response at all
      ▪ because gc did frequently


   Out of Swap
     Java process crash
   GC overhead limit exceeded
    Java Heap Space
     monitor app log & gc log
     heap dump or jmap –histo|-dump when oom
     mat analyze heap dump
     app developer or btrace to find where cause
   Out of swap
     crash log
     google-perftools
     then btrace to find where cause


     ps: maybe u can try delete -XX:+DisableExplicitGC
     if exist,reason can be found in this link。
   use ThreadLocal carefully;
   limit Collection/StringBuilder etc. size;
   limit batch ops size;
   limit the data size return by database;
   avoid infinite loop;
   OOM
     java.lang.OutOfMemoryError: {$reason}
   Full GC Frequently
   CMS GC
     promotion failed
     concurrent mode failure
   Most Cases
     because memory consumes too much
   a special case
     a special frequently GC Case
   slow response or no response at all
   According gc log to see if the reason is
    consuming too much memory
     if yes then add -XX:+HeapDumpBeforeFullGC or
      write a shell file to dump memory when full gc;
     if no then use pstack & source code to see why full
      gc executes;
   OOM
     java.lang.OutOfMemoryError: {$reason}
   Full GC Frequently
   CMS GC
     promotion failed
     concurrent mode failure
   promotion failed Case I
     363439.937: [Full GC 363439.938: [ParNew
      (promotion failed): 523840K->523840K(523840K),
      0.3404490 secs] 1690606K->1712399K(1834560K),
      0.3412880 secs]
     Taobao JDK help to find the reason
      ▪ ==WARNNING== allocating large array: thread_id[0x00002aaac2d85800],
        thread_name[ajp-0.0.0.0-8009-48], array_size[782611408 bytes],
        array_length[195652847 elememts]
   promotion failed Case II
     1768.432: [GC 1768.432: [ParNew (promotion
      failed): 1572827K->1572827K(1572864K),
      0.2845480 secs]
     1972.208: [CMS: 10545160K-
      >5853155K(14680064K), 8.5355320 secs]
      12105682K->5853155K(16252928K), [CMS Perm :
      20907K->20873K(34952K)], 212.3113910 secs]
      [Times: user=53.38 sys=22.45, real=212.28 secs]
     because cms gc fragmention
   four cases
     promotion failed case III
      ▪ because CMSInitiatingOccupancyFraction too high...
     concurrent mode failure case I
      ▪ because CMSInitiatingOccupancyFraction too high...
     concurrent mode failure case II
      ▪ because old gen is too small
     concurrent mode failure case III
      ▪ because permgen
   CMS GC problem will cause serial full gc,so
    the app response time will be very slow...
   According gc log to see if the reason is
    because CMSInitiatingOccupancyFraction is
    too high,if yes then adjust the option;
   if no,then
     promotion failed
      ▪ if because memory ran out,then dump memory;
      ▪ if because cms gc fragmention,currently we only can do
        is execute jmap –histo:live at regular time;
     concurrent mode failure
      ▪ the same as promotion failed...
   OOM
     java.lang.OutOfMemoryError: {$reason}
   Full GC频繁
   CMS GC
     promotion failed
     concurrent mode failure
   Young GC slow
   CMS GC Concurrent-Mark causes app stop...
   ...
Ad

More Related Content

What's hot (20)

Pick diamonds from garbage
Pick diamonds from garbagePick diamonds from garbage
Pick diamonds from garbage
Tier1 App
 
Shared Memory Performance: Beyond TCP/IP with Ben Cotton, JPMorgan
Shared Memory Performance: Beyond TCP/IP with Ben Cotton, JPMorganShared Memory Performance: Beyond TCP/IP with Ben Cotton, JPMorgan
Shared Memory Performance: Beyond TCP/IP with Ben Cotton, JPMorgan
Hazelcast
 
Am I reading GC logs Correctly?
Am I reading GC logs Correctly?Am I reading GC logs Correctly?
Am I reading GC logs Correctly?
Tier1 App
 
NoSQL 동향
NoSQL 동향NoSQL 동향
NoSQL 동향
NAVER D2
 
7 jvm-arguments-v1
7 jvm-arguments-v17 jvm-arguments-v1
7 jvm-arguments-v1
Tier1 app
 
Java 어플리케이션 성능튜닝 Part1
Java 어플리케이션 성능튜닝 Part1Java 어플리케이션 성능튜닝 Part1
Java 어플리케이션 성능튜닝 Part1
상욱 송
 
Jvm problem diagnostics
Jvm problem diagnosticsJvm problem diagnostics
Jvm problem diagnostics
Danijel Mitar
 
Thanos - Prometheus on Scale
Thanos - Prometheus on ScaleThanos - Prometheus on Scale
Thanos - Prometheus on Scale
Bartłomiej Płotka
 
7 jvm-arguments-Confoo
7 jvm-arguments-Confoo7 jvm-arguments-Confoo
7 jvm-arguments-Confoo
Tier1 app
 
Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)
Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)
Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)
Red Hat Developers
 
Redis ndc2013
Redis ndc2013Redis ndc2013
Redis ndc2013
DaeMyung Kang
 
9.4 the hard way
9.4 the hard way9.4 the hard way
9.4 the hard way
Samantha Billington
 
Jvm & Garbage collection tuning for low latencies application
Jvm & Garbage collection tuning for low latencies applicationJvm & Garbage collection tuning for low latencies application
Jvm & Garbage collection tuning for low latencies application
Quentin Ambard
 
Become a Garbage Collection Hero
Become a Garbage Collection HeroBecome a Garbage Collection Hero
Become a Garbage Collection Hero
Tier1app
 
Cram
CramCram
Cram
JamesBonfield
 
JVM Tuning and Profiling
JVM Tuning and ProfilingJVM Tuning and Profiling
JVM Tuning and Profiling
Bhuvan Rawal
 
7 Ways To Crash Postgres
7 Ways To Crash Postgres7 Ways To Crash Postgres
7 Ways To Crash Postgres
PostgreSQL Experts, Inc.
 
Exactly once with spark streaming
Exactly once with spark streamingExactly once with spark streaming
Exactly once with spark streaming
Quentin Ambard
 
Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)
Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)
Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)
Ontico
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 
Pick diamonds from garbage
Pick diamonds from garbagePick diamonds from garbage
Pick diamonds from garbage
Tier1 App
 
Shared Memory Performance: Beyond TCP/IP with Ben Cotton, JPMorgan
Shared Memory Performance: Beyond TCP/IP with Ben Cotton, JPMorganShared Memory Performance: Beyond TCP/IP with Ben Cotton, JPMorgan
Shared Memory Performance: Beyond TCP/IP with Ben Cotton, JPMorgan
Hazelcast
 
Am I reading GC logs Correctly?
Am I reading GC logs Correctly?Am I reading GC logs Correctly?
Am I reading GC logs Correctly?
Tier1 App
 
NoSQL 동향
NoSQL 동향NoSQL 동향
NoSQL 동향
NAVER D2
 
7 jvm-arguments-v1
7 jvm-arguments-v17 jvm-arguments-v1
7 jvm-arguments-v1
Tier1 app
 
Java 어플리케이션 성능튜닝 Part1
Java 어플리케이션 성능튜닝 Part1Java 어플리케이션 성능튜닝 Part1
Java 어플리케이션 성능튜닝 Part1
상욱 송
 
Jvm problem diagnostics
Jvm problem diagnosticsJvm problem diagnostics
Jvm problem diagnostics
Danijel Mitar
 
7 jvm-arguments-Confoo
7 jvm-arguments-Confoo7 jvm-arguments-Confoo
7 jvm-arguments-Confoo
Tier1 app
 
Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)
Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)
Shenandoah GC: Java Without The Garbage Collection Hiccups (Christine Flood)
Red Hat Developers
 
Jvm & Garbage collection tuning for low latencies application
Jvm & Garbage collection tuning for low latencies applicationJvm & Garbage collection tuning for low latencies application
Jvm & Garbage collection tuning for low latencies application
Quentin Ambard
 
Become a Garbage Collection Hero
Become a Garbage Collection HeroBecome a Garbage Collection Hero
Become a Garbage Collection Hero
Tier1app
 
JVM Tuning and Profiling
JVM Tuning and ProfilingJVM Tuning and Profiling
JVM Tuning and Profiling
Bhuvan Rawal
 
Exactly once with spark streaming
Exactly once with spark streamingExactly once with spark streaming
Exactly once with spark streaming
Quentin Ambard
 
Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)
Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)
Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)
Ontico
 
Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.
Alexey Lesovsky
 

Viewers also liked (19)

VMIL keynote : Lessons from a production JVM runtime developer
VMIL keynote : Lessons from a production JVM runtime developerVMIL keynote : Lessons from a production JVM runtime developer
VMIL keynote : Lessons from a production JVM runtime developer
Mark Stoodley
 
账务系统设计及应用
账务系统设计及应用账务系统设计及应用
账务系统设计及应用
wang zaixiang
 
Git 实战
Git 实战Git 实战
Git 实战
简放 视野
 
高性能的Java代码编写及常见问题排查
高性能的Java代码编写及常见问题排查高性能的Java代码编写及常见问题排查
高性能的Java代码编写及常见问题排查
bluedavy lin
 
C1000K高性能服务器构建技术
C1000K高性能服务器构建技术C1000K高性能服务器构建技术
C1000K高性能服务器构建技术
Feng Yu
 
Implementing a JavaScript Engine
Implementing a JavaScript EngineImplementing a JavaScript Engine
Implementing a JavaScript Engine
Kris Mok
 
Java常见问题排查
Java常见问题排查Java常见问题排查
Java常见问题排查
bluedavy lin
 
线上问题排查交流
线上问题排查交流线上问题排查交流
线上问题排查交流
简放 视野
 
沒有 GUI 的 Git
沒有 GUI 的 Git沒有 GUI 的 Git
沒有 GUI 的 Git
Chia Wei Tsai
 
Strata Beijing - Deep Learning in Production on Spark
Strata Beijing - Deep Learning in Production on SparkStrata Beijing - Deep Learning in Production on Spark
Strata Beijing - Deep Learning in Production on Spark
Adam Gibson
 
美团点评技术沙龙14:美团四层负载均衡
美团点评技术沙龙14:美团四层负载均衡美团点评技术沙龙14:美团四层负载均衡
美团点评技术沙龙14:美团四层负载均衡
美团点评技术团队
 
如何针对业务做DB优化
如何针对业务做DB优化如何针对业务做DB优化
如何针对业务做DB优化
Jinrong Ye
 
程序猿都该知道的MySQL秘籍
程序猿都该知道的MySQL秘籍程序猿都该知道的MySQL秘籍
程序猿都该知道的MySQL秘籍
Jinrong Ye
 
寫給大家的 Git 教學
寫給大家的 Git 教學寫給大家的 Git 教學
寫給大家的 Git 教學
littlebtc
 
MySQL技术分享:一步到位实现mysql优化
MySQL技术分享:一步到位实现mysql优化MySQL技术分享:一步到位实现mysql优化
MySQL技术分享:一步到位实现mysql优化
Jinrong Ye
 
20120613联动优势数据访问层DAL架构和实践4(刘胜)最新特性
20120613联动优势数据访问层DAL架构和实践4(刘胜)最新特性20120613联动优势数据访问层DAL架构和实践4(刘胜)最新特性
20120613联动优势数据访问层DAL架构和实践4(刘胜)最新特性
liu sheng
 
2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShare2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShare
SlideShare
 
What to Upload to SlideShare
What to Upload to SlideShareWhat to Upload to SlideShare
What to Upload to SlideShare
SlideShare
 
Getting Started With SlideShare
Getting Started With SlideShareGetting Started With SlideShare
Getting Started With SlideShare
SlideShare
 
VMIL keynote : Lessons from a production JVM runtime developer
VMIL keynote : Lessons from a production JVM runtime developerVMIL keynote : Lessons from a production JVM runtime developer
VMIL keynote : Lessons from a production JVM runtime developer
Mark Stoodley
 
账务系统设计及应用
账务系统设计及应用账务系统设计及应用
账务系统设计及应用
wang zaixiang
 
高性能的Java代码编写及常见问题排查
高性能的Java代码编写及常见问题排查高性能的Java代码编写及常见问题排查
高性能的Java代码编写及常见问题排查
bluedavy lin
 
C1000K高性能服务器构建技术
C1000K高性能服务器构建技术C1000K高性能服务器构建技术
C1000K高性能服务器构建技术
Feng Yu
 
Implementing a JavaScript Engine
Implementing a JavaScript EngineImplementing a JavaScript Engine
Implementing a JavaScript Engine
Kris Mok
 
Java常见问题排查
Java常见问题排查Java常见问题排查
Java常见问题排查
bluedavy lin
 
线上问题排查交流
线上问题排查交流线上问题排查交流
线上问题排查交流
简放 视野
 
Strata Beijing - Deep Learning in Production on Spark
Strata Beijing - Deep Learning in Production on SparkStrata Beijing - Deep Learning in Production on Spark
Strata Beijing - Deep Learning in Production on Spark
Adam Gibson
 
美团点评技术沙龙14:美团四层负载均衡
美团点评技术沙龙14:美团四层负载均衡美团点评技术沙龙14:美团四层负载均衡
美团点评技术沙龙14:美团四层负载均衡
美团点评技术团队
 
如何针对业务做DB优化
如何针对业务做DB优化如何针对业务做DB优化
如何针对业务做DB优化
Jinrong Ye
 
程序猿都该知道的MySQL秘籍
程序猿都该知道的MySQL秘籍程序猿都该知道的MySQL秘籍
程序猿都该知道的MySQL秘籍
Jinrong Ye
 
寫給大家的 Git 教學
寫給大家的 Git 教學寫給大家的 Git 教學
寫給大家的 Git 教學
littlebtc
 
MySQL技术分享:一步到位实现mysql优化
MySQL技术分享:一步到位实现mysql优化MySQL技术分享:一步到位实现mysql优化
MySQL技术分享:一步到位实现mysql优化
Jinrong Ye
 
20120613联动优势数据访问层DAL架构和实践4(刘胜)最新特性
20120613联动优势数据访问层DAL架构和实践4(刘胜)最新特性20120613联动优势数据访问层DAL架构和实践4(刘胜)最新特性
20120613联动优势数据访问层DAL架构和实践4(刘胜)最新特性
liu sheng
 
2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShare2015 Upload Campaigns Calendar - SlideShare
2015 Upload Campaigns Calendar - SlideShare
SlideShare
 
What to Upload to SlideShare
What to Upload to SlideShareWhat to Upload to SlideShare
What to Upload to SlideShare
SlideShare
 
Getting Started With SlideShare
Getting Started With SlideShareGetting Started With SlideShare
Getting Started With SlideShare
SlideShare
 
Ad

Similar to Java memory problem cases solutions (20)

JVM memory management & Diagnostics
JVM memory management & DiagnosticsJVM memory management & Diagnostics
JVM memory management & Diagnostics
Dhaval Shah
 
Mastering java in containers - MadridJUG
Mastering java in containers - MadridJUGMastering java in containers - MadridJUG
Mastering java in containers - MadridJUG
Jorge Morales
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryError
Tier1 app
 
Performance tuning jvm
Performance tuning jvmPerformance tuning jvm
Performance tuning jvm
Prem Kuppumani
 
7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx
Tier1 app
 
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
Jelastic Multi-Cloud PaaS
 
How to Check and Optimize Memory Size for Better Application Performance
How to Check and Optimize Memory Size for Better Application PerformanceHow to Check and Optimize Memory Size for Better Application Performance
How to Check and Optimize Memory Size for Better Application Performance
Tier1 app
 
16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem
Tier1 app
 
Gc algorithms
Gc algorithmsGc algorithms
Gc algorithms
Michał Warecki
 
QCon 2017 - Java/JVM com Docker em produção: lições das trincheiras
QCon 2017 - Java/JVM com Docker em produção: lições das trincheirasQCon 2017 - Java/JVM com Docker em produção: lições das trincheiras
QCon 2017 - Java/JVM com Docker em produção: lições das trincheiras
Leonardo Zanivan
 
Cold fusion is racecar fast
Cold fusion is racecar fastCold fusion is racecar fast
Cold fusion is racecar fast
ColdFusionConference
 
Garbage collection in JVM
Garbage collection in JVMGarbage collection in JVM
Garbage collection in JVM
aragozin
 
Spotting Trouble Early: What Java GC Patterns Can Tell You
Spotting Trouble Early: What Java GC Patterns Can Tell YouSpotting Trouble Early: What Java GC Patterns Can Tell You
Spotting Trouble Early: What Java GC Patterns Can Tell You
KumarNagaraju4
 
Jvm gc那点事
Jvm gc那点事Jvm gc那点事
Jvm gc那点事
boboj
 
millions-gc-jax-2022.pptx
millions-gc-jax-2022.pptxmillions-gc-jax-2022.pptx
millions-gc-jax-2022.pptx
Tier1 app
 
Elastic JVM for Scalable Java EE Applications Running in Containers #Jakart...
Elastic JVM  for Scalable Java EE Applications  Running in Containers #Jakart...Elastic JVM  for Scalable Java EE Applications  Running in Containers #Jakart...
Elastic JVM for Scalable Java EE Applications Running in Containers #Jakart...
Jelastic Multi-Cloud PaaS
 
Troubleshooting Memory Problems in Java Applications
Troubleshooting Memory Problems in Java ApplicationsTroubleshooting Memory Problems in Java Applications
Troubleshooting Memory Problems in Java Applications
Poonam Bajaj Parhar
 
Garbage collection in JVM
Garbage collection in JVMGarbage collection in JVM
Garbage collection in JVM
aragozin
 
this-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptxthis-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptx
Tier1 app
 
Inside the JVM - Performance & Garbage Collector Tuning in JAVA
Inside the JVM - Performance & Garbage Collector Tuning in JAVA Inside the JVM - Performance & Garbage Collector Tuning in JAVA
Inside the JVM - Performance & Garbage Collector Tuning in JAVA
Rafael Monteiro e Pereira
 
JVM memory management & Diagnostics
JVM memory management & DiagnosticsJVM memory management & Diagnostics
JVM memory management & Diagnostics
Dhaval Shah
 
Mastering java in containers - MadridJUG
Mastering java in containers - MadridJUGMastering java in containers - MadridJUG
Mastering java in containers - MadridJUG
Jorge Morales
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryError
Tier1 app
 
Performance tuning jvm
Performance tuning jvmPerformance tuning jvm
Performance tuning jvm
Prem Kuppumani
 
7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx
Tier1 app
 
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
State of Java Elasticity. Tuning Java Efficiency - GIDS.JAVA LIVE 2020
Jelastic Multi-Cloud PaaS
 
How to Check and Optimize Memory Size for Better Application Performance
How to Check and Optimize Memory Size for Better Application PerformanceHow to Check and Optimize Memory Size for Better Application Performance
How to Check and Optimize Memory Size for Better Application Performance
Tier1 app
 
16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem
Tier1 app
 
QCon 2017 - Java/JVM com Docker em produção: lições das trincheiras
QCon 2017 - Java/JVM com Docker em produção: lições das trincheirasQCon 2017 - Java/JVM com Docker em produção: lições das trincheiras
QCon 2017 - Java/JVM com Docker em produção: lições das trincheiras
Leonardo Zanivan
 
Garbage collection in JVM
Garbage collection in JVMGarbage collection in JVM
Garbage collection in JVM
aragozin
 
Spotting Trouble Early: What Java GC Patterns Can Tell You
Spotting Trouble Early: What Java GC Patterns Can Tell YouSpotting Trouble Early: What Java GC Patterns Can Tell You
Spotting Trouble Early: What Java GC Patterns Can Tell You
KumarNagaraju4
 
Jvm gc那点事
Jvm gc那点事Jvm gc那点事
Jvm gc那点事
boboj
 
millions-gc-jax-2022.pptx
millions-gc-jax-2022.pptxmillions-gc-jax-2022.pptx
millions-gc-jax-2022.pptx
Tier1 app
 
Elastic JVM for Scalable Java EE Applications Running in Containers #Jakart...
Elastic JVM  for Scalable Java EE Applications  Running in Containers #Jakart...Elastic JVM  for Scalable Java EE Applications  Running in Containers #Jakart...
Elastic JVM for Scalable Java EE Applications Running in Containers #Jakart...
Jelastic Multi-Cloud PaaS
 
Troubleshooting Memory Problems in Java Applications
Troubleshooting Memory Problems in Java ApplicationsTroubleshooting Memory Problems in Java Applications
Troubleshooting Memory Problems in Java Applications
Poonam Bajaj Parhar
 
Garbage collection in JVM
Garbage collection in JVMGarbage collection in JVM
Garbage collection in JVM
aragozin
 
this-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptxthis-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptx
Tier1 app
 
Inside the JVM - Performance & Garbage Collector Tuning in JAVA
Inside the JVM - Performance & Garbage Collector Tuning in JAVA Inside the JVM - Performance & Garbage Collector Tuning in JAVA
Inside the JVM - Performance & Garbage Collector Tuning in JAVA
Rafael Monteiro e Pereira
 
Ad

More from bluedavy lin (9)

Java内存管理问题案例分享
Java内存管理问题案例分享Java内存管理问题案例分享
Java内存管理问题案例分享
bluedavy lin
 
HBase@taobao for 技术沙龙
HBase@taobao for 技术沙龙HBase@taobao for 技术沙龙
HBase@taobao for 技术沙龙
bluedavy lin
 
Hbase简介与实践分享
Hbase简介与实践分享Hbase简介与实践分享
Hbase简介与实践分享
bluedavy lin
 
菜鸟看Hbase
菜鸟看Hbase菜鸟看Hbase
菜鸟看Hbase
bluedavy lin
 
Sun JDK 1.6内存管理 -调优篇
Sun JDK 1.6内存管理 -调优篇Sun JDK 1.6内存管理 -调优篇
Sun JDK 1.6内存管理 -调优篇
bluedavy lin
 
Sun jdk 1.6内存管理 -使用篇
Sun jdk 1.6内存管理 -使用篇Sun jdk 1.6内存管理 -使用篇
Sun jdk 1.6内存管理 -使用篇
bluedavy lin
 
Sun jdk 1.6 gc
Sun jdk 1.6 gcSun jdk 1.6 gc
Sun jdk 1.6 gc
bluedavy lin
 
并发编程交流
并发编程交流并发编程交流
并发编程交流
bluedavy lin
 
OSGi理论与实战
OSGi理论与实战OSGi理论与实战
OSGi理论与实战
bluedavy lin
 
Java内存管理问题案例分享
Java内存管理问题案例分享Java内存管理问题案例分享
Java内存管理问题案例分享
bluedavy lin
 
HBase@taobao for 技术沙龙
HBase@taobao for 技术沙龙HBase@taobao for 技术沙龙
HBase@taobao for 技术沙龙
bluedavy lin
 
Hbase简介与实践分享
Hbase简介与实践分享Hbase简介与实践分享
Hbase简介与实践分享
bluedavy lin
 
Sun JDK 1.6内存管理 -调优篇
Sun JDK 1.6内存管理 -调优篇Sun JDK 1.6内存管理 -调优篇
Sun JDK 1.6内存管理 -调优篇
bluedavy lin
 
Sun jdk 1.6内存管理 -使用篇
Sun jdk 1.6内存管理 -使用篇Sun jdk 1.6内存管理 -使用篇
Sun jdk 1.6内存管理 -使用篇
bluedavy lin
 
并发编程交流
并发编程交流并发编程交流
并发编程交流
bluedavy lin
 
OSGi理论与实战
OSGi理论与实战OSGi理论与实战
OSGi理论与实战
bluedavy lin
 

Recently uploaded (20)

GDG Cloud Southlake #42: Suresh Mathew: Autonomous Resource Optimization: How...
GDG Cloud Southlake #42: Suresh Mathew: Autonomous Resource Optimization: How...GDG Cloud Southlake #42: Suresh Mathew: Autonomous Resource Optimization: How...
GDG Cloud Southlake #42: Suresh Mathew: Autonomous Resource Optimization: How...
James Anderson
 
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Markus Eisele
 
fennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solutionfennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solution
shallal2
 
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Cyntexa
 
An Overview of Salesforce Health Cloud & How is it Transforming Patient Care
An Overview of Salesforce Health Cloud & How is it Transforming Patient CareAn Overview of Salesforce Health Cloud & How is it Transforming Patient Care
An Overview of Salesforce Health Cloud & How is it Transforming Patient Care
Cyntexa
 
UiPath Automation Suite – Cas d'usage d'une NGO internationale basée à Genève
UiPath Automation Suite – Cas d'usage d'une NGO internationale basée à GenèveUiPath Automation Suite – Cas d'usage d'une NGO internationale basée à Genève
UiPath Automation Suite – Cas d'usage d'une NGO internationale basée à Genève
UiPathCommunity
 
Challenges in Migrating Imperative Deep Learning Programs to Graph Execution:...
Challenges in Migrating Imperative Deep Learning Programs to Graph Execution:...Challenges in Migrating Imperative Deep Learning Programs to Graph Execution:...
Challenges in Migrating Imperative Deep Learning Programs to Graph Execution:...
Raffi Khatchadourian
 
AI x Accessibility UXPA by Stew Smith and Olivier Vroom
AI x Accessibility UXPA by Stew Smith and Olivier VroomAI x Accessibility UXPA by Stew Smith and Olivier Vroom
AI x Accessibility UXPA by Stew Smith and Olivier Vroom
UXPA Boston
 
DevOpsDays SLC - Platform Engineers are Product Managers.pptx
DevOpsDays SLC - Platform Engineers are Product Managers.pptxDevOpsDays SLC - Platform Engineers are Product Managers.pptx
DevOpsDays SLC - Platform Engineers are Product Managers.pptx
Justin Reock
 
Shoehorning dependency injection into a FP language, what does it take?
Shoehorning dependency injection into a FP language, what does it take?Shoehorning dependency injection into a FP language, what does it take?
Shoehorning dependency injection into a FP language, what does it take?
Eric Torreborre
 
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Mike Mingos
 
RTP Over QUIC: An Interesting Opportunity Or Wasted Time?
RTP Over QUIC: An Interesting Opportunity Or Wasted Time?RTP Over QUIC: An Interesting Opportunity Or Wasted Time?
RTP Over QUIC: An Interesting Opportunity Or Wasted Time?
Lorenzo Miniero
 
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Safe Software
 
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
João Esperancinha
 
Developing System Infrastructure Design Plan.pptx
Developing System Infrastructure Design Plan.pptxDeveloping System Infrastructure Design Plan.pptx
Developing System Infrastructure Design Plan.pptx
wondimagegndesta
 
AI 3-in-1: Agents, RAG, and Local Models - Brent Laster
AI 3-in-1: Agents, RAG, and Local Models - Brent LasterAI 3-in-1: Agents, RAG, and Local Models - Brent Laster
AI 3-in-1: Agents, RAG, and Local Models - Brent Laster
All Things Open
 
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
SOFTTECHHUB
 
May Patch Tuesday
May Patch TuesdayMay Patch Tuesday
May Patch Tuesday
Ivanti
 
Top-AI-Based-Tools-for-Game-Developers (1).pptx
Top-AI-Based-Tools-for-Game-Developers (1).pptxTop-AI-Based-Tools-for-Game-Developers (1).pptx
Top-AI-Based-Tools-for-Game-Developers (1).pptx
BR Softech
 
Slack like a pro: strategies for 10x engineering teams
Slack like a pro: strategies for 10x engineering teamsSlack like a pro: strategies for 10x engineering teams
Slack like a pro: strategies for 10x engineering teams
Nacho Cougil
 
GDG Cloud Southlake #42: Suresh Mathew: Autonomous Resource Optimization: How...
GDG Cloud Southlake #42: Suresh Mathew: Autonomous Resource Optimization: How...GDG Cloud Southlake #42: Suresh Mathew: Autonomous Resource Optimization: How...
GDG Cloud Southlake #42: Suresh Mathew: Autonomous Resource Optimization: How...
James Anderson
 
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Markus Eisele
 
fennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solutionfennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solution
shallal2
 
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Cyntexa
 
An Overview of Salesforce Health Cloud & How is it Transforming Patient Care
An Overview of Salesforce Health Cloud & How is it Transforming Patient CareAn Overview of Salesforce Health Cloud & How is it Transforming Patient Care
An Overview of Salesforce Health Cloud & How is it Transforming Patient Care
Cyntexa
 
UiPath Automation Suite – Cas d'usage d'une NGO internationale basée à Genève
UiPath Automation Suite – Cas d'usage d'une NGO internationale basée à GenèveUiPath Automation Suite – Cas d'usage d'une NGO internationale basée à Genève
UiPath Automation Suite – Cas d'usage d'une NGO internationale basée à Genève
UiPathCommunity
 
Challenges in Migrating Imperative Deep Learning Programs to Graph Execution:...
Challenges in Migrating Imperative Deep Learning Programs to Graph Execution:...Challenges in Migrating Imperative Deep Learning Programs to Graph Execution:...
Challenges in Migrating Imperative Deep Learning Programs to Graph Execution:...
Raffi Khatchadourian
 
AI x Accessibility UXPA by Stew Smith and Olivier Vroom
AI x Accessibility UXPA by Stew Smith and Olivier VroomAI x Accessibility UXPA by Stew Smith and Olivier Vroom
AI x Accessibility UXPA by Stew Smith and Olivier Vroom
UXPA Boston
 
DevOpsDays SLC - Platform Engineers are Product Managers.pptx
DevOpsDays SLC - Platform Engineers are Product Managers.pptxDevOpsDays SLC - Platform Engineers are Product Managers.pptx
DevOpsDays SLC - Platform Engineers are Product Managers.pptx
Justin Reock
 
Shoehorning dependency injection into a FP language, what does it take?
Shoehorning dependency injection into a FP language, what does it take?Shoehorning dependency injection into a FP language, what does it take?
Shoehorning dependency injection into a FP language, what does it take?
Eric Torreborre
 
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Mike Mingos
 
RTP Over QUIC: An Interesting Opportunity Or Wasted Time?
RTP Over QUIC: An Interesting Opportunity Or Wasted Time?RTP Over QUIC: An Interesting Opportunity Or Wasted Time?
RTP Over QUIC: An Interesting Opportunity Or Wasted Time?
Lorenzo Miniero
 
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Integrating FME with Python: Tips, Demos, and Best Practices for Powerful Aut...
Safe Software
 
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
João Esperancinha
 
Developing System Infrastructure Design Plan.pptx
Developing System Infrastructure Design Plan.pptxDeveloping System Infrastructure Design Plan.pptx
Developing System Infrastructure Design Plan.pptx
wondimagegndesta
 
AI 3-in-1: Agents, RAG, and Local Models - Brent Laster
AI 3-in-1: Agents, RAG, and Local Models - Brent LasterAI 3-in-1: Agents, RAG, and Local Models - Brent Laster
AI 3-in-1: Agents, RAG, and Local Models - Brent Laster
All Things Open
 
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
SOFTTECHHUB
 
May Patch Tuesday
May Patch TuesdayMay Patch Tuesday
May Patch Tuesday
Ivanti
 
Top-AI-Based-Tools-for-Game-Developers (1).pptx
Top-AI-Based-Tools-for-Game-Developers (1).pptxTop-AI-Based-Tools-for-Game-Developers (1).pptx
Top-AI-Based-Tools-for-Game-Developers (1).pptx
BR Softech
 
Slack like a pro: strategies for 10x engineering teams
Slack like a pro: strategies for 10x engineering teamsSlack like a pro: strategies for 10x engineering teams
Slack like a pro: strategies for 10x engineering teams
Nacho Cougil
 

Java memory problem cases solutions

  • 2. Java Runtime Data Area -Xss Local Vars Native method Stack Operator -XX:PermSize – PC Method Area XX:MaxPermSize Stack Frame Method Stack Heap -Xms -Xmx
  • 3. Oracle JDK Java Heap -Xmn New Generation Eden S0 S1 Old Generation -XX:SurvivorRatio
  • 4. Oracle JDK startup options for memory manage  -Xms –Xmx –Xmn –XX:PermSize –XX:MaxPermSize  -XX:SurvivorRatio  -XX:+UseParallelGC || -XX:+UseParallelOldGC ▪ -XX:MaxTenuringThreshold -XX:ParallelGCThreads ▪ -XX:-UseAdaptiveSizePolicy  -XX:+UseConcMarkSweepGC ▪ -XX:CMSInitiatingOccupancyFraction - XX:+UseCMSInitiatingOccupancyOnly  -XX:+DisableExplicitGC – XX:+ExplicitGCInvokesConcurrent
  • 5. Startup options or tools for memory problem solution  -XX:+PrintGCDateStamps –XX:+PrintGCDetails -Xloggc  -XX:+HeapDumpOnOutOfMemoryError  -XX:+HeapDumpBeforeFullGC  -XX:+PrintFlagsFinal(JDK 6u21+)  for CMS GC ▪ -XX:+PrintPromotionFailure -XX:+CMSDumpAtPromotionFailure  jinfo  jstat  jmap  MAT  btrace  google perf-tools
  • 6. OOM  java.lang.OutOfMemoryError: {$reason}  Full GC Frequently  CMS GC  promotion failed  concurrent mode failure
  • 12. Then to find who use Deflater.init
  • 13. java.lang.OutOfMemoryError: {$reason}  GC overhead limit exceeded  Java Heap Space  Unable to create new native thread  PermGen Space  Direct buffer memory  request {} bytes for {}. Out of swap space?
  • 14. GC overhead limit exceeded Java Heap Space  if code didn’t catch OutOfMemoryError,thread will exit;  slow response or no response at all ▪ because gc did frequently  Out of Swap  Java process crash
  • 15. GC overhead limit exceeded Java Heap Space  monitor app log & gc log  heap dump or jmap –histo|-dump when oom  mat analyze heap dump  app developer or btrace to find where cause
  • 16. Out of swap  crash log  google-perftools  then btrace to find where cause  ps: maybe u can try delete -XX:+DisableExplicitGC if exist,reason can be found in this link。
  • 17. use ThreadLocal carefully;  limit Collection/StringBuilder etc. size;  limit batch ops size;  limit the data size return by database;  avoid infinite loop;
  • 18. OOM  java.lang.OutOfMemoryError: {$reason}  Full GC Frequently  CMS GC  promotion failed  concurrent mode failure
  • 19. Most Cases  because memory consumes too much  a special case  a special frequently GC Case
  • 20. slow response or no response at all
  • 21. According gc log to see if the reason is consuming too much memory  if yes then add -XX:+HeapDumpBeforeFullGC or write a shell file to dump memory when full gc;  if no then use pstack & source code to see why full gc executes;
  • 22. OOM  java.lang.OutOfMemoryError: {$reason}  Full GC Frequently  CMS GC  promotion failed  concurrent mode failure
  • 23. promotion failed Case I  363439.937: [Full GC 363439.938: [ParNew (promotion failed): 523840K->523840K(523840K), 0.3404490 secs] 1690606K->1712399K(1834560K), 0.3412880 secs]  Taobao JDK help to find the reason ▪ ==WARNNING== allocating large array: thread_id[0x00002aaac2d85800], thread_name[ajp-0.0.0.0-8009-48], array_size[782611408 bytes], array_length[195652847 elememts]
  • 24. promotion failed Case II  1768.432: [GC 1768.432: [ParNew (promotion failed): 1572827K->1572827K(1572864K), 0.2845480 secs]  1972.208: [CMS: 10545160K- >5853155K(14680064K), 8.5355320 secs] 12105682K->5853155K(16252928K), [CMS Perm : 20907K->20873K(34952K)], 212.3113910 secs] [Times: user=53.38 sys=22.45, real=212.28 secs]  because cms gc fragmention
  • 25. four cases  promotion failed case III ▪ because CMSInitiatingOccupancyFraction too high...  concurrent mode failure case I ▪ because CMSInitiatingOccupancyFraction too high...  concurrent mode failure case II ▪ because old gen is too small  concurrent mode failure case III ▪ because permgen
  • 26. CMS GC problem will cause serial full gc,so the app response time will be very slow...
  • 27. According gc log to see if the reason is because CMSInitiatingOccupancyFraction is too high,if yes then adjust the option;  if no,then  promotion failed ▪ if because memory ran out,then dump memory; ▪ if because cms gc fragmention,currently we only can do is execute jmap –histo:live at regular time;  concurrent mode failure ▪ the same as promotion failed...
  • 28. OOM  java.lang.OutOfMemoryError: {$reason}  Full GC频繁  CMS GC  promotion failed  concurrent mode failure
  • 29. Young GC slow  CMS GC Concurrent-Mark causes app stop...  ...
  翻译: