Computer preemption and TotalView have made debugging Pascal much more seamless

1© 2017 Rogue Wave Software, Inc. All Rights Reserved. 1
Computer
preemption and
TotalView have
made debugging
Pascal much more
seamlessS7507
GPU Technology Conference

Debugging challenges

Target domains
What are people doing with GPUs
• Physical Modeling
• Biological Modeling
• Financial Modeling
• Deep Learning
• Signal Processing
• Oil and Gas

Common challenges
What makes GPU debugging hard?
• Hardware differences
– Vectors
• Data decomposition
– Ghost regions
• Buffer overruns
– MPI communication
• Pointers
– Memory spaces
• Concurrency
– Racy bugs

Challenges continued
• What about directive based approaches?
– OpenACC
– OpenMP 4 target device and SIMD directives
– OpenCL
• Compiler manages data migration
• Some challenges remain
– Variable annotation correctness
• Performance impact of choices
– Pointers
– MPI
– Error propagation

Strategies and techniques

Debugging techniques
• Timing
– What threads run
where
• Shared memory
– CUDA Memcheck
• Testing at scale
– Record test session

Observations - multithreaded
debugging
• Multithreaded applications
– Data can be shared
– Separate stacks
• Compared with multiprocess parallelism
– Similar challenges
– Communication not as localized
– Shared memory management

Debugging multicore applications
• Print debugging is limiting
• Need thread aware debugger
• Multithread requirements
– Asynchronous interface
– Access to all threads
– Complete control over threads
– Display of thread states
– Display of data across threads
Basics for multithreaded debugging

Debugging strategies
Reproduce in a thread
Vary # threads
Examine synch points
Use watchpoints
ReplayEngine
Multi-core adds complexity

TotalView for the NVIDIA ® GPU
Accelerator
• CUDA 6.0, 6.5, 7.0, 7.5, 8.0
• Cray, OpenPOWER, Linux, OpenACC
• Compiler: GCC, PGI
• Features
– MPI-based clusters
– Flexible display
– Physical (device, SM, Warp, Lane)
– Logical (Grid, Block) tuples
– Viewing of memory address spaces
– Leverages CUDA memcheck
– Dynamic parallelism support

Situation before advancements
Only one process per node can be debugged
But it can use all the GPUs
Due to limitation in debug API - one debug process per node

Current TotalView advancements
Multiple processes can be debugged but need to be exclusive
Achieving this by mapping 1 debug process to each GPU

Before Pascal
• Stepping code on a GPU stops everything else
– What happens if graphics id using the GPU
• Tips
– Needed to debug code on a GPU separate from graphing
– Software preemption slows everything down

What is Compute Preemption
Built into Pascal and post-Pascal hardware
Prevents long-running applications from monopolizing
• Tasks can be interrupted
• Dynamic load balancing
• Pixel and thread level preemption
• Instruction Level granularity
• For TotalView
• Can debug on same GPU
• No need to change application to work with GUI
• Things keep going even - breakpoints are hit

Other general tips
• Retrieves data from all threads easily
• Support for aggregated data collection
• On CUDA still prints for each thread
TotalView dprint()
dprint -g agg_1
Focus: 64:32000[0-63.1-500]
0x00000000 (0) : 64:16000[0-63.1, 0-63.3, 0-63.5, 0-63.7, 0-63.9, ...]
0x00000001 (1) : 64:16000[0-63.2, 0-63.4, 0-63.6, 0-63.8, 0-63.10, ...]
MRNet: A Multicast/Reduction Network
OpenSource tree-based multicast network
• broadcast/receive
• Supports debugging on large cluster

S7506: Debugging machine learning call
stacks
Shows Python Vs C++
C++ data
Py data

TotalView

TotalView
• Comprehensive multithreaded analysis and debug environment
– Thread specific breakpoints
– Control individual thread execution
– View thread specific stack and data
– View complex data types easily
• Integrated reverse debugging
• Track memory leaks in running applications
• Supports C/C++ on Linux
– New mixed language Pascal – C/C++
• Allowing the business to have
– Predictable development schedules
– Less time spent debugging
• Come to Thursday’s session
– S7506 - ROLLING IN THE DEEP: DEBUG MACHINE LEARNING CALL STACKS

HRL Laboratories
• “In the first full day of using TotalView, we were quickly able to solve the
bug that had us stumped for weeks. “
• Kirill Minkovich, HRL Laboratories
• Neural Network Simulation in the Defense Industry
• Need to adopt new technology and achieve massive
scaling on a deadline
• Project went from failing to over-achieving

Interested in learning more
about TotalView for HPC:
www.roguewave.com

Computer preemption and TotalView have made debugging Pascal much more seamless

Recommended

More Related Content

What's hot (20)

Similar to Computer preemption and TotalView have made debugging Pascal much more seamless (20)

More from Rogue Wave Software (20)

Recently uploaded (20)

Computer preemption and TotalView have made debugging Pascal much more seamless

Editor's Notes