Foreword Prefa e. About the Authors A knowledgments PART ONE Observability Methods Chapter 1 Introdu tion to Observability Tools 1.1 Observability Tools 1.2 Drill-Down Analysis 1.3 About Part One Chapter 2 CPUs 2.1 Tools for CPU Analysis 2.2 vmstat Tool 2.3 CPU Utilization 2.4 CPU Saturation 2.5 psrinfo Command 2.6 uptime Command 2.7 sar Command 2.8 Clock Tick Woes 2.9 mpstat Command 2.10 Who Is Using the CPU? 2.11 CPU Run Queue Latency 2.12 CPU Statisti s Internals 2.13 Using DTrace to Explain Events from Performance Tools 2.14 DTrace Versions of runq- sz, %runo 2.15 DTrace Probes for CPU States Chapter 3 Pro esses 3.1 Tools for Pro ess Analysis 3.2 Process Statistics Summary: prstat 3.3 Process Status: ps 3.4 Tools for Listing and Controlling Pro esses 3.5 Process Introspection Commands 3.6 Examining User-Level Locks in a Process 3.7 Tracing Processes 3.8 Java Processes Chapter 4 Disk Behavior and Analysis 4.1 Terms for Disk Analysis 4.2 Randomcvs. Sequential I/O 4.3 Storage Arrays 4.4 Sector Zoning 4.5 Max I/O Size 4.6 iostat Utility 4.7 Disk Utilization 4.8 Disk Saturation 4.9 Disk Throughput 4.10 iostat Referen e 4.11 Reading iostat 4.12 iostat Internals 4.13 sar -d 4.14 Trace Normal Form TNF) Tracing for I/O 4.15 DTrace for I/O 4.16 Disk I/O Time 4.17 DTraceToolkit Commands 4.18 DTraceTazTool Chapter 5 File Systems 5.1 Layers of File System and I/O 5.2 Observing Physi al I/O 5.3 File System Latency 5.4 Causes of Read/Write File System Latency 5.5 Observing File System “Top End” Activity 5.6 File System Caches 5.7 NFS Statistics Chapter 6 Memory 6.1 Tools for Memory Analysis 6.2 vmstat(1M) Command 6.3 Types of Paging 6.4 Physical Memory Allocation 6.5 Relieving Memory Pressure 6.6 Scan Rate asca Memory Health Indicator 6.7 Process Virtual and Resident Set Size 6.8 Using pmap to Inspect Process Memory Usage 6.9 Calculating Pro ess Memory Usage with ps and pmap 6.10 Displaying Page-Size Information with pmap 6.11 Using DTrace for Memory Analysis 6.12 Obtaining Memory Kstats 6.13 Using the Perl Kstat API to Look at Memory Statistics 6.14 System Memory Allo ation Kstats 6.15 Kernel Memow with kstat 6.16 System Paging Kstats 6.17 Observing MMU Performance Impact with trapstat 6.18 Swap Space Chapter 7 Networks 7.1 Terms for Network Analysis 7.2 Packets Are Not Bytes 7.3 Network Utilization 7.4 Network Saturation 7.5 Network Errors 7.6 Misconfigurations 7.7 Systemwide Statistics 7.8 Per-Process Network Statistics 7.9 TCP Statistics 7.10 IP Statistics 7.11 ICMP Statistics Chapter 8 Performance Counters 8.1 Introducing CPU Caches 8.2 pustat Command 8.3 putrack Command 8.4 busstat Command Chapter 9 Kernel Monitoring 9.1 Tools for Kernel Monitoring 9.2 Profiling the Kernel and Drivers 9.3 Analyzing Kernel Locks 9.4 DTrace lockstat Provider 9.5 DTrace Kernel Profiling 9.6 Interrupt Statistics: vmstat -i 9.7 Interrupt Analysis: intrstat PART TWO Observability Infrastructure Chapter 10 Dynami Tracing 10.1 Introduction to DTrace 10.2 The Basics 10.3 Inspecting Java Appli ations with DTrace 10.4 DTrace Architecture 10.5 Summary 10.6 Probe Reference 10.7 MDB Reference Chapter 11 Kernel Statistics 11.1 C-Level Kstat Interface 11.2 Command-Line Interface 11.3 Using Peri to A ess kstats 11.4 Snooping a Program's kstat Use with DTrace 11.5 Adding Statistics to the Solaris Kernel 11.6 Additional Information PART THREE Debugging Chapter 12 The Modular Debugger 12.1 Introduction to the Modular Debugger 12.2 MDB Concepts Chapter 13 An MDB Tutorial 13.1 Invoking MDB 13.2 MDB Command Syntax 13.3 Working with Debugging Targets 13.4 GDB-to-MDB Referen e 13.5 dcmd and Walker Reference Chapter 14 Debugging Kernels 14.1 Working with Kernel Cores 14.2 Examining User Process Stacks within a Kernel Image 14.3 Swit hing MDB to Debugca Specifi Process 14.4 kmdb, the Kernel Modular Debugger 14.5 Kernel Built-In MDB dcmds APPENDICES Appendix A Tunables and Settings Appendix B DTrace One-Liners Appendix C Java DTrace Scripts Appendix D Sample Peri Kstat Utilities Bibliography Index