Advanced Sun Fire™ Mid-Range Troubleshooting IES-443 Student Guide With Instructor Notes Sun Microsystems, Inc UBRM05-104 500 Eldorado Blvd Broomfield, CO 80021 U.S.A Revision A Copyright 2002 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303, U.S.A All rights reserved This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any Third-party software, including font technology, is copyrighted and licensed from Sun suppliers Sun, Sun Microsystems, the Sun Logo, Sun Enterprise, Sun StorEdge, Sun Fire, Netra, Sun Enterprise and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc in the U.S and other countries UNIX is a registered trademark in the U.S and other countries, exclusively licensed through X/Open Company, Ltd U.S Government approval might be required when exporting the product RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015 (b)(6/95) and DFAR 227.7202-3(a) DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID THIS MANUAL IS DESIGNED TO SUPPORT AN INSTRUCTOR-LED TRAINING (ILT) COURSE AND IS INTENDED TO BE USED FOR REFERENCE PURPOSES IN CONJUNCTION WITH THE ILT COURSE THE MANUAL IS NOT A STANDALONE TRAINING TOOL USE OF THE MANUAL FOR SELF-STUDY WITHOUT CLASS ATTENDANCE IS NOT RECOMMENDED ECCN Date – August 2002 Copyright 2002 Sun Microsystems Inc., 901 San Antonio Road, Palo Alto, California 94303, Etats-Unis Tous droits réservés Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution, et la décompilation Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun Sun, Sun Microsystems, the Sun Logo, Sun Enterprise, Sun StorEdge, Sun Fire, Netra, Sun Enterprise et Solaris sont des marques de fabrique ou des marques déposées de Sun Microsystems, Inc aux Etats-Unis et dans d’autres pays UNIX est une marques déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd L’interfaces d’utilisation graphique OPEN LOOK et Sun™ a été développée par Sun Microsystems, Inc pour ses utilisateurs et licenciés Sun reconnaît les efforts de pionniers de Xerox pour larecherche et le développement du concept des interfaces d’utilisation visuelle ou graphique pour l’industrie de l’informatique Sun détient une licence non exclusive de Xerox sur l’interface d’utilisation graphique Xerox, cette licence couvrant également les licenciés de Sun qui mettent en place l’interface d’utilisation graphique OPEN LOOK et qui en outre se conforment aux licences écrites de Sun L’accord du gouvernement américain est requis avant l’exportation du produit LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON CE MANUEL DE RÉFÉRENCE DOIT ÊTRE UTILISÉ DANS LE CADRE D’UN COURS DE FORMATION DIRIGÉ PAR UN INSTRUCTEUR (ILT) IL NE S’AGIT PAS D’UN OUTIL DE FORMATION INDÉPENDANT NOUS VOUS DÉCONSEILLONS DE L’UTILISER DANS LE CADRE D’UNE AUTO-FORMATION Please Recycle Please Recycle Table of Contents About This Course Preface-xxiii Course Goals Preface-xxiii Course Map .Preface-xxiv Topics Not Covered Preface-xxv How Prepared Are You? Preface-xxv Introduction Preface-xxvi How to Use Course Materials Preface-xxvi Conventions Preface-xxvi Icons Preface-xxvi Typographical Conventions Preface-xxviii Reviewing the Sun Fire Servers 1-1 Objectives 1-1 Relevance 1-2 Additional Resources 1-2 Sun Fire Server Models 1-3 Server Naming 1-3 Sun Fire Features 1-5 Interconnect Capabilities 1-7 Peak System Bandwidth 1-8 System Board Physical Locations 1-9 Sun Fire 6800 Server 1-9 Sun Fire 4810 and 4800 Servers 1-10 Sun Fire 3800 Server 1-10 Sun Fire I/O Assemblies 1-11 I/O Assembly Locations 1-12 Sun Fire 6800 Server 1-12 Sun Fire 48x0 Server 1-13 Sun Fire 3800 1-14 Compact PCI I/O 1-15 Sun Fire 4800/4810/6800 Four-Slot cPCI Board 1-15 Sun Fire 3800 Six-Slot cPCI Board 1-17 Sun Fireplane Switch Boards 1-18 Sun Fire 6800 Server 1-18 v Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Sun Fire 4800, 4810, and 3800 Servers 1-18 Sun Fireplane Switch Board Physical Locations 1-19 Agent IDs 1-21 CPU Locations and Agent IDs 1-21 Memory Controller Mapping 1-22 IOC AID 1-23 Locating I/O Devices in Sun Fire 6800/4810/ 4800 Systems 1-24 Four Slot cPCI I/O Assembly 1-27 Sun Fire 3800 I/O Device Location Mapping 1-29 Six Slot cPCI I/O Assembly 1-30 Sun Fireplane Interconnect 1-31 How the Sun Fireplane Interconnect Works 1-32 Troubleshooting Tools 1-33 Service Mode 1-33 System Logging 1-35 Explorer 1-37 Parity 1-38 Parity Checking 1-39 Problems with Parity 1-39 ECC 1-40 Check Your Progress 1-41 Power Management and the Frame Manager 2-1 Objectives 2-1 Relevance 2-2 Additional Resources 2-2 Power 2-3 The RTU and RTS 2-4 Redundant Power 2-6 The AC/DC Power Supplies 2-8 Housekeeping Voltage 2-9 Power Distribution 2-10 Sun Fire 3800 2-11 Sun Fire 48x0 2-12 Sun Fire 6800 2-14 DC/DC Component Power Supplies 2-16 Board Voltage Requirements 2-17 The Frame Manager 2-18 Exercise: Managing Power 2-20 Objective 2-20 Description 2-20 Preparation 2-20 Problem Presentation 2-20 Task 2-22 Exercise Summary 2-35 vi Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Check Your Progress 2-36 Domains and Segments .3-1 Objectives 3-1 Relevance 3-2 Additional Resources 3-2 Virtual Servers 3-3 Domains 3-4 Segments 3-6 Domains and Segments in the Sun Fire 6800 3-8 Impact of Multiple Segments and Domains 3-11 Power Grid Segmentation 3-12 Power Grids on Sun Fire 48x0/3800 3-12 Power Grids on Sun Fire 6800 3-13 Domain Recovery 3-15 Sun Fire 3800, 48x0, and 6800 Board Configurations 3-15 Segment and Domain Summary 3-19 Exercise: Identifying Domains and Segments 3-20 Objective 3-20 Preparation 3-20 Task 3-20 Task 3-22 Exercise Summary 3-31 Check Your Progress 3-32 The System and I/O Boards .4-1 Objectives 4-1 Relevance 4-2 Additional Resources 4-2 System Board 4-3 System Board Interconnects 4-4 CPU Placement Rules 4-6 Sun Fire I/O 4-7 I/O Assembly Configurations 4-7 The PCI I/O Board 4-9 PCI Card Support 4-10 PCI Board Slot Configuration 4-10 The cPCI Board Assembly 4-12 cPCI Card Slot Configuration 4-14 cPCI Operation 4-14 The 3800 cPCI Board 4-15 3800 cPCI Card Slot Configuration 4-17 I/O Board Power 4-18 PCI Card Power 4-18 Fireplane Bus Interface 4-19 Failure on a PCI Board 4-20 vii Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Exercise: Explaining System and I/O Board Operation 4-21 Objective 4-21 Preparation 4-21 Task 4-21 Task 4-23 Task 4-25 Exercise Summary 4-28 Check Your Progress 4-29 The Sun Fireplane Interconnect Bus 5-1 Objectives 5-1 Relevance 5-2 Additional Resources 5-2 Fireplane Bus Introduction 5-3 Fireplane Bus Design 5-4 The Fireplane Switch Overview 5-5 Address Interconnect Overview 5-8 Data Interconnect Overview 5-9 L1 Data Flow 5-11 System Board Port Configuration 5-11 I/O Board Port Configuration 5-12 The ASIC Tree 5-13 Circuitry on L1 System Boards 5-16 Safari Ports 5-17 Console Bus 5-17 Fireplane Bus Flow Control Circuitry 5-18 The Fireplane Buses in the Sun Fire 3800, 4800, and 4810 5-19 Address Interconnect 5-19 SDC Management 5-23 Data Interconnect 5-27 The Fireplane Buses in the Sun Fire 6800 5-32 Address Interconnect 5-32 SDC Management Interconnect 5-34 Data Interconnect 5-36 Fireplane Bus Data Paths 5-38 Quadword Data Line Structure 5-39 Data Pathing 5-40 System Board to L2 Data Path 5-40 I/O Board to L2 Data Path 5-41 Sun Fire 3800, 4800, and 4810 Data Path 5-42 Sun Fire 6800 Data Path 5-43 Sun Fire 6800 Double Pump Mode 5-44 L1 System Board Data Path Bit Assignment 5-46 L1 I/O Board Data Path Bit Assignment 5-48 Exercise: Identifying the Safari Bus 5-51 viii Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Objective 5-51 Preparation 5-51 Task 5-51 Task 5-53 Exercise Summary 5-58 Check Your Progress 5-59 Parity and ECC Detection and Recovery 6-1 Objectives 6-1 Relevance 6-2 Additional Resources 6-2 Error Detection 6-3 Parity 6-4 Parity Detection in the Data Path 6-5 Parity Detection in the Address Path 6-8 Parity Detection in the Control Path 6-9 Error Correction Code (ECC) 6-11 ECC Error Types 6-11 ECC Creation 6-13 End-to-End ECC Protection 6-14 ECC Syndromes 6-16 The ECC Syndrome Table 6-16 Signaling ECC Syndrome Codes 6-18 ECC Error Identification 6-20 Locating a DX ECC Error 6-20 The DX ECC Status Register 6-22 CPU-Caused Interconnect ECC Error Example 6-24 ECC Errors from Memory 6-26 Data Bit Identification 6-28 ECC Error Reporting 6-32 Asynchronous Fault Status Register 6-33 The AFSR Table 6-33 AFT Labels 6-36 Asynchronous Fault Address Register 6-37 Physical Address Space 6-38 AFAR Addressing 6-39 AFSR Overwrite Policy 6-43 AFSR/AFAR Overwrite Policy 6-43 ECC and MTAG Syndrome Fields Overwrite Policy 6-44 Exercise: Identifying and Diagnosing Parity and ECC Errors 6-45 Preparation 6-45 Task 6-45 Task 6-47 Task 6-49 ix Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Exercise Summary 6-55 Check Your Progress 6-56 Caching and Interconnect Operations 7-1 Objectives 7-1 Relevance 7-2 Additional Resources 7-2 Introduction 7-3 The UltraSPARC III Caches 7-4 The Instruction Cache 7-4 The Data Cache 7-4 The External Cache 7-4 The Write Cache 7-5 The Prefetch Cache 7-6 Cache Snooping 7-7 Snoopy Coherency 7-8 Fireplane Address Bus Snooping Operation 7-10 Snoop Response Signals 7-10 Cache Data State Tags 7-13 CTags 7-13 MOESI State Transitions 7-14 The DTags 7-15 Fireplane Bus Transactions 7-16 Interconnect Signal Groups 7-16 Address Interconnect Operation 7-18 Address Transaction (ATrans) 7-18 Sun Fire Transaction Request Codes 7-19 Requests for Data 7-20 The Data Interconnect 7-22 Data Transaction (DTrans) 7-22 Request Flow 7-23 Address Read /Write Transaction 7-23 Data Transaction 7-25 Read-to-Share Cache Example 7-29 Arbitration on AR and SDC ASICS 7-31 Address Interconnect Arbitration 7-31 Data Bus Arbitration and Operation 7-33 Direction of ECC Error Reporting 7-35 The SDC ECC Error Register 7-39 General Notes on L1 DX ECC Errors & SC Messages 7-41 Exercise: Interconnecting Operation 7-43 Objective 7-43 Preparation 7-43 Task 7-43 Task 7-45 x Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Index Numerics 3800 address interconnect 5-19 cPCI location 1-14 data control interconnect 5-23 Fireplane bus 5-19 Fireplane switch 1-18 power 2-11 diagram 2-12 power supplies 2-8 RTS/RTU 2-6 system board location 1-10 4800 address interconnect 5-19 data control interconnect 5-23 Fireplane switch 1-18 power supplies 2-8 switch location 1-20 system board location 1-10 4810 address interconnect 5-19 data control interconnect 5-23 Fireplane switch 1-18 I/O assembly location 1-13 power 2-12 diagram 2-13 power supplies 2-8 switch location 1-20 system board location 1-10 48V_AUX 2-9 56V_AUX 2-9 6800 address interconnect 5-32 data control interconnect 5-34 data interconnect 5-34 domains 3-8 Fireplane switch 1-18 I/O assembly location 1-12 power 2-14 diagram 2-15 power segmentation 3-13 power supplies 2-8 segments 3-8 switch location 1-19 system board location 1-9 A A145 2-8 A152 2-8 A153 2-8 AC-DC power supplies 2-8 AddrArbIn 7-32 AddrArbOut 7-32 address ecache data A-10 ecache tag A-11 ECC protection 6-8 interconnect 7-18 6800 5-32 level 5-19 overview 5-8 local arbitration 7-31 parity detection 6-9 Index-1 Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A parity error 6-8 physical 6-39 request codes 7-19 SAM B-20 Transaction Request Group 7-18 AddrPty 6-3 AFAR 6-37 overwrite 6-43 AFSR 6-33 AFAR 6-37 overwrite 6-43 agent ID 1-21 PCI IOC 1-23 AID 1-21 PCI IOC 1-23 range 1-21 AR connections 5-19 CPU connections 4-6 error pause 9-42 level 5-19 6800 5-32 system board 4-4 arbitration bus 7-31 level data 7-33 local address 7-31 Arbitration Group 7-31 architecture DRAM A-4 ASIC DCDS 5-27 DX 5-28 error reporting 9-34 ASR 5-18 assembly I/O 1-11 Asynchronous Fault Address Register 6-37 Asynchronous Fault Status Register 6-33 ATransID 7-18 B bandwidth Fireplane 1-7 Index-2 peak 1-8 bank determining 8-14 DIMM 8-3 memory 8-4 BIST 6-3 board Fireplane Switch 5-5 ID 9-31 local I2C bus 9-28 repeater 5-5 RP 5-5 switch locations 1-31 system 4-3 Boot Bus Controller 9-7 BootBus 9-12 CPU interface B-24 JTAG 9-19 boundary scan 9-19 Built-In Self Test 6-3 bus address parity error 6-8 address parity detection 6-9 arbitration 7-31 BootBus 9-12 console 9-5 console hub 9-5 data 5-9 data paths 5-38 ECC error source 6-20 Fireplane goals 5-3 Fireplane operation 1-32 I2C 9-20 global 9-21 I/O board 9-28 local 9-28 JTAG 9-19 parking 7-33 pause 5-18 performance domains 3-19 PROM 9-17 Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A SC 9-3 snoopy 1-32 transaction interrupt 7-26 read 7-23 write 7-26 C cache characteristics C-2 CPU latency B-18 cpustat C-10 cputrack C-11 data 7-4, B-18 Data Cache Unit B-18 direct mapped C-3 ECC B-25 ECU B-21 external 7-4 hardware 7-3 Harvard C-5 hit rate C-6 instruction 7-4 parity B-25 performance C-6, C-7 physical address C-4 prefetch B-18, B-19 relationships B-19 SAM addressing B-20 set associative C-3 statistics C-10 sub-block 7-4 thrashing C-8 indentifying C-8 UltraSPARC III 7-4 unified C-5 VIPT C-4 virtual address C-3 write 7-4, B-18 Cache Data State Tags 7-13 capabilities Sun Fire 1-5 Sun Fire interconnect 1-7 card PCI power 4-18 CAS A-4, A-8 CBH error pause 9-42 CE 6-11 centerplane errors 5-18 power 2-10 cfgadm PCI 4-15 characteristics cache C-2 check bit ECC 6-16 Cheetah 1-3 chip ecache A-10 bit assignment A-11 HPC-3130 4-14 Ichip 9-17 memory A-6 Chorus OS Frame Manager 2-18 clock 9-44 console 9-47 consumers 9-47 domain 3-5 doubling 9-49 EP111 fanout 9-45 failover 9-50 fanout 9-49 local distribution 9-48 PLL 9-45 reference 9-48 requirements 9-45 SC swap 9-50 selection 9-48 source selection 9-49 system 9-47 users 9-47 Clock Group 7-17 command address interconnect codes 7-19 cfgadm PCI 4-15 Index Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Index-3 cpustat C-10 cputrack C-11 nvci 8-16 prtdiag 8-16 setkeyswitch error pause 9-43 setupdomain 8-16 Compact PCI 1-15 component power supplies 2-16 components internal names 1-3 configuration I/O board SBBC 9-13 interleave 8-12, 8-14 SC SBBC 9-9 System board SBBC 9-11 console bus 9-5 hub 9-5 JTAG 9-19 console clock 9-47 context MMU C-4 converter AC-DC 2-8 correctable error 6-11 cPCI 1-15 slot assembly 1-15 slot assembly 1-17 DR 4-14 location 3800 1-14 CPU B-2 AR connections 4-6 BootBus 9-12 cache 7-4 relationships B-19 statistics C-11 cache latency B-18 DCDS connection 5-46 interface B-24 ECC B-25 placement 4-6 cpustat C-10 cputrack C-11 Index-4 CTags 7-13 D D$ C-5 D103 2-16 D105 2-16 D106 2-16 D107 2-16 D108 power supply 4-18 D109 2-16 D110 2-16, 9-20 D133 2-16 D134 2-16 data cache 7-4 ecache address A-10 ecache location A-11 external cache B-21 interconnect 6800 5-34 starvation 7-33 level arbitration 7-33 line structure 5-39 data cache B-18 data control interconnect 6800 5-34 level 5-23 data interconnect 5-9 data path bit assignment 5-42 double pump 5-43, 5-44 bus 5-38 bus ECC 6-20 CPU control B-24 ECC 6-11 I/O board 5-41 system board 4-4 Data Switch 5-28 DC 1-3 DC-DC power supplies 2-16 DCDS 5-27 CPU connection 5-46 Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A CPU interface B-24 data flow 8-9 diagram 5-27 Fireplane port 5-46 system board 4-4 DCU B-18 device Fireplane 7-18 diagram 9-11 DIMM 8-3 bank 8-3 capacity 8-3 data flow 8-9 direct mapped cache C-3 DLAT 7-3 domain 3-3, 3-4 6800 3-8 bandwidth 3-5 clocking 3-5 configuration 3-4 Enterprise 10000 3-4 error isolation 3-4 hardware 3-5 numbers 3-4 operation 3-4 requirements 3-4 segment comparison 3-19 double bit error 1-39 double pump 5-43, 5-44 DR cPCI 4-14 interleave 8-4 DRAM A-2 cell A-4 controller B-21, B-22 refresh A-5 DTags 7-15 DTLB B-18 DTransID 7-22 Dual CPU Data Switch 5-27 DX 5-28 diagram 5-28 ECC error source 6-20 level 6800 5-34 data path 5-28 PCI IOC connection 5-41 queue 5-28 system board 4-4 E E$ C-5 EBus 9-3 ecache address mapping A-10 chip assignment A-11 data location A-11 tag addressing A-11 ECC 6-12 AFSR 6-33 algorithm A-12 bus 6-11 checking 6-20 bus error source 6-20 cache B-25 calculation A-12 check bits 6-16 correctable error 6-11 CPU B-25 error 6-5 reporting 6-32 type 6-11 interconnect 6-11 memory error 6-26 MTAG syndrome 6-35 reporting 6-32 syndrome example 6-16 signalling 6-18 table 6-16, A-12 uncorrectable error 6-11 EChip Fireplane Switch 9-38 I/O board 9-37 SC board 9-39 System board 9-36 Echip 9-35 Index Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Index-5 ECU B-21 EP111 9-45, 9-49 EPLD 9-35 error address parity 6-8 AFSR 6-33 ASIC reporting 9-34 CE 6-11 classes 10-3 domain isolation 3-4 double bit 1-39 ECC 6-5 persistence 6-12 type 6-11 Fireplane Switch path 9-38 I/O board path 9-37 logging 9-34 masking 9-34 memory ECC 6-26 parity 1-38, 6-5 pause 9-42 register 9-34 reporting ECC 6-32 SC board path 9-39 source ECC bus 6-20 System board path 9-36 UE 6-11 error chip 9-35 error pause 9-42 AR 9-42 CBH 9-42 error persistence 6-12 even parity 1-38 external cache 7-4, B-21 F failover clock 9-50 failure power 2-6 fan I2C 9-21 unswitched power 2-9 fanout buffer chip 9-45 Index-6 fault ID board 9-32 features Sun Fire 1-5 FGU B-17 latency B-17 Fireplane 3800 5-19 Address Transaction Request Group 7-18 arbitration 7-31 Arbitration Group 7-31 ATransID 7-18 capabilities 1-7 CPU interface B-24 data bit assignment 5-42 double pump 5-43, 5-44 data paths 5-38 design 5-4 device 7-18 device AID 1-21 DTransID 7-22 DX and PCI IOC 5-48 ECC 6-11 ECC error source 6-20 error detection 6-3 fault isolation 5-18 Flow Control 5-18 goal 5-3 interconnect 5-3 operation 1-32 overview 1-31 port 1-21 switch boards 1-18 transaction interrupt 7-26 read 7-23 write 7-26 TTargID 7-22 TTransID 7-22 Fireplane Bus Transactions 7-16 Fireplane Switch board 5-5 clock users 9-47 diagram 5-6 error path 9-38 Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A first error 9-34 Flash Prom 9-17 Flow Control 5-18 FP execution B-10 Frame Manager 2-18 components 2-18 FRU error source 10-3 G global I2C bus 9-21 glue_FPGA 9-39 goals Fireplane bus 5-3 grid 6800 power 2-14, 3-13 H hardware error logging 9-34 hardware caches 7-3 Harvard cache C-5 hierarchy interconnect 5-4 hit rate cache C-6 hot swap PCI 4-14 housekeeping voltage 2-9 HPC PCI 4-14 HPC-3130 4-14 hub console bus 9-5 I I$ C-5 I/O assembly 1-11 slot cPCI 1-15 4810 location 1-13 slot cPCI 1-17 6800 location 1-12 cPCI 1-15 power limit 4-18 error path 9-37 I/O board clock users 9-47 data path 5-41 I2C bus 9-28 Ichip 9-17 overview 1-11 power 4-18 SBBC 4-20 SBBC configuration 9-13 SBBC diagram 9-13 I2C bus local 9-28 fans 9-21 global 9-21 housekeeping voltage 2-9 I/O board 9-28 ID board 9-31 multiplexor 9-21 SEEPROM 9-21 I2C bus 9-20 Ichip 9-17 ID board 9-31 diagram 9-32 faults 9-32 IDLE 7-33 IEEE 1149.1 9-19 IEU B-15 IIU B-13 instruction cache 7-4 instruction fetch B-8 instruction issue B-9 integer execution B-9 interconnect address 7-18 6800 5-32 level 5-19 overview 5-8 request codes 7-19 Address Transaction Request Group 7-18 arbitration 7-31 bandwidth Index Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Index-7 peak 1-8 capabilities 1-7 console bus 9-5 data 5-9 6800 5-34 data bit assignment 5-42 double pump 5-43, 5-44 data control 6800 5-34 level 5-23 domains 3-4 ECC 6-11 error detection 6-3 errors 6-5 failure source 6-7 fault isolation 5-18 Flow Control 5-18 levels 5-4 memory B-22 operation 1-32 overview 1-31 parity checking 6-5 quad pump 5-43, 5-44 transaction interrupt 7-26 read 7-23 write 7-26 Interconnect Signal Groups 7-16 Inter-IC bus 9-20 interleave 8-4, 8-10 checking 8-16 configuration 8-12 cross-board 8-4 DR 8-4 nvci 8-16 POST 8-12 prtdiag 8-16 scope 8-12 setupdomain 8-16 intermittant ECC 6-12 interrupt acknowledge 7-29 Ichip 9-17 Index-8 J Joint Test Action Group 9-19 JTAG 9-19 access 9-19 interface 9-19 ring 9-19 SBBC 9-7 JTAG+ 9-19 L L1 Error Mask and Latch FPGA 9-35 L1_FPGA 9-35 layout UltraSPARC III B-4 line structure 5-39 local I2C bus 9-28 location 3800 cPCI 1-14 PCI IOC 1-23 logging error 9-34 logical bank memory 8-4 M Mapped 7-11 mapping address 6-39 PCI IOC 1-23 masking error 9-34 MCU B-22 data flow 8-9 MD 1-3 ME 1-3 memory access cycle A-8 bank 8-3 determining 8-14 logical 8-4 physical 8-4 Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A chip properties A-6 controller B-22 correctable error 6-11 DIMM 8-3 DRAM A-2 ECC error 6-26 interconnect B-22 interleave 8-10 MCU B-22 read A-9 refresh A-2, A-5 SIMM 8-3 SRAM A-2 subsystem 8-9 uncorrectable error 6-11 write A-9 micro-tags B-20 MMU context C-4 MOESI State Transitions 7-14 MPC850DC 2-18 MTAG syndrome 6-35 N N+1 power 2-8 names Sun Fire 1-3 noncacheable address mapping 6-39 nvci interleave 8-16 O odd parity 1-38 Other Interconnect Signal Groups 7-17 overview I/O board 1-11 overwrite AFAR 6-43 AFSR 6-43 OwnedIn 7-12 OwnedOut 7-12 P parity 1-38 address detection 6-9 address error 6-8 bus checking 6-20 cache B-25 checking 1-39 error 6-5 source 6-7 interconnect checking 6-5 problems 1-39 Parity Group 6-3 ParityBidi 6-3 ParitySingle 6-3 parking 7-33 partition 3-3 path Fireplane Switch error 9-38 I/O board error 9-37 SC board error 9-39 System board error 9-36 pause error 9-42 PauseIn 5-18 PauseOut 5-18 PCI hot swap 4-14 SBBC SC bus 9-9 PCI IOC AID 1-23 DX connection 5-41 HPC PCI 4-14 locations 1-23 quad pump 5-43, 5-44 SBBC 4-20 PDC 7-3 pending queue B-24 performance cache C-6, C-7 domain 3-19 interleave 8-10 segment 3-19 Index Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Index-9 persistent ECC 6-12 physical address cache C-4 physical bank memory 8-4 pipeline B-6 A0/A1 B-15 FGU latency B-17 Floating Point B-17 FP execution B-10 graphics B-17 instruction fetch stages B-8 instruction issue stages B-9 integer execution stages B-9 load/store B-15 MS B-16 parallel B-5 snoop B-24 stages B-8 trap and done B-10 PLL 9-48 clock 9-45 port AR 5-19 console bus 9-5 Fireplane 1-21 POST interleave 8-12 power 3800 2-11 diagram 2-12 4810 2-12 diagram 2-13 6800 2-14 diagram 2-15 6800 segmentation 3-13 centerplane 2-10 configuration 2-6 D108 supply 4-18 distribution 2-3 failure 2-6 fan 2-9 I/O board 4-18 management 7-17 N+1 2-8 redundant 2-6 Index-10 RTS 2-4 RTS/RTU connections 2-5 RTU 2-4 SC calculation 2-9 sufficiency 2-9 power supply AC-DC 2-8 ratings 2-8 DC-DC 2-16 naming 2-16 PPP Frame Manager 2-19 prefetch cache B-18, B-19 PROM bus 9-17 prtdiag interleave 8-16 Q quad pump 5-43, 5-44 quadword 5-39 queue pending B-24 store queue B-18 R RAS A-4, A-8 read bus transaction 7-23 memory A-9 Read Response 7-25 Redundant Transfer Switch 2-4 Redundant Transfer Unit 2-4 reference clock 9-48 refresh A-2, A-5 register AFAR 6-37 AFSR 6-33 repeater 5-5 request address codes 7-19 requirements Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A board voltage 2-17 clock 9-45 reset SBBC control 9-8 Reset Group 7-17 ring JTAG 9-19 RP 5-5 RTS 2-4 configuration 2-6 connections 2-5 TopCap 2-19 RTU 2-4 connections 2-5 TopCap 2-19 rule interleave 8-14 RW1C 9-34 S SAM B-20 SBBC 9-7 BootBus 9-12 console bus 9-5 error pause 9-42 error reporting 9-34 Fireplane Switch error path 9-38 I/O board 4-20, 9-13 error path 9-37 I/O board diagram 9-13 Ichip 9-17 JTAG 9-19 PCI bus 9-7 PCI interrupts 9-10 PCI IOC 4-20 PROM bus 9-17 reset control 9-8 SC 9-9 SC board error path 9-39 SC diagram 9-9 SC PCI bus 9-9 System board 9-11 error path 9-36 SC block diagram 9-3 bus configuration 9-3 clock failover 9-50 generation 9-44 users 9-47 error pause 9-42 failover clock 9-50 power calculation 2-9 SBBC configuration 9-9 SBBC diagram 9-9 SBBC error path 9-39 SBBC PCI bus 9-9 swap clock 9-50 SC board error path 9-39 ScApp ID board 9-33 Schizo 1-3 SCL 9-20 scope interleave 8-12 SCPOST ID board 9-32 SDA 9-20 SDC Hi Priority 7-33 level 5-23 6800 5-34 Low Priority 7-33 system board 4-4 SDRAM A-7 DIMM 8-3 SEEPROM I2C 9-21 ID board 9-31 ID board contents 9-31 segment 3-3 6800 3-8 6800 power 3-13 domain Index Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Index-11 comparison 3-19 numbers 3-4 select PLL 9-48 Serengeti 1-3 set associative cache C-3 setkeyswitch error pause 9-43 setupdomain interleave 8-16 signal AddrArbIn 7-32 AddrArbOut 7-32 Address Transaction Request Group 7-18 AddrPty 6-3 Arbitration Group 7-31 ATransID 7-18 clock 9-44 DTransID 7-22 Flow Control 5-18 Hi Priority 7-33 Low Priority 7-33 OwnedIn 7-12 OwnedOut 7-12 Parity Group 6-3 ParityBidi 6-3 ParitySingle 6-3 PauseIn 5-18 PauseOut 5-18 timing 9-44 TTargID 7-22 TTransID 7-22 signalling ECC syndrome 6-18 SIMM 8-3 SIU B-24 snoop pipeline B-24 Snoop Response Signals 7-10 Snoopy bus 1-32 source clock 9-49 SP 1-3 SpeedCtl Group 7-17 SRAM A-2, A-6 cell A-6 stages Index-12 pipeline B-8 starvation data bus 7-33 statistics cache C-11 CPU C-10 cputrack C-11 STICK 7-17 sticky ECC 6-12 store queue B-18 sub-block 7-4 Sum Addressed Memory B-20 Sun Fire interconnect 1-7 internal names 1-3 model comparison 1-5 models Sun Fire 1-3 power 2-3 superscalar B-5 switch 4800 location 1-20 4810 location 1-20 6800 location 1-19 board locations 1-31 boards 1-18 syndrome ECC example 6-16 MTAG 6-35 signalling 6-18 table 6-16, A-12 syslog 6-5 System board error path 9-36 SBBC configuration 9-11 SBBC diagram 9-11 system board 3800 location 1-10 4800 location 1-10 4810 location 1-10 6800 location 1-9 clock users 9-47 contents 4-3 CPU placement 4-6 interconnects 4-4 Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A system clock 9-47 T table ECC syndrome 6-16 tag ecache B-21 addressing A-11 micro-tags B-20 thrashing cache C-8 identifying C-8 timer STICK 7-17 timing 9-44 TLB 7-3 data B-18 TopCap 2-18 diagram 2-19 transaction Fireplane interrupt 7-26 read 7-23 write 7-26 trap and done B-10 TTargID 7-22 TTransID 7-22 U UE 6-11 UltraSPARC III B-2 cache B-18 properties 7-4 relationships B-19 construction B-3 data cache unit B-18 ECC B-25 external cache unit B-21 FP execution B-10 FP Graphics unit B-17 instruction fetch stages B-8 instruction issue stages B-9 instruction issue unit B-13 integer execution stages B-9 integer execution unit B-15 layout B-4 memory control unit B-22 pipeline B-6 stages B-8 placement 4-6 requests outstanding B-24 superscalar B-5 system interface unit B-24 trap and done B-10 units B-11 uncorrectable error 6-11 unified cache C-5 unit data cache B-18 external cache B-21 FP Graphics B-17 instruction issue B-13 integer execution B-15 memory control B-22 system interface B-24 UltraSPARC III B-11 V VIPT C-4 virtual address cache C-3 virtual server 3-3 voard voltage requirements 2-17 voltage board requirements 2-17 housekeeping 2-9 W W$ 7-4 write bus transaction 7-26 memory A-9 write cache 7-4, B-18 writeback buffer B-21 Index Copyright 2002 Sun Microsystems, Inc All Rights Reserved Enterprise Services, Revision A Index-13 [...]... 1-1 0 Figure 1-5 Sun Fire Mid-Range I/O Assembly 1-1 1 Figure 1-6 Sun Fire 6800 Server PCI I/O Assembly Locations 1-1 2 Figure 1-7 Sun Fire 4800 Server I/O Assembly Locations 1-1 3 Figure 1-8 Sun Fire 3800 cPCI I/O Assembly Locations 1-1 4 Figure 1-9 Four-Slot cPCI I/O Board Logical Block Diagram 1-1 5 Figure 1-1 0 Six-Slot cPCI I/O Board Logical Block Diagram 1-1 7 Figure 1-1 1 Sun Fireplane... Assembly 4-9 Figure 4-4 cPCI I/O Board 4-1 3 Figure 4-5 cPCI Board and TI HPC-3130 HPC-PCI Chip 4-1 5 Figure 4-6 3800 cPCI I/O Board 4-1 6 Figure 4-7 PCI Bus-Fireplane Bus Relationship 4-1 9 Figure 4-8 I/O Board SBBC Interfaces 4-2 0 Figure 5-1 Bus Hierarchy Levels 5-4 Figure 5-2 Fireplane Switches in the Sun Fire Mid-Range Platforms 5-6 Figure 5-3 Fireplane... B-13 Figure B-5 Integer Execute Unit B-15 Figure B-6 Data Cache Unit B-19 Figure B-7 Prefetch Cache Data Flow B-19 Figure B-8 Data Cache SAM Addressing B-20 Figure B-9 Memory Subsystem Interconnect B-22 Figure C-1 Performance Loss to Cache Misses C-7 Figure C-2 Example of Cache Thrashing C-9 xviii Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems,... Figure 5-1 9 Sun Fire 6800 Data Interconnect 5-3 6 Figure 5-2 0 Fireplane Data Path Bandwidth 5-3 8 Figure 5-2 1 Data Line Quadword Structure 5-3 9 Figure 5-2 2 Sun Fire 3800 and 48x0 Data Paths 5-4 2 Figure 5-2 3 Sun Fire 6800 Single Segment Data Path 5-4 3 Figure 5-2 4 System Data Order and Bit Slicing in Double Pump Mode Fireplane Switch ( Part A) 5-4 4 xvi Advanced Sun Fire™ Mid-Range Troubleshooting. .. Table 1-1 Sun Fire Family Names 1-3 Table 1-2 Sun Fire Family Component Names 1-4 Table 1-3 Sun Fire Family Maximum Configurations 1-5 Table 1-4 Sun Fire Family System Interconnect Specifications 1-7 Table 1-5 Sun Fire ASIC List 1-1 6 Table 1-6 CPU Numbering 1-2 1 Table 1-7 IOC AID Numbering 1-2 3 Table 1-8 Device Path to I/O Card Slot Location Mapping 1-2 5... Error Register 9-1 5 Table 9-2 Global I2C Bus Assignments 9-2 1 Table 9-3 Global I2C Bus Device Address 9-2 3 Table 9-4 Local I2C DIMM Locations 9-2 7 Table 9-5 First Error Register on CPU board 9-3 3 Table 9-6 First Error Register on System Controller 9-3 8 Table 9-7 Sun Fire Clock Distribution 9-4 5 xx Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems,... 2-1 9 Figure 3-1 Bus Clocking with Two Domains 3-5 Figure 3-2 Segments in a Sun Fire 3800, 4800, or 4810 Server 3-7 Figure 3-3 Two Domains in a Sun Fire 6800 3-8 Figure 3-4 Two Segments in a Sun Fire 6800 3-9 Figure 3-5 Domains and Segments in a Sun Fire 6800 3-1 0 Figure 4-1 System Board Major Components 4-4 Figure 4-2 System Board Logical Components 4-5 Figure 4-3 ... Figure 7-9 Write Transaction 7-2 8 Figure 7-1 0 Read-to-Share from Memory on Another Board 7-2 9 Figure 7-1 1 SDC Arbiter 7-3 3 Figure 7-1 2 Inbound and Outbound Transfers on the DX Switch 7-3 5 Figure 8-1 Memory Subsystem 8-9 Figure 9-1 Console Bus Structure 9-4 Figure 9-2 Console Bus Hub 9-5 Figure 9-3 SC SBBC Block Diagram 9-9 Figure 9-4 System... 80 5-7 367 ● Sun Microsystems, Inc., Sun Fire 3800 Installation Guide, Part Number 80 6-2 941 ● Sun Microsystems, Inc., Sun Fire Midrange Systems Hardware Reference Manual, Part Number 80 5-7 363 ● Sun Microsystems, Inc., Sun Enterprise Serengeti-12i, 12, 8, and SunRack Cabinet Reference Manual, Part Number 80 5-7 37 1-0 4 Advanced Sun Fire™ Mid-Range Troubleshooting Copyright 2002 Sun Microsystems, Inc All Rights... 6800/48x0 Servers Four-Slot cPCI Physical Slot Designations 1-2 8 Figure 1-1 8 Example I/O Device Path for Sun Fire 3800 Systems 1-2 9 Figure 1-1 9 Sun Fire 3800 System Six-Slot cPCI Physical Slot Designations 1-3 0 Figure 1-2 0 Sun Fireplane Interconnect Operational View 1-3 1 Figure 2-1 RTU and RTS Power Connections 2-5 Figure 2-2 RTS and RTU Units 2-6 Figure 2-3 Sun Fire 3800 Logical