The UL 4600 Guidebook

What to Include in an Autonomous Vehicle Safety Case

Philip Koopman, Ph.D.

Carnegie Mellon University

Thanks to Deb, Jackie, Heather & Uma

First Edition, 2022. (Version 1.0.00)

ISBN: 9798365303065 Trade Paperback ISBN: 9798365303249 Hardcover

No part of this publication may be reproduced or transmitted in any form or by any means, including photocopy, scanning, recording, or any information storage and retrieval system, without permission in writing from the copyright holder. This book has neither been approved nor endorsed by UL Standards and Engagement, which publishes the ANSI/UL 4600 standard.

INFORMATION IN THIS BOOK IS PROVIDED “4S IS” AND ANY EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR PUBLISHER BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, CONSEQUENTIAL, OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE INFORMATION IN THIS BOOK IS INTENDED TO ONLY PARTIALLY SUPPORT SAFETY PRACTICES, AND MORE IS REQUIRED TO ACHIEVE ACCEPTABLE SAFETY. YOU ARE RESPONSIBLE FOR THE SAFETY OF THE SYSTEMS YOU DESIGN AND OPERATE REGARDLESS OF THE CONTENT OF THIS BOOK. THIS BOOK IS NOT A SUBSTITUTE FOR THE OFFICIALLY ISSUED STANDARD. BY OPENING THIS BOOK, THE READER AGREES TO THESE TERMS AND ACCEPTS SOLE RESPONSIBILITY FOR ANY AND ALL DIRECT AND INDIRECT USES OF ITS CONTENTS.

Contents iii

Contents

PreLaCe ss sscsiicccsncceascewssed saseectecdenbtsdeasusebecsscabeceaseesbecdsesdseessecesdeetsadodeaseentesieestass vii L. IMtrOdUctiON.............ccscccssccscsccsscssscescsssnsssnsssnsssnessesssesssesssesssssessseees 1 VD QUICK HOU: 52.665 ei vnleths ceded cae Stade acl ee as sa es OR Gc ise ale NER tse ha 1 V2. RESOUPCES jis cia diey oe denchiobedee, ieee Austeteceios iesck peices sai Ligsntees th aeecks othe 2

2. Overview and applicability of UL 4600 ..............ccccsscsssesstscssscssscneeoes 3 2k: History OF UE 4000 & asiecavess oisdevissahenndschdszevvecdbvesd goats as saheathase eeoaveainwss ce 3

2 Aol WEA600 Meine 3.24.04 GeO oie ektets ee aed nite ian chetnideeeee tae backs 3 2.1.2. Other AV safety standards ..........cccecccescessceseceesceeeeeeseeeeeeseeeseeenaes 4

2.2. Scope and applicability of UL 4600.00.00... cc cecceescessceeceeeeeteeeteeenseesaes 5 2:2 1. WL A600 SCOPES sisesiets cisbedehoeadeteadeageua iiedd cca sotacbaad Gia tanegons beebieda tees 5 2.2.2. UL 4600 applicability 0... cc cecceseceseceseceseceeeceeeeeeseeeseeeeeeeeesaes 6 2.2.3. UL 4600 and the standards ecosystem. ........c.ceccccseesseeseeetseeseesees 8

2.3. Frequently asked questions and misconceptions. ..........escceceeseeeeeeeeeees 9 2A: RESOUICES i /Raccadeds hoes tetede ee eve se. ete es hides cides tiaeee areas 12

3. Requirements and prompt element .................sccsssccssscccccssceersserees 13 3.1. Requirement Structure ........ccccecccescceseceseceseceseceeeceeeeeeseseseeeseeeseeeseeenaes 13 3.2, Prompt Clements ziscesenstishsiseee tetiee Wiiehedh cena eeokaaea aes 14 3:3: DIEVIAUONS shez Stecsasarravacsreds aadeeretia a iibarestiaesecri aa tava etierenlateees 16

Ae. SVOTMIMOlOGY. sessccccscscacconssscssescccusecssavesonscssosocsntseswsscodesesssesedeccedeveossasesce 19 4.1. Terminology approach .0......cccecsceesseeseeseeessecseeceaecnsecesecseeeseeeseeeeneeeses 19 4.2. Key conceptual terms ..........cccceccseeseeseesseeseesecesecesecsseceeeseeeseeeeeeeeses 20

5... THe Safety CASE. sscessccssiciscsscscsesscsseseccnasssonsostessccsasedeasessissooveseseasscevesens 23 5.1. Structure of a safety CaS... ccceccseeseeseesecssecneceseceseeseeeeeeseeeeeneenaes 23 5.2. Safety Case PLOPCrties..... ee eeceesceeseesseesceeseecseecneceseceseeeeeseeeseeeeeseeeaes 26 5.2.1. Safety case scope and the item... cece ceecceseeeseeeseeeeeeeeeeeseeenes 26 5.2.2. Concept Of Operations ........ cc cecccesecssecsteceteceteceseeeeeeeeeeeeeeeeeeenseeeaes 27 5.2.3. Element Out Of Context interfaces... eeceeceeseeeseeeeeeeeeeeeeees 27 5:2: A Ty Pes: OF CLAIMS ia xieeees Pav digstoleve sovsalede ol iate vies vaste ticae oot vent es Saeed 28

5.3. Safety case tool SUPPOTT...... eee ccceeceeseetecsecsseceseceseceseceseeseeeseeeeeeeeenes 29 5.4. Argument SUffiCieny.......eeccceccceeseesseesseessecseeesseceeceseeseeeeeeseeeeeseeenes 31 5.4.1. Inapplicable prompt elements «0.0.00... cecceecceseceeseeeeeeeeeeeeeeeseeeees 31 Di4.2 ASSUMPUIONS: jeccecessse cavkvoetcnees oaesestbancvved dase Teaethaeveatdaseesseghvess ctbsaes 32 5.4.3. Difficult-to-reproduce aspects ........:ccsccesccessceseeeeeeeeeeeeeeeeeeenseeeees 33 SAA ACCepted TSK Sif cscditecses vous alee sa cebte settee ca sacs hanes bea aeek saa cetestaas 34 5.4.5. Confirmation bias and defeaters ...........ccccccesceesseeseeeseeeseesseenees 35 5.4.6. Non-deductive arguments and unknOWDS .........c:cccsseesseesteesteeees 36

Dede malety: CUMUCE: c.s5 s8 ocak vesdevess teddbestbatceceschs aeaveelawers cous Gaia at. dessesvelees 37 DG RESOULCES #5 i505; <- ecb oscaeete lanes So dat, ade iains aetonteeedboouesteleds had eae. colo seaete eset 38

6. - Hazards and risks .............ccssccssscssevssevsscssscsssessscssscssscssssssssssesseessoesees 39 61, Fault Model sic cecdesc Neher cae os eatin on Seationtelden ted bah ba dared daeetat 40

(SPO F-42106 (ce er aA Rn En en Al

iv Koopman

6.3 Risk: Cvalviati Ome. sce ie2. deed sa catedeotanicva ds seas tovhe decd dodetsess Waieass dana deeaateckd aoe 43 6:4 Risk iti Paton ic.2..se..ccedecs conseb cz cede saan sPustaddlagul sda te beaceOhiarasPeateaingehatans 44 6.4.1. Criticality levels and risk mitigation ...........cccccceesseeseeseeteesees 44 6.4.2. Going beyond risk-based mitigation ...........cccceesceeseeteeteeteeees 45 6.4.3. Acceptable item-level risk... ceecesccessceseceeeceeeeesseeeeeeeeeaeesaes 46 6:5 RESOUNCES .5...scieie da civngcatelonsecndeouetulnuad eceieessluearte ative dade easels 46 7. Interaction with people and road USELS.............ccssccssosssessseseseeseeones 49 7.1. Communication fUNCtIONS.........cccceecceesceeeeeeeeseeetseessecssecnsecnseceseeeseees 49 7.2. Interacting with people and animals .............cceceececeeceeeeeeeeeeeeeeeaeens 50 7.2.1. Occupant INteractiONS .........ccceesceseceteceseceeeceeeeeeeeeeeeeeseeeeeeeeesaes 50 7.2.2. Passenger-settable parameters ...........:cccscccssceesceeeeeeeeteeeteeesseesaes 51 7.2.3. Demographic profile.........ccccccesccsseceseceseceeeceeeceeeeeseeesseeeseeeaeesaes 52 7.2.4. Safety contribution by people...........ccescesscesseeeseeeteeeteeeteeeseesaes 52 TDD MOde Chan SOS 6: caesar: ccvnccs ty lu csnge an ek ankeaeenrecdacectsteuieiee cietdesttenlb ities 53 7.3. Interacting with other Vehicles.........eccecsceesseeseeseeeteeseceecnteeeeeeaeens 53 7.3.1. Hazards involving other vehicles ..........c.cccceesceeseeeseeeteeetseenseesaes 54 1232s RUNG DrCAKING ws 255% sethis vas, teseves shesgessecevs sandeseateagengs ons See sesgansenss 54 TAGRESOURCES sisccsci vestvaietogn set cides HeSedgetvaaeans atecgneal Gdop vba tteasee deters oe 55 8. Autonomy functions and SUPPOFt .............cccsccsccscecccecscssesccessscerssesees 57 8.1. Autonomy architecture eSCTIPtiONn.........:ccceeceseceteceeeceeeeeteeeseeeeneeees 57 8.2. Operational Design Domain ........... cc cccceceeseceteeeteceteceeceeeeeeeeeneeeeneeaes 58 BBs SCNSING ss: ccccciTealeect sis sages bangnedescedesed snsadods aacalda vengdates undeand oaeduetbcaelvones 60 8.3.1. Individual SeNSOTs..........ceccceescessceesceeseeeseeeseeeseeeeecaecssecneenseeeaeees 61 81322 “SenSOF LUSION as steeacstderecs lessees raise etaceteeeR tales tates 61 B.A POTCE DU OM Jceacc sexes Sa deet cs fe cavs aah cen rede faces cola ld vaste cevead devs eota dering: 62 8:5--Machine learning escape ocedegi cnet thse cewhcdecd oetedceaeve lub vecdcaakeutveakvinns 63 8:6, PLANNING. 33 225. .cgeec sie adee ke bbedi a tecncbeswndauge Sd Shane vorsdudesnsetule tandeesadondageadands 64 B27. PEECICHON s cscs sscaehensp Fics easdetas bo diced ad lanhpeedeobancdyacs]y divessaeableabediocuagoee 64 8.8. Trajectory, motion control, actuation, and status...........ceeeseerees 65 BO. TAIN Gos aise. cases sandy adv since bet once ate none bes Sindh ais ous saver clave wivovuseVea cba tues vbegenwerts 66 Si LOssRESOUICES eiiee ve ccccteeenipe ceeds vetshbet a tenbtieei ee inieiee babe dubia de 67 9. Software & system engineering PLOCeSS ............ccccccesccscssceressseeseess 69 9.1. Software and system engineering key Points...........::ceseeseceteeeteeeeeees 69 O72. RESOULCES $ii20s<21eseciten festeae ce eaten ieyeteescesddedageagea so etgelbrdgstete Lieceeweean ented 70 10. Dependability .................cssscsccsrccsscsccccesccessscrsscscescccsesccesssssessesceseeees 71 10.1. Fault detection and mitigation ..........cceceeseescesseeeneeeteeeteeetseesseensees 71 10.1.1. Fault detection coverage & diagnoOSis.........ccccceseeseesteesteensees 71 LOW: 2.-Remte oration -.cscsiiesecivecscccessecotb ee edecstcushcatvans ceastvaatdanpesdeeestees 72 10.2. Redundancy management ...........cccccccessceesceeeceeeeeeseeeeeeeeeeeesseesees 73 10.3. Operational MOES 00... ceceescceseceseceseceeeceeeceeeeeseeeeneeeseeeseeeaeeeaeesaees 74 10.3.1. Identifying MOdeS....... cece ceeceseceseceeeceeeeeeeeeeseecseeeseeeeeesaeenaees 75 10.3.2. Degraded operations ...........ccccsccesccsssceeeceeeeeeeeeeseeeseeeseeceeeseeensees 75

LOA termi FODUStHN SS si sicc. cecsccavecsteets oneds cas uabcdecsonsavcsVoalevensoneds es uadedetedeece es 78

Contents Vv

10:5. Incident: reSpOnse sei vc seve ilaccceceass Leake chedased is Levies si teanees lesa bed Deletlvnts 79 LOs6:, System tMINg feist okie ee cae Glaielta Rite thse eGheelete eons 81 10.7. Cybersecurity .......ccccccccesccesscsesceeeceeseeeseeeseecseecsaecaecesecnseeseeseeeeeneeses 81 10:8. RESOULCES ..5:2.5:5cocebeciehupeeblanshegdethsouicoes Loeddechaptablesss, levbecdecealepgiecbare 82 11. Data and networking ................ccssssccssscsscccescesssccecceccescceseseesscsseseess 83 11.1. Why data safety matters 00.0.0. cccecccecsceeseeseeeseceseceseceeeseeeseeeseeeeses 83 11.2%, Data transmission 3s. c0tic cece evcasvva vik cons cccsivenh Cobtean chests daitees stonzecststevdeasts 84 11.2.1. Data transfer properties... cccecccecsseesseesteceteceteceeeeeeeeeeeeseeenes 84 11.2.2. Remote Operation .........eccecccecscesseeseeessecseeeseecseeceecneeeeeceeeeeeneeaes 85

1133. Data StOra ge ic. .ieiico a eccccssctessestaetslaecite. teedeaciiacecbestbeajsteisen tes anccbise 86 11.4. Infrastructure (data and otherwise) .0......cccccccesscsteceseceseeeteeeeeeeseeeses 87 VS 3, RESOURCES i shircc 2 2eccnacnlovrns shies Ataginncns caus athens sdds austere Heagatenbeee: 88 12. Verification, validation, and test...............ccsscssssssscssscssscsssesssscssones 89 VQ al. VGCV Cth OS® osicsscces tierce cicatieae ad tecdndenkven’ Cab teey ce onktd teen oe oieeetstendeen 89 T2220 SENG COVELA BCH i522 Sa cceds waladtebiha adele heed aesaguanes hdeedar shgueectobecdeustinereeads 90 12:3. TOStUI Sven cs cessed card e6 8 cdevsda seed Ohcadaaulade aed hh dostte Wit abl needa nether 90 12.4. Run-time MOMItOTING .... eee eeceeceeseeetceeceeseeeseecaecesecsseeseeeseeeseneeses 92 12:5. Satety: case: updates ..veacsecdciavasciactteet doce esdedee eden nesvoguae senate: 93 1276: RESOUNCES 4 j-teciavalavers bdiat aiotars Addeaaresianes itnas tien Tecauel ene a 93 13. Tools, COTS, and legacy qualification................scccssssssscssscsssesseees 95 V3 el LOOMS? octet cenveceeietat cob tens cUaedives thivees ndcakotebsbuecg hantvehdeh Can centolse eevbteas caved 95 13D SiMUAtONS cccs cieeiteeeacdssdgeete ee sghe de buedgned doeeddgosbseeloe ovedduassandeael doneldeete 96 13.3. COTS and legacy COMpoOMNENtS ..........ccceeseesteeeteceteceteceeeceeeeeeeeeeneeeses 97 13745 RESOUNCES :332265. 5. 22och PE eaith petites eee aeeei pete dee aces 99 14. Lifecycle COMCEITS .............scccscssccrccsscesccccescceseceesseccesceesesceessessesceseese 101 14.1. Requirements and design validation ...........ecceceeseeeeceeeseeeeceeeeaeees 101 14.2. Build, design release, and manufacturing ............:cecceseeteeeteeeeeeees 102 P43 Supply chain oui ccc scosrcects ivceivecdvecctuveuds tee cnencesitendubeiedediekten lub eecs tod 103 14.4. Field modifications and updates... ccccsseesseesteeetecetecetecnteeeeeeees 103 14:5): QperathOne::25:si es ecliovisccaseveleeasei gligidiarielaahh dae taiee Sl gineieiwelss 104 14.6. Retirement and disposal 00.0.0... ccceccceeseeseeeseeesseeseeenecseceecneeseeeees 105 V4], RESOUTCES "isi, oceisbcnscSeieeceetcsenteel fag seul oben Ateadea tease echoes sen rast 106 15. Maintenance) .............ccsccsscccsssssccssscssccssscsssssssscssscsessnsssnsssnesensssecseeness 107 15.1. Maintenance and inspection requirement ...........::ccccececeteeeteeeeees 107 15.2. Prompting maintenance and inspection activities «00.0... eee 108 15.3. Maintenance and inspection faults 0.0.0.0... cccecsseesseesteestecetecnteeeeeees 109 15.4. Non-operational safety ..........ccceeccecscessceeseeeseeeteeeseeesseesecseenteesseees 109 ESA. Tmacti ve vehicle i. sccccss css seteacahaeessccgareess counts schasstascddensssabeeaeab vey 110 15.4.2. Equipment degradation ..........ccccsceesseesseeseeetseesecsecstecnteeeaeens 110

15,52, RESOULCES sii. 2 tieiethd ies aebnd Sipe enti er Wend Abe eta 111 16. Safety Performance Indicators ..............cccscsccsssscceccsssesscscssceesssserees 113

16:2. Threshold Values esc.c. cock vec cateccicvcoctveccugesbessuwerveccusbesbesedetisccugeveocseeets 113

vi Koopman

16.2. Dangerous behaviors ..........cccceesseesseesseesseessecssecnsecnecnseeseeeeeeeeeeeeses 114 VO:3: SUPPTISES ss. cbacccste.dan cont ii ceriel a ceBiactaete feroceelaeiebstaa WO a oaate tea onetads 115 16.4. SPI collection and feedback... ecccccccsseestecsteceteceseeeseeseeeeeeeeees 116 16.5. A proposal for tying SPIs to the safety case oo... ee eeeeeeseeeeeeeeeee 117 FG. 6: RESOUECES Sesrecchsseccts ans rece es cock wvcgavees cobanchs vaueese ceadeasvvadouean cease Dheshatees 118 17. ASSOESSMENE ..........cccccsscssscssecssscsssensssnessesssssssessscsssssesssescsesssessnessnesees 121 17.1. Conformance planning.........ccecccecsceesseesecsteestecneceeceeeeseeseeeeseeeses 121 17:2, Self-asSeSSIMent ge. scsecdes we fezeveavachans deees cased dete ik ebat deine 122 17.3. Independent assessment...........ccccsceesseesceesteeseceteceseceeeneeeseeeeseeeses 123 17.4. Continual re-assesSMent ........:.eccceescesseeeeeeseeeseeececseeessecneeneesseenes 125 L8., “WEAP=UP scccissvassdcsaceccstssnsnccenaceceensousnscevasecsenscenvossnavocecasesensdsnesodseascoenad 127 18.1. Additional information ..........cccceccceeceecseesseeseeesecseeensecneeneeseeees 127

18.2. About the author wo... cccccessessssscescescsesessssesesceseeessnseaseesees 128

Preface vii

Preface

Autonomous vehicles will not be viable for real-world use on public roads unless we can make them acceptably safe. Not perfectly safe to the point of zero crashes — although that is a worthy goal. Rather, acceptably safe will do for commercial deployment, with a hope of providing a substantive improvement over the current mishap rate of human drivers.

An acceptably safe outcome for autonomous vehicle (AV) deployment is not a foregone conclusion. Safety engineering does not happen all by itself, not even if super-smart engineers have the best intentions to create safe AVs. Simply being smart will not ensure all aspects of AV safety are covered any more than being smart will necessarily instill the skills and experience needed to make a safe aircraft. Achieving a safe outcome requires having strong safety engineering skills, as well as applying lessons learned across many domains in how to create safe systems.

Creating a safe system design has always required tremendous attention to detail. That in turn involves the use of specific safety engineering approaches such as hazard analysis, risk mitigation, and careful implementation of redundancy architectures. Following an industry-created safety standard helps ensure that the right approaches are used in the right way to achieve acceptable safety.

Because of the novelty of machine learning technology and lack of a human driver to display an approximation of common sense, autonomous vehicles present significantly different and dramatically more challenging issues for ensuring safety than traditional vehicles. Different companies are trying different approaches tuned for different applications and different implementation architectures. We have not yet arrived at a fixed design approach for building a safe AV. Nonetheless, such vehicles are deployed on public roads, and safety remains a pressing question.

The industry consensus at this point seems to be that safety will not be ensured by following a building code-style recipe for how to build an AV. Maybe that will happen someday, but not today. Rather, the industry has converged on the concept of a safety case as a way to argue that an AV is acceptably safe for its intended operations. The idea of a safety case for cars is not a new one. The decade-old ISO 26262 automotive functional safety standard requires a safety case.

ANSI/UL 4600 extends the safety case approach to its logical conclusion for AVs, resulting in the most comprehensive standard for autonomous vehicle safety currently available. It describes how to assess that the AV’s safety case includes everything it should, to support a credible claim of acceptable safety.

This book covers the background of the standard, how the standard is structured, key terminology, and a clause-by-clause summary of the standard. It is not a detailed restatement, but rather a high-level overview. Ideally, the

viii Koopman

reader will go through this book, get the big picture, and then be ready to dive into the details of the standard itself.

To keep things concise, this is a guided tour of the standard rather than an in-depth text on system safety. The style of writing is intended to be a descriptive narrative rather than an academic text. This should make concepts more accessible to those who are not safety engineering experts. (Those looking for the humorous footnote style seen in my previous book on “how safe is safe enough” will be disappointed. This is more of a just-the-facts guided tour.)

If you are familiar with system safety concepts and functional safety, especially in the automotive industry, you might find yourself nodding along as you run down through the topics. If so, that’s great, because it means you’ re getting the big picture in mind as preparation to dive into the standard.

If you hit a chapter on a topic you’ve not dealt with before, that is a great opportunity to expand your breadth in system safety before diving into the details of that part of the standard. Most chapters have a reference section with places to get started if a topic is new to you. If you are new to safety engineering for AVs in general, chapter 1 lists some getting-started resources.

Creating UL 4600 has been quite a journey. I personally wrote the proposed text (200+ pages of it) of the initial draft that kicked things off. After submitting that draft, we followed an ANSI-conformant consensus process to ensure robust engagement with stakeholders. Hundreds of comments (many hundreds) arrived from all over the world. Suggestions, and sometimes complaints, were resolved. That feedback improved clarity, added essential elements, and resolved controversy.

I have remained closely involved with the revision process for each edition via submitting change proposals, performing technical reviews, and commenting on proposed changes. But by no means am I the only one at work on this standard.

As is appropriate for an industry standard, the entire revision and approval process is public. I get one vote out of 30-40 allocated to members of the Standards Technical Panel (STP) voting committee for eventual approval of each edition of the standard. The issued standard represents the results of an accredited industry standard consensus process that reflects inputs from vehicle makers, component suppliers, regulators, consumers, assessment organizations, safety researchers, and more.

From time to time some industry politics have come into play — as they do with every standard. But a delightful thing about his process has been that the participants were overwhelmingly not there for the politics, but rather to get the job done. Many in the industry doubted that UL 4600 could be issued on our stretch-goal timeline of about a year from proposal to issued standard, but indeed that is how it turned out. It could never have been done without the common efforts, willingness to have frank discussions, and helpful contributions of so many Standards Technical Panel members and other stakeholders. Thank you so much to everyone who contributed!

Preface ix

If you wonder how I came to know enough about such a wide variety of topics to put into the draft proposal, chapter 18 has a brief bio. Suffice it to say that I’ve had a really broad range of experiences across many different industries, and seen an awful lot of stuff that is relevant to a standard like this. That includes doing hundreds of design reviews for products and components not only automotive, but also rail, industrial controls, building automation, power systems, and even a bit of work on aviation control networking safety. Much of UL 4600 falls into the bins of “don’t make this mistake because that turned out badly for someone else,” “this is how safety is done in other industries in addition to automotive” and “did you think of that?”

While many stakeholders made valuable contributions, there are a few special contributors I want to thank in particular. Deb Prince wrangled everyone (including me) through the standard process and supported my sometimes unconventional approaches. Jackie Erickson provided invaluable contributions to stakeholder outreach and messaging, especially with regulators and media. Heather Sakellariou did the heavy lifting on logistics, editing, comment management, and production for the standard. Uma Ferrell provided pivotal feedback on the early outline as well as lessons drawn from her extensive aviation safety experience. Thanks also to Frank Fratrik, Jason Smith, and Mahmood Tabaddor for their contributions to the drafting process. Jack Weast, Rafael Zalman, Finch Fulton, Nat Beuse, Junko Yoshida, Roger Cohen, Aaron Kane, and Chuck Weinstock also provided particularly important discussions, support, and other contributions.

Nothing is ever perfect, and everything can be improved. But fortunately, both this book and the UL 4600 standard itself can be updated with comparatively little pain. If you see something that should be fixed, please let me know via an e-mail to AVSafety@Koopman.us

Meanwhile, happy reading!

Philip Koopman Pittsburgh, PA, November 2022.

Introduction 1

1. Introduction Welcome to the world of UL 4600!

This book boils a lengthy, dense, and complex standard down as much as can be done while still being a comprehensive treatment for something that is — well — lengthy, dense, and complex.

UL 4600 takes a detailed, thorough approach because its purpose is to make sure nothing important gets left out of the safety case for an autonomous vehicle (AV). AVs are incredibly complex, so there is a lot of ground to cover.

This book, in contrast, uses a more narrative approach. General themes, ideas, and considerations are described, often in an order that flows better from a narrative point of view. While most chapters correspond to the sections of UL 4600, this book’s subsections do not necessarily follow the exact flow of the standard’s subsections. Rather, this book follows an order better suited to telling the high-level story of each corresponding clause (chapter) in UL 4600.

Not every detail can be in this book. Rather, the main ideas are discussed in a general sense. Think of this book as a way to understand the main themes and get an orientation to what is going on, without getting bogged down in the mechanics of the standard itself. After all, if you really want the gory details, that is what the standard is for.

1.1. Quick tour

Here is a quick tour of the rest of this book:

e Chapter 2 covers the history and scope of UL 4600. Briefly, it deals with how to know that an autonomous vehicle (AV) safety case has what it needs to ensure that an AV will be acceptably safe. There is also a Frequently Asked Questions section that answers common questions and clarifies some common misconceptions regarding UL 4600.

e Chapter 3 describes the structure of UL 4600, which emphasizes the use of “prompt elements” to help remind both safety engineers and safety case assessors what should be addressed by the safety case. It is important to be oriented to this approach before diving into the standard itself.

e Chapter 4 covers key terminology and concepts. Every standard has some defined terms with nuances not necessarily easy to interpret without a little introductory guidance. Read this chapter if you want to know what UL 4600 might mean by “acceptable,” “item,” and “argue,” among other terms.

2 Koopman

e Chapters 5-17 cover the corresponding clauses in UL 4600, ranging from safety cases in chapter 5 to the assessment process in chapter 17.

e Chapter 18 has some pointers to additional information that might prove useful.

1.2. Resources

e UL 4600 information launch page: https://users.ece.cmu.edu/~koopman/ul4600/index.html

e Video tutorial on UL 4600 (23 minutes):

o YouTube version: https://youtu.be/Zx VMX8SjPvw o Archive.org version: https://archive.org/details/L109-ul-4600 e Video tutorial series on AV safety to provide background © https://users.ece.cmu.edu/~koopman/lectures/index.html#av (includes slides, YouTube videos, and archive.org mirrors)

e A graduate-level course on embedded system and software safety taught by the author at Carnegie Mellon University, with all lecture videos freely available online: https://course.ece.cmu.edu/~ece642/

e Some historical notes on the evolution of UL 4600: https://www.eetimes.com/safe-autonomy-ul-4600-and-how-it-grew/