First made available in 2014. Released under the terms of The lead author would like to thank Maryse (May) Beauregard for proof reading this text throughout her term and to Colin MacPherson, among many other students, who made suggestions for how the original course could be modified. I would also like to thank the many MTE graduates and employees at Clearpath Robotics, Inc., especially Ilya Baranov. Finally, and most importantly, the lead author would like to thank the many authors who have published research and textbooks on this and related fields. While many of those papers and texts are excellent references, few are appropriate as a textbook for a junior undergraduate course in real-time systems, hence the reason for authoring this text. Typographic conventions This text uses a 10 pt Times New Roman font where italics indicates new terms and names of books. 9 pt Consolas is used for program listings and console commands with output, and within paragraphs for keywords, variables and function names. Disclaimer This document is intended for the instruction and examination of MTE 241 Introduction to Computer Structures and Real-time Systems at the University of Waterloo. The material in it reflects the authors’ best judgment in light of the information available to them at the time of preparation. Any reliance on this document by any party for any other purpose is the responsibility of such parties. The authors accepts no responsibility for errors or omissions, or for damages, if any, suffered by any party as a result of decisions made or actions based on the contents of this text for any other purpose than that for which it was intended. Printed in Canada. A Practical Introduction to Real-time Systems for Undergraduate Engineering Douglas Wilhelm Harder, Jeff Zarnett, Vajih Montaghami and Allyson Giannikouris University of Waterloo Version 0.2015.09.91 To Sherry E. Robinson, Bill S. Lin and Jakub Dworakowski vii Preface This is an introduction to real-time systems for engineering students who are not focused on computer or software engineering. This document is intended for MTE 241 Introduction to Computer Structures and Real-time Systems . The material in it reflects the authors best judgment in light of the information available to them at the time of preparation. Any reliance on this document by any party for any other purpose are the responsibility of such parties. The authors accepts no responsibility for damages, if any, suffered by any party as a result of decisions made or actions based on these course slides for any other purpose than that for which it was intended. A one paragraph summary of this text is as follows: This course will being by introducing the requirements and constraints of real-time and embedded systems, together with examples. Next, we will consider various programming paradigms, appropriate characteristics for embedded and real-time programming languages, and an introduction to the C programming language and contrast it with C++. We continue by describing the organization of a computer, including descriptions of Turing machines, register machines, main memory, processor architecture and operating systems. Continuing from here, we describe static memory allocation and the call stack, and then consider dynamic memory allocation, including numerous variable-sized-block strategies, their appropriateness for real-time systems and various implementations in FreeRTOS. We also consider automatic memory allocation, including garbage collection. Following this, we discuss the idea of threads and tasks running in parallel, looking at examples of sorting in parallel, and the data structures necessary to maintain threads. We then consider the issue of scheduling threads, first with multi-programming, non-preemptive scheduling and the concept of context switching, and then considering multitasking with preemptive scheduling, focusing on real-time schedulers such as earliest-deadline-first and rate- monotonic scheduling. Next, we consider the issue of the communication of other devices with the processor and the concept of interrupts and interrupt service routines as well as the impact of interrupts on real-time systems. Next we consider synchronization issues between cooperating threads and tasks, including issues of serialization and synchronization. We describe semaphores and consider a number of synchronization problems and consider solutions using semaphores. Additionally, we consider other means of automatic synchronization. With this, we consider the application of semaphores and synchronization in general to resource management, looking specifically at priority inversion. The most serious issue, however, is deadlock, when tasks and threads holding resources are mutually blocked on subsequent requests and how to avoid this issue. Next, inter-process communication is discussed together with how synchronization can be achieved through messaging. Next, we consider fault tolerance, specifically considering error detection and correction, the synchronization of clocks, and fault-tolerant message passing. Having considered all this, we now consider how resource management can be centralized in an operating system protected with fault tolerance through kernel modes and space. Having achieved this, we now consider the problem of software simulating including client-server models and distributions, and then software verification, including a look at propositional logic, predicate logic, linear temporal logic, computation tree logic and model checkers. We conclude the course by consider issues of data management and file management, virtual memory and caching, digital signal processing and digital control theory, and finishing with an introduction to security and a look at what is ahead. viii The following is a brief summary of the topics with brief justifications as to the order. Note that based on the strategy, the order of these topics differs greatly from a general operating systems course and that the emphasis on topics will differ based on the focus on real-time systems as opposed to the design of general operating systems, per say. 1. Introduction to real-time systems An introduction to what a real-time system is and is not. This will be supported by various definitions and requirements as well as examples including anti-lock braking systems and issues with real-time systems such as the Mars Spirit rover. 2. Real-time programming MTE students will, up to this point, have only taken C++ courses with a focus on object-oriented programming. We will discuss desirable characters of real-time programming languages and consider some of the shortcomings of C for this purpose. Never-the-less, this lecture will be designed to introduce the paradigms of an imperative language including a discussion on the design of data structures. This will lead to Laboratory 1 which will see the students author a data structure, compile it, and download it onto the Keil board. We will conclude with a discussion on memory allocation, both static and dynamic, by viewing the consequence of each in a C program. This will lead to the topic following a high-level description of computer organization. 3. Computer organization This next topic will introduce models such as register machines and the relevance of the processor and main memory by a quick description of a Turing machine. Next we will visit various architectures, including the Harvard and von Neumann architectures, but will also look at the Cortext-M3 core design. We will also consider resources and conclude by the functionality offered by operating systems and the constraints placed on real-time operating systems. 4. Static memory allocation Based off the discussions in both the previous topics, we proceed with discussing static memory allocation, including data and a call stack. This topic is meant more a higher level overview of these topics, but we will look at the call stack in the RTX RTOS works, as an example. The detailed implementation of a call stack is likely not necessary for a mechatronics student (comments please?). 5. Dynamic memory allocation Again, based on the discussions in Topics 2 and 3, we will now proceed to discussing dynamic memory allocation strategies. We discuss the various approaches and strategies, and determine which of these would be most appropriate for real-time systems. Having read through the literature, there are a number of dynamic memory allocators that are appropriate for real-time systems but are not discussed in many of the undergraduate text books on operating systems due to their specialized nature. This topic will include discussions on the C implementation of these strategies and look at the rt_MemBox implementation. This will lead to Laboratory 2 which will look at implementing a memory allocation strategy in a program that will be downloaded onto the Keil board. ix 6. Threads and tasks Next we discuss the concept of tasks and parallel execution. In this introduction, we will focus on executing tasks in parallel on separate cores or processors with access to shared memory. We will discuss concepts relevant to solving tasks in parallel, beginning with parallelizing algorithms such as quicksort and merge sort, topics covered in any introduction to algorithms and data structures. We may consider graph algorithms, and then consider the general characteristics of algorithms that can be parallelized. We may discuss the existence of NC, the class of problems that can be solved in polylogarithmic time on parallel computers — I do not currently see the value in it, but I am open to changing this if it is felt that theory on this level is appropriate for mechatronics students, at least to make them aware that such theory exists. This will lead to a Laboratory on converting a serial algorithm into one that is executed in parallel on the Keil board. 7. Scheduling Next we consider the problem of executing multiple tasks on the same processor. This will introduce the idea of a scheduling algorithm and we will consider all necessary tools necessary to perform such tasks. We will first look at multiprogramming, followed by an introduction to hardware interrupts, followed by a discussion on multitasking. This will lead to the concept of scheduling tasks and we will first look at earliest-deadline first and least-slack-time first scheduling, and some of the issues associated with these. This will be followed by considering real-time scheduling and the use of priorities to overcome some of the issues with the two previous algorithms. We will then consider periodic scheduling, specifically the rate- monotonic scheduling algorithm as well as dealing with sporadic interrupts. We will conclude by considering multitasking together with periodic timing interrupts and schedulers such as round-robin. 8. Hardware interrupts How hardware interrupts work, interrupt service routines, interrupt vectors and masks. 9. Synchronization For synchronization, we will consider a number of problems in both serialization and mutual exclusion. We will look back and see how the scheduler can be used to make the implementation of semaphores efficient by the blocking of tasks. This was a relatively brief topic in the previous offering, and deserves more significant focus. We will use semaphores to model solutions to most of the problems we will look at, but we will also discuss monitors and other solutions, specifically looking at the use of the Java keyword synchronized The Ada rendezvous will also be described. We will conclude with a discussion on the problem of priority inversion where a lower-priority task inadvertently blocks the execution of a higher- priority task and some solutions for this. Again, some of these topics are not even covered in operating systems text books due to their specialized applications. This will lead to a Laboratory where students will solve a problem not covered in class using semaphores. 10. Resources management Now that we have considered semaphores as a specific resource that can be used by tasks, we will next consider the allocation of resources in general. This is appropriate at this time, as all of the issues we have seen with semaphores and synchronization also apply to the allocation and sharing of resources. We will discuss various mechanisms that can be used in conjunction with the scheduler to ensure the efficient execution of tasks in environments where different tasks will compete for resources. The Mars Pathfinder as well as other cases will be considered where resource allocation strategies can lead to deadlock or missed deadlines and how all the strategies used to, for example, prevent priority inversion with semaphores, automatically apply to resources. This will lead to a laboratory using other resources on the Keil board. 11. Deadlock We now will consider the issue of deadlock with respect to synchronization by considering examples where deadlock can occur quite easily. We will look at various algorithms for deadlock detection and recovery. Note: we will not consider the banker’s algorithm, as this is, to my understanding, hardly ever used even in embedded systems. x 12. Inter-process communication In this topic, we will consider various means of inter-task communication, including buffers and pipes and the use of messaging and mailboxes. Unlike the previous offering of this course, a significant focus will be made as to how inter-process communication can be used for the synchronization of tasks executing on independent processors. 13. Fault tolerance How do we deal with faulty systems? 14. Operating systems Up to this point, we were focused on the separate components necessary to have tasks run in parallel and to synchronize in order to achieve various goals. We will now wrap these tasks together into a kernel and discuss the benefits and costs of having this functionality provided by a series of functions executing in a protected kernel mode (together with a discussion of software interrupts). We will discuss various operating systems and their appropriateness with respect to them being used for real-time environments. We will also observe that essentially all of the functionality that we have discussed in class is associated with functionality available in the RTX real-time operating system. 15. Simulating physical systems How do we simulate a physical system? That is, how do we validate a system? 16. Software verification How do we verify that software does what it should do? 17. File management This topic will look at the design of various file management systems as an overview. We will consider how such a system can be built on top of the flash memory available on the Keil evaluation board. 18. Data management An overview of appropriate data structures and data management for real-time systems. 19. Virtual memory and caching We conclude with two other topics which mechatronics students should be aware of, but are not critical to real-time systems and some of the provisos that should be made if either of these is used in a real-time system. We discuss these together because the algorithms that are appropriate for one are also appropriate for the other. 20. Digital signal processing An introduction to digital signal processing, including the definition of signals, signal processing, a discussion on causal linear time-independent filters, digital filters and discrete transforms. 21. Digital control theory An introduction to digital control theory. 22. Security To be completed. 23. Looking ahead Looking ahead to see what you can expect in future courses and research in real-time systems. xi Contents Preface ..........................................................................................................................................................................7 1 Introduction to real-time systems ............................................................................................................................ 1 1.1 What is a real-time system? .................................................................................................................................1 1.2 Case study: anti-lock braking system .................................................................................................................4 1.3 Components of real-time systems ........................................................................................................................5 1.4 The history of real-time programming ............................................................................................................... 12 1.5 Topic summary .................................................................................................................................................. 12 Problem set .............................................................................................................................................................. 13 2 Real-time, embedded and operating-system programming languages .............................................................. 15 2.1 Programming languages .................................................................................................................................... 15 2.2 The C programming language ........................................................................................................................... 28 2.3 Summary of real-time programming ................................................................................................................. 48 Problem set .............................................................................................................................................................. 49 3 Computer organization .......................................................................................................................................... 51 3.1 The Turing machine ........................................................................................................................................... 52 3.2 Register machines .............................................................................................................................................. 53 3.3 Main memory .................................................................................................................................................... 57 3.4 Processor architecture ........................................................................................................................................ 63 3.5 Operating systems .............................................................................................................................................. 69 3.6 Computer organization summary....................................................................................................................... 72 Problem set .............................................................................................................................................................. 73 4 Static memory allocation ........................................................................................................................................ 75 4.1 The requirements of a function .......................................................................................................................... 76 4.2 The Cortex-M3 design ....................................................................................................................................... 81 4.3 Set jump and long jump ..................................................................................................................................... 81 4.4 Summary of static memory allocation ............................................................................................................... 82 Problem set .............................................................................................................................................................. 83 5 Dynamic memory allocation .................................................................................................................................. 85 5.1 Abstract dynamic memory allocator .................................................................................................................. 85 5.2 Allocation strategies .......................................................................................................................................... 94 5.3 Case study: FreeRTOS ................................................................................................................................... 107 5.4 Other features: clearing and reallocation ........................................................................................................ 112 5.5 Summary of dynamic memory allocation ........................................................................................................ 113 Problem set ............................................................................................................................................................ 114 6 Threads and tasks ................................................................................................................................................. 117 6.1 Weaknesses in single threads ........................................................................................................................... 117 6.2 Creating threads and tasks ............................................................................................................................... 118 6.3 Applications of threads and tasks .................................................................................................................... 125 6.4 Maintaining threads ......................................................................................................................................... 131 6.5 The volatile keyword in C .......................................................................................................................... 142 6.6 Summary of threads and tasks ......................................................................................................................... 143 Problem set ............................................................................................................................................................ 144 7 Scheduling ............................................................................................................................................................. 147 7.1 Background: waiting on tasks and resources .................................................................................................. 147 7.2 Introduction to multitasking ............................................................................................................................ 148 7.3 Non-preemptive scheduling algorithms ........................................................................................................... 173 7.4 Preemptive scheduling algorithms ................................................................................................................... 178 7.5 Issues with scheduling ..................................................................................................................................... 200 7.6 Summary of scheduling ................................................................................................................................... 203 Problem set ............................................................................................................................................................ 204 8 Hardware interrupts ............................................................................................................................................ 209 8.1 Sources of interrupts ........................................................................................................................................ 209 xii 8.2 The mechanism of interrupts ........................................................................................................................... 210 8.3 Ignoring and nested interrupts ......................................................................................................................... 214 8.4 Waiting for an interrupt ................................................................................................................................... 217 8.5 System design .................................................................................................................................................. 218 8.6 Watchdog timers .............................................................................................................................................. 220 8.7 Implementation of interrupts ........................................................................................................................... 221 8.8 Apollo 11: an interrupt overload ..................................................................................................................... 223 8.9 Summary hardware interrupts .......................................................................................................................... 223 Problem set ............................................................................................................................................................ 224 9 Synchronization .................................................................................................................................................... 225 9.1 The need for synchronization .......................................................................................................................... 225 9.2 Petri nets — describing synchronizations graphically ....................................................................................... 231 9.3 Synchronization through token passing ........................................................................................................... 237 9.4 Test-and-set — a crude signal with polling ....................................................................................................... 238 9.5 Semaphores — a better signal without polling .................................................................................................. 240 9.6 Problems in synchronization ............................................................................................................................ 258 9.7 Automatic synchronization .............................................................................................................................. 281 9.8 Summary of synchronization ........................................................................................................................... 286 Problem set ............................................................................................................................................................ 287 10 Resource management........................................................................................................................................ 291 10.1 Semaphores .................................................................................................................................................... 291 10.2 Classification of resources ............................................................................................................................. 291 10.3 Device management ....................................................................................................................................... 293 10.4 Resource managers ........................................................................................................................................ 294 10.5 Priority and deadline inversion ...................................................................................................................... 294 10.6 Summary of resource management ................................................................................................................ 297 Problem set ............................................................................................................................................................ 298 11 Deadlock .............................................................................................................................................................. 299 11.1 Requirements for deadlock ............................................................................................................................ 299 11.2 Deadlock modeling ........................................................................................................................................ 300 11.3 Techniques for preventing deadlock during the design ................................................................................. 303 11.4 Deadlock detection and recovery ................................................................................................................... 308 11.5 Deadlock avoidance ....................................................................................................................................... 323 11.6 Summary ........................................................................................................................................................ 323 Problem set ............................................................................................................................................................ 324 12 Inter-process communication ............................................................................................................................ 325 12.1 Classification of communications .................................................................................................................. 325 12.2 Solutions for communication ......................................................................................................................... 327 12.3 Priorities of messages .................................................................................................................................... 330 12.4 Synchronization ............................................................................................................................................. 330 12.5 Network communications .............................................................................................................................. 333 12.6 Summary of inter-process communication .................................................................................................... 336 Problem set ............................................................................................................................................................ 337 13 Fault tolerance .................................................................................................................................................... 339 13.1 Failures in real-time systems ......................................................................................................................... 340 13.2 Redundancy ................................................................................................................................................... 344 13.3 Error detection and correction in signals ....................................................................................................... 345 13.4 Clocks ............................................................................................................................................................ 350 13.5 Byzantine generals’ problem ......................................................................................................................... 358 13.6 Summary of fault tolerance............................................................................................................................ 366 Problem set ............................................................................................................................................................ 367 14 Operating systems............................................................................................................................................... 369 14.1 Operating systems as resource managers ....................................................................................................... 370 14.2 Processor modes ............................................................................................................................................ 370 14.3 Memory management .................................................................................................................................... 373 14.4 Microkernels .................................................................................................................................................. 373 xiii 14.5 Real-time operating systems .......................................................................................................................... 374 14.6 Examples of real-time operating systems ...................................................................................................... 374 14.7 Summary of operating systems ...................................................................................................................... 380 Problem set ............................................................................................................................................................ 381 15 Software simulation ............................................................................................................................................ 383 15.1 Physics engines .............................................................................................................................................. 383 15.2 Modelling client-server systems .................................................................................................................... 383 15.3 15.3 Simulating variation ............................................................................................................................... 393 15.4 Summary of simulating physical systems ...................................................................................................... 396 Problem set ............................................................................................................................................................ 397 16 Software verification .......................................................................................................................................... 401 16.1 The scenario and user or client needs ............................................................................................................ 402 16.2 Propositional logic ......................................................................................................................................... 403 16.3 Predicate logic ............................................................................................................................................... 411 16.4 Linear temporal logic (LTL) .......................................................................................................................... 412 16.5 Computation tree logic (CTL) ....................................................................................................................... 418 16.6 Model checkers .............................................................................................................................................. 418 16.7 Modelling software ........................................................................................................................................ 419 16.8 Summary of software verification ................................................................................................................. 419 Problem set ............................................................................................................................................................ 420 17 File management ................................................................................................................................................. 425 17.1 Block addressable .......................................................................................................................................... 425 17.2 Files ............................................................................................................................................................... 426 17.3 Organization .................................................................................................................................................. 427 17.4 File systems ................................................................................................................................................... 427 17.5 Data formats .................................................................................................................................................. 439 17.6 The file abstraction ........................................................................................................................................ 443 17.7 Keil RTX RTOS ............................................................................................................................................ 444 17.8 Summary ........................................................................................................................................................ 444 Problem set ............................................................................................................................................................ 445 18 Data management ............................................................................................................................................... 446 18.1 Linear data structures ..................................................................................................................................... 446 18.2 Hash tables ..................................................................................................................................................... 448 18.3 Graphs............................................................................................................................................................ 451 18.4 Non-relational databases ................................................................................................................................ 451 18.5 Relational databases....................................................................................................................................... 454 18.6 Summary of data management ...................................................................................................................... 454 19 Virtual memory and caching ............................................................................................................................. 455 19.1 Caches and virtual memory ........................................................................................................................... 455 19.2 Multiple levels of cache ................................................................................................................................. 456 19.3 Using solid-state drives as caches ..............................................................