Towards Automotive Embedded Systems with Self-X Properties 427 Nowadays there are three major vehicle network systems (cp. Figure 6): The most common network technology used in vehicles is the Controller Area Network (CAN) bus (Robert Bosch GmbH, 1991). CAN is a multi-master broadcast bus for connecting ECUs without central control, providing real-time capable data transmission. FlexRay (FlexRay Consortium, 2005) is a fast, deterministic and fault-tolerant automotive network technology. It is designed to be faster and more reliable than CAN. Therefore, it is used in the field of safety-critical applications (e.g. active and passive safety systems). The Media Oriented Systems Transport (MOST) (MOST Cooperation, 2008) bus is used for interconnecting multimedia and infotainment components proving high data rates and synchronous channels for the transmission of audio and video data. Fig. 6. In-vehicle network topology of a BMW 7-series (Source: BMW AG, 2005) The vehicle features reach from infotainment functionalities without real-time requirements over features with soft real-time requirements in the comfort domain up to safety-critical features with hard real-time requirements in the chassis or power train domain. Therefore, various requirements and very diverse system objectives have to be satisfied during runtime. By using a multi-layered control architecture it is possible to manage the complexity and heterogeneity of modern vehicle electronics and to enable adaptivity and self-x properties. To achieve a high degree of dependability and a quick reaction to changes, we use different criteria for partitioning the automotive embedded system into clusters (see Figure 7): New Trends and Developments in Automotive System Engineering 428 FunctionFunctionFunctionFunctionFunction Vehicle Cluster Safety Cluster SIL 1 Safety Cluster SIL 3 Safety Cluster SIL 4 Safety Cluster SIL 2 Network Cluster PT-CAN Network Cluster FlexRay Network Cluster MOST Feature Cluster (Engine Control) Feature Cluster (ESP) Feature Cluster (Keyless Entry) Feature Cluster (AuxIn) Service Cluster Service Cluster Layer 1 Top Layer Layer 2 Layer 3 Layer 4 Network Cluster K-CAN Safety Cluster SIL 0 Feature Cluster (Parking Assistant) Layer 0 Function Service Cluster FunctionFunctionFunction Service Cluster Fig. 7. Example of a hierarchical multi-layered architecture for today’s automotive embedded systems In a first step, the whole system (Vehicle Cluster on the top layer) is divided into the five Safety Integrity Levels (SIL 0-4) (International Electrotechnical Commission (IEC), 1998), because features with the same requirements on functional safety can be managed using the same algorithms and reconfiguration mechanisms. Nowadays, this classification is more appropriate than the traditional division into different automotive software domains because most new driver-assistance features do not fit into this domain-separated classification anymore. In a second partitioning, the system is divided into the physical location of the vehicle’s features according to the network bus the feature is designed for. This layer is added, so that all features with the same or similar communication requirements (e.g. required bandwidth) and real-time requirements can be controlled in the same way. On the next layer, each Network Cluster is divided into the different features which are communicating using this vehicle network bus. Hence, each feature is controlled by its own control loop, managing its individual requirements and system objectives. Most features within the automotive domain are composed of several software components as well as sensors and actuators. One example is the Adaptive Cruise Control (ACC) feature which can automatically adjust the car’s speed to maintain a safe distance to the vehicle in front. This is achieved through a radar headway sensor to detect the position and the speed of the leading vehicle, a digital signal processor and a longitudinal controller for calculating New Trends and Developments in Automotive System Engineering 430 https: //www.autosar.org. Cai, L. & Gajski, D. (2003). Transaction level modeling: an overview, Proceedings of the 1 st IEEE/ACM/IFIP international conference on Hardware/software Codesign and system synthesis (CODES+ISSS ’03), pp. 19–24. CAR 2 CAR Communication Consortium (2010). http://www.car-to-car.org. Cattrysse, D. & Van Wassenhove, L. (1990). A survey of algorithms for the generalized assignment problem, Erasmus University, Econometric Institute. Cimatti, A., Pecheur, C. & Cavada, R. (2003). Formal verification of diagnosability via symbolic model checking, In Proceedings of the 18th International Joint Conference on Artificial Intelligence IJCAI03, pp. 363–369. Cuenot, P., Frey, P., Johansson, R., Lönn, H., Reiser, M., Servat, D., Koligari, R. & Chen, D. (2008). Developing Automotive Products Using the EASTADL2, an AUTOSAR Compliant Architecture Description Language, Embedded Real-Time Software Conference, Toulouse, France. Czarnecki, K. & Eisenecker, U. (2000). Generative programming: methods, tools, and applications, Addison-Wesley. Dinkel, M. (2008). A Novel IT-Architecture for Self-Management in Distributed Embedded Systems, PhD thesis, TU Munich. Dinkel, M. & Baumgarten, U. (2007). Self-configuration of vehicle systems - algorithms and simulation, WIT ’07: Proceedings of the 4th International Workshop on Intelligent Transportation, pp. 85–91. EAST-ADL2 (2010). Profile Specification 2.1 RC3, http://www.atesst.org/home/liblocal/docs/ATESST2_D4.1.1_EAST-ADL2- Specification_ 2010-06-02.pdf. FlexRay Consortium (2005). The FlexRay Communications System Specifications Version 2.1. http://www.flexray.com/. Fürst, S. (2010). Challenges in the design of automotive software, Proceedings of Design, Automation, and Test in Europe (DATE 2010). Geihs, K. (2008). Selbst-adaptive Software, Informatik Spektrum 31(2): 133–145. Hardung, B., Kölzow, T. & Krüger, A. (2004). Reuse of software in distributed embedded automotive systems, Proceedings of the 4th ACM international conference on Embedded software pp. 203 – 210. Hofmann, P. & Leboch, S. (2005). Evolutionäre Elektronikarchitektur für Kraftfahrzeuge (Evolutionary Electronic Systems for Automobiles), it-Information Technology 47(4/2005): 212–219. Hofmeister, C. (1993). Dynamic reconfiguration of distributed applications, PhD thesis, University of Maryland, Computer Science Department. Horn, P. (2001). Autonomic computing: IBM’s perspective on the state of information technology, IBM Corporation 15. IEEE (2005). IEEE Standard 1666-2005 - System C Language Reference Manual. International Electrotechnical Commission (IEC) (1998). IEC 61508: Functional safety of Electrical/ Electronic/Programmable Electronic (E/E/PE) safety related systems. Towards Automotive Embedded Systems with Self-X Properties 431 Kephart, J. O. & Chess, D. M. (2003). The vision of autonomic computing, Computer 36(1): 41– 50. McKinley, P. K., Sadjadi, S. M., Kasten, E. P. & Cheng, B. H. (2004). Composing adaptive software, IEEE Computer 37(7): 56–64. Mogul, J. (2005). Emergent (Mis)behavior vs. Complex Software Systems, Technical report, HP Laboratories Palo Alto. MOST Cooperation (2008). MOST Specification Rev. 3.0. http://www.mostcooperation. com/. Mühl, G., Werner, M., Jaeger, M., Herrmann, K. & Parzyjegla, H. (2007). On the definitions of self-managing and self-organizing systems, KiVS 2007 Workshop: Selbstorganisierende, Adaptive, Kontextsensitive verteilte Systeme (SAKS 2007). Müller-Schloer, C. (2004). Organic computing: on the feasibility of controlled emergence, CODES+ISSS ’04: Proceedings of the 2nd IEEE/ACM/IFIP international conference on Hardware/software codesign and system synthesis, ACM, pp. 2–5. Open SystemC Initiative (OSCI) (2010). SystemC, http://www.systemc.org. OSEK VDX Portal (n.d.). http://www.osek-vdx.org. Pretschner, A., Broy, M., Kruger, I. & Stauner, T. (2007). Software engineering for automotive systems: A roadmap, Future of Software Engineering (FOSE ’07) pp. 55–71. Robert Bosch GmbH (1991). CAN Specification Version 2.0. http://www. semiconductors.bosch.de/pdf/can2spec.pdf. Robertson, P., Laddaga, R. & Shrobe, H. (2001). Self-adaptive software, Proceedings of the 1st international workshop on self-adaptive software, Springer, pp. 1–10. Schmeck, H. (2005). Organic computing - a new vision for distributed embedded systems, ISORC ’05: Proceedings of the Eighth IEEE International Symposium on Object-Oriented Real-Time Distributed Computing, IEEE Computer Society, pp. 201–203. Serugendo, G., Foukia, N., Hassas, S., Karageorgos, A., Mostéfaoui, S., Rana, O., Ulieru, M., Valckenaers, P. & Aart, C. (2004). Self-organisation: Paradigms and Applications, Engineering Self-Organising Systems pp. 1–19. Teich, J., Haubelt, C., Koch, D. & Streichert, T. (2006). Concepts for self-adaptive automotive control architectures, Friday Workshop Future Trends in Automotive Electronicsand Tool Integration (DATE’06). Trumler, W., Helbig, M., Pietzowski, A., Satzger, B. & Ungerer, T. (2007). Self-configuration and self-healing in autosar, 14th Asia Pacific Automotive Engineering Conference (APAC- 14). Urmson, C. & Whittaker, W. R. (2008). Self-driving cars and the urban challenge, IEEE Intelligent Systems 23: 66–68. Weiss, G., Zeller, M., Eilers, D. & Knorr, R. (2009). Towards self-organization in automotive embedded systems, ATC ’09: Proceedings of the 6th International Conference on Autonomic and Trusted Computing, Springer-Verlag, Berlin, Heidelberg, pp. 32–46. Williams, B. C., Nayak, P. P. & Nayak, U. (1996). A model-based approach to reactive self- configuring systems, In Proceedings of AAAI-96, pp. 971–978. Wolf, T. D. & Holvoet, T. (2004). Emergence and self-organisation: a statement of similarities and differences, Lecture Notes in Artificial Intelligence, Springer, pp. 96–110. New Trends and Developments in Automotive System Engineering 432 Zadeh, L. (1963). On the definition of adaptivity, Proceedings of the IEEE 51(3): 469–470. Zeller, M., Weiss, G., Eilers, D. & Knorr, R. (2009). A multi-layered control architecture for self-management in adaptive automotive systems, ICAIS ’09: Proceedings of the 2009 International Conference on Adaptive and Intelligent Systems, IEEE Computer Society, Washington, DC, USA, pp. 63–68. 0 4D Ground Plane Estimation Algorithm for Advanced Driver Assistance Systems Faisal Mufti 1 , Robert Mahony 1 and Jochen Heinzmann 2 1 Australian National University 2 Seeing Machines Ltd. Australia 1. Introduction Over the last two decades there has been a significant improvement in automotive design, technology and comfort standards along with safety regulations and requirements. At the same time, growth in population and a steady increase in the number of road users has resulted in a rise in the number of accidents involving both automotive users as well as pedestrians. According to World Health Organization, road traffic accidents, including auto accidents and personal injury collisions account for the deaths of an estimated 1.2 million people worldwide each year, with 50 million or more suffering injuries (Organization, 2009). These figures are expected to grow by 20% within the next 20 years (Peden et al., 2004). In the European Union alone the imperative need for Advanced Driver Assistance Systems (ADAS) sensors can be gauged from the fact that every day the total number of people killed on Europe’s roads are almost the same as the number of people killed in a single medium-haul plane crash (Commission, 2001) with 3 rd party road users (pedestrian, cyclist, etc) comprising the bulk of these fatalities (see Figure 1 for proportion of road injuries) (Sethi, 2008). This transforms into a direct and indirect cost on society, including physical and psychological damage to families and victims, with an economic cost of 160 billion euros annually (Commission, 2008). These statistics provide a strong motivation to improve the ADAS ability of automobiles for the safety of both passengers and pedestrians. The techniques to develop vision based ADAS depend heavily on the imaging device technology that provides continuous updates of the surroundings of the vehicle and aid ϯϮй ϰϳй ϭϲй ϱй WĞĚĞƐƚƌŝĂŶƐ ĂƌƐ DŽƚŽƌĐLJĐůĞƐͬLJĐůĞƐ KƚŚĞƌƐ Fig. 1. Proportion of road traffic injury deaths in Europe (2002-2004). 22 2 Trends and Developments in Automotive Engineering drivers in safe driving. In general these sensors are either spatial devices like monocular CCD cameras, stereo cameras or other sensor devices such as infrared, laser and time-of-flight sensors. The fusion of multiple sensor modalities has also been actively pursued in the automotive domain (Gern et al., 2000). A recent autonomous vehicle navigation competition DARPA (US Defense Advanced Research Projects Agency) URBAN Challenge (Baker & Dolan, 2008) has demonstrated a significant surge in efforts by major automotive companies and research centres in their ability to produce ADAS that are capable of driving autonomously in an urban terrain. Range image devices based on the principle of time-of-flight (TOF) (Xu et al., 1998) are robust against shadow, brightness and poor visibility making them ideal for use in automotive applications. Unlike laser scanners (such as LIDAR or LADAR) that traditionally require multiple scans, 3D TOF cameras are suitable for video data gathering and processing systems especially in automotive that often require 3D data at video frame rate. 3D TOF cameras are becoming popular for automotive applications such as parking assistance (Scheunert et al., 2007), collision avoidance (Vacek et al., 2007), obstacle detection (Bostelman et al., 2005) as well as the key task of ground plane estimation for on-road obstacle and obstruction avoidance algorithms (Meier & Ade, 1998; Fardi et al., 2006). The task of obstacle avoidance has normally been approached as by either (a) directly detecting obstacles (or vehicles) and pedestrian or (b) estimating ground plane and locating obstacles from the road geometry. Ground plane estimation has been tackled using methods such as least squares (Meier & Ade, 1998), partial weighted eigen methods (Wang et al., 2001), Hough Transforms (Kim & Medioni, 2007), and Expectation Maximization (Liu et al., 2001), amongst others. Computationally expensive semantic or scene constraint approaches (Cantzler et al., 2002; N ¨uchter et al., 2003) have also been used for segmenting planar features. However, these methods work well for dense 3D point clouds and are appropriate for laser range data. A statistical framework of RANdom SAmple Concensus (RANSAC) for segmentation and robust model fitting using range data is also discussed in literature (Bolles & Fischler, 1981). Existing work in applying RANSAC to 3D data for plane fitting uses single frame of data (Bartoli, 2001; Hongsheng & Negahdaripour, 2004) or tracking of data points (Yang et al., 2006), and does not exploit the temporal aspect of 3D video data. In this work, we have formulated a spatio-temporal RANSAC algorithm for ground plane estimation using 3D video data. The TOF camera/sensor provides 3D spatial data at video frame rate and is recorded as a video stream. We model a planar 3D feature comprising two spatial directions and one temporal direction in 4D. We consider a linear motion model for the camera. In order that the resulting feature is planar in the full spatio-temporal representation, we require that the camera rotation lies in the normal to the ground plane, an assumption that is naturally satisfied for the automotive application considered. A minimal set of data consisting of four points is chosen randomly amongst the spatio-temporal data points. From these points, three independent vector directions, lying in the spatio-temporal planar feature are computed. A model for the 3D planar feature is obtained by computing the 4D cross product of the vector directions. The resulting model is scored in the standard manner of RANSAC algorithm and the best model is used to identify inlier and outlier points. The final planar model is obtained as a Maximum likelihood (ML) estimation derived from inlier data where the noise is assumed to be Gaussian. By utilizing data from a sequence of temporally separated image frames, the algorithm robustly identifies the ground plane even when the ground plane is mostly obscured by passing pedestrians or cars and in the presence of walls (hazardous planar surfaces) and other obstructions. The fast segmentation of the obstacles 434 New Trends and Developments in Automotive System Engineering 4D Ground Plane Estimation Algorithm for Advanced Driver Assistance Systems 3 CMOS correlation in sensor matrix 3D data dispaly upto 25 frames/sec IR source Modulated signal 3D scene Reflected signal Signal Processing Within same housing unit Signal Generator/ Modulator 0 o 90 o 180 o 270 o t d = 2 c r r Fig. 2. Basic principle of TOF 3D imaging system. is achieved using the statistical distribution of the feature and then employing a statistical threshold. The proposed algorithm is simple as no spatio-temporal tracking of data points is required. It is computationally inexpensive without the need of image/feature selection, calibration or scene constraint and is easy to implement in fewest possible steps. This chapter is organized as follows: Section 2 describes the time-of-flight camera/sensor technology, Section 3 presents the structure and motion model constraints for planar feature, Section 4 describes formulation of spatio-temporal RANSAC algorithm, Section 5 describes application of the framework and Section 6 presents experimental results and discussion, followed by conclusion in Section 7. 2. Time-of-flight camera Time-of-Flight (TOF) sensors estimate distance to a target using the time of flight of a modulated infrared (IR) wave between the sender and the receiver (see Fig. 2). The sensor illuminates the scene with a modulated infrared waveform that is reflected back by the objects and a CMOS (Complementary metal-oxide- semiconductor) based lock in CCD (charge-coupled device) sensor samples four times per period. With the precise knowledge of speed of light c,eachofthese(64 ×48) smart pixels, known as Photonic Mixer Devices (PMD) (Xu et al., 1998), measure four samples a 0 , a 1 , a 2 , a 3 at quarter wavelength intervals. The phase ϕ of the reflected wave is computed by (Spirig et al., 1995) ϕ = arctan a 0 − a 2 a 1 − a 3 . The amplitude A (of reflected IR light) and the intensity B representing the gray scale image returned by the sensor are respectively given by A =  (a 0 − a 2 ) 2 +(a 1 − a 3 ) 2 2 , B = a 0 + a 1 + a 2 + a 3 4 . With measured phase ϕ, known modulation frequency f mod and precise knowledge of speed of light c it is possible to measure the un-ambiguous distance r from the camera, r = c.ϕ 4π f mod .(1) 435 4D Ground Plane Estimation Algorithm for Advanced Driver Assistance Systems 4 Trends and Developments in Automotive Engineering Y i X i Z i y x r 3D Point PMD (x,y,r) f Y w Z w X w Image projection Fig. 3. Time-of-Flight sensor geometry With a modulation wavelength of λ mod , this leads to a maximum possible unambiguous range of (λ mod /2). For a typical camera such as PMD 3k-S (PMD, 2002), f mod =20Mhz and with a speed of light c given by 3 × 10 8 m/s, the non-ambiguous range r max of the TOF camera is given as r max = c 2 f mod = 3 ×10 8 2 ·20 ×10 6 = 7.5meters. The sensor returns a range r value for each pixel as a function of pixel coordinates (x, y) as shown in Fig. 3. The range values are used to compute 3D position X X X (X,Y, Z) of the point Z = r(x, y). f  f 2 + x 2 + y 2 ; X = Z x f ; Y = Z y f ,(2) where f is the focal length of the camera. 3. Structure and motion constraints In the following section we will discuss the motion model and the planar feature parameters essential to derive the spatio-temporal RANSAC formulation for a planar feature. 3.1 Motion model Consider a TOF camera moving in space. Let {i} denote the frame of reference at time stamp i,1 ≤i ≤ n, attached to the camera. Let {W}denote the fixed world reference frame. The rigid body transformation W i M : R 3 → R 3 ; X X X i → X X X W := W i RX X X i + W T i (3) is defined as the coordinate mapping from frame {i}to world frame {W}with rotation ( W i R) and translation ( W T i ) respectively. Let ¯ X X X ∈R 4 denote the homogenous coordinates of X X X ∈R 3 , then the transformation (3) in matrix form is given by W i ¯ M : R 4 → R 4 ;(4) ¯ X X X W =  X X X W 1  =  W i R W T i 01  X X X i 1  = W i ¯ M ¯ X X X i .(5) Let i j ¯ M be the rigid body mapping from frame {j} to frame {i} then, i j ¯ M = i W ¯ M W j ¯ M =( W i ¯ M) −1W j ¯ M. Hence i j ¯ M =  ( W i R  )( W j R)( W i R  )( W T j − W T i ) 01  .(6) 436 New Trends and Developments in Automotive System Engineering [...]... 10 frames (g-h) Cars, wall and a person as obstacles at turning (i) Corresponding spatio-temporal ground plane fit (j-k) Pedestrians (l) Ground plane fit 12 444 Trends Developments in Automotive System Engineering New Trends andand Developments in Automotive Engineering (a) (b) Fig 7 Using data from sequence 5, (a) Standard RANSAC plane fitting algorithm picks the wall with a single frame data (b) Spatio-temporal... Keppeler, N (2007) Free space determination for parking slots using a 3D PMD sensor, Proc IEEE Intelligent Vehicles Symposium, pp 154–159 16 448 Trends Developments in Automotive System Engineering New Trends andand Developments in Automotive Engineering Sethi, D (2008) Road traffic injuries among vulnerable road users Shaw, R (1987) Vector cross products in n dimensions, Int J Math Educ Sci Technol 18(6):... constraint (GPC) Sullivan (1994) (see Figure 4) In real environments for motion captured at nearly video frame rate, the piecewise linear velocity along the normal direction can be assumed constant as evident from the experiments 6 438 Trends Developments in Automotive System Engineering New Trends andand Developments in Automotive Engineering yaw roll pitch Fig 4 Vehicle with roll, pitch and dominant... planar surface estimation that is otherwise susceptible to noisy data in any algorithm developing a single frame data Further improvement in computation cost can be achieved through dedicated hardware implementation 14 446 Trends Developments in Automotive System Engineering New Trends andand Developments in Automotive Engineering 1 Time in seconds 0.8 Seq 1 Seq 2 Seq 3 Seq 4 Seq 5 0.6 0.4 0.2 0 5 6 7... normal distribution in observed depth ¯ η,h 10 442 Trends Developments in Automotive System Engineering New Trends andand Developments in Automotive Engineering An obstacle detection algorithm can be applied once a robust estimation of planar ground surface is available In the proposed framework, the algorithm evaluates each spatio-temporal data point and categorizes traversable and non-traversable... information, navigation systems offer intelligent route guidance, road status and helpful information on and along the way In the recent years the development of navigation systems is advancing So the pro- and contras for portable navigators in contrast to integrated systems are discussed 452 New Trends and Developments in Automotive System Engineering 2 History of vehicular entertainment From year 1920... depth parameter h can be determined by h 1 = X i , η1 − α ( i − 1 ) (25) However, the parameter h is not required for the robust estimation phase of the RANSAC algorithm and is evaluated in the second phase where a refined model is estimated 8 440 Trends Developments in Automotive System Engineering New Trends andand Developments in Automotive Engineering 1600 1400 Frequency 120 0 1000 800 600 400 200 0... subsections, explaining the tool chain of vehicular entertainment systems from reception of radio waves, demodulation and distribution of signal to the audio and visual end We begin with a brief historical overview of car entertainment and show modern installations in contrast In order to understand the evolution better, we give a list of existing broadcasting standards and explain tuner concepts and diversity... organized in an individual bus The interconnection of information from one bus-ring to another can be organized by gateways These gateways filter the required information from one bus, re-code messages - if required - and transmit them to the other bus ring 454 New Trends and Developments in Automotive System Engineering A typical bus architectures can be classified in 3 main groups and a variety of... stream and correct transmission errors The sound quality is intuitively better than analog broadcasting but coverage is found to be reduced That is mainly due to the fact that digital wireless systems have an abrupt go-nogo border, while analog systems degrade gradually until signals disappear in noise [Koch, 2008] 460 New Trends and Developments in Automotive System Engineering 5.3 VHF-band and UHF-band . use different criteria for partitioning the automotive embedded system into clusters (see Figure 7): New Trends and Developments in Automotive System Engineering 428 FunctionFunctionFunctionFunctionFunction Vehicle. its application to finding cylinders in range data, Proc. Seventh Int. Joint Conf. Artificial Intelligence, pp. 637–643. 446 New Trends and Developments in Automotive System Engineering . comes from incorporating the temporal dimension along with the data available by incorporating multiple images from the 440 New Trends and Developments in Automotive System Engineering 4D Ground