Designing An Arm-Based Multithreaded Audio/Visual/Motion Recording System: Part 1

* application timers
* threads
* message queues
* mutexes
* memory byte pools

Designed for use in motorized vehicle fleets this case study depends heavily on the use of application timers. One reason for using so many timers is because we need to schedule the copying of data from the temporary memory to the protected memory whenever any one of four events occurs.

Our design provides the ability to record several events within each 24-second time frame, rather than just one. Application timers play a major role in providing this feature. We also used application timers to simulate interrupts that signify the occurrence of events, and we used one timer to display periodic system statistics.

How the VAM works
The VAM system, based on the ThreadX RTOS and ARM processor, features a small recording device that could be attached to a vehicle's windshield directly behind the rear-view mirror to avoid intrusion into the driver's field of vision.

When triggered by an accident or unsafe driving, the VAM system automatically records everything the driver sees and hears in the 12 seconds preceding and the 12 seconds following the event. Events are stored in the unit's digital memory, along with the level of G-forces on the vehicle. In the event of an accident, unsafe driving, warning, or other incident, the VAM system provides an objective, unbiased account of what actually happened.

To complete the system, there should be a driving feedback system that downloads the data from the VAM system unit and provides playback and analysis. This system could also be used to create a database of incidents for all the drivers in the vehicle fleet. We will not consider that system; instead, we will focus on the capture of real-time data for the VAM system unit.

As noted earlier, most of the VAM unit could be located behind the rear view mirror, so it would not obscure the vision of the driver. The unit would have to be installed so that the lenses have a clear forward view and rear view. The system includes a readily accessible emergency button so the driver can record an unusual, serious, or hazardous incident whenever necessary. The VAM system is constantly recording everything the driver sees, hears, and feels. That is, the VAM system records all visual activities (front and rear of the vehicle), audible sounds, and G-forces.

As illustration of how the VAM system could be used, consider the following scenario: A driver has been somewhat inattentive and has failed to stop at a red traffic light. By the time the driver realizes the error, the vehicle has already entered the intersection and is headed toward an oncoming vehicle.

The driver vigorously applies the brakes and swerves to the right. Typical G-forces for this incident are about "0.7 (forward) and +0.7 (side). Thus, the VAM system detects this incident and records it as an unsafe driving event in the protected memory.

When we download and analyze the data from the VAM system, we should be able to clearly see that the driver ran a red light and endangered passengers and other people on the highway, as well as the vehicle itself. We would have been legally liable for the driver's actions if this incident had resulted in a collision.

In this scenario, no collision resulted from this incident. However, this recording would show that this driver was clearly at fault and perhaps needs some refresher training. Figure 1 below illustrates the G-forces that can be detected, where the front of the vehicle appears at the top of the illustration.

Figure 1: Directions of G-forces.

The system stores the 24 seconds of video, audio, and motion recording that surround the time of this incident in protected memory and illuminates a red light that indicates a driving incident has occurred. This light can be turned off only when the special downloading process has been performed; the driver cannot turn it off. For simplicity, we will omit certain details that are not important to the development of this system, such as file-handling details.

Statement of Problem
The VAM system is based on a set of sensors that measure G-forces experienced by a driver in a motorized vehicle. The system uses two sets of measurements. One set indicates forward or backward motion of the vehicle. Negative forward values indicate deceleration, or G-forces pushing against the driver's front side, while positive forward values indicate acceleration, or G-forces pushing against the driver's back.

The other set of measurements indicates sideways motion of the vehicle. Negative side values indicate acceleration to the right, or G-forces pushing against the driver's left side, while positive side values indicate acceleration to the left, or G-forces pushing against the driver's right side. For example, if a vehicle makes a hard left turn, then the sensors produce a positive side value.

Figure 2: Events and corresponding priorities

The VAM system detects and reports four categories of events. We assign each category a priority, indicating the importance of the event. Figure 2 above lists the event categories and their corresponding priorities. (This event priority is not the same as a ThreadX thread or interrupt priority. We use event priorities to classify the relative importance of the events; we do not use them to affect the time when the events are processed.)

Event priorities serve two primary purposes. First, a priority indicates the severity of an event. Second, an event priority determines whether the current event can overwrite a previously stored event in the protected memory.

For example, assume that the protected memory is full and the driver hits the emergency button, thereby creating a manually triggered event. The only way that this event can be saved is if a previous manually triggered event has already been stored. Thus, a new event can overwrite a stored event of the same or lower priority, but it cannot overwrite a stored event with a higher priority.

If the G-force sensors detect an accident, unsafe driving, or warning, the VAM system generates an interrupt so that ThreadX can take appropriate action and archive that event. Figure 3 below contains a graphical representation of the G-forces in this system, in which the event labeled by the letter "W" is a warning event.

Figure 3: Graphical classification of events by G-forces.

The driver may hit the emergency button at any time to generate an interrupt, signifying a manually triggered event. Figure 4 below  contains the actual G-force values that are used to detect and report these events. We assume symmetry in how we classify forward and side G-forces, but we could easily modify that assumption without affecting our design.

Figure 4: G-Forces and event classifications.

To add some perspective about G-forces, consider a vehicle accelerating from zero to 60 miles per hour (0 to 96 kilometers/hour) in six seconds. This produces a G-force of about 0.4—not enough to trigger an unsafe incident report, but enough to trigger a warning event.

However, if a driver is applying "hard braking" to a vehicle, it could produce a Gforce of about 0.8, which would trigger an unsafe driving event. If a vehicle crashes into a solid wall while traveling at 62 mph (100 km/hr), this produces a G-force of almost 100!

The VAM system uses two non-volatile memory systems: a temporary memory system and a protected memory system. The protected memory system stores only detected or manually triggered incidents, while the other system is a temporary memory that records video, audio, and G-forces. It's not necessary to retain ordinary driving activities, so the temporary memory system is overwritten after some period of time, depending on the size of the temporary memory.

As noted previously, the protected memory system stores all crash events, unsafe driving events, warnings, and manually triggered events, plus associated audio and video, as long as memory space is available. The protected memory system could be available in several different sizes, and our design will be able to accommodate those different memory sizes.

Figure 5 below  illustrates the temporary memory system used for continuous recording. This is actually a circular list where the first position logically follows the last position in the list.

Figure 5: Temporary memory (Circular list).

This system provides temporary storage that is overwritten repeatedly. Its main purpose is to provide data storage for an event that needs to be saved in the protected memory.When an event occurs, the 12 seconds preceding the event have already been stored in the temporary memory. After the 12 seconds of data following the event have been stored, the system stores this 24 seconds of data in protected memory.

The actual size of the temporary memory can be configured to the needs of the user. Figure 6 below illustrates the protected memory that is used to store the automatically detected or manually triggered events.

Figure 6: Protected memory.

The size of the protected memory can also be configured according to the needs of the user. We arbitrarily assume that this memory can store 16 events, although we can change this value without affecting our design. In addition to the actual data for the event, the protected memory must store the priority and time of the event's occurrence.

This information is essential in the case where the protected memory becomes full and another event is detected or manually triggered. An event can never overwrite a higher priority event, but it can overwrite an event with the same or lower priority. Figure 7 below summarizes the event overwrite rules.

Figure 7: Event overwrite rules.

As stated previously, when the G-force sensors detect a crash, the system generates an interrupt with event priority 1. Unsafe driving events are logged as event priority 2, warnings as event priority 3, and manual events (pushing the emergency button) as event priority 4. Our objective is to respond to these events (which will appear to the system as interrupts), as well as to handle initialization and to process routine data.

Our design will be simplified and will concentrate on control issues. Next we will consider thread design issues, and public resources design. We will assume that other processes will handle the actual storing and copying of data. (These data handling tasks could be performed by the software package FileX, which is a companion product to ThreadX.)

Thread Design
We need a thread to perform various initialization duties such as setting pointers and variables, opening the files for the temporary memory and the protected memory, and establishing communication paths and buffers. We will assign the name initializer to this thread. 

We need a thread to coordinate the capture and storage of data from the VAM system unit to the temporary memory. This thread manages the transfer of audio, video, and G-force data from the VAM system unit to an internal buffer, and then to the temporary memory. We will assign the name data_capture to this thread. Figure 8 below, illustrates this thread's operation.

Figure 8: Capturing data from the VAM unit.

We need four ISRs to handle the detected and triggered events. We need one message queue called event_notice to store information about an event until the copying process begins. We also need one thread to process the copying of data from the temporary memory to the protected memory. We will assign the name event_recorder to this thread.

This thread is activated 12 seconds after an event has been detected. At this time, the temporary memory contains the 12 seconds of data preceding the event and 12 seconds of data following the event.

The event_recorder thread then takes identifying information from the event_notice queue and then copies the audio, video, G-force, and timing data for this event from the temporary memory to the protected memory, including the event priority. If protected memory is full, the event recorder thread employs the overwrite rules shown in Figure 7, earlier.

We will use four timers to simulate the arrival of external interrupts for the detectable and triggered events. To simulate such an interrupt, the timer in question saves the thread context, invokes the corresponding ISR, and then restores the thread context.

We will assign the names crash_interrupt, unsafe_interrupt, warning_interrupt, and manual_interrupt to these four timers. Figure 9 below presents an overview of the interaction between the timers that simulate interrupts, their associated ISRs, the scheduling timers, and the event recorder thread that actually performs the copying.

Figure 9: Overview of event interrupts and data recording.

Public Resources Design
We will use one mutex to provide protection for the protected memory while copying data from the temporary memory. To begin copying data, the event_recorder thread must obtain ownership of this mutex. We will assign the name memory_mutex to this mutex.

We need four application timers to schedule the copying of data from temporary memory to protected memory. When an event is detected, the corresponding application timer is activated and expires precisely 12 seconds later. At this point in time, the event_recorder thread begins copying data. We will assign the names crash_timer, unsafe_timer, warning_timer, and manual_timer to these four application timers. Figure 10, below  illustrates how these application timers generate interrupts and schedule the processing of events.

Figure 10: Event processing.

As illustrated in Figure 10 above, an event interrupt is simulated by one of the four timers created for this purpose. One of four corresponding ISRs is invoked and it sends event information to the message queue, activates one of four corresponding scheduling timers, and sets it to expire in 12 seconds. When that timer expires, it resumes the event_recorder thread, which should be in a suspended state unless it is in the process of copying information from temporary memory to protected memory.

When the event_recorder thread resumes execution, it takes one message from the queue. This message contains two pieces of information about an event: the frame index and the event priority. The frame index is a location in temporary memory that specifies where information about the event begins.

The event_recorder thread then obtains the memory_mutex and if successful, proceeds to copy information from temporary memory to protected memory. When the thread completes copying, it releases the mutex and suspends itself. Figure 11 below contains a summary of the public resources needed for this case study.

Figure 11: Summary of public resources used for the VAM system.

Figure 12 below contains a summary of the definitions, arrays, and variables used. We will simulate the temporary memory file system and the protected memory file system with arrays.

Figure 12: Definitions, arrays, and counters used in the VAM system.

The event counters in Figure 12, above are used when printing periodic statistics about the state of the system. We also need a collection of functions to perform the actions of the timers and the threads. Figure 13 below summarizes all thread entry functions and timer expiration functions used in this system.

Figure 13: Entry and expiration functions used in the VAM system.

Next in Part 2 on Implementing the VAM system, we will develop the complete program for our system as well as for the individual components of our system.

Dr. Edward L. Lamie is professor emeritus and former department chair of California State University's Computer Science Department at the Stanislaus Campus. He directs educational services at Express Logic, where he is responsible for development and delivery of customer training.

Editor's note: A complete listing of the system described here is included in Lamie's book as well as on the CD in the back of the book.