Getting started with Real-Time Linux Michael Barabanov (baraban@fsmlabs.com) Copyright VJY Associates L.L.C. 1999. All rights reserved. Introduction Real-Time Linux is a hard real-time operating system that can coexist with the Linux OS on the same computer. With Real-Time Linux, it is possible to create real-time POSIX.1b threads that will run at precisely specified moments of time. The white paper in rtl/doc/design.pdf explains the basic architecture in more detail. This document provides a quick starting guide for new users of Real-Time Linux (RTL). Other documents useful in this context are: - The Real-Time Linux whitepaper (URL) - The Single UNIX specification (http://www.opengroup.org/onlinepubs/7908799/index.html) - The LinuxThreads library documentation (included with glibc2). You can try running man 3 pthread_create to see if it is installed on your system - Getting Started With POSIX Threads (by Thomas Wagner and Don Towsley) http://centaurus.cs.umass.edu/~wagner/threads_html/tutorial.html - Other documents or books describing POSIX threads. The Real-Time Linux distribution contains UNIX manual pages for functions specific to Real-Time Linux in rtl/man directory. You can modify the MANPATH environment variable accordingly. The html/man directory contains manual pages converted to HTML. The best way to get started with Real-Time Linux is to browse through the included example programs. In particular, examples/measurements and examples/fp are recommended. Be sure to also read the Frequently Asked Questions file included with the distribution. General Notes Since Real-Time programs in RTL are executed in the kernel space, special care must be taken when programming real-time tasks. Programming errors may easily bring the system down. The future versions of RTL will provide ways to debug RT-threads in user space, and then gradually switch to the hard real-time mode. Linux Modules This version of Real-Time Linux supports loading RT-programs in the form of Linux modules. A Linux module source file is an ordinary C language file, with the main() function replaced by a pair of init/cleanup functions: int init_module(); This function is called when the module is loaded in the kernel. It should return 0, on failure, a negative value. void cleanup_module(); This function is called when the module is unloaded. The compiled module is loaded in the kernel with the insmod(1) command. See also the Linux Kernel-HOWTO. Writing Real-Time Linux POSIX threads Threads are light-weight processes sharing a common address space. Conceptually, the Linux kernel threads of control (one for each CPU in the system) are also RTL threads. In RTL, all threads share the Linux kernel address space. To create a new Real-Time thread, use pthread_create (3) function. #include int pthread_create(pthread_t * thread, pthread_attr_t * attr, void * (*start_routine)(void *), void * arg); Note. This function must only be called from the Linux kernel thread (e.g., from init_module()). The thread is created using the attributes specified in the "attr" thread attributes object. If attr is NULL, default attributes are used. See POSIX functions pthread_attr_init(3), pthread_attr_setschedparam(3), pthread_attr_getschedparam(3), and RTL-specific functions pthread_attr_getcpu_np(3), pthread_attr_setcpu_np(3) The id of the created thread is stored in the location pointed to by "thread". The function pointed to by start_routine is taken to be the thread code. It is passed the "arg" argument. To kill a thread, the (non-portable) pthread_kill_np RTL function can be used: int pthread_delete_np (pthread_t thread); The thread stops its execution immediately. Scheduling threads RTL provides a way to request the thread code to be run at particular moments of time (scheduling). Real-Time Linux provides several clocks that can be used as a reference for thread scheduling. To obtain the current clock reading, use the clock_gettime(3) function. int clock_gettime(clockid_t clock_id, struct timespec *ts); clock_id is the clock to be read. ts is the location to store the read value. struct timespec { time_t tv_sec; /* seconds */ long tv_nsec /* nanoseconds */ }; Note. The currently supported clocks are: CLOCK_UST. This stands for "Unadjusted System Time". This clock is never adjusted or reset. It provides a nice source of timestamps. CLOCK_REALTIME. This is the standard POSIX realtime clock. Currently it is the same as CLOCK_UST. It is planned that this clock will give the world time in future versions of RTL. CLOCK_8254. Used on non-SMP machines for scheduling. CLOCK_APIC. Used on SMP machines. CLOCK_APIC corresponds to the local APIC clock of the processor that executes clock_gettime. You can not read or set the APIC clock of other processors. You can get the clock that the scheduler uses for task scheduling by calling the rtl_getschedclock(3) function. Real-time Linux provides a pure priority-driven scheduler. In this scheduler, the highest priority ready thread is always chosen to run. If the two threads have the same priority, which one is chosen is undefined. The thread priority can be modified at thread creation time using pthread_attr_setschedparam(3) or afterwards using pthread_setschedparam(3). int pthread_setschedparam(pthread_t thread, int policy, const struct sched_param *param); The policy argument is currently not used in RTL, but should be specified as SCHED_FIFO for compatibility with future versions. The struct sched_param contains the sched_priority member. Higher values correspond to higher priorities. Use sched_get_priority_max(3) and sched_get_priority_min(3) to determine possible values of sched_priority. To make a real-time thread execute periodically, the following function can be used: int pthread_setperiod_np(pthread_t thread, const struct itimerspec *its); struct itimerspec { struct timespec it_interval; /* timer period */ struct timespec it_value; /* timer expiration */ }; pthread_setperiod_np marks the thread as periodic. The timing is specified by the "its" parameter. The it_value member of the passed struct itimerspec specified the time of the first invocation; the it_interval is the thread period. The period can be 0, which corresponds to a one-time task invocation. The actual execution timing is done with the pthread_wait_np(3) function int pthread_wait_np(void); This function suspends the execution of the calling thread until the time specified by pthread_setperiod_np(). The simplest "Hello World" RTL program #include #include #include pthread_t thread; void * start_routine(void *arg) { struct sched_param p; p . sched_priority = 1; pthread_setschedparam (pthread_self(), SCHED_FIFO, &p); pthread_make_periodic_np (pthread_self(), gethrtime(), 500000000); while (1) { pthread_wait_np (); rtl_printf("I'm here; my arg is %x\n", (unsigned) arg); } return 0; } int init_module(void) { return pthread_create (&thread, NULL, start_routine, 0); } void cleanup_module(void) { pthread_delete_np (thread); } Write this program to a file called hello.c, copy any rtl.mk file from the distribution to the same directory and compile the program with make -f rtl.mk hello.o Note. You should have the RTL compiled and installed for this to succeed. Note. This program is also found in examples/hello Once you have the timer (rtl_time.o) and the scheduler (rtl_sched.o) modules loaded, you can use the following command to start the thread: insmod hello.o You should see the thread printing messages. To stop the program, run rmmod hello. Using Real-Time FIFOs The general philosophy of Real-Time Linux is that the real-time part of an application should be fast, small, and simple. You should try to split your application in a way where most work is done in user space. This approach makes for easier debugging and better understanding of the real-time part of the system. Several communication mechanisms exist to allow real-time threads and user space Linux processes to communicate. The most important ones are Real-Time FIFOs and shared memory. Real-Time FIFOs are First-In-First-Out queues that can be read and written by Linux processes and Real-Time threads. FIFOs are uni-directional. You can use a pair of FIFOs for bi-directional data exchange. To use the FIFOs, system/rtl_posixio.o and fifos/rtl_fifo.o Linux modules must be loaded in the kernel. RT-FIFOs are Linux character devices with the major number of 150. The device file names are /dev/rtf0, /dev/rtf1, etc through /dev/rtf63 (the maximum number of RT-FIFOs in the system is configurable during system compilation). The device entries in /dev are created during system installation. Before a real-time FIFO can be used, it needs to be initialized. #include int rtf_create(unsigned int fifo, int size); int rtf_destroy(unsigned int fifo); rtf_create allocates the buffer of the specified size for the fifo buffer. The fifo argument corresponds to the minor number of the device. rtf_destroy deallocates the FIFO. Note. These functions must only be called from the Linux kernel thread (e.g., from init_module()). After the FIFO is created, the following calls can be used to access it from Real-Time threads: open(2), read(2), write(2), close(2). The support is planned for all of the stdio functions support. Note. The current implementation requires the FIFOs to be opened in the non-blocking mode (O_NONBLOCK) by Real-Time threads. Linux processes can use UNIX file IO functions without restrictions. See the examples/measurement/rt_process.c example program to see how RT-FIFOs can be used. Using shared memory For the shared memory, you can you the excellent mbuff driver by Tomasz Motylewski . It is included in the distribution and is located in the drivers/mbuff directory. The manual is included with the package. Here I just briefly describe the basic mode of operation. First, the mbuff.o module must be loaded in the kernel. Two functions are used to allocate blocks of shared memory, connect to them and eventually deallocate them. #include void * mbuff_alloc(const char *name, int size); void mbuff_free(const char *name, void * mbuf); The first time mbuff_alloc is called with the given name, a shared memory block of the specified size is allocated. The reference count for this block is set to 1. On success, the pointer to the newly allocated block is returned. NULL is returned on failure. If the block with the specified name already exists, this function returns the pointer that can be used to access this block and increases the reference count. mbuff_free deassociates mbuf from the specified buffer. The reference count is decreased by 1. When it reaches 0, the buffer is deallocated. These functions are available for use in both Linux processes and the Linux kernel threads. Note. mbuff_alloc and mbuff_free can not be used from Real-Time threads. You should call them from init_module and cleanup_module only. Accessing physical memory and IO ports from Real-Time threads These capabilities are essential for programming hardware devices in the computer. RTL, just like ordinary Linux, supports the /dev/mem device (man 4 mem) for accessing physical memory from RTL threads. The rtl_posixio.o module must be loaded. The program opens /dev/mem, mmaps it, and then proceeds to read and write the mapped area. The program in examples/mmap shows how to do that. The IO port access functions are described specifically for the x86 architecture #include void outb(unsigned int value, unsigned short port); /* output a byte to a port */ void outw(unsigned int value, unsigned short port); /* output a word to a port */ char inb(unsigned short port); /* read a byte from a port */ short inw(unsigned short port); /* read a word from a port */ The functions with the _p suffix (e.g. outb_p) provide a small delay after reading or writing the port. This delay is needed for some slow ISA devices on fast machines. (See also Linux I/O port programming mini-HOWTO). Look in examples/sound to see how some of these functions are used to program the PC Real-Time clock and the speaker. Using Floating Point Operations in RTL POSIX threads By default, the use of floating-point operations in RTL POSIX threads is prohibited. You can use the following RTL-specific function to change the status floating-point operations: int pthread_setfp_np (pthread_t thread, int flag); To enable FP operations in the thread, use the flag of 1. To disable FP operations, pass 0. The examples/fp directory contains several examples of tasks using floating point and the math library. Symmetric Multi-Processing Considerations From the point of view of thread scheduling, the Real-Time Linux implements a separate UNIX process for each active CPU in the system. In general, thread control functions can only be used for threads running on the local CPU. The notable exceptions are: int pthread_wakeup_np (pthread_t thread); /* wake up suspended thread */ int pthread_delete_np (pthread_t thread); /* destroy the thread */ Interfacing RTL components to Linux (RTL white paper discusses this topic in more detail). RTL threads, sharing the common address space with the Linux kernel, in principle can call the Linux kernel functions. However, in general, this is not safe to do, because RTL threads may run even while Linux has interrupts disabled. Only functions that do not modify Linux kernel data structures (e.g., vsprintf), may be called from RTL threads. RTL provides two mechanisms of delayed execution to overcome this limitation: soft interrupts and task queues. Using Interrupts. Soft interrupts and shared hard interrupts There are two types of interrupts in Real-Time Linux: hard and soft. The soft interrupts are normal Linux kernel interrupts. They have an advantage that some Linux kernel functions can safely be called from them (which ones, depends on the kernel version). However, for many tasks, they do not provide hard real-time performance: they may be delayed for considerable periods of time. Hard interrupts (or real-time interrupts), on the other hand, have much lesser latency. However, just like for real-time threads, only a very limited set of kernel functions may be called from the hard interrupt handlers. The solution is to split the functionality between hard and soft interrupt handlers. #include int rtl_request_irq(unsigned int irq, unsigned int (*handler)(unsigned int, struct pt_regs *)); int rtl_free_irq(unsigned int irq); These two functions are used for installing and uninstalling hard interrupt handlers for specific interrupts. The manual pages describe their operation in detail. Interrupt-driven RTL threads can be created using thread wakeup/suspend functions: int pthread_wakeup_np (pthread_t thread); int pthread_suspend_np (void); An interrupt-driven thread calls pthread_suspend_np and blocks. Later the interrupt handler calls pthread_wakeup_np for this thread. The thread will run until the next call to pthread_wakeup_np. Soft interrupts int rtl_get_soft_irq( void (*handler)(int, void *, struct pt_regs *), const char * devname); This function allocates a virtual irq number and installs the handler function for it. This virtual interrupt can later be triggered using rtl_global_pend_irq(3). void rtl_global_pend_irq(int ix); rtl_global_pend_irq is safe to use from real-time threads and real-time interrupts. void free_irq(unsigned int irq, void *dev_id); free_irq is the Linux kernel function that uninstalls the handler for the interrupt "irq". For the interrupts requested with rtl_get_soft_irq the dev_id should be 0. Soft interrupts are used in the Real-Time Linux FIFO implementation (fifos/rtl_fifo.c). RTL Task Queues TODO Writing RTLinux device drivers. The POSIXIO module TODO RT-Linux serial driver (rt_com) rt_com is a driver for 8250 and 16550 families of UARTs commonly used in PCs (COM1, COM2, etc). Please refer to drivers/rt_com for examples of using serial IO from real-time programs. Writing RTLinux schedulers and modifying the standard scheduler This is an advanced topic to be written later. Here are some points: the scheduler is implemented in the scheduler/rtl_sched.c and architecture-dependent files in include/arch-i386 and scheduler/i386. The scheduling decision is taken in the rtl_schedule() function. By modifying this function, it is possible to change the scheduling policy.