FreeBSD SMP Project

Project Goal

The FreeBSD SMP project, often referred to as SMPng (SMP next generation), is focused on implementing fine-grained SMP support for the FreeBSD 5.0 kernel (scheduled for November 2002). Due to FreeBSD's history, this is much like trying to fit a square peg into a round hole, and as such, the intermediate results aren't pretty in many ways. We are specifically not attempting to rewrite the kernel from scratch, nor are we on a crusade to fix all the architectural nits currently present in the kernel. In fact, we expect to leave a trail of architectural nits that will still be evident in many ways when FreeBSD 5.0 is released. This is a pragmatic project rather than a theoretical one; we need to have the kernel working and stable in under a year, so time restraints require that we be realistic about what to do when.

Project Plan

This web page contains information related to the effort to improve SMP support in FreeBSD. In general, this project uses what it can from the BSD/OS 5.0 development kernel, and re-implements what cannot be directly used due to divergence in the code bases.

As with any free software project, a detailed schedule is not possible. We expect to have significant performance and stability issues that need to be worked through over the first several months of the project, though every effort will be made to keep -current running as well as possible.

Resources and Links

Status

Following is an incomplete list of general tasks.

Task Responsible Last updated Status
Convert the giant lock from spinning to blocking, add the scheduler lock, add per-CPU idle processes. Matt Dillon 25 June 2000 Done
Port the BSD/OS locking primitives (i386). Jake Burkholder 3 July 2000 Done
Implement heavy-weight interrupt threads (i386). Greg Lehey 3 August 2000 Done
Rewrite the low level interrupt code (i386 UP). Greg Lehey 3 August 2000 Done
Demonstrated reasonable stability (self-hosted buildworld) (i386 UP). -smp developers 12 August 2000 Done
Port the BSD/OS locking primitives (alpha). Doug Rabson 24 August 2000 Done
Stub out (disable) spl()s. Greg Lehey 30 August 2000 Done
Port the BSD/OS ktr code. Greg Lehey, John Baldwin 30 August 2000 Done
Rewrite the low level interrupt code (i386 SMP). John Baldwin 1 September 2000 Done
Demonstrated reasonable stability (self-hosted buildworld) (i386 SMP). -smp developers 6 September 2000 Done
Demonstrated reasonable stability (self-hosted buildworld) (alpha). -smp developers 6 September 2000 Done
Make malloc and friends thread-safe. Jason Evans 10 September 2000 Done
Implement msleep(), make tsleep() an msleep() wrapper. Jake Burkholder 11 September 2000 Done
Make fxp driver thread-safe. Chuck Paterson 17 September 2000 Done
Make mbuf's thread-safe. Bosko Milekic 29 September 2000 Done
Lock manager re-work. Jason Evans 3 October 2000 Done
Implement heavy-weight interrupt threads (alpha). John Baldwin, Doug Rabson 5 October 2000 Done
Rewrite the low level interrupt code (alpha). Doug Rabson, John Baldwin 5 October 2000 Done
Process accounting. Tor Egge, John Baldwin 5 October 2000 Done
Make ethernet drivers thread-safe. Bill Paul 15 October 2000 Done
Make the mutex headers mostly machine-independent. John Baldwin 20 October 2000 Done
Rename SMP_DEBUG to MUTEX_DEBUG. John Baldwin 20 October 2000 Done
Give each soft interrupt its own thread. Chuck Paterson 25 October 2000 Done
Make sf_bufs (sendfile(2)) thread-safe. Bosko Milekic 5 November 2000 Done
Make the witness code work correctly. John Baldwin 18 November 2000 Done
Split the ktr-specific code out of db_interface.c. John Baldwin 15 December 2000 Done
Convert the sio driver to using a spin mutex. John Baldwin 18 December 2000 Done
Implement condition variables. Jake Burkholder, Jason Evans 15 January 2001 Done
Add a flag to mtx_init() (MTX_RECURSE) that denotes whether a mutex is allowed to recurse. Bosko Milekic 19 January 2001 Done
Make the zone allocator thread-safe. Dag-Erling Smorgrav 21 January 2001 Done
Convert simplelocks to mutexes. Jason Evans 24 January 2001 Done
Make kernel preemptive with respect to interrupts. Jake Burkholder 31 January 2001 Done
Cleanup of mutex API. Bosko Milekic 8 February 2001 Done
Remove COM_LOCK. Mark Murray 11 February 2001 Done
Merge various scheduling classes into one run queue. Modify scheduler to support preemptable kernel. Jake Burkholder 11 February 2001 Done
Make priority propagation work correctly. Jake Burkholder 11 February 2001 Done
Make most of the interrupt thread code MI and shared between hardware and software interrupts. John Baldwin 18 February 2001 Done
Implement sx (shared/exclusive) locks. Jason Evans 5 March 2001 Done
Generalize/improve witness to handle more complex locking primitives (mtx, sx). John Baldwin 28 March 2001 Done
Convert the allproc and proctree locks from lockmgr locks to sx locks. John Baldwin 28 March 2001 Done
Make mbuf system use condition variables instead of msleep()/wakeup(). Bosko Milekic 2 April 2001 Done
Remove <sys/mutex.h> includes from other kernel headers such as <vm/vm_zone.h>, <sys/resourcevar.h>, <sys/ucred.h>, and <sys/mbuf.h>. Mark Murray 15 May 2001 Done
Cleanup the various mp_machdep.c's, unify various SMP API's such as IPI delivery, etc. John Baldwin 15 May 2001 Done
Make most of the forward_* and forwarded_* functions MI. John Baldwin 15 May 2001 Done
Complete the MD support for SMP on the Alpha platform. Andrew Gallatin, Doug Rabson, John Baldwin 15 May 2001 Done
Convert select() to use condition variables. Seigo Tanimura 15 May 2001 Done
Add a "giant" lock around the VM subsystem. Alfred Perlstein 13 June 2001 Done
Introduce a modified slab allocator for the mbuf subsystem. Bosko Milekic 21 June 2001 Done
Add a witness_assert() function to handle lock assertions. John Baldwin 27 June 2001 Done
Extend sx locks to support try lock operations. John Baldwin 27 June 2001 Done
Document KTR. John Baldwin 28 June 2001 Done
Make fork_return, fork_exit, ast, and userret MI. John Baldwin 29 June 2001 Done
Make sched_lock's savecrit a per-process property saved and restored in mi_switch and initialized in fork_exit. John Baldwin 30 June 2001 Done
Make ast() loop. John Baldwin 10 August 2001 Done
Add upgrade/downgrade sx lock operations. Alexander Kabaev, Jason Evans 13 August 2001 Done
Implement semaphores. Jason Evans 14 August 2001 Done
Add support for upgrade/downgrades in witness. John Baldwin 23 August 2001 Done
Remove the MP safe syscall flag from the syscall table and add explicit mtx_lock/unlock's of Giant to all syscalls. Matt Dillon 7 September 2001 Done
Make most of cpu_wait() and cpu_exit() MI. Peter Wemm 9 September 2001 Done
Split NFS into client and server. Peter Wemm 18 Oct 2001 Done
Lock taskqueues. Andrew Reiter, John Baldwin 25 October 2001 Done
Add a per-thread ucred reference. John Baldwin 25 October 2001 Done
Make most of the per-CPU stuff MI. John Baldwin 11 December 2001 Done
Make critical section saved state per-thread instead of per-lock so that interlocking spin locks work properly. John Baldwin 17 December 2001 Done
Replace the APIC-specific imen_mtx with a MI-named icu_lock to protect interrupt controllers and associated data within the kernel for both i386 and alpha. John Baldwin 20 December 2001 Done
Use the per-thread critical section nesting level in the mutex and interrupt thread code to automatically determine when to not preempt. This makes the MTX_NOSWITCH, SWI_SWITCH, and SWI_NOSWITCH flags obslete as the kernel will be able to figure out the proper behavior on its own. John Baldwin 5 January 2002 Done
Lock struct filedesc and struct file. Seigo Tanimura, Alfred Perlstein 12 January 2002 Done
Lock struct proc. John Baldwin 20 February 2001 In progress
Lock struct pgrp, struct session, and struct sigio. Seigo Tanimura 13 June 2001 In progress
Make KTRACE write into tracefiles asynchronously. John Baldwin 6 September 2001 In progress
Make the kernel fully preemptive. John Baldwin 7 September 2001 In progress
Convert vm_map locks to sx locks. Brian Feldman 18 October 2001 In progress
Lock down the tty subsystem. Dick Garner, Jeremy Scofield 16 November 2001 In progress
Fix clock locking to be the same on all platforms. John Baldwin 16 November 2001 In progress
Make sleep mutexes spin if the current lock holder is executing on another CPU. John Baldwin 16 November 2001 In progress
Add support for the IA32 pause instruction to spin loops in locks. John Baldwin 16 November 2001 In progress
Optimize UP support by changing spin locks to only perform critical section enter and exits. John Baldwin 20 December 2001 In progress
Implement lazy interrupt thread switching (context stealing) on i386. Bosko Milekic 4 January 2002 In progress
Fix synchronization of TLB flushes and invlpg() on x86 SMP. Peter Wemm 4 January 2002 In progress
Create mechanism in cdevsw structure to protect thread-unsafe drivers. John Baldwin 15 May 2001 Stalled
Make printf() safe to call in almost any situation to avoid deadlocks. Chuck Paterson 15 May 2001 Stalled
Implement trap handler for cmpxchgl on 80386 to support generic userland atomic operations. Jake Burkholder 15 May 2001 Stalled
Add locking to NFS.   15 May 2001 Not Started
Use per-CPU buffers for ktr to reduce synchronization.   5 September 2000 Not Started
Remove priority argument from tsleep(), msleep(), cv_*wait*().   12 January 2001 Not Started
Reimplement kqueue using condition variables. Jonathan Lemon 15 March 2001 Not Started
Conditionalize atomic ops in the SMP code that are used for debugging statistics. Peter Wemm 15 March 2001 Not Started
Add a new witness check for exiting processes to verify that an exiting process holds no locks. John Baldwin 13 June 2001 Not Started
Make cpu_core MI.   13 June 2001 Not Started
Lock struct pargs.   20 June 2001 Not Started
Specify priorities for condition variables, semaphores, and sx locks.   7 September 2001 Not Started
Fix SIGXPU and other #if 0'd things in mi_switch().   7 September 2001 Not Started
Axe schedpu() in favor of event driven priority updates as much as possible.   7 September 2001 Not Started
Fix PHOLD() so that it blocks to guarantee PS_INMEM.   7 September 2001 Not Started
Make grow_stack() MI. Possibly even a macro or inline.   7 September 2001 Not Started
Fix *hold (e.g. crhold) to return reference to object.   7 September 2001 Not Started
Fix various procfs_machdep.c to use PHOLD, not sched_lock.   7 September 2001 Not Started
Convert eventhandlers to sx locks.   7 September 2001 Not Started
Add witness checking for lockmgr locks.   7 September 2001 Not Started
Add ICU spin locks on ia64 and sparc64.   4 January 2002 Not Started

This table lists the todo subtasks for multithreading the network stack.

Task Responsible Last updated Status
Protect network interface queues. Jonathan Lemon 24 November 2000 Done
Lock struct ifnet.   19 January 2001 Not Started

Known Issues

Issue Last updated Status
Idle processor time is not charged to the idle processes. 20 September 2000 Resolved
microuptime creeps backwards. 4 October 2000 Resolved
microuptime() went backwards 4 October 2000 Resolved
Process accounting is not accurate (the more CPUs, the closer to correct it is). 5 October 2000 Resolved
M_DEVBUF is probably the wrong memory pool for interrupt stuff and we should think about creating a new malloc pool for that stuff. 9 February 2001 Resolved
PC card eject panics due to a race condition in the interrupt thread code. 15 March 2001 Resolved
SMP x86 boxes are seeing NCPU * 100 clk interrupts and NCPU * 128 rtc interrupts. 15 May 2001 Resolved
Witness will infinitely recurse when it acquires Giant after sleeping with a sleepable lock. 27 June 2001 Resolved
Serial gdb does not work if boot_ddb and boot_gdb options are specified. 5 September 2000 Unresolved
Serial gdb does not work at 115200 baud. 5 September 2000 Unresolved
Profiling is broken. 20 February 2001 Unresolved

News

The remainder of this page is structured as a reverse-chronological log.

15 May 2001 22 March 2001 5 March 2001 24 January 2001 12 January 2001 11 October 2000 8 September 2000 6 September 2000 5 September 2000 1 September 2000 30 August 2000 12 August 2000 3 August 2000 6 July 2000 5 July 2000 3 July 2000 26 June 2000 25 June 2000 19 June 2000

\http://www.freebsd.org/mailto.html
Copyright © 1995-2002 The FreeBSD Project. All rights reserved.
Last modified: 2002/01/15 12:53:29