The Evolution of Darwin

Technology Profile

The Evolution of Darwin

The evolution that successfully put a Unix engine under the Mac OS hood began with the acquisition of NeXT by Apple in December 1996. It came to fruition in Mac OS X, a completely rebuilt implementation of the Mac OS that, for the first time, combined Macintosh ease-of-use with the power and stability of Unix.

At the heart of Mac OS X one finds Darwin, an open source core that integrates a diverse collection of powerful technologies in a robust, flexible architecture. Darwin is like Linux with a day job: By day, it stays discreetly in the background, running Mac OS X. By night, Darwin shows its open source roots: hackable, extensible, and the product of the same community, culture, and traditions that created Apache, sendmail, GNU/Linux, Mozilla, and UNIX itself. Users are happy, but engineers also get to have their fun.

That dual identity -- part reliably thrumming engine, part developer's toy chest -- was not a foregone conclusion. Even after the announcement was made to recreate Mac OS on a Unix foundation, thorny questions remained. Would the newly acquired technology be integrated quietly, without fanfare, or would the BSD UNIX-based code be openly acknowledged as part of the product development strategy?

Many wondered whether the NeXT acquisition would be a simple technology transfer or something more significant. After all, Unix is not just a set of technologies -- it's also a philosophy, a culture, and a community. Did moving Mac OS toward Unix also mean embracing the community and philosophy that spawned Unix in the first place? Just as important as the decision to merge Mac OS and Unix was the question of how that merging would evolve.

Fortunately, Apple soon made it clear that as part of this metamorphosis, it would adopt two radical policies: an evolutionary approach to architecture and a revolutionary approach to software development. That evolutionary approach now informs the Darwin core operating system, and through it, Mac OS X. And that revolutionary developmental approach is open source.

APPLE GOES OPEN SOURCE

In 1999, Apple became the first major computer company to make open source development a fundamental part of its software strategy. Apple released the source code to practically all of the components of Darwin to the developer community, enabling developers to see how Apple has implemented it and to use portions of the code within their own software. After an exhaustive examination of the whole gamut of open source licenses, and after consultation with the Open Source Initiative (OSI), Mozilla, and others, the Apple Public Source License was created to put Darwin in the hands of the open source community. Darwin also incorporates code distributed under other open source licenses (such as the Apache web server).

Apple chose open source over the closed corporate approach for a number of reasons, both philosophical and practical. One of the most important practical reasons was the richer pool of talent made available by opening up the development process. Another was the superiority of the ultimate product, due both to the inherent virtues of the open source development paradigm and the ease of incorporating other open source technologies. Apple's historical relationship with educational IT departments, research organizations, and the teaching profession also made the open source approach a natural choice.

Committed to full participation in the open source process, Apple created a CVS repository containing the source code to Darwin available on the Internet (with few exceptions only available as binaries). The development infrastructure also includes mailing lists, a tracking system, on-call Apple engineers, and a documentation project under the Common Documentation License.

DARWIN'S STRUCTURE

The final architectural scheme for Mac OS X, unveiled at the Macworld Expo in January 2000, combines nine basic elements in a layered, component-based model. At its foundation is Darwin, which actually contains two layers of its own: the Mach kernel and the BSD subsystem wrapped around it. The time-tested Mach 3.0 kernel provides the rock-solid stability that is indispensable in consumer and production environments. The move to a kernel-based architecture (a first for the consumer Mac OS) also makes porting to other platforms much more practical.

Darwin also incorporates a full implementation of BSD (Berkeley Software Distribution) UNIX, welded on top of the Mach kernel. The hybrid BSD architecture adopted by Darwin embodies its historical association with the BSD code base and underscores both the project's strong relationship with the various BSD organizations and its strong cultural affinity with the open source developer community. Indeed, Darwin itself constitutes a full-blown operating system, complete in and of itself, made available as open source by Apple -- a fact that's both astonishing and under-appreciated.

Darwin is comprised of five main components: the Mach microkernel and BSD subsystem, the file system, networking, and the I/O Kit.

The Mach microkernel at the heart of Darwin (based on Carnegie-Mellon University's Mach 3.0) manages processor resources, scheduling, and memory protection, and gives other OS layers a messaging-centered infrastructure.
Darwin wraps a customized version of 4.4 BSD-Lite2 kernel and userspace around Mach. It includes many of the POSIX APIs, exporting them to user-space, and abstracts Darwin's file system and networking. Darwin's BSD also provides the process model, basic security policies, and threading support for Mac OS X.
Supporting both Mac and Unix file systems, Darwin's file system -- based on an refined VFS design that allows the addition of new file systems and enhances those already supported -- is a key element in Mac OS X's versatility.
Darwin makes industry-standard TCP/IP -- based on the original, time-tested BSD code -- the primary fundamental networking infrastructure for Mac OS X.
Darwin incorporates the I/O Kit, an object-oriented framework for developing device drivers that also provides much of the infrastructure that drivers require to support SMP and real-time preemption

It's worth emphasizing that even without the other elements of Mac OS X -- the powerful graphics layer (QuickTime, OpenGL, Quartz), the array of flexible application environments (Carbon, Cocoa, Java), and the eye-catching Aqua user interface -- Darwin is a complete, functional operating system.

Darwin implements an industrial-strength security architecture based on open standards, building on The Open Group's Common Data Security Architecture (CDSA). Open source was the preferred approach here partly because peer review is simply the best way to validate security technologies. Beyond that, improving extensibility is far easier with open source than with commercial software. While some elements of CDSA are based on proprietary technology licensed from third-parties, one of Darwin's principal goals is to make the open source effort functionally complete, which means creating an open source reference implementation of its encryption library.

DARWIN'S ROOTS

The Darwin team is indebted to a diverse collection of open source projects, including the following:

- Mach, which was originally developed by Project Mach at Carnegie-Mellon University, and later enhanced by the Open Software Foundation (now The Open Group).

- 4.4BSD-Lite2, originated in UC Berkeley's Computer Systems Research Group and developed by a large number of contributors:

FreeBSD, the primary reference platform for Darwin's BSD kernel development.
NetBSD, the upstream source for a significant portion of Darwin's user-space commands and tools.
OpenBSD, with its focus on robustness and security and its integrated cryptography, provides OpenSSH for secure remote access.

- Apache HTTPD, the world's most popular web server, is included as part of the Darwin distribution, making Apple the largest distributor of Apache.

WHERE TO FIND DARWIN

Ever since Darwin's initial release in March 1999, its development (and the development of other open source development projects, such as the Darwin Streaming Server, OpenPlay, and the CDSA) has been managed and hosted by Apple. Partly owing to its historical associations with BSD UNIX and partly for pragmatic reasons, the Darwin project takes a more centralized, more closely synchronized BSD-style approach to managing the development process -- distinct from the more freewheeling style that characterizes Linux development. Since Mac OS X releases are built directly on the live Darwin CVS repository, a fairly comprehensive procedural framework for registering and managing Darwin developers has been put into place to ensure a high level of quality control.

To broaden the collaboration between Apple and the development community beyond that more constrained model, OpenDarwin.org was founded in April 2002 by the Internet Software Consortium (ISC), with help from Apple. Hosted by the ISC, OpenDarwin.org is a development "sandbox" where riskier changes to Darwin can be explored and tested without affecting the reference community. OpenDarwin.org complements Apple's developer infrastructure by allowing greater participation from the community and greater opportunity to innovate and explore new technologies. While the project is fully independent of Apple -- with its own website, project news, bug tracking services, and CVS repository -- Apple will continue to work closely with OpenDarwin to bring enhancements back into Mac OS X.

Darwin has been a groundbreaking project on many fronts. Its hybrid BSD architecture is a technical triumph that makes life immeasurably better for both users and developers. What's more, putting the core of its commercial offering into the hands of the open source community has made Apple the first major computer manufacturer to fully embrace the open source movement. For Mac OS X developers and users alike, that move opens the door to the many benefits -- from better code to a stronger community -- offered by the open source model.