From: dil...@apollo.backplane.com (Matthew Dillon)
Subject: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8io8gf$2n12$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636866750
X-Trace: FreeBSD.csie.NCTU.edu.tw 961520975 89130 140.113.209.200 (20 Jun 2000 17:09:35 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 17:09:35 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
All SMP/FreeBSD/BSDI work should move to the freebsd-smp list.
(I've BCC'd this to -current).
--
I now have mutexes 99% working in an SP build. I will be making
SP (single processor) patch sets available tonight or tomorrow
morning. (99% == I can boot the machine normally but after a
while it locks up due to an interrupt nesting issue which I haven't
dealt with yet).
In order to make an MP build work, we need interrupt threads (either
light or heavy weight).
Greg, you expressed an interest in doing the light-weight interrupt
threads. Do you want me to do the heavy-weight interrupt threads
first? It's a lot of nasty assembly and it is a prerequisit for being
able to do the light weight interrupt threads.
I think once the heavy weight interrupt threads are done that the
light weight interrupts can be implemented entirely in C.
Right now the SP build works because I am allowing (unthreaded) interrupts
to steal the idleproc's context, and that only works because they can
get the giant mutex without blocking (remember, the BSDI giant mutex
is a blocking mutex, not a spin mutex). In the MP build the interrupts
need to be able to block getting the giant mutex which means we need
to implement heavy-weight interrupt threads at the very least before
we can get anything working, because we are not allowed to block in
the idleproc.
I believe I have done almost everything necessary to implement heavy
weight interrupt threads, including moving the kthreads initialization
code to an earlier point in the boot process (so much so that I think
the device init stuff can run with working kernel processes rather then
with all the delayed-thread stuff).
I also think that people can start hacking on the code with the SP
build once I fix this last little problem, then I can work on the MP
build in parallel. The MP build requires a significant amount of
additional work including having to rewrite most of the APIC assembly
code (just as I had to rewrite most of the ICU assembly code for the SP
build). It's another weekend at least for that. Using the SP build
does not exercise all the MP characteristics of the mutexes but it
exercises enough that it can be used to vett for dumb mistakes.
-Matt
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: mja...@feral.com (Matthew Jacob)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8io8ui$2o3a$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636871441
X-Trace: FreeBSD.csie.NCTU.edu.tw 961521426 90219 140.113.209.200 (20 Jun 2000 17:17:06 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 17:17:06 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
Matt- is this all on i386 now? I can start throwing some cycles into trying to
clean up the alpha side- that is, if Doug isn't doing it already.
-matt
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: dil...@apollo.backplane.com (Matthew Dillon)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8iobru$2tb9$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636887604
X-Trace: FreeBSD.csie.NCTU.edu.tw 961524414 95594 140.113.209.200 (20 Jun 2000 18:06:54 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 18:06:54 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
:
:Matt- is this all on i386 now? I can start throwing some cycles into trying to
:clean up the alpha side- that is, if Doug isn't doing it already.
:
:-matt
Yes, all i386. At the meeting we decided to break the alpha port
temporarily.
I think alpha work can begin when I make the SP (single processor)
patches available. I would recommend sticking to SP alpha builds
probably for a few months until our i386 MP work (which is mostly
machine independant) is reasonably stable. (does alpha even have an MP
build yet?).
-Matt
Matthew Dillon
<dil...@backplane.com>
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: i...@village.org (Warner Losh)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8ioea2$9j$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636903580
X-Trace: FreeBSD.csie.NCTU.edu.tw 961526914 308 140.113.209.200 (20 Jun 2000 18:48:34 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 18:48:34 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
In message <200006201708.KAA87...@apollo.backplane.com> Matthew Dillon writes:
: I now have mutexes 99% working in an SP build. I will be making
: SP (single processor) patch sets available tonight or tomorrow
: morning. (99% == I can boot the machine normally but after a
: while it locks up due to an interrupt nesting issue which I haven't
: dealt with yet).
May I request that you don't check in the new MP core until the SP
issues have been resolved? I don't care if the MP boxes have major
pain after the commit date (that's rather the point of all the new MP
work), but have some serious issues if the SP boxes have major pain.
It wasn't clear from the orignial message if the major pain and
bleeding is for all users, or just MP users.
It would be extremely hard to make forward progress on anything while
the pain is there. For a few days or even a couple of weeks this
would be OK, but for months that is not acceptible to me. I'm trying
to get NEWCARD finished, as well as start the cardbus work and those
two projects are being done in -current. There's no pain to
non-laptop users (in fact, there's currently no pain to laptop users
that use the old system). It is extremely desirable to have the
current tree buildable and stable for SP machines during this
transition.
Please note, this isn't an attempt to disparage your work, or to try
to get overly in the way of it. I'm merely trying to keep the SP
world relatively stable while the MP work is going on. Ther'es a 6
year tradition of having -current be basically usable for developers
to develop in that I'd like to continue.
Warner
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: mja...@feral.com (Matthew Jacob)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8iocfo$2ufd$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636893091
X-Trace: FreeBSD.csie.NCTU.edu.tw 961525048 96750 140.113.209.200 (20 Jun 2000 18:17:28 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 18:17:28 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
> Yes, all i386. At the meeting we decided to break the alpha port
> temporarily.
I guess it burns me a bit that this meeting excluded some of us who have a
strong interest in seeing FreeBSD succeed.
> I think alpha work can begin when I make the SP (single processor)
> patches available. I would recommend sticking to SP alpha builds
> probably for a few months until our i386 MP work (which is mostly
> machine independant) is reasonably stable. (does alpha even have an MP
> build yet?).
Doug was 'real close' last I heard.
Okay, I'll drop working on FreeBSD until next year then. 'so long.
-matt
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: dil...@apollo.backplane.com (Matthew Dillon)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8ioggb$2a89$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636914590
X-Trace: FreeBSD.csie.NCTU.edu.tw 961529163 76042 140.113.209.200 (20 Jun 2000 19:26:03 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 19:26:03 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
:> I think alpha work can begin when I make the SP (single processor)
:> patches available. I would recommend sticking to SP alpha builds
:> probably for a few months until our i386 MP work (which is mostly
:> machine independant) is reasonably stable. (does alpha even have an MP
:> build yet?).
:
:Doug was 'real close' last I heard.
:
:Okay, I'll drop working on FreeBSD until next year then. 'so long.
:
:-matt
All I am saying here is that the BSDI SMP merge work will turn the
MP build into a moving target for the next month at least, and it
wouldn't be a very good use of time trying to keep the alpha build
in sync for the MP build until after the API's have settled down. The
SP build is another story... I expect those API's to settle down within
a week.
-Matt
Matthew Dillon
<dil...@backplane.com>
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: dil...@apollo.backplane.com (Matthew Dillon)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8iohbe$2lgn$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636917982
X-Trace: FreeBSD.csie.NCTU.edu.tw 961530030 87576 140.113.209.200 (20 Jun 2000 19:40:30 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 19:40:30 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
:
:In message <200006201708.KAA87...@apollo.backplane.com> Matthew Dillon writes:
:: I now have mutexes 99% working in an SP build. I will be making
:: SP (single processor) patch sets available tonight or tomorrow
:: morning. (99% == I can boot the machine normally but after a
:: while it locks up due to an interrupt nesting issue which I haven't
:: dealt with yet).
:
:May I request that you don't check in the new MP core until the SP
:issues have been resolved? I don't care if the MP boxes have major
:pain after the commit date (that's rather the point of all the new MP
:work), but have some serious issues if the SP boxes have major pain.
:It wasn't clear from the orignial message if the major pain and
:bleeding is for all users, or just MP users.
:
:It would be extremely hard to make forward progress on anything while
:the pain is there. For a few days or even a couple of weeks this
:would be OK, but for months that is not acceptible to me. I'm trying
:to get NEWCARD finished, as well as start the cardbus work and those
:...
The BSDI SMP merge work is not just an MP issue. It changes the
way the kernel operates at a fundamental level. What it means,
in short, is that the MP mechanisms are going to become 'native' to
the kernel, even for SP builds.
I'll give you an example: The SPL mechanism is inherently a
single-processor algorithm. It just doesn't work on an MP system.
Rather then hack SPL's for MP builds we are going to (eventually)
remove them entirely from the system ... SP or MP build.
So, for example, the Scheduler will no longer be protected by
splhigh()/spl0(). It will instead be protected by SchedMutex... the
same protection mechanism will be used for both the SP and MP builds.
Another example: Interrupts are going to become interrupt threads
natively (on both the SP and MP builds). This means that the kernel
must be preemptable while sitting in kernel mode. On both the SP
and MP builds.
So, as you can see, most of the mechanisms we need to implement apply
to both SP and MP builds.
This means that SP builds are going to be unstable for a while, not
just MP builds, as we work out the issues. I am fairly confident that
I can produce a stable set of changes in the first patch set, but
the fact of the matter is that much of the work is going to entail
ripping out the SPL compatibility mechanisms one at a time and replacing
them with mutexes. This will be an ongoing process over the next 6
months and that means that -current is going to be less stable for
the entire time -- for both the SP and MP builds.
:Please note, this isn't an attempt to disparage your work, or to try
:to get overly in the way of it. I'm merely trying to keep the SP
:world relatively stable while the MP work is going on. Ther'es a 6
:year tradition of having -current be basically usable for developers
:to develop in that I'd like to continue.
:
:Warner
-Matt
Matthew Dillon
<dil...@backplane.com>
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: i...@village.org (Warner Losh)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8ioi2f$2m5p$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636925840
X-Trace: FreeBSD.csie.NCTU.edu.tw 961530768 88250 140.113.209.200 (20 Jun 2000 19:52:48 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 19:52:48 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
In message <200006201936.MAA88...@apollo.backplane.com> Matthew Dillon writes:
: This means that SP builds are going to be unstable for a while, not
: just MP builds, as we work out the issues. I am fairly confident that
: I can produce a stable set of changes in the first patch set, but
: the fact of the matter is that much of the work is going to entail
: ripping out the SPL compatibility mechanisms one at a time and replacing
: them with mutexes. This will be an ongoing process over the next 6
: months and that means that -current is going to be less stable for
: the entire time -- for both the SP and MP builds.
It sounds like there will be a large step function of instability that
will be introduced right away. Then, over time, thigns will become
more or less stable as the work progresses. Some days will be good
tree days, others bad. So long as I can boot my laptop with -current
to do pccard work (which means it must last 15 minutes of light usage,
ideally more), then it meets my lower bounds of acceptible. It would
be nice if I could also build kernels and such on my laptop during
that time as well, but that isn't as critical to me since I can reboot
with the older kernel. There's some pain, but on the whole it would
be workable if I can do these things.
I know that it is hard, but can you characterize the level of
instability that I'd see on a regular basis? Where in the continum of
"we talking uptimes of 1 minute or less" to "don't put this into
production" do you see things most of the time?
Warner
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: g...@lemis.com (Greg Lehey)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8ioimq$2mo6$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636926914
X-Trace: FreeBSD.csie.NCTU.edu.tw 961531418 88839 140.113.209.200 (20 Jun 2000 20:03:38 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 20:03:38 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
On Tuesday, 20 June 2000 at 10:08:30 -0700, Matthew Dillon wrote:
> All SMP/FreeBSD/BSDI work should move to the freebsd-smp list.
>
> (I've BCC'd this to -current).
>
> --
>
> I now have mutexes 99% working in an SP build. I will be making
> SP (single processor) patch sets available tonight or tomorrow
> morning. (99% == I can boot the machine normally but after a
> while it locks up due to an interrupt nesting issue which I haven't
> dealt with yet).
Great.
> In order to make an MP build work, we need interrupt threads (either
> light or heavy weight).
>
> Greg, you expressed an interest in doing the light-weight interrupt
> threads.
Yes, I've committed to both tasks, first the heavyweight processes and
then the light-weight threads.
> Do you want me to do the heavy-weight interrupt threads first?
> It's a lot of nasty assembly and it is a prerequisit for being
> able to do the light weight interrupt threads.
Thanks, I don't have any problem with assembler.
> I think once the heavy weight interrupt threads are done that
> the light weight interrupts can be implemented entirely in C.
Interestingly, I was expecting more assembler in the light-weight
threads. But in both cases I would expect the assembler component to
be minimal in either case.
> Right now the SP build works because I am allowing (unthreaded)
> interrupts to steal the idleproc's context, and that only works
> because they can get the giant mutex without blocking (remember,
> the BSDI giant mutex is a blocking mutex, not a spin mutex). In
> the MP build the interrupts need to be able to block getting the
> giant mutex which means we need to implement heavy-weight
> interrupt threads at the very least before we can get anything
> working, because we are not allowed to block in the idleproc.
Right, that was my understanding last week.
> I believe I have done almost everything necessary to implement
> heavy weight interrupt threads, including moving the kthreads
> initialization code to an earlier point in the boot process (so
> much so that I think the device init stuff can run with working
> kernel processes rather then with all the delayed-thread stuff).
Why don't we take this one off line and discuss just how it should be
done? I'm at USENIX at the moment and don't have much time to invest
on this topic (as promised last week), but we also agreed that it
didn't have to be ready this week.
Greg
--
Finger g...@lemis.com for PGP public key
See complete headers for address and phone numbers
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
From: dil...@apollo.backplane.com (Matthew Dillon)
Subject: Re: SMP discussion moving to freebsd-smp
Date: 2000/06/21
Message-ID: <8ioi72$2m97$1@FreeBSD.csie.NCTU.edu.tw>#1/1
X-Deja-AN: 636928643
X-Trace: FreeBSD.csie.NCTU.edu.tw 961530915 88360 140.113.209.200 (20 Jun 2000 19:55:15 GMT)
Organization: NCTU CSIE FreeBSD Server
NNTP-Posting-Date: 20 Jun 2000 19:55:15 GMT
Newsgroups: mailing.freebsd.smp
X-Complaints-To: usenet@FreeBSD.csie.NCTU.edu.tw
The kernel will always be buildable. It should be stable enough to
last for 5 minutes. But there are going to be a lot of legacy issues
that need to be fixed -- drivers that get broken. What will probably
happen is that a base system with only core features enabled (disk,
serial, console, network) will always be more stable during the mergework
then a system with the kitchen sink in. A system with the kitchen
sink in may not even compile!
I don't expect the work required to make drivers compile and run again
to be all that difficult, that's the whole reason for keeping the SPL
compatibility code in place during the merge work. But it will also
be non-zero, and we will want the SPL compatibility code *gone* by
release time.
I'll give you one example of this: Drivers that create kernel threads
(there are two or three) have to use the new mutex code on entry now,
and the kthread_create() has been renamed to mp_kthread_create() to
guarentee that. Fixing them will not be difficult, but is also non-zero.
Another example: cpu_switch() has been renamed to mp_cpu_switch() and
mi_switch() has been renamed to mp_switch(). These interfaces may now
only be called from mutex-aware code. I've adjusted all the core code
to deal with it but there are probably a few drivers that try to poke
too deep into the kernel that need to be adjusted as well. The
adjustments required are straightforward. For example, in order to be
able to call mp_switch() the caller must hold the scheduler mutex
and release Giant (there are two helper routines to do all the dirty
work here).
-Matt
To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message
|