[gentoo-user] OT: pthreads condition variable/mutex question

[gentoo-user] OT: pthreads condition variable/mutex question

[Grant Edwards]

This is not Gentoo specific, and while I'm doing my prototyping and
development on a Gentoo system, the eventual target is not going to be
running Gentoo -- so feel free to ignore this thread or throw things
at me.

I'm trying to figure out how to synchronize threads which may be in
different processes.  Basically, I want thread A to be able to wake up
any number of other threads B, C, D, ... who are all blocking until A
says "go" (and who may or may not be in other processes).

Other (mostly embedded) OSes I've used had some sort of "event flag"
API that did exactly what I'm looking for, but I can't seem to find
such a thing in pthreads.

A condition variable in shared memory is the closest thing I have
found, and my test applications are working OK (so far).  But, I'm
unclear on the purpose of the mutex whose address you pass to
pthread_cond_wait().

Is it to prevent race conditions when manipulating the condition
variable's internal state? I don't see how that can be the case, since
the signal/broadcast call would have to be aware of it and it isn't.

The mutex appears to be there to serialize access to some user-defined
variable(s) (outside the condition variable itself) which I don't
have. So all the mutex locking/unlocking and resultant blocking of B,
C, D is just wasted overhead and pointless latency.

pthread_cond_wait(3) says

   When using condition variables there is always a Boolean predicate
   involving shared variables associated with each condition wait that
   is true if the thread should proceed. Spurious wakeups from the
   pthread_cond_timedwait() or pthread_cond_wait() functions may
   occur. Since the return from pthread_cond_timedwait() or
   pthread_cond_wait() does not imply anything about the value of this
   predicate, the predicate should be re-evaluated upon such return.

I have no "Boolean predicate" (which presumably comprises the
"user-defined variables outside the condition variable" I mentioned
above), and I don't want spurious wakeups, so a pthreads condition
variable would appear to be the wrong thing to use.

Is there something like an "event flag" similar to a condition
variable without spurious wakeup problem and without the extra
overhead of the mutex and "Boolean predicate".

Or am I expected to build my own "event flag" using the aforesaid 
"boolean predicate" just to avoid the spurious wakeup problem?  [I'm
guessing this is the case...]

-- 
Grant Edwards               grant.b.edwards        Yow! I'm DESPONDENT ... I
                                  at               hope there's something
                              gmail.com            DEEP-FRIED under this
                                                   miniature DOMED STADIUM ...

Re: [gentoo-user] OT: pthreads condition variable/mutex question

[Andrés Becerra Sandoval]

2014-08-13 12:21 GMT-05:00 Grant Edwards <grant.b.edwards@gmail.com>:
> This is not Gentoo specific, and while I'm doing my prototyping and
> development on a Gentoo system, the eventual target is not going to be
> running Gentoo -- so feel free to ignore this thread or throw things
> at me.
>
> I'm trying to figure out how to synchronize threads which may be in
> different processes.  Basically, I want thread A to be able to wake up
> any number of other threads B, C, D, ... who are all blocking until A
> says "go" (and who may or may not be in other processes).
>
> Other (mostly embedded) OSes I've used had some sort of "event flag"
> API that did exactly what I'm looking for, but I can't seem to find
> such a thing in pthreads.
>
> A condition variable in shared memory is the closest thing I have
> found, and my test applications are working OK (so far).  But, I'm
> unclear on the purpose of the mutex whose address you pass to
> pthread_cond_wait().
>
> Is it to prevent race conditions when manipulating the condition
> variable's internal state? I don't see how that can be the case, since
> the signal/broadcast call would have to be aware of it and it isn't.
>
> The mutex appears to be there to serialize access to some user-defined
> variable(s) (outside the condition variable itself) which I don't
> have. So all the mutex locking/unlocking and resultant blocking of B,
> C, D is just wasted overhead and pointless latency.
>
> pthread_cond_wait(3) says
>
>    When using condition variables there is always a Boolean predicate
>    involving shared variables associated with each condition wait that
>    is true if the thread should proceed. Spurious wakeups from the
>    pthread_cond_timedwait() or pthread_cond_wait() functions may
>    occur. Since the return from pthread_cond_timedwait() or
>    pthread_cond_wait() does not imply anything about the value of this
>    predicate, the predicate should be re-evaluated upon such return.
>
> I have no "Boolean predicate" (which presumably comprises the
> "user-defined variables outside the condition variable" I mentioned
> above), and I don't want spurious wakeups, so a pthreads condition
> variable would appear to be the wrong thing to use.
>
> Is there something like an "event flag" similar to a condition
> variable without spurious wakeup problem and without the extra
> overhead of the mutex and "Boolean predicate".
>
> Or am I expected to build my own "event flag" using the aforesaid
> "boolean predicate" just to avoid the spurious wakeup problem?  [I'm
> guessing this is the case...]
>
> --
> Grant Edwards               grant.b.edwards        Yow! I'm DESPONDENT ... I
>                                   at               hope there's something
>                               gmail.com            DEEP-FRIED under this
>                                                    miniature DOMED STADIUM ...
>
>

Hi Grant,

The best explanation I have read is this chapter:
http://pages.cs.wisc.edu/~remzi/OSTEP/threads-cv.pdf

from the book:

http://pages.cs.wisc.edu/~remzi/OSTEP/

I know its 17 pages, but it is worth it!


-- 
  Andrés Becerra Sandoval

Re: [gentoo-user] OT: pthreads condition variable/mutex question

[Andrés Becerra Sandoval]

2014-08-13 12:36 GMT-05:00 Andrés Becerra Sandoval <andres.becerra@gmail.com>:
> 2014-08-13 12:21 GMT-05:00 Grant Edwards <grant.b.edwards@gmail.com>:
>> This is not Gentoo specific, and while I'm doing my prototyping and
>> development on a Gentoo system, the eventual target is not going to be
>> running Gentoo -- so feel free to ignore this thread or throw things
>> at me.
>>
>> I'm trying to figure out how to synchronize threads which may be in
>> different processes.  Basically, I want thread A to be able to wake up
>> any number of other threads B, C, D, ... who are all blocking until A
>> says "go" (and who may or may not be in other processes).
>>
>> Other (mostly embedded) OSes I've used had some sort of "event flag"
>> API that did exactly what I'm looking for, but I can't seem to find
>> such a thing in pthreads.
>>
>> A condition variable in shared memory is the closest thing I have
>> found, and my test applications are working OK (so far).  But, I'm
>> unclear on the purpose of the mutex whose address you pass to
>> pthread_cond_wait().
>>
>> Is it to prevent race conditions when manipulating the condition
>> variable's internal state? I don't see how that can be the case, since
>> the signal/broadcast call would have to be aware of it and it isn't.
>>
>> The mutex appears to be there to serialize access to some user-defined
>> variable(s) (outside the condition variable itself) which I don't
>> have. So all the mutex locking/unlocking and resultant blocking of B,
>> C, D is just wasted overhead and pointless latency.
>>
>> pthread_cond_wait(3) says
>>
>>    When using condition variables there is always a Boolean predicate
>>    involving shared variables associated with each condition wait that
>>    is true if the thread should proceed. Spurious wakeups from the
>>    pthread_cond_timedwait() or pthread_cond_wait() functions may
>>    occur. Since the return from pthread_cond_timedwait() or
>>    pthread_cond_wait() does not imply anything about the value of this
>>    predicate, the predicate should be re-evaluated upon such return.
>>
>> I have no "Boolean predicate" (which presumably comprises the
>> "user-defined variables outside the condition variable" I mentioned
>> above), and I don't want spurious wakeups, so a pthreads condition
>> variable would appear to be the wrong thing to use.
>>
>> Is there something like an "event flag" similar to a condition
>> variable without spurious wakeup problem and without the extra
>> overhead of the mutex and "Boolean predicate".
>>
>> Or am I expected to build my own "event flag" using the aforesaid
>> "boolean predicate" just to avoid the spurious wakeup problem?  [I'm
>> guessing this is the case...]
>>
>> --
>> Grant Edwards               grant.b.edwards        Yow! I'm DESPONDENT ... I
>>                                   at               hope there's something
>>                               gmail.com            DEEP-FRIED under this
>>                                                    miniature DOMED STADIUM ...
>>
>>
>
> Hi Grant,
>
> The best explanation I have read is this chapter:
> http://pages.cs.wisc.edu/~remzi/OSTEP/threads-cv.pdf
>
> from the book:
>
> http://pages.cs.wisc.edu/~remzi/OSTEP/
>
> I know its 17 pages, but it is worth it!
>
>
> --
>   Andrés Becerra Sandoval

In short:

Withouth the use of the lock, the condition variable and a shared
variable in concert you can get in trouble!


-- 
  Andrés Becerra Sandoval

Re: [gentoo-user] OT: pthreads condition variable/mutex question

[Alec Ten Harmsel]

2014-08-13 12:21 GMT-05:00 Grant Edwards <grant.b.edwards@gmail.com>:
> This is not Gentoo specific, and while I'm doing my prototyping and
> development on a Gentoo system, the eventual target is not going to be
> running Gentoo -- so feel free to ignore this thread or throw things
> at me.

You're close enough ;) I'll try to answer to the best of my ability,
which is admittedly not much.

> I'm trying to figure out how to synchronize threads which may be in
> different processes.  Basically, I want thread A to be able to wake up
> any number of other threads B, C, D, ... who are all blocking until A
> says "go" (and who may or may not be in other processes).

Without knowing what you're doing, this sounds like a bad idea; if you
*need* to synchronize threads, why aren't they running in the same process?

If it's going to be running on Linux, you can send signals through the
kernel's signal API; specifically HUP, USR1, and USR2 might be of
interest to you.

> A condition variable in shared memory is the closest thing I have
> found, and my test applications are working OK (so far).  But, I'm
> unclear on the purpose of the mutex whose address you pass to
> pthread_cond_wait().

I'm too much of a rookie to know how to do this; how are you sharing
memory between processes?

> The mutex appears to be there to serialize access to some user-defined
> variable(s) (outside the condition variable itself) which I don't
> have. So all the mutex locking/unlocking and resultant blocking of B,
> C, D is just wasted overhead and pointless latency.

This is definitely not a task for mutexes, a boolean or signaling would
work much better.

I'm not sure what exactly you're trying to do, but I hope this helps. If
you can be more specific about the relationships between processes, I
might be able to help more.

Alec

[gentoo-user] Re: OT: pthreads condition variable/mutex question

[Grant Edwards]

On 2014-08-13, Alec Ten Harmsel <alec@alectenharmsel.com> wrote:
> 2014-08-13 12:21 GMT-05:00 Grant Edwards <grant.b.edwards@gmail.com>:

> Without knowing what you're doing, this sounds like a bad idea; if
> you *need* to synchronize threads, why aren't they running in the
> same process?

I'm trying to decouple different portions of a system as much as
possible.  Currently, the different parts communicate via Unix domain
sockets.  That works OK, but for a few of the high-frequence
operations I'm trying to find a way to eliminate the overhead that's
involved in sockets (system calls, context switches, copying data from
userspace to kernel space and back to user space).

> If it's going to be running on Linux, you can send signals through
> the kernel's signal API; specifically HUP, USR1, and USR2 might be of
> interest to you.

I may have to look into that.  Unfortunately signals are a real pain
to use in a multi-threaded application: all signals get delivered to a
single thread, when then has to "distribute" them somehow to the
thread that really cares.

>> A condition variable in shared memory is the closest thing I have
>> found, and my test applications are working OK (so far).  But, I'm
>> unclear on the purpose of the mutex whose address you pass to
>> pthread_cond_wait().
>
> I'm too much of a rookie to know how to do this; how are you sharing
> memory between processes?

The standard Posix way: shm_open() et al.

>> The mutex appears to be there to serialize access to some
>> user-defined variable(s) (outside the condition variable itself)
>> which I don't have. So all the mutex locking/unlocking and resultant
>> blocking of B, C, D is just wasted overhead and pointless latency.
>
> This is definitely not a task for mutexes, a boolean or signaling
> would work much better.

I'm not sure how one would use a boolean to wake up a thread.

I may have to stick with sockets when I want to block until some event
happens.

-- 
Grant Edwards               grant.b.edwards        Yow! It's a hole all the
                                  at               way to downtown Burbank!
                              gmail.com            

[gentoo-user] Re: OT: pthreads condition variable/mutex question

[Grant Edwards]

On 2014-08-13, Andrés Becerra Sandoval <andres.becerra@gmail.com> wrote:

> In short:
>
> Withouth the use of the lock, the condition variable and a shared
> variable in concert you can get in trouble!

That is often true, but 

 1) I don't have a shared variable that I want to associate with the
    condition variable.  There is some shared data, but I'm going to
    use means other than a mutex to control access to it.
    
 2) In most of the examples presented in that article the shared
    variable is an int that is set to 0 or 1 by one thread and is
    tested for non-zero value by the other.  I'm not aware of any CPU
    on which Linux runs where those operations can result in a race
    condition thus requiring a mutex.  [Yes I know that isn't
    guaranteed by the C language definiton, but one can verify that
    it's true.]

If I create a shared mutex just to make pthread_cond_wait happy, and
one of the "clients" (the ones calling pthread_cond_wait) dies while
holding the mutex, then it deadlocks all of the other clients.  To be
safe it seems one should create a private mutex for each thread that's
going to call pthread_cond_wait().  Private mutexes should be very
little overhead in the uncontended case, so that's an option.
    
-- 
Grant Edwards               grant.b.edwards        Yow! I wonder if I should
                                  at               put myself in ESCROW!!
                              gmail.com            

Re: [gentoo-user] Re: OT: pthreads condition variable/mutex question

[Alec Ten Harmsel]

On Wed 13 Aug 2014 03:23:21 PM EDT, Grant Edwards wrote:
> On 2014-08-13, Alec Ten Harmsel <alec@alectenharmsel.com> wrote:
>> 2014-08-13 12:21 GMT-05:00 Grant Edwards <grant.b.edwards@gmail.com>:
>
>> Without knowing what you're doing, this sounds like a bad idea; if
>> you *need* to synchronize threads, why aren't they running in the
>> same process?
>
> I'm trying to decouple different portions of a system as much as
> possible.  Currently, the different parts communicate via Unix domain
> sockets.  That works OK, but for a few of the high-frequence
> operations I'm trying to find a way to eliminate the overhead that's
> involved in sockets (system calls, context switches, copying data from
> userspace to kernel space and back to user space).

Decoupling == great. Is it possible that you could do something like 
this:

Thread 'a' in process 'a':
1. Event in thread 'a' is generated or whatever
2. if low-frequency event, send a message over a socket to a different 
thread/process
3. if high-frequency event, push event into shared memory thread-safe 
queue

Thread 'b' in process 'a':
1. infinite loop reading from shared memory queue and processing events

Ideally, the high-frequency events should be handled in the same 
thread, then in the same process, and only lastly in a different 
process. While decoupling is great, it seems like you're losing the 
benefits of it by tightly coupling all of your "decoupled" threads and 
processes.

> I may have to stick with sockets when I want to block until some event
> happens.

To be clear, do you want to block or sleep/yield until an event happens?

I'm sorry for not being too helpful. Just one last question: Can you 
describe what exactly your code is supposed to do, or is it something 
that you can't talk about because it's a work thing? I don't care 
either way, but I'm just curious because it seems you need to optimize 
quite a bit.

Alec

Re: [gentoo-user] OT: pthreads condition variable/mutex question

[Alan McKinnon]

On 13/08/2014 19:21, Grant Edwards wrote:
> This is not Gentoo specific, and while I'm doing my prototyping and
> development on a Gentoo system, the eventual target is not going to be
> running Gentoo -- so feel free to ignore this thread or throw things
> at me.
> 
> I'm trying to figure out how to synchronize threads which may be in
> different processes.  Basically, I want thread A to be able to wake up
> any number of other threads B, C, D, ... who are all blocking until A
> says "go" (and who may or may not be in other processes).


Sounds like you a "one talker - many listeners" model. Have you
considered a really simple solution like dbus?



> 
> Other (mostly embedded) OSes I've used had some sort of "event flag"
> API that did exactly what I'm looking for, but I can't seem to find
> such a thing in pthreads.
> 
> A condition variable in shared memory is the closest thing I have
> found, and my test applications are working OK (so far).  But, I'm
> unclear on the purpose of the mutex whose address you pass to
> pthread_cond_wait().
> 
> Is it to prevent race conditions when manipulating the condition
> variable's internal state? I don't see how that can be the case, since
> the signal/broadcast call would have to be aware of it and it isn't.
> 
> The mutex appears to be there to serialize access to some user-defined
> variable(s) (outside the condition variable itself) which I don't
> have. So all the mutex locking/unlocking and resultant blocking of B,
> C, D is just wasted overhead and pointless latency.
> 
> pthread_cond_wait(3) says
> 
>    When using condition variables there is always a Boolean predicate
>    involving shared variables associated with each condition wait that
>    is true if the thread should proceed. Spurious wakeups from the
>    pthread_cond_timedwait() or pthread_cond_wait() functions may
>    occur. Since the return from pthread_cond_timedwait() or
>    pthread_cond_wait() does not imply anything about the value of this
>    predicate, the predicate should be re-evaluated upon such return.
> 
> I have no "Boolean predicate" (which presumably comprises the
> "user-defined variables outside the condition variable" I mentioned
> above), and I don't want spurious wakeups, so a pthreads condition
> variable would appear to be the wrong thing to use.
> 
> Is there something like an "event flag" similar to a condition
> variable without spurious wakeup problem and without the extra
> overhead of the mutex and "Boolean predicate".
> 
> Or am I expected to build my own "event flag" using the aforesaid 
> "boolean predicate" just to avoid the spurious wakeup problem?  [I'm
> guessing this is the case...]
> 


-- 
Alan McKinnon
alan.mckinnon@gmail.com

[gentoo-user] Re: OT: pthreads condition variable/mutex question

[Grant Edwards]

On 2014-08-13, Alec Ten Harmsel <alec@alectenharmsel.com> wrote:

>> I may have to stick with sockets when I want to block until some event
>> happens.
>
> To be clear, do you want to block or sleep/yield until an event
> happens?

I don't see the difference -- isn't that what a blocking call does:
sleep/yield until some event happens?

> I'm sorry for not being too helpful. Just one last question: Can you 
> describe what exactly your code is supposed to do, or is it something
> that you can't talk about because it's a work thing? I don't care 
> either way, but I'm just curious because it seems you need to
> optimize quite a bit.

One process implements a communications protocol that is maintaining
communications links with a handful of slave devices connected to
serial ports running at baud rates up to 230400 baud (38400 is the
most common).  There are typically one (or maybe two) hundred messages
per second being exchanged with each slave device.  I'll call that
process the server.

There are other client processes that want to access the slaves and
the inforamation being received from them. Some of the clients just
want to do low-frequency transactions for configuration/diagnostic
purposes, and Unix domain sockets work fine for that.

Other clients may want to wake up every time a certain high frequency
event happens.  That's where I'm trying to use condition variables.

Other clients will periodically (typically once every 5-50 ms) want to
see the most recent copy of a particular received message.  I'm
thinking about using shared memory and rwlocks for that, but I haven't
figured out how to deal with the case where a process aborts while
holding a lock.

-- 
Grant Edwards               grant.b.edwards        Yow! Kids, don't gross me
                                  at               off ... "Adventures with
                              gmail.com            MENTAL HYGIENE" can be
                                                   carried too FAR!

[gentoo-user] Re: OT: pthreads condition variable/mutex question

[Grant Edwards]

On 2014-08-13, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 13/08/2014 19:21, Grant Edwards wrote:

>> This is not Gentoo specific, and while I'm doing my prototyping and
>> development on a Gentoo system, the eventual target is not going to
>> be running Gentoo -- so feel free to ignore this thread or throw
>> things at me.
>> 
>> I'm trying to figure out how to synchronize threads which may be in
>> different processes.  Basically, I want thread A to be able to wake up
>> any number of other threads B, C, D, ... who are all blocking until A
>> says "go" (and who may or may not be in other processes).
>
> Sounds like you a "one talker - many listeners" model.

For that particular case, yes.

> Have you considered a really simple solution like dbus?

I don't know if I would call dbus "really simple".  :)

My current implementation uses Unix domain sockets (which is what dbus
usually uses, isn't it?), and I'm trying to figure out how to reduce
overhead and latency.  dbus would add even more overhead (it has code
to deal with byte ordering, serial/cookies, and various other features
and abstractions).  I'm not sure it's really practical for
high-frequency events (e.g 100-200 events per second).

-- 
Grant Edwards               grant.b.edwards        Yow! I've got a COUSIN
                                  at               who works in the GARMENT
                              gmail.com            DISTRICT ...

Re: [gentoo-user] Re: OT: pthreads condition variable/mutex question

[Alan McKinnon]

On 13/08/2014 22:28, Grant Edwards wrote:
>> Have you considered a really simple solution like dbus?
> I don't know if I would call dbus "really simple".  :)
> 
> My current implementation uses Unix domain sockets (which is what dbus
> usually uses, isn't it?), and I'm trying to figure out how to reduce
> overhead and latency.  dbus would add even more overhead (it has code
> to deal with byte ordering, serial/cookies, and various other features
> and abstractions).  I'm not sure it's really practical for
> high-frequency events (e.g 100-200 events per second).


To be honest, I'm somewhat out of my depth in this area. Sure, I can
discuss it but you want details, and that I can't really provide.

I mentioned dbus really just as a way to encourage you to think out of
the box a little. Maybe it fits your needs, maybe not. But at least you
would have given it some thought :-)

Reading through the rest of the thread, perhaps dbus isn't suitable for
this. You have lots of messages and they don't seem to be text-based.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: OT: pthreads condition variable/mutex question

[Alec Ten Harmsel]

On Wed 13 Aug 2014 04:19:19 PM EDT, Grant Edwards wrote:
> On 2014-08-13, Alec Ten Harmsel <alec@alectenharmsel.com> wrote:
>
>>> I may have to stick with sockets when I want to block until some event
>>> happens.
>>
>> To be clear, do you want to block or sleep/yield until an event
>> happens?
>
> I don't see the difference -- isn't that what a blocking call does:
> sleep/yield until some event happens?

My bad, got a little confused. Trying to think a little too hard ;) 
Also, like Mr.McKinnon mentioned above, I'm a little out of my depth as 
well, although I have written some fancy multi-threaded computer vision 
code before.

>> I'm sorry for not being too helpful. Just one last question: Can you
>> describe what exactly your code is supposed to do, or is it something
>> that you can't talk about because it's a work thing? I don't care
>> either way, but I'm just curious because it seems you need to
>> optimize quite a bit.
>
> One process implements a communications protocol that is maintaining
> communications links with a handful of slave devices connected to
> serial ports running at baud rates up to 230400 baud (38400 is the
> most common).  There are typically one (or maybe two) hundred messages
> per second being exchanged with each slave device.  I'll call that
> process the server.

Alright, following you so far.

> There are other client processes that want to access the slaves and
> the inforamation being received from them. Some of the clients just
> want to do low-frequency transactions for configuration/diagnostic
> purposes, and Unix domain sockets work fine for that.

Seems legit.

> Other clients may want to wake up every time a certain high frequency
> event happens.  That's where I'm trying to use condition variables.

I think you should step away from the fancy decoupling and process the 
high frequency events in a separate thread in the server; if passing 
them over sockets is too much overhead, I don't see any other way to do 
this. I don't want to be extremely critical, but if the whole point of 
the server process is *only* the message passing, what does that gain 
you? Why bother only passing messages in the server? Might as well do 
some processing as well.

> Other clients will periodically (typically once every 5-50 ms) want to
> see the most recent copy of a particular received message.  I'm
> thinking about using shared memory and rwlocks for that, but I haven't
> figured out how to deal with the case where a process aborts while
> holding a lock.

Assuming there's not too many different message types, you could cache 
the most recent of each type in a hash map in the server process, 
making retrieving the most recent copy just a query to the server and 
no problems with locking on the client side.

In summation, I think you should process high-frequency messages in the 
server process, cache most recent messages in the server, and you can 
keep a separate client for checking the most recent messages and a 
separate client for the occasional debugging/configuration tasks.

IIRC, I'm pretty sure that's what made nginx so much more performant 
than Apache initially; instead of launching a new thread for every 
request (Apache did/does this) or doing some sort of fancy message 
passing, nginx pushes requests into a queue in shared memory and a 
thread pool processes the queue.

Alec

Re: [gentoo-user] Re: OT: pthreads condition variable/mutex question

[J. Roeleveld]

On Wednesday, August 13, 2014 08:19:19 PM Grant Edwards wrote:
> On 2014-08-13, Alec Ten Harmsel <alec@alectenharmsel.com> wrote:
> >> I may have to stick with sockets when I want to block until some event
> >> happens.
> > 
> > To be clear, do you want to block or sleep/yield until an event
> > happens?
> 
> I don't see the difference -- isn't that what a blocking call does:
> sleep/yield until some event happens?
> 
> > I'm sorry for not being too helpful. Just one last question: Can you
> > describe what exactly your code is supposed to do, or is it something
> > that you can't talk about because it's a work thing? I don't care
> > either way, but I'm just curious because it seems you need to
> > optimize quite a bit.
> 
> One process implements a communications protocol that is maintaining
> communications links with a handful of slave devices connected to
> serial ports running at baud rates up to 230400 baud (38400 is the
> most common).  There are typically one (or maybe two) hundred messages
> per second being exchanged with each slave device.  I'll call that
> process the server.

1 process (server)
Do you have seperate threads per slave device? Then each thread can handle the 
incoming messages when they arrive.

> There are other client processes that want to access the slaves and
> the inforamation being received from them. Some of the clients just
> want to do low-frequency transactions for configuration/diagnostic
> purposes, and Unix domain sockets work fine for that.

Have the client processes connect to a seperate thread for the communication. 
Then set up a shared memory between the relevant slave-device-thread and the 
client-process-thread(s)
I would create 2 different memory sets. One that is written only by the slave-
device-thread. And the other only by the client-process-thread(s)

That way, the client-process-threads can read that to get the current status.
And the slave-device can read the "commands" from the client-process-thread(s)

> Other clients may want to wake up every time a certain high frequency
> event happens.  That's where I'm trying to use condition variables.
> 
> Other clients will periodically (typically once every 5-50 ms) want to
> see the most recent copy of a particular received message.  I'm
> thinking about using shared memory and rwlocks for that, but I haven't
> figured out how to deal with the case where a process aborts while
> holding a lock.

This sounds similar to an application I wrote a long time ago to deal with 
connecting a desktop application to a telecomms server dealing with all the 
phone connections in a company.

--
Joost