[gentoo-user] Reinventing the wheel

["Steven J. Long" <slong@rathaus.eclipse.co.uk>]

Alan McKinnon wrote:
> Steven J. Long wrote:
> > "POSIX 4: Programming for the Real World" (Gallmeister, 1995)
> > "UNIX Network Programming vol 2: Interprocess Communications" (Stevens, 1999)
> > 
> > More here:
> > https://foss.aueb.gr/posix/
> 
> I'll look into those, but do take note those books are 14 and 18 years old

Yes I am aware of that ;) The age is not the point. The content and its relevance are.
Further, you want Lewine 1994, first published 1991, if you're at all concerned about
portability, so make it 20 years; and that doesn't get you the deep insight that really
matters: read the books on the site in order if you want that, doing the exercises if
you want to actually implement stuff. And ask in #friendly-coders for some more books ;)

> - that's eternity in our world.

It's only really an eternity in compute-time, afaic. Calling something "innovation"
doesn't make it innovative. And it certainly doesn't make it an invention.

Nor is the speed with which fads and modern capitalism can move, any indication of
progress. Sure a lot's "happened". But not much has changed. "One True Way" is still
around, it's just mutated into "N+1 True Way", as we read something about "Plan to throw
one away, you will anyway" and we've "innovated" that to "Throw every version away" as
we don't know what an ABI promise means, and it's soo easy just to push a binary update,
when you don't have to deal with the consequent service failure.

> Basics never change, details do. Some features are here for the long
> haul and I doubt anything will really change them: pipes, named pipes,
> unix sockets and things of that ilk.

Which is why a 14-year old book describing them is still valid. There's actually a
decade of other books by Stevens before that, and APUE (on the site) was updated in 2005
by Rago (who was writing about SysV networking at the same time as the first UNP and
APUE.) Stevens himself died unfortunately: a _great_ loss.

If you look on the site, you'll see vol 1 of UNP was also updated. And that's where the
"eternity of changes" have really taken place: in remote networking, not in local IPC,
which is a solved problem. If you know the background. And a programmer always should
(or get another job, if learning it is too much.)

There are newer versions of both APUE and UNP vol 1, but I hear they're not as good.
So I'll get them when they're a bit cheaper, and I have some idle time.

> The real bugbear with IPC is people
> reinventing the wheel over and over and over to do simple messaging -

Which is exactly why it pays to know about the existing mechanisms instead of trying
to reinvent them. What you're saying here is exactly my point.

> writing little daemons that do very little except listen for a small
> number of messages from localhost and react to them.
> 
> Use a generic message bus for that!

There's no need. Most apps have a select/poll routine (or the equivalent) in any case,
especially the ones that respond to events, including pretty much all desktop apps. So
either you respond to the IPC channel in the main event routine, or you do some in a
thread. There's several mechanisms, and several methods to do different things. POSIX
gives you the standardised components: it's up to you to put them together.

wrt "a generic message bus" that's called a "message queue". And a programmer who finds
them "too difficult to use" is basically a nub. I've read people say that because it's
not an fd, it's not worth using. Which is completely amateur afaic: that's an awfully
small comfort-zone. By all means push for an eventfd in POSIX: but in the meantime, be
capable of more than one thing.

AF_UNIX datagram sockets are fine too, and are in fact what dbus uses. As I said, I never
actually criticised dbus itself: I'm fine with a desktop-session bus, to multiplex and
broadcast the various events of user interest, and I quite liked the protocol when I first
read up on it.

To use that as basic plumbing for other things, is a bad idea imo. All you're doing is
implementing a central point of failure, that has the additional borkage of being involved
with user desktop events. A complete encapsulation nightmare imo. But I don't really
care what other people do with their boxes: it doesn't affect me, so why should I? As
others have pointed out, dbus is certainly not required for a production server, and I
sincerely doubt it ever will be. There's too many experienced admins and coders who quietly
earn a living off clean systems, without ever getting involved in mailing-list debates.

> It fits nicely in the grand Unix tradition of do one job and do it well,

See below wrt filesystems.

> and few apps have passing messages around as their core function. Hand it off to the
> system, that's what it's there for.

Exactly: the operating-system. It's such a common problem, and it has wider implications
to do with scheduling and priority that come up around synchronisation, that it's all been
provided several times already.

> One day I might well do an audit of a typical server base system and
> count all the apps that have a hidden roll-your-own message process in
> place. I'm certain the results will be scary.

So long as you don't think using OS IPC counts as "roll-your-own." It doesn't: dbus is the
"roll-your-own", or rather "use someone else's purportedly ready-rolled", only its developers
want it to be the only mechanism, and used for everything, much like their plans for systemd.

I gave up reading code for a lot of stuff, about a year after I started using Gentoo. I
found it too disappointing, especially for apps I loved using. The real problem, imo,
is that people don't pass around, review and patch code, in the way that they used to in
the early days: every system was built from source, as Gentoo is now. The trend more and
more is to obfuscated idiot-boxes where "everything is taken care of for you"; until you
need something they didn't think of, when it gets tricky. The idiot-software keeps getting
in your way, thinking it knows what you need better than you do, there's no plain-text config
in sight, data is all kept in "efficient" binary (aka proprietary) format so it's impossible
to debug, and the 'devs' keep whining about "use-cases" and wanting to question your motives,
instead of helping you get the job done and out the door.

People can deride me as living in some fantasy world, dreaming of days when there were only
500 machines or w/e. It really doesn't bother me: to me it's like deriding the Calculus
because it was derived in the 18th Century, and saying Arabic numerals are "too complex
for the average user" so let's "simplify" and use IIIIII for 6. It merely displays a lack of
fundamental knowledge, about the basics of CS: which is no longer what's taught in the IT
Marketing depts of today. And fretting about ignorant opinion of the task in front of you is a
recipe for failure.

In this case, reinventing the basic OS is like saying let's have a userland daemon to mediate
access to the hard-drive. "There's far too many apps that are basically small daemons that do
little but wait and respond to filesystem activity, reacting to the contents of the files. Use
a generic daemon for that!"

You see my point? That's what libc is for, and it has an awful lot more specified in POSIX
than Standard C.

We all owe a deep debt to RMS for POSIX, and its public availability; without it Linux would
never have got off the ground.

"Good Programming is not learned from generalities, but by seeing how significant programs
 can be made clean, easy to read, easy to maintain and modify, human-engineered, efficient
 and reliable, by the application of common sense and good programming practices.
 Careful study and imitation of good programs leads to better writing."
  -- Kernighan & Plauger (1978)

It's a shame that so many youngsters think "new" == "good", and don't want to learn from the
people who were in their shoes 30-50 years ago, and passed down the craft. (What else would
they have done with it? ;) Personally I love reading the old simple stuff (and I mean really
basic), as it keeps me grounded. But there's an awful lot more that isn't so simple, and the
reason it's still around is obvious: "Damn, that shit works". Much like calculus still works.

Regards,
igli
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)