[gentoo-embedded] High Speed Serial Problem

[gentoo-embedded] High Speed Serial Problem

[David Relson]

G'day,

I'm porting a DOS application to Linux and have encountered problems
with the high speed serial devices from which the app receives data.

The hardware consists of a PC-104 board with a 486 compatible SOC
(system on a chip) providing 4 serial ports.  Ports 2 and 4 connect to
external devices that communicate at 115,200 baud.  The device sends a
42 byte record (containing a CRC), asserts a break to indicate end of
record, then does it again.  Every hour or so the app sends a message
to the device.  The message is sent immediately after the break is
recognized.

As an additional complication, the device connection is RS-485.  Being
half duplex, the code enables receive mode and transmit mode as needed.

The DOS app has an ISR to handle the communication.  Measured
throughput is approx 4,230 characters per second.  There is an error
rate of approx 2.5 errors per 1,000 records.

The Linux app has a separate thread for each of the 2 serial ports.
Data is received using select() and read().  Read is called with a 32
byte buffer.  As neither BRKINT nor IGNBRK is set, the input thread
recognizes 0xFF, 0x00, 0x00 sequences as breaks.  When the encoded
break is recognized, any pending message is sent.  

The Linux app receives approx 5,169 bytes per second and has an error
rate of 100 errors per 1,000 records.

Errors generally show up as dropped bytes.  Usually only 1 or 2 bytes.

The first (and most obvious) problem is that the Linux error rate is
40X higher rate than the DOS error rate.

 The second problem is that output from Linux isn't working.  After
 writing to the device, reading back doesn't show any change.  It's
 suspected that there's a timing issue, i.e. a timing window for
 receiving messages after the device sends a break.

I'm using a 2.6.29 kernel and the 486 SOC is running at 300mhz.
Processor speed is not believed to be the root because a newer 500mhz
version of the PC-104 board exhibits the same symptoms.

Any suggestions of what to do or where to look to solve the problems?

Thank you.

David

Re: [gentoo-embedded] High Speed Serial Problem

[Manuel Lauss]

David,

I suggest you post this question to linux-kernel@vger.kernel.org and
linux-serial@vger.kernel.org
instead.

One question though:  does your target have a real (physically real) uart chip?
There are some x86 systems out there which emulate a 8250 through SMM code,
which exhibit these kinds of failures at high baud rates.  Do lower
baudrates work?

Best regards,
       Manuel Lauss

Re: [gentoo-embedded] High Speed Serial Problem

[David Relson]

On Sun, 14 Mar 2010 17:34:50 +0100
Manuel Lauss wrote:

> David,
> 
> I suggest you post this question to linux-kernel@vger.kernel.org and
> linux-serial@vger.kernel.org
> instead.
> 
> One question though:  does your target have a real (physically real)
> uart chip? There are some x86 systems out there which emulate a 8250
> through SMM code, which exhibit these kinds of failures at high baud
> rates.  Do lower baudrates work?
> 
> Best regards,
>        Manuel Lauss

Hello Manuel,

The UARTs are real and work very very well with DOS (remember the 2.5
errors per 1000 records).

The high speed serial device has a fixed 115KB rate and is
transferring 4000+ characters per second.  With the fixed baud rate,
lower baudrates aren't usable, hence can't be tested.

David

Re: [gentoo-embedded] High Speed Serial Problem

[Peter Stuge]

David Relson wrote:
> Any suggestions of what to do or where to look to solve the
> problems?

Look to the hardware. Linux is not introducing these problems.

Start with tapping the RS-485 signal with two other receivers, and
look at what bytes are being sent when.

Linux can do plenty of buffering in it's RX interrupt so you're not
going to lose any incoming data.


//Peter

RE: [gentoo-embedded] High Speed Serial Problem

[Peter Bell]

G'day,

I'm porting a DOS application to Linux and have encountered problems
with the high speed serial devices from which the app receives data.

Any suggestions of what to do or where to look to solve the problems?


OK, this sounds like a timing problem to me - I assume you are controlling
the bus transceiver from your userland code?  The Linux serial handler has a
lot of buffering in it and it enables the FIFOs on the UART chip, so you may
only get an interrupt some time after the data has been received.

My first approach would be to get a scope and look at the signal lines (TX,
RX and transmit enable) and see if the timing looks plausible.  If you have
a spare I/O line you can play with, then set that up so that it pulses or
toggles when you detect a break - you can then see how long it's taking for
the code to recognize the condition.

Pete

Re: [gentoo-embedded] High Speed Serial Problem

[Bob Dunlop]

Hi,

> The Linux app has a separate thread for each of the 2 serial ports.
> Data is received using select() and read().  Read is called with a 32
> byte buffer.  As neither BRKINT nor IGNBRK is set, the input thread
> recognizes 0xFF, 0x00, 0x00 sequences as breaks.  When the encoded
> break is recognized, any pending message is sent.  

There should be no problem handling 115K2 data at full speed on a 300MHz
processor although some SOC UARTs are realy gnarly.  Using line break as
a protocol signalling element is so 1970s but again shouldn't be a problem.

What are your VMIN, VTIME settings ? They can have a big impact on read
performance/timing.  VMIN=VTIME=0 makes a mockery of using select for
example.

What size of return are you actually seeing from the read ?

If you are seeing reasonably full blocks, does you code handle the three char
break sequence spanning the read boundy ?  Sorry if this is teaching grandma
to suck eggs.

-- 
        Bob Dunlop

RE: [gentoo-embedded] High Speed Serial Problem

[Relson, David]

Hello Bob,

VMIN and VTIME are the default values of 1 and 0, respectively.  If I
understand correctly, these settings will provide data as rapidly as
possible.

At present, I am using a 256 byte buffer for reading data.  I've added
some statistics gathering.  Every 1000 reads, the program logs the total
bytes read and the maximum read.  The program was run for 30 seconds to
provide 6 statistics records

With _no_ processing of data, 1000 reads got from 27,545 to 28,520
chars.  The max read was 128 chars.  3 of the 1000 read samples had max
reads of 56 or 57 and 3 had max reads of 110, 126, and 128.

With copying of data to a buffer (which involves several function calls,
incrementing counters, etc), 1000 reads got from 27,187 to 28,241 chars
and each of the 6 sample sets had a max read of 110 to 128 bytes.

One would expect processing to increase the total bytes received, but
this was not the case.

The code _is_ handling the 3 char break sequence and the (undocumented)
doubling (escaping) of 0xFF chars (to distinguish them from break
sequences).

Regards,
 
David
-----Original Message-----
From: Bob Dunlop [mailto:bob.dunlop@xyzzy.org.uk] 
Sent: Monday, March 15, 2010 9:35 AM
To: gentoo-embedded@lists.gentoo.org
Subject: Re: [gentoo-embedded] High Speed Serial Problem

Hi,

> The Linux app has a separate thread for each of the 2 serial ports.
> Data is received using select() and read().  Read is called with a 32
> byte buffer.  As neither BRKINT nor IGNBRK is set, the input thread
> recognizes 0xFF, 0x00, 0x00 sequences as breaks.  When the encoded
> break is recognized, any pending message is sent.  

There should be no problem handling 115K2 data at full speed on a 300MHz
processor although some SOC UARTs are realy gnarly.  Using line break as
a protocol signalling element is so 1970s but again shouldn't be a
problem.

What are your VMIN, VTIME settings ? They can have a big impact on read
performance/timing.  VMIN=VTIME=0 makes a mockery of using select for
example.

What size of return are you actually seeing from the read ?

If you are seeing reasonably full blocks, does you code handle the three
char break sequence spanning the read boundy ?  Sorry if this is
teaching grandma to suck eggs.

-- 
        Bob Dunlop

[gentoo-embedded] SOLUTION (partial) was: High Speed Serial Problem

[David Relson]

On Mon, 15 Mar 2010 13:47:14 -0400
Relson, David wrote:

> Hello Bob,
> 
> VMIN and VTIME are the default values of 1 and 0, respectively.  If I
> understand correctly, these settings will provide data as rapidly as
> possible.
> 
> At present, I am using a 256 byte buffer for reading data.  I've added
> some statistics gathering.  Every 1000 reads, the program logs the
> total bytes read and the maximum read.  The program was run for 30
> seconds to provide 6 statistics records
> 
> With _no_ processing of data, 1000 reads got from 27,545 to 28,520
> chars.  The max read was 128 chars.  3 of the 1000 read samples had
> max reads of 56 or 57 and 3 had max reads of 110, 126, and 128.
> 
> With copying of data to a buffer (which involves several function
> calls, incrementing counters, etc), 1000 reads got from 27,187 to
> 28,241 chars and each of the 6 sample sets had a max read of 110 to
> 128 bytes.
> 
> One would expect processing to increase the total bytes received, but
> this was not the case.
> 
> The code _is_ handling the 3 char break sequence and the
> (undocumented) doubling (escaping) of 0xFF chars (to distinguish them
> from break sequences).
> 
> Regards,
>  
> David

The serial input problem has been solved!  While talking with the
project manager, it occurred to me to check whether all byte codes,
i.e. 0x00 to 0xFF, were being received.  I quickly found that 2 were
missing -- 0x11 and 0x13 -- which are the usual XON and XOFF
characters.  Simply setting termios.c_iflag=PARMK (rather than or'ing
the flag) solved the lost character input problem.

Even with a dedicated thread, the program is still receiving multiple
characters per read.  Calculating average and maximum byte counts for
each 10,000 reads, I've often seen maximums of 150 and have never seen
an average under 12.  These numbers occur with VMIN=1 and VTIME=0.  

Is there a setting so that the serial port driver will release
characters sooner (rather than later)?

My basic loop looks like:

   while (1) {
     char buffer[256];
     select(...)
     count = read(fd, buffer, sizeof(buffer))
     ... save characters.
   }

Regards,

David

Re: [gentoo-embedded] SOLUTION (partial) was: High Speed Serial Problem

[James Ausmus]

On Sat, Mar 20, 2010 at 12:02 PM, David Relson <relson@osagesoftware.com> wrote:
<snip>
> Even with a dedicated thread, the program is still receiving multiple
> characters per read.  Calculating average and maximum byte counts for
> each 10,000 reads, I've often seen maximums of 150 and have never seen
> an average under 12.  These numbers occur with VMIN=1 and VTIME=0.
>
> Is there a setting so that the serial port driver will release
> characters sooner (rather than later)?
<snip>

Wasn't involved in the other thread, so my apologies if this has
already been answered/stated, but are you using the realtime kernel
patches? If you run RT-Preempt, then you can set the priority of the
kernel thread that is handling the serial interrupt high than all the
other kernel threads, which should improve your latencies... See
https://rt.wiki.kernel.org/index.php/Main_Page for more info.

HTH-

James


> Regards,
>
> David
>
>

Re: [gentoo-embedded] High Speed Serial Problem

[wireless]

David Relson wrote:

> I'm porting a DOS application to Linux and have encountered problems
> with the high speed serial devices from which the app receives data.



Well here is link to a home brewed Serial Chip buffering
circuit, that may better allow you to look at the signals
and protocol interactions on your serial ports. I have found
the hardware ckt to help (stabalize) bit sniffing on serial
communications. The keyword is 'eavesdropping' as used
by embedded designers. What good about this project, is it
is an easy, practical work that allows you to learn the art
of eavesdropping on a com link, rs232C being the vintage
form. RS485 will come much easier into enlightenment for you....


You can download the tar file from here:

http://sourceforge.net/projects/serialsniffer/



If you look at the file: LinuxSPA.png (after unpacking the
tar file), you'll see the circuits; I used gimp, then you
can see the chips used to buffer the signals are the old
reliable 1488 and 1489.

I'm not sure about the max speed, I believe its around
500kbps, so you might have to find newer, faster
replacements chips for the 1488 and the 1489.....


When you go to test your hardware/software between the new
embedded linux system and a tried and proven dos based
system, this sort of buffering circuit is very valuable to
keep from "loading/distorting" the signals between the 2
different devices, particularly as the speed of
communications increases.


You can look at both ends of the serial communications
with 2 different windows on each side of the comm link, to
discerning transmission issues, such as timing sync, skew
etc etc.


hth,
James