VxWorks Network Buffer Fills Causing Loss of Communication

This KnowledgeBase covers a behavior of the VxWorks operating system involving a known limitation to how it handles network buffer allocation. Most networking problems with LabVIEW Real-Time Targets are not related to this issue but are due to High Priority Loops forcing the operating system to stop network communication to insure loop execution. Common solutions to that issue is to check CPU usage is not 100% and if possible slow the high priority loops and see if the issue is resolved. The following is a detailed discussion of the VxWorks buffer limitation, its causes, and possible workarounds.

Opening a TCP or UDP port in LabVIEW opens that port for both reading and writing. In rare cases, an open port that is only used for writing may accumulate incoming data (often UDP multicast packets) in the network receive buffer. This is an issue because VxWorks uses a single, shared buffer (divided into a fixed number of clusters of different sizes) for all network communication. If all clusters of a certain size become full, the network stack will hang and the Real-Time target will experience symptoms such as:

• All network communication is lost
  
• Unable to find the target in Measurement & Automation Explorer (MAX)
• Unable to ping the target from a host computer
  
• RTEXE hangs
  
• Depending on application, it may continue to run without error
• Rebooting the target returns it to normal operation

Note: This issue only affects VxWorks targets. Refer to KnowledgeBase 4LRA4IQ0: What Operating System is my Real-Time Controller Running and Why? to determine what operating system your Real-Time target uses.

To determine if you are experiencing this issue, perform the following steps:

1. View the console out by following the steps in KnowledgeBase 3M87HMD7: Usage of Console Out Switch on CompactRIO, Single-Board RIO, cFP-21xx, and cFP-22xx Controllers
2. Type inetstatShow into the terminal program (without quotes, case sensitive) to display a list of all open connections
   
3. Look at the Recv-Q column to see how many bytes are in the receive buffer for each connection.
   
4. Repeat Step 2 and 3 to determine if Recv-Q is increasing for any connection.

Please note that running out of any size of cluster can cause the network stack to hang. Therefore, there is no specific size cutoff for this condition.


Two tested workarounds that prevent the network receive buffer from overflowing are:

Read in Parallel from the Open Port
Using a TCP Read or UDP Read function in parallel with the Write function removes packets from the network receive buffer. A Read option needs to occur for each packet in the buffer, so the timeout ms function input (especially the default value of 25 seconds) is a good option for controlling timing. This workaround is best suited for continual write processes, and was benchmarked to only increase CPU load by .65% on a cRIO 9012.


Open, Read, Close
Closing the TCP or UDP port clears out the network receive buffer. Therefore, an "open, read, close" process can be used to ensure unwanted packets do not accumulate. Opening and closing actions can be decimated to occur every 5 or 10 writes, but this increases the time that packets could accumulate. This workaround is best suited to occasional write operations, as benchmarking indicated that a closed port cannot be reopened for up to 450 μs.

The Workarounds.zip folder attached below contains LabVIEW code that demonstrates both workarounds.