Ethernet stops working on heavy loads

[Spekadyon]

Hi,

Since I got my wandboard quad two years ago, I face recurring problems with the ethernet interface. After some (variable) time, the ethernet interface becomes completely stuck. It appears to be working from a "software" point of view (no error logged, interface still "UP"), but no packets are emitted or received (impossible to emit or respond to ping, for example). The only way I found to recover from the freeze is to reboot the board.

I compile the kernel directly from the sources (rev. 4.5.0-rc6+, from https://github.com/RobertCNelson/armv7- ... atform.git). I observed the same behaviour with various revisions of the kernel from this repository (4.5.0-rc6+, 4.4, 4.0, and older ones). One reproducible way to trigger the freeze is to transfer large files (> GB) by NFS. Strangely, these events becomes really rare when the files are exchanged by rsync. The problem seems to be NFS-related in some obscure way.

I tried one kernel from the wandboard repository (3.14.28-01573-ge2df042-dirty, https://github.com/wandboard-org/linux.git) with less success: the interface did not freeze but files transferred by NFSv4 were invariably corrupted (and still no errors logged by the kernel). I also observed some weird behavior of rsync during transfers, especially during gentoo portage-tree sync. The transfer was interrupted for various reasons, for example

Code: Select all

 rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(1633)

I'm back on a CNelson kernel, and watchdog is taking care of the periodical reboots, but it is not very convenient...
Has anyone ever encountered this bug or has an idea to track it down?

[Tapani]

So you suspect this is a network issue, rather than an NFS issue?

When the network goes down, does bringing the interface up/down help?
How about disconnecting/reconnecting the ethernet cable?
Or using ethtool to change link speed?

When it comes to NFS ... maybe changes to rsize/wsize can help or using NFSv2/v3 instead of v4?

Can this issue be reproduced with any of our "official" software images?

[Spekadyon]

Thanks for your answer.
I don't really know where the problem comes from, but the ethernet interface is completely dead when this happens, not only NFS, that's why I suspect the problem to be a network issue.

I have already tried to reduce the speed of the network, from 1GB to 100M, but the problem still happens. It only takes longer for it to occurs…
Next time the freeze occurs, I'll try to plug/unplug the cable and try to up/down the interface.

I've already tried NFSv3, with the same results, as far as I can remember.

I'll also try an official image... Salvaging the kernel+modules from the last ubuntu 14.04 sd card image should be sufficient?

[ClayMontgomery]

I worked on a project about 2 years ago where we were trying to stream large videos to an i.MX6 using UDP and we saw that a lot of the packets were being dropped (according to ifconfig). We took this problem to Freescale where their factory engineers told us that the FIFOs built into the ethernet interface were not large enough to handle such heavy loads. We developed a work-around (at the app level) that reduced the problem substantially by reducing the load on the i.MX6's memory interface. I suggest you take a look at the specs on the ethernet interface. Hope this helps.

Regards, Clay