Ethernet stops working on heavy loads

This is the place for software discussions not having a dedicated subforum

Ethernet stops working on heavy loads

Postby Spekadyon » Thu Mar 03, 2016 12:55 pm

Hi,

Since I got my wandboard quad two years ago, I face recurring problems with the ethernet interface. After some (variable) time, the ethernet interface becomes completely stuck. It appears to be working from a "software" point of view (no error logged, interface still "UP"), but no packets are emitted or received (impossible to emit or respond to ping, for example). The only way I found to recover from the freeze is to reboot the board.

I compile the kernel directly from the sources (rev. 4.5.0-rc6+, from https://github.com/RobertCNelson/armv7- ... atform.git). I observed the same behaviour with various revisions of the kernel from this repository (4.5.0-rc6+, 4.4, 4.0, and older ones). One reproducible way to trigger the freeze is to transfer large files (> GB) by NFS. Strangely, these events becomes really rare when the files are exchanged by rsync. The problem seems to be NFS-related in some obscure way.

I tried one kernel from the wandboard repository (3.14.28-01573-ge2df042-dirty, https://github.com/wandboard-org/linux.git) with less success: the interface did not freeze but files transferred by NFSv4 were invariably corrupted (and still no errors logged by the kernel). I also observed some weird behavior of rsync during transfers, especially during gentoo portage-tree sync. The transfer was interrupted for various reasons, for example
Code: Select all
 rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(1633)


I'm back on a CNelson kernel, and watchdog is taking care of the periodical reboots, but it is not very convenient...
Has anyone ever encountered this bug or has an idea to track it down?
Spekadyon
 
Posts: 3
Joined: Thu Mar 03, 2016 11:10 am

Re: Ethernet stops working on heavy loads

Postby Tapani » Fri Mar 04, 2016 10:37 am

So you suspect this is a network issue, rather than an NFS issue?

When the network goes down, does bringing the interface up/down help?
How about disconnecting/reconnecting the ethernet cable?
Or using ethtool to change link speed?

When it comes to NFS ... maybe changes to rsize/wsize can help or using NFSv2/v3 instead of v4?

Can this issue be reproduced with any of our "official" software images?
Tapani
Site Admin
 
Posts: 705
Joined: Tue Aug 27, 2013 8:32 am

Re: Ethernet stops working on heavy loads

Postby Spekadyon » Fri Mar 04, 2016 10:56 am

Thanks for your answer.
I don't really know where the problem comes from, but the ethernet interface is completely dead when this happens, not only NFS, that's why I suspect the problem to be a network issue.

I have already tried to reduce the speed of the network, from 1GB to 100M, but the problem still happens. It only takes longer for it to occurs…
Next time the freeze occurs, I'll try to plug/unplug the cable and try to up/down the interface.

I've already tried NFSv3, with the same results, as far as I can remember.

I'll also try an official image... Salvaging the kernel+modules from the last ubuntu 14.04 sd card image should be sufficient?
Spekadyon
 
Posts: 3
Joined: Thu Mar 03, 2016 11:10 am

Re: Ethernet stops working on heavy loads

Postby ClayMontgomery » Thu Mar 24, 2016 3:50 pm

I worked on a project about 2 years ago where we were trying to stream large videos to an i.MX6 using UDP and we saw that a lot of the packets were being dropped (according to ifconfig). We took this problem to Freescale where their factory engineers told us that the FIFOs built into the ethernet interface were not large enough to handle such heavy loads. We developed a work-around (at the app level) that reduced the problem substantially by reducing the load on the i.MX6's memory interface. I suggest you take a look at the specs on the ethernet interface. Hope this helps.

Regards, Clay
ClayMontgomery
 
Posts: 70
Joined: Tue Nov 19, 2013 9:14 pm


Return to Software - General

Who is online

Users browsing this forum: Bing [Bot] and 15 guests