Now it worked again. It even survived a restart. I already was able to remove and replug the cable while pinging and he survived that. Then it was gone again. I restarted the system, replugged the cable and it was back after some restarts. Seems to be some hardware problem and not a problem with the drivers itself
Sigh.
Are there any verbosity settings I can enable in this driver? Or some output to the log which shows init/negotiation details?