One of the main reasons I chose the ML605 over a less expensive spartan6 development board was the built-in hard tri-mode ethernet MAC. The ML605 supports a variety of links between the MAC and the M88E1111 PHY on the board. Different interface choices required a bit more work. There is also a “ten-bit interface” and “reduced ten-bit interface” TBI/RTBI supported by the PHY, but these aren’t options in xps_ll_temac.
EDK Setup
Base System Builder for LL TEMAC
Setting up the hard temac can be done by adding the xps_ll_temac to an EDK project. The base system builder allows this to be done easily. There are two interfaces to software — the lightweight local-link fifo, and the high performance local-link dma interface.
The DMA interface allows for interrupt coalescing, and affords the larger memory advantage of DDR3. Combined, these allow the microblaze to buffer up several packets and then receive one interrupt to determine what to do with the group of packets. This increases latency, as a packet might not be processed immediately. It also improves bandwidth and CPU utilization as the CPU will not need as many context switches as if it had been interrupted on a more frequent basis.
The base system builder can do either. The next question is which interface to use between the MAC and the PHY. For the ML605 and hobbyist, there really isn’t a need to use anything other than GMII. The pins are already there, and the base system builder will already have the UCF set up correctly.
MIIM
In addition to the GMII/RGMII/SGMII signals, there is a management bus. Two wires, MDC and MDIO. It’s a simple bidirectional, synchronous serial bus that is used to access control and status information on the PHY. In the case of SGMII (or 1000BASE-X), the EMAC also has a register space. The EMAC will intercept accesses to the device numbered C_PHY_ADDR. Thus the configuration of the ll_temac should use a number other than 7 (the M88E1111’s address on the ML605).
As an individual, its a bit more difficult to use this bus. There isn’t currently a publicly available datasheet that describes the registers for the M88E1111 and how to use them. Such data is available if your company signs an NDA with Marvell. Luckily, many of the useful registers are either completely or partially defined in various appnotes and Linux kernel modules. There’s probably a few more defined in other open-source OS’s, like FreeBSD or OpenSolaris. Some others can be estimated by looking for the register space of PHYs in the same family. For example, you might find a dual-PHY or quad-PHY version that happens to have an OSS driver.
Connection Using GMII
GMII is intended to be a robust bus — less influenced by signal integrity issues. It’s main issue is the number of IO signals, 22 (23 with the MII tx clock). The bus uses an 8b interface running at 125MHz. This interface is the easiest to get to work. It also is nice because the lower clock rate makes it easier to analyze using a moderately high speed oscilloscope or logic analyzer.
Beyond that, when using the base system builder, the UCF file will already have the correct constraints. Here are the UCF constraints for the GMII interface. I suggest this for most hobbyists, as its the easiest to get working. RGMII and SGMII are used to allow PCB and backplane/cabling density improvements, as well as to free up device pins on the FPGA. With the ML605, you don’t get any advantage using RGMII/SGMII over GMII as the traces are already routed, and the pins already used.
Connection Using RGMII
RGMII Ports
Getting RGMII to work was the most difficult of the three IO busses I tested. GMII had sane defaults, and with SGMII most of the work is done with hard-IP. With RGMII, the user has to do a bit of configuration to get the IO to work. Worse yet, there is conflicting advice and more than one way to do this. EDK also requires that the RGMII ports are made external, as in the picture.
RGMII v1.3 was defined by HP. It specifies that both devices will send data and clock at the same instant. The clock trace is made longer in order to provide setup/hold time. RGMII v2 allows the devices to provide internal delays. Thus there is a mix of RGMII v1.3 and RGMII v2 settings/advice. Both the PHY and the FPGA can delay clock lines, Xilinx has mismatched traces to the PHY for the rx data, and Xilinx allows delays of the data lines on a per-line basis.
In a strange twist, I was able to get the Rx path to work fairly easily. There is an issue in that different appnotes provide different advice on the OFFSET-IN constraints. Thus the input delay taps I used don’t match every appnote. Still, it was easy enough to get the Rx path to work with good estimates of what the OFFSET-IN constraints should be.
The Tx path was different. I’m still a bit concerned about it as well. The MAC was receiving ARP requests, and attempting to reply. These replies appeared to be transmitted with errors. I eventually found an appnote that suggested LOC’ing 6 IDELAYCTRLs in EDK, as well as using an ODELAY of six for TXC0. I’m suspicious because six seems very low, I would have expected a value in the 20-25 range. It turns out that LOC’ing the six IDELAYCTRLs also gives three errors, so I LOC’d (using EDK):
IDELAYCTRL_X2Y1-IDELAYCTRL_X2Y3-IDELAYCTRL_X1Y3
This is my current, working UCF for RGMII. I don’t suggest using it blindly though, there’s still too much unexplained about the tx path for me to recommend it. Further, it assumes that the FPGA will perform the tx/rx delays. The default kernel module isn’t set up for that when using the “M88E1111 with RGMII” option.
Connection Using SGMII
SGMII Settings
SGMII is a serial bus — two differential pairs running at 1250MHz. The 8b data is encoded in a nonsystematic fashion into 10b codewords. This is why the interface runs at a higher rate than the minimum 1000MHz. The 8b/10b coding is chosen to provide a large number of transitions, and provides unique synchronization sequences. SGMII reduces the number of connections to just four.
One of the issues with SGMII is that some PHYs will use a very low transmit strength for the SGMII connection from the PHY to the MAC. In such cases, the data will have bit errors unless the correct MDIO-accessible registers are set. For the ML605, the PHY is close to the FPGA, so this isn’t an issue.
The EMAC has a “SGMII PHY” register space. The EMAC will intercept accesses to this devices PHY address. This is why the C_PHY_ADDR is not set to the same value as the M88E1111 PHY’s address of 7. This is because the SGMII link between the MAC and the PHY has its own configuration and autonegotiation parameters.
I had issues with ISE12.2 and SGMII. There was an odd build error where IBUFDS_GTXE1_X0Y8 wasn’t able to route to BUFR_X1Y9. This is a valid statement, but I’m not sure why the tools chose these two locations. It was easy enough to fix by LOC’ing the BUFR to BUFR_X2Y9, which is in the same clock region.
The UCF was also updated to provide LOC and clock constraints for the GTXE1. The GTX tile doesn’t need to be LOC’d, as these IO pins are directly connected to the GTX tile, so there is only one place to place the tile. I also added some clock constraints. Here is the UCF constraints for the SGMII interface.
Updated Kernel Modules
I made some changes to the kernel module to get things to run smoothly. I added an M88E1111 with SGMII config option. Luckily, nothing specials needed to be done to set up the PHY, other than use the correct config pins on the M88E1111. This was true for both SGMII and GMII.
For the SGMII speed detection, I decided to use the speed detection logic in the SGMII-PHY. This gives me access to determine if the SGMII link is established and autonegotiation of the link is complete. This can provide valuable debug information.
I’m mixed about RGMII. The default kernel module enables rx/tx delay, as well as used generic speed detection. It also issued a soft-reset within phy_setup. This was a bit interesting because the speed is detected directly after phy_setup, and the reset would cause the link to go down, then come back up, which re-calls the speed detection. In my module, I eventually just decided to use the default RGMII settings, and use the phy-specific status register (same as GMII uses) for speed detection.
The modified drivers/net/xilinx_lltemac/xlltemac_main.c and drivers/net/Kconfig are included.