OHW2013 workshop

An open source PCIe device virtualization framework

Plan

• Context and objectives • Design and implementation • Future directions • Questions

Context - ESRF and the ISDD electronic laboratory

The ESRF is an XRAY light source for Europe located in Grenoble, France. The ISDD electronic laboratory mission is to develop and investigate XRAY instrumentation electronics.

Context - RASHPA

• • • •

high bandwidth data acquisition framework for 2D XRAY detectors Intel X86 64 workstation KC705 prototype board, PCIe over cable see ICALEPCS poster session

Context - DCORE

• generic data acquisition and control boards • Intel ATOM CPU • Xilinx Spartan6 FPGA

Context - DUB

• project specific readout electronics • Intel ATOM CPU • Xilinx Virtex6 FPGA

VPCIe - rationale

The above boards rely on PCIe as the default CPU to FPGA communication link. VPCIe is a framework made to virtualize these platforms on a standard desktop PC.

VPCIe - objectives

• CPU software must run unmodified (including drivers) • few modifications allowed for functional VHDL simulation • run multiple PCIe devices concurrently • performances not critical

VPCIe - benefits

• hardware software codesign • reduce development cycle time • platform scaling (multiple PCIe devices) • test with different CPU architectures (INTEL, ARM ...) • investigate unavailable technologies (NVM EXPRESS ...) • testing (fault injection ...)

VPCIe - concepts

Applications on a host machine access hardware via interfaces. By instrumenting these interfaces, one can redirect the accesses to a software implementing the device. The device is said to be virtualized.

VPCIe - building blocks

VPCIe relies on opensource projects QEMU • http://wiki.qemu.org/Main Page • architecture emulator (X86 64, ARM ...) • used to trap PCIe hardware accesses

GHDL • http://ghdl.free.fr • VHDL frontend for GCC • used to implement device in VHDL

VPCIe - CPU virtualization

Full featured LINUX system • runs in a QEMU virtual machine • PCIe accesses are trapped and sent over TCP to the devices • PCIe forwarder is available as a QEMU patch I maintainers contacted for a merge

VPCIe - device virtualization

Virtual devices • can be implemented in C or VHDL I I

GHDL is used to compile VHDL into a native executable a glue interfaces the executable to the VPCIe runtime

• run as a LINUX processes, can be duplicated at will • PCIe made simple, focus on device logic I but close to the XILINX PCIe transaction layer

VPCIe - implementation

VPCIe - EBONE based example

EBONE is a PCIe centric FPGA core interconnect developped at the ESRF and recently released on OHR (http://www.ohwr.org/projects/e-bone). Excluding the PCIe layer, most of the VHDL remains unchanged in a typical EBONE based design.

VPCIe - virtualized device VHDL interface

entity endpoint i s port ( rst : in std ulogic ; clk : in std ulogic ; req req req req req

en : wr : bar : addr : data :

out out out out out

std std std std std

ulogic ulogic ulogic ulogic ulogic

; ; v e c t o r ( p c i e . BAR WIDTH − 1 downto 0 ) ; v e c t o r ( p c i e . ADDR WIDTH − 1 downto 0 ) ; v e c t o r ( p c i e . DATA WIDTH − 1 downto 0 ) ;

rep en : rep data :

in std ulogic ; i n s t d u l o g i c v e c t o r ( p c i e . DATA WIDTH − 1 downto 0 ) ;

mwr mwr mwr mwr

in in in in

en : addr : data : size :

msi en : ); end e n t i t y ;

std std std std

ulogic ulogic ulogic ulogic

in std ulogic

; v e c t o r ( p c i e . ADDR WIDTH − 1 downto 0 ) ; v e c t o r ( p c i e . PAYLOAD WIDTH − 1 downto 0 ) ; v e c t o r ( p c i e . SIZE WIDTH − 1 downto 0 ) ;

VPCIe - virtualized device C interface

/∗ r u n t i m e i n i t i a l i z a t i o n ∗/ int pcie init net ( pcie dev t ∗, . . . ) ; int p c i e f i n i ( pcie dev t ∗); int pcie loop ( pcie dev t ∗); /∗ m i s c c o n f i g b y t e a c c e s s o r s ∗/ int pcie set deviceid ( pcie dev t ∗, . . . ) ; int pcie set vendorid ( pcie dev t ∗, . . . ) ; /∗ PCIe typedef typedef int

BAR a c c e s s h a n d l e r s ∗/ v o i d (∗ p c i e r e a d f n t ) ( u i n t 6 4 t , v o i d ∗ , s i z e t , v o i d ∗ ) ; v o i d (∗ p c i e w r i t e f n t ) ( u i n t 6 4 t , c o n s t v o i d ∗ , s i z e t , v o i d ∗ ) ; pcie set bar ( pcie dev t ∗, . . . , pcie readfn t , pcie writefn t ,

/∗ h o s t memory r e a d w r i t e o p e r a t i o n s ∗/ int pcie write host mem ( pcie dev t ∗, uint64 t , s i z e t ∗); int pcie read host mem ( pcie dev t ∗, uint64 t , s i z e t ∗); /∗ s e n d an MSI ∗/ int pcie send msi ( pcie dev t ∗);

...);

VPCIe - future directions

• GHDL no longer maintained • merge PCIe forwarder in QEMU • XILINX AXI stream compatible PCIe layer • reimplement PCIe forwarding and protocol • NVM Express integration testing • licensing

VPCIe - availability

VPCIe source is available online • https://github.com/texane/vpcie • documentation still poor, but clear examples • feedbacks or contributions are welcome

VPCIe - questions

Thanks for your attention. Any question?