j-marjanovic.io - Jan Marjanovic

Exploring the PS-PL AXI interfaces on Zynq UltraScale+ MPSoC

2021-12-29T18:00:00+01:00

I recently held a presentation at the 10th MicroTCA Workshop for Industry and Research, where I presented some hardware we have developed, discussed the advantages of using SoCs, and showed a couple of examples where we successfully leveraged the features of these devices.

At my day job the dataflow through the system is relatively simple, the data is captured in large chunks and the direction is always well defined, i.e. for data acquisition systems from FPGA to CPU and then into the network/storage.

On the other hand, there are a lot of applications that can benefit from a tighter coupling between the CPU and the FPGA, for example: HW-accelerated compression and decompression, HW-accelerated regex, HW-accelerated graph traversal, machine learning accelerators,...

This is why I have decided to explore all ports available in Zynq US+ MPSoC in my free time and to describe this adventure in this blog post.

Introduction

Xilinx Zynq UltraScale+ MPSoC provides four different types of interfaces between the so-called Processing System (PS) and Programmable Logic (PL), leveraging the wide variety of different protocols standardized in Advanced Microcontroller Bus Architecture.

In this blog post I will explore the performance characteristics of three different interfaces:

Accelerator Coherency Port interface
High-Performance Coherent interface
High-Performance interface

AXI Coherency Extension port is the most interesting of all and it deserves a dedicated blog post.

The purpose of this blog post is to gather information on how to make the interfaces work correctly and to measure the performance in different scenarios. The information on various interfaces is scattered across different documents (Zynq UltraScale+ Device Technical Reference Manual, CoreLink CCI-400 Cache Coherent Interconnect Technical Reference Manual, and Arm Cortex-A53 MPCore Processor Technical Reference Manual), Xilinx wiki pages, Xilinx Answer Records and various forum threads and GitHub issues.

PL-PS Interfaces

Shown in the figure below are the interfaces between PS and PL.

In this blog post I will focus on the following three interfaces:

High-Performance interface

This interface connects directly to the DDR Memory Subsystem and completely bypasses the Cache Coherent Interconnect and the APU. According to UG1085, this interface is ideal for large datasets. The software needs to bypass the cache when accessing the data.

High-Performance Coherent interface

This interface is connected to a port on CCI-400 interconnect. When configured accordingly the memory transactions are communicated to the APU, providing tighter integration with software.

AR 69446 mentions that:

The HPC ports are preferable to the ACP port in most applications as they provide higher bandwidth and do not disturb the contents of the processor L2 cache.

Accelerator Coherency Port interface

Cortex-A53 TRM describes the ACP interface in the following way:

The ACP is provided to reduce software cache maintenance operations when sharing memory regions with other masters, and to allow other masters to allocate data into the L2 cache.

According to UG1085:

[...] ACP is optimal for medium-grain acceleration, such as a block-level crypto accelerator and video macro-block level processing.

Motivation

Cache-aware interfaces allow tighter integration between SW and HW which is of particular interest for accelerating workloads with FPGA. Instead of explicitly copying the data to the FPGA, doing the computation in the FPGA, and copying data back from the FPGA, with cache-aware interfaces we can modify the data directly in the program memory, thus skipping the unnecessary copying.

I have prepared a Jupyter notebook that demonstrates this approach. An AXI Proxy (explained in detail in the next chapter) is used to read from and to a Numpy array.

The screenshot below shows the most important part of the notebook, where the AXI Proxy is used to modify the content of the Numpy array from the FPGA side.

Test setup

Hardware

All measurements were performed on an Ultra96-V2 board. The board contains an XCZU3EG, 2GB of DDR4 memory connected to the Processing System, and not much more.

The blue LED near the USB connector is used to indicate activity on AXI PL-PS ports:

AXI Proxy

AXI Proxy IP acts (as the name suggests) as a proxy between an AXI4-Lite subordinate port and an AXI4 manager port. With this IP we can measure read and write latency on all aforementioned ports.

The subordinate port provides registers where the software can prepare the data to be written, retrieve the data which was read, start read and write transactions and measure the time an individual transaction took. The transaction time is measured in clock cycles between an address being provided (AxVALID going high) and a full response is received. To match the requirements of the ACP interface, all transactions are 64-byte long (4 beats, 128-bit wide). The software has also the possibility to set some of the AXI side-band signals which affect the caching properties: AxCACHE, AxPROT, and AxUSER.

Shown in the figure below is the Vivado block diagram used to perform the tests with AXI Proxy. There are three instances of the IP, each connected to one of the ports on the Zynq MPSoC block. System ILA is used to provide additional visibility of the connections between AXI Proxy and PL-PS ports on the Zynq UltraScale+ MPSoC block.

AXI Traffic Generator

AXI Traffic Generator can generate a sequence of AXI bursts with an incrementing address and a known pattern, which simulates behavior of a DMA. With this IP we can measure the read and write throughput of transactions of different sizes.

Also here each AXI burst contains 64 bytes (4 beats, 128-bit). The IP measures the number of clock cycles the entire transfer took.

Shown in the figure below is the Vivado block diagram used to perform the tests with Jan's AXI Traffic Generator. As with the AXI Proxy, there are three instances and a System ILA to observe the transactions.

Yocto layer

To generate the SD card image containing:

the FPGA bitstreams,
FPGA manager (to download the bitstreams),
device tree overlays (the description of FPGA),
programs to control the IPs,
the required Python libraries and
Jupyter notebooks

I have created a Yocto layer, available on my GitHub: meta-zynqmp-pl-ps-interfaces. The usage description is available in the README.md.

Configuration

u-dma-buf

For use with ACP and HPC port, the u-dma-buffer needs to have the dma-coherent flag set. This is achieved with an entry in the device tree.

HP port

HP port requires no special configuration on the FPGA side, and the u-dma-buf needs to be opened with the O_SYNC flag.

HPC port

Addressing

The HPC by default uses physical addressing, although the SMMU can be configured to use the virtual addressing, as mentioned in UG1085:

"Comparably, S_AXI_HPCx_FPD uses a virtual address [...]"

FSBL and reg.init

A couple of patches for the FSBL configure CCI in the correct configuration so that the transactions on the HPC port are considered shareable.

ACP port

The ACP interface is in detail described in ARM Cortex-A53 MPCore Processor Technical Reference Manual.

Since this interface provides direct access into L2 cache, and therefore the transfer needs to follow certain restrictions: to achieve the best performance the transfers should be 64 bytes long (one cache line), and AxCACHE and AxPROT signals need to be set to certain values.

regs.init

regs.init is used to write to the APU Configuration Register (LPD_SLCR), which enables the broadcasting of the transactions towards the CCI, as described on Xilinx Wiki on Cache Coherence.

Measurements

Throughput/interface utilization

The throughput was measured with the AXI Traffic Generator, described in one of the previous chapters. What we measure here is not strictly throughput but interface utilization, i.e. in what percentage of the clock cycles was there a beat transmitted on the port. One can easily derive the throughput (in B/s) from the interface parameters (16-byte wide, running at 250 MHz). During the tests, only one interface was active at the time.

The entire measurement procedure is documented in a Jupyter notebook: 00-traffic-gen.ipynb
The FPGA project for Ultra96-V2 board is available here: example-13-axi-traffic-gen/ultra96v2_prj

The final results are presented in the graph below.

The HP port exhibits a typical behavior of a process where the start-up time plays a significant role - the interface utilization is lower for smaller transactions, but approaches an asymptotical limit at a certain size. All ports use the same settings (4 transactions in flight, 64 bytes per transaction) to make the comparison between different ports fair; this value is clearly too low for the long latencies of DDR4. To achieve higher throughput, one should increase the burst length.

CCI adds additional latency to the HPC port and the throughput on this port is even lower. Interestingly, once the transfer size is larger than the L2 cache, the interface utilization starts getting higher. This would indicate that the cache switches to write-through "mode" once it has seen a certain access pattern; the CCI400 TRM is quite vague in this regard.

The ACP benefits from the L1 and L2 caches and the interface utilization is the highest of all ports for small transfers (below 1 MB). Once the transfer size gets larger than the L2 cache size the interface utilization is severely affected.

Latency

The latency was measured with AXI Proxies. A single measurement sample was obtained with the following procedure:

the SW generates a random value and instructs AXI Proxy to write this random value into a shared buffer through one of the ports
the SW then reads the data from the buffer and compares it to what AXI Proxy has written
the SW then generates another random value and writes it directly to the buffer
the SW then instructs the AXI Proxy to read the data and then compares the value to the previously generated value

With this procedure we can check that the writes and reads are visible in both directions, and this procedure also mimics a typical usage of an FPGA accelerator. During the tests, only one interface was active at the time.

The entire measurement procedure is documented in a Jupyter notebook: 02-axi-proxy-on-repeat.ipynb
The FPGA project for Ultra96 board is available here: example-12-axi-proxy/ultra96v2_prj

The results of the latency measurements are presented in the graph below.

On the cached interfaces (ACP and HPC) we see, expectedly, very uniform latency values. On the other side, on the HP interface we see that the latency to the DDR4 is quite high and also quite variable, presumably because the transaction needs to be scheduled after some other transactions or memory refresh is currently being performed.

There is an interesting start-up behavior on the ACP and HPC interfaces. I am assuming that the L2 cache is an inclusive cache and contains all lines from the L1 cache. When a write request arrives at the L2 cache and the line is in Invalid state, the L2 can immediately decide that it can acknowledge the write transaction. When the write request arrives at the L1 cache, however, the L1 cache needs to first inform the L2 cache to invalidate the line, and only then proceed by acknowledging the write transaction.

Conclusion

In this blog post I have explored three different types of interfaces between PL and PS in Zynq UltraScale+ MPSoC. Different types of interfaces provide different trade-offs in terms of coupling between SW/HW, ease of use, throughput, and latency. Cache-aware interfaces can be used to access the data structures in running software and can therefore be used to seamlessly accelerate certain parts of the program.

Some edge cases were not explored (e.g. increasing the burst length for the interface utilization measurement), but I decided to skip those to make the blog post short(er). With Zynq US+ MPSoC boards being ubiquitous, this task is "left as an exercise to the reader".

I hope that this blog post, alongside the Vivado projects and the Yocto layer can be used as a reference on how to use those ports.

Appendix

System ILA waveforms

AXI Proxy

HP port

root@u96v2-sbc:~# app-axi-proxy --interface hp --use-osync
Udmabuf
  name = axi:udmabuf@0x0
  virt addr = 0xffff9c0c7000
  phys addr = 0x5e100000
  size = 33554432
  flags = 0x101002
UioDevice{number =  5, name =             AxiProxy, addr = 0xa0020000, size = 65536, note = hp}
UioDevice{number =  6, name =             AxiProxy, addr = 0xa0030000, size = 65536, note = hpc}
UioDevice{number =  4, name =             AxiProxy, addr = 0xa0040000, size = 65536, note = acp}
UioDevice{number =  1, name =             axi-pmon, addr = 0xfd0b0000, size = 65536, note = }
UioDevice{number =  2, name =             axi-pmon, addr = 0xfd490000, size = 65536, note = }
UioDevice{number =  0, name =             axi-pmon, addr = 0xffa00000, size = 65536, note = }
UioDevice{number =  3, name =             axi-pmon, addr = 0xffa10000, size = 65536, note = }
AxiProxy info:
  id reg = 0xa8122081
  version = 0.3.1
SW read, HW written:
  10, 10
  20, 20
  30, 30
  40, 40
readback, expected:
  10, 10
  20, 20
  30, 30
  40, 40
stats: dur_wr = 36, dur_rd = 70

HPC port

root@u96v2-sbc:~# app-axi-proxy --interface hpc --axi-cache 15 --axi-prot 2 --axi-user 1
Udmabuf
  name = axi:udmabuf@0x0
  virt addr = 0xffff931d8000
  phys addr = 0x5e100000
  size = 33554432
  flags = 0x2
UioDevice{number =  5, name =             AxiProxy, addr = 0xa0020000, size = 65536, note = hp}
UioDevice{number =  6, name =             AxiProxy, addr = 0xa0030000, size = 65536, note = hpc}
UioDevice{number =  4, name =             AxiProxy, addr = 0xa0040000, size = 65536, note = acp}
UioDevice{number =  1, name =             axi-pmon, addr = 0xfd0b0000, size = 65536, note = }
UioDevice{number =  2, name =             axi-pmon, addr = 0xfd490000, size = 65536, note = }
UioDevice{number =  0, name =             axi-pmon, addr = 0xffa00000, size = 65536, note = }
UioDevice{number =  3, name =             axi-pmon, addr = 0xffa10000, size = 65536, note = }
AxiProxy info:
  id reg = 0xa8122081
  version = 0.3.1
SW read, HW written:
  10, 10
  20, 20
  30, 30
  40, 40
readback, expected:
  10, 10
  20, 20
  30, 30
  40, 40
stats: dur_wr = 46, dur_rd = 40

ACP port

root@u96v2-sbc:~# app-axi-proxy --interface acp --axi-cache 15 --axi-prot 2 --axi-user 1
Udmabuf
  name = axi:udmabuf@0x0
  virt addr = 0xffffa1356000
  phys addr = 0x5e100000
  size = 33554432
  flags = 0x2
UioDevice{number =  5, name =             AxiProxy, addr = 0xa0020000, size = 65536, note = hp}
UioDevice{number =  6, name =             AxiProxy, addr = 0xa0030000, size = 65536, note = hpc}
UioDevice{number =  4, name =             AxiProxy, addr = 0xa0040000, size = 65536, note = acp}
UioDevice{number =  1, name =             axi-pmon, addr = 0xfd0b0000, size = 65536, note = }
UioDevice{number =  2, name =             axi-pmon, addr = 0xfd490000, size = 65536, note = }
UioDevice{number =  0, name =             axi-pmon, addr = 0xffa00000, size = 65536, note = }
UioDevice{number =  3, name =             axi-pmon, addr = 0xffa10000, size = 65536, note = }
AxiProxy info:
  id reg = 0xa8122081
  version = 0.3.1
SW read, HW written:
  10, 10
  20, 20
  30, 30
  40, 40
readback, expected:
  10, 10
  20, 20
  30, 30
  40, 40
stats: dur_wr = 25, dur_rd = 14

HP port

root@u96v2-sbc:~# app-axi-traffic-gen --count 32 --interface hp --use-osync
UioDevice{number =  4, name =        AxiTrafficGen, addr = 0xa0040000, size = 65536, note = acp}
UioDevice{number =  5, name =        AxiTrafficGen, addr = 0xa0050000, size = 65536, note = hp}
UioDevice{number =  6, name =        AxiTrafficGen, addr = 0xa0060000, size = 65536, note = hpc}
UioDevice{number =  1, name =             axi-pmon, addr = 0xfd0b0000, size = 65536, note = }
UioDevice{number =  2, name =             axi-pmon, addr = 0xfd490000, size = 65536, note = }
UioDevice{number =  0, name =             axi-pmon, addr = 0xffa00000, size = 65536, note = }
UioDevice{number =  3, name =             axi-pmon, addr = 0xffa10000, size = 65536, note = }
AxiTrafficGen info:
  id reg = 0xa8172a9e
  version = 0.9.7
Udmabuf
  name = axi:udmabuf@0x0
  virt addr = 0xffff954fe000
  phys addr = 0x5e100000
  size = 33554432
  flags = 0x101002
Transfering 32 bursts
Memory check (size = 32 bursts) successfully completed
Memory check (size = 32 bursts) successfully completed
stats: rd cyc = 670, wr cyc = 265, rd_ok = 128

HPC port

root@u96v2-sbc:~# app-axi-traffic-gen --count 32 --interface hpc --axi-cache 15 --axi-prot 2 --axi-user 1
UioDevice{number =  4, name =        AxiTrafficGen, addr = 0xa0040000, size = 65536, note = acp}
UioDevice{number =  5, name =        AxiTrafficGen, addr = 0xa0050000, size = 65536, note = hp}
UioDevice{number =  6, name =        AxiTrafficGen, addr = 0xa0060000, size = 65536, note = hpc}
UioDevice{number =  1, name =             axi-pmon, addr = 0xfd0b0000, size = 65536, note = }
UioDevice{number =  2, name =             axi-pmon, addr = 0xfd490000, size = 65536, note = }
UioDevice{number =  0, name =             axi-pmon, addr = 0xffa00000, size = 65536, note = }
UioDevice{number =  3, name =             axi-pmon, addr = 0xffa10000, size = 65536, note = }
AxiTrafficGen info:
  id reg = 0xa8172a9e
  version = 0.9.7
Udmabuf
  name = axi:udmabuf@0x0
  virt addr = 0xffffa781e000
  phys addr = 0x5e100000
  size = 33554432
  flags = 0x2
Transfering 32 bursts
Memory check (size = 32 bursts) successfully completed
Memory check (size = 32 bursts) successfully completed
stats: rd cyc = 391, wr cyc = 756, rd_ok = 128

ACP port

root@u96v2-sbc:~# app-axi-traffic-gen --count 32 --interface acp --axi-cache 15 --axi-prot 2 --axi-user 1     
UioDevice{number =  4, name =        AxiTrafficGen, addr = 0xa0040000, size = 65536, note = acp}
UioDevice{number =  5, name =        AxiTrafficGen, addr = 0xa0050000, size = 65536, note = hp}
UioDevice{number =  6, name =        AxiTrafficGen, addr = 0xa0060000, size = 65536, note = hpc}
UioDevice{number =  1, name =             axi-pmon, addr = 0xfd0b0000, size = 65536, note = }
UioDevice{number =  2, name =             axi-pmon, addr = 0xfd490000, size = 65536, note = }
UioDevice{number =  0, name =             axi-pmon, addr = 0xffa00000, size = 65536, note = }
UioDevice{number =  3, name =             axi-pmon, addr = 0xffa10000, size = 65536, note = }
AxiTrafficGen info:
  id reg = 0xa8172a9e
  version = 0.9.7
Udmabuf
  name = axi:udmabuf@0x0
  virt addr = 0xffff9469a000
  phys addr = 0x5e100000
  size = 33554432
  flags = 0x2
Transfering 32 bursts
Memory check (size = 32 bursts) successfully completed
Memory check (size = 32 bursts) successfully completed
stats: rd cyc = 140, wr cyc = 248, rd_ok = 128

FSBL output

Xilinx Zynq MP First Stage Boot Loader 
Release 2021.1   Jun  6 2021  -  07:07:32
MultiBootOffset: 0x0
Reset Mode      :       System Reset
Platform: Silicon (4.0), Running on A53-0 (64-bit) Processor, Device Name: XCZU3EG
SD0 Boot Mode 
PMU Firmware 2021.1     Jun  6 2021   07:07:32
PMU_ROM Version: xpbr-v8.1.0-0
Protection configuration applied
EL = 3
CCI_REG: register dump
  offset 0 = 0
  offset 10 = 0
  offset 14 = 8000003F
  offset 18 = 0
  offset 1C = 0
  offset 40 = 0
CCI_REG: debug enable
CCI_REG: register dump
  offset 0 = 0
  offset 10 = 0
  offset 14 = 8000003F
  offset 18 = 0
  offset 1C = 0
  offset 40 = 3
CCI: enable snoop, ctrl before = C0000000
CCI: enable snoop, ctrl after = C0000001
CCI: shareable override reg - before = 0
CCI: shareable override reg - after = 3
Exit from FSBL

Xilinx, Inc. Xilinx, the Xilinx logo, Vivado, Zynq are trademarks of Xilinx in the United States and other countries.

AMBA, ARM, Cortex and TrustZone are registered trademarks of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. CoreLink is a trademark of ARM Limited (or its subsidiaries) in the EU and/or elsewhere.

All trademarks and registered trademarks are the property of their respective owners.

Notes on "A Primer on Memory Consistency and Cache Coherence"

2021-12-23T16:00:00+01:00

A Primer on Memory Consistency and Cache Coherence, Second Edition

Authors:

Vijay Nagarajan, University of Edinburgh
Daniel J. Sorin, Duke University
Mark D. Hill, University of Wisconsin, Madison
David A. Wood, University of Wisconsin, Madison

"This primer is intended for readers who have encountered memory consistency and cache coherence informally, but now want to understand what they entail in more detail."

[this book talks to me]

Chapter 1

memory consistency - effects of stores and loads on the memory, correct behavior
cache coherence - HW implementation to ensure correct behavior when caches are involved
memory (consistency) model - the specification about allowed behavior of MT programs
snooping - cache controller performs a broadcast

Chapter 2

memory-side cache (last-level cache) - not a concern in regards to coherence, just reduces the latency
"Informally, a coherence protocol must ensure that writes are made visible to all processors."
consistency-agnostic coherence (atomic writes) vs consistency-directed coherence ([posted] writes, ordering rules)
single-writer-multiple-reader (SWMR) invariant
data-value invariant
other definitions of invariants

Chapter 3

reordering, write buffer
MC (memory consistency) model
core pipeline can have an impact on the consistency

Sequential consistency

L. Lamport: How to Make a Correct Multiprocess Program Execute Correctly on a Multiprocessor
program order, memory order
op1 <m op2, op1 <p op2
for SC: op1 <p op2 ==> op1 <m op2 (==> = implies)
requirement: LL, LS, SS, SL dependencies, loads get the value of the last store, atomic RWM
naive implementation: the switch
speculative loads are just discarded, speculative stores only present the address and not the data --> cache can check if (or inform) other caches have the datum
multi-threading: other threads must have independent write buffer (so to behave like other processors); normal write buffers (with queue) are anyway not possible in SC
[ Mod-01 Lec-32 Case study: MIPS R10000 ]

Chapter 4

total store order (TSO)
"[TSO] model astonishes some people..."
TSO behaves equally for programs that: 1. store data first, 2. then store done flag
write buffer, FIFO
FENCE instr

Chapter 5

relaxed (weak) memory consistency
most application do not require strong consistency
accesses can be re-ordered, Coalescing Write Buffer

Example Relaxed Consistency Model (XC)

FENCE instruction
TSO for accessing the same address (LL, LS, SS)
data race - two or more programs accessing the same memory, at least one access is a write

Release Consistency

ACQUIRE, RELEASE instead of FENCE

RVWMO

dependency-induced ordering
same address ordering
Atomic Memory Operation (AMO) and Load Reserve/Store Conditional

IBM POWER

"On a first pass of this primer, readers may wish to skim or skip this section; this memory model is significantly more complicated than the models presented thus far in this primer."

ARMv7, ARMv8

ARMv7 similar to POWER
ARMv8 provides a total memory order, has ACQUIRE and RELEASE

High-level Language Models

SC for DRF

Chapter 6

two invariants: SWMR, data-value
coherence controller: a FSM, one per each cache (cache controller) or memory (memory controller)
coherence protocol: communication between FSM

"Other agents, such as I/O devices, may behave like cache controllers, memory controllers, or both depending upon their specific requirements."

States

four characteristics:
validity
dirtiness
exclusivity
ownership
stable states:
Modified: valid and potentially dirty
Owned: valid and owned, possibly stale, read-only
Exclusive: no other cache has a copy, read-only
Shared: valid but not exclusive and not dirty, read-only
Invalid: not present in the cache or stale
transient states:
$$ XY^Z $$
snooping vs directory
invalidate vs update

Chapter 7

"all coherence controllers observe (snoop) coherence requests in the same order and collectively "do the right thing" to maintain coherence."

transactions need to be ordered
serialization (ordering) point, e.g. bus arbitration

baseline: MSI

atomic requests and atomic transactions
processor loads go from Invalid to Shared
processor stores go from Invalid to Modified
both loads and stores wait for the data before transitioning to the steady state
if Modified, GetS and GetM from other cores transition to Shared and Invalid, respectively

Non-atomic requests, atomic transactions

queue or buffer between the cache controller and the bus (or interconnect)

Exclusive state

handles case when the block is first read and then written -> no need to inform other cores
GetS brings the block state to S or E (depends on states of other controllers)

Owned state

when a block is in state M or E and receives GetS

Non-atomic bus

in-order vs out-of-order
FIFO queues between the controller and request and response bus

Interconnect network

tree topology
Timestamp Snooping

Chapter 8

"Directory protocols were originally developed to address the lack of scalability of snooping protocols."

directory with a global view
transactions typically involve 2 or 3 steps:
a request by the controller and a reply by the directory
a request by the controller, requests to other controllers and replies from other controllers
4th step is also possible
directory as the serialization point
for some requests (e.g. GetM) all caches with a block in state S must ACK

MSI

an entry (per each block) consists of the state bits, owner id (if in state M) and sharer list (bit vector, if in state S)
I or S to M: directory returns to the requestor the number of controllers which must ACK
PutM: contains data (from the controller to the directory)
virtual networks (message class) to avoid deadlocks (requests blocking responses)

Exclusive state

GetS and not shared by other controllers --> E
a core can silently (no coherence request required) upgrade E to M
E block: owned vs not-owned
PutE (directory from E to I)

Owned state

[how did it got dirty if it is read-only?]
Owned is basically a frozen Modified state

Directory state

full representation only for smaller number of cores
coarse directory: more controllers are "grouped" in a same bit
limited pointer directory: broadcast after I sharers, invalidate one of the sharers, trap to a SW handler

Directory organization

one entry per each block of the memory
directory cache: accesses have (usually) good locality, smaller datums ([ROMANES EUNT DOMUS])--> even small caches have a high hit rate
backed by DRAM
inclusive: only for blocks cached, a miss indicates the I state
Null Directory Cache: [this sounds more like a snooping protocol]

Optimizations

"The typical, general solution to the problem of a centralized bottleneck is to distribute the resource."

distributed directories
non-stalling directory protocols
eviction of blocks in S state

Chapter 9

Instruction caches

"[...] truly self-modifying code is rare,"

core writes to D$, I$ only observers writes
icbi (instruction cache block invalidate) instruction on POWER

Virtual caches

advantage: no need for address translation (on the critical path)
require reverse address translation
synonyms: same virtual address mapped to multiple physical addresses

Write-true caches

two-state protocol (VI)
simpler evictions
some issues with multi-threading

Coherent DMA

"DMA controllers have very different locality patterns than conventional cores" - GetMs are wasteful, the entire block is typically overwritten --> special requests

Multi-level caches

inclusive cache: e.g. L2 contains a superset of L1
multiple multi-processors: LLC as a memory-side or core-side cache
hierarchical protocol: intra-chip, inter-chip (e.g. intra-chip snooping, inter-chip directory)

Performance optimization

Migratory sharing

one thread reads and writes, then another thread reads and writes, ...
E state already helps
HW to detect GetS and then GetM
alternatively, add Migratory M state (I -> S -> M)

False sharing

more cores accessing the same block without actually dependencies; solutions: sub-block coherence, speculation

Liveness

deadlock - cyclical dependencies
protocol deadlock
cache resource deadlock
virtual networks
livelock - a special case of starvation
starvation (solved with fair arbitration)

Token Coherence

a third method (alongside snooping and directory-based protocols)
tokens instead of status bits
cores exchange tokens
a core with one token can read, a core with all tokens can write to a block

Chapter 10

"This is the age of specialization."

SoC share one physical memories, e.g. GPUs have two separate physical memories
cooperative thread array (CTA)
SIMT - Single-Instruction-Multiple-Thread
http://www0.cs.ucl.ac.uk/staff/j.alglave/papers/asplos15.pdf
GPU: relaxed memory order
one approach: consistency-agnostics coherence protocol - not suitable for GPUs (large L1 caches, many threads = many transactions)

Temporal coherence

lease (limited amount of time), block gets automatically invalidated
global notion of time
GetV(t), Write, WriteV(t)
stalled writes (until the lease expires)
GWCT (Global Write Completion Time) - all FENCE must stall until this time
performance sensitive to the selection of lease time
timestamp complexity (e.g. rollover)

Release consistency-directed coherence

atomic operations which order memory accesses in one direction (in contrast with FENCE)
scope (CTA vs GPU)

Heterogeneous systems

more devices with different memory consistency models
intuition: the weaker model of the two
OpenCL: SC for HRF (Heterogeneous-Race-Free)

Heterogeneous coherence protocols

global controller
local controllers contain shims (or translators)

"the global coherence interface must disambiguate CPU sharers from GPU sharers"

coherence tracking at a larger granularity (e.g. page instead of a cache line)
scratchpads in GPUs (programmer-controlled memories)

Chapter 11

specification = contract between the user and the implementation
input actions, internal actions, output actions
safety and liveness

Operational specification

state machine
liveness ensured with temporal logic
linearizability
CMurphi and TLA

Axiomatic specification

Alloy, Herd

Litmus tests

https://www.cl.cam.ac.uk/~sf502/regressions/rmem/
https://github.com/nvlabs/litmustestgen

Validation

Formal methods

manual methods (e.g. assigning timestamps as values)
model checkers (state space becomes quickly very large)

Testing

Off-line testing
On-line testing (checker implemented in HW)

Stratix V accelerator card from eBay, part 8

2021-10-31T11:00:00+01:00

My last blog post on this topic explored the PCI Express connections on this board. I have determined that the board contains a custom FPGA with two Hard IPs for PCIe, where the commercially-available part only contains one. At the end of the post I explored different options on how to access/enable the second IP, and none of my attempts were successful. The post concluded on an optimistic note, with an expectation that this obstacle will somehow be solved.

Luckily, a couple of weeks after my post, while I was still contemplating how to approach this obstacle, @gatecatte managed to figure out how to convince Quartus to let us generate a bitstream with two Hard IP blocks; one just needs to override DEV_DIE_INFO::is_global_id_enabled function to return true. @rombik_su then prepared an easy-to-use patch, available here.

This discovery enabled me to continue with the preparation of an example design for the Pikes Peak and Storey Peaks boards.

In this blog post I describe the work done to get the PCIe interface (including a high-performance DMA) up and running and present some results of integration tests with two different computers.

Introduction

Existing IPs

Quartus contains several IPs for PCI Express, where the Hard IP is wrapped with different interfaces. The easiest one to use would be the Avalon-MM Stratix V Hard IP for PCI Express; this IP provides an Avalon-MM Host interface to handle the Memory Reads and Memory Writes as well as an Avalon-MM Agent interface to provide a Direct Memory Access (DMA) port. Another IP already contained in Quartus, called Modular Scatter-Gather DMA can be directly connected to this port to transfer the data between the on-board DDR3 memory and the system memory.

Unfortunately, for some reason, this IP does not support the operation in Gen3 x8 mode.

Development of a new IP

At this point I have decided to have some fun and develop a PCIe endpoint IP from scratch. The following were my requirements/guidelines:

integration with Stratix V Hard IP
high throughput: the IP should be able to operate with a 256-bit-wide interface at 250 MHz, and achieve high interface utilization
relatively simple: it is reasonable to sacrifice a small amount of performance to keep the IP simple. For example I do not use the "Multiple packets per cycle" option - this reduces the maximum throughput by roughly 6% (16 bytes per 256-byte packet), but makes the logic much simpler.
use of standardized interfaces: e.g. Avalon-MM and Avalon-ST
a base for custom developments: I want to reuse this IP for other applications with this board, and the IP should be general enough to allow different use cases
written in Chisel HDL

After several weekends of development I am proud to present my creation: PCIe endpoint with a high-performance DMA and an Avalon-MM interface

The IP is currently in version 0.7 - it is usable and it works, but some edge cases are not yet handled (one is described in this blog post) and some smaller improvements are also expected in the future.

Architecture

The figure below shows the main building block of the IP; several modules are responsible for the reception part and several modules responsible for the transmission part.

MemoryReadWriteCompl parses the received packet and forwards it to the next corresponding module
AvalonAgent converts MWr and MRd PCIe packets into transactions on Avalon-MM interface; it is able to handle both 32-bit and 64-bit requests (by performing two accesses on the 32-bit interface)
CompletionRecv parses the received completion packets and forwards them on an Avalon-ST interface. It informs the BusManagerEngine about the received completion packets so that the later can keep the track of the number of dwords in flight.
BusManagerRegs provides a read and write access to registers (similar to AvalonAgent) which are needed for the DMA engine (e.g. destination address, number of bytes to transfer, ...)
BusManagerEngine is the main module which performs the DMA function - it generates either MWr or MRd (depending on the instructions from software). When the transfer is complete it informs the InterruptCtrl
CompletionGen generates a completion packet after a MRd packets with the information provided by a corresponding module
TxArbiter manages the access to the tx_st port
InterruptCtrl generates interrupt requests on the behalf of other modules and from the external interface

Test suite

The project includes a non-very-extensive, but still very useful test suite:

Here it is clear that this is a hobby project, a serious IP would have more unit tests and also some more high-level tests.

Data generator and data checker

To validate the IP in real hardware, I have developed two auxiliary IPs: Avalon-ST Generator and Avalon-ST Checker. The first one generates a 256-bit-wide data stream composed of 16-bit counter values and the second one checks the received data stream against a reference counter and stores the results in internal registers. Both IPs also contain Avalon-MM interface which can be used to control the IPs. Shown in the figure below are the connections between all relevant IPs for the PCIe test.

Integration with Quartus Platform Designer

Below is an screenshot of the PCIe endpoint IP connected to the Stratix V Hard IP for PCI Express in Quartus Platform Designer:

Linux driver, test program and TUI

The FPGA design is only one half of this story, the second half is a Linux driver. The driver is also available on my GitHub page: Linux driver for Unofficial Pikes Peak/Storey Peak reference design.

The driver is a relatively standard Linux PCIe driver, it registers its probe function with the PCIe subsystem based on vendor ID/device ID for Altera Stratix V and subsystem vendor ID of 0x01a2 (for JAN) and devices IDs of 0x0001 for the first interface, 0x0002 for the second interface and 0x0a00 for custom applications.

Once the match is found, it performs the house-keeping tasks (allocating memory, claiming BARs, allocating DMA buffer, enabling the device) and it finally creates a character device (in /dev) with a name derived from the PCIe address.

The user-space programs interact with the driver in two ways. To access the registers on BAR0 (i.e. reads and writes) the programs can mmap() the char device and dereference a pointer to uint32_t or uint64_t. Two ioctls can be used to exchange the DMA buffer with the user-space program, one for retrieving the buffer content and another for setting it. Finally, another ioctl can be used to start the DMA transfer; the user-space program should provide the transfer size and the direction, and the driver will perform the desired request. To reduce the CPU utilization, the thread is put to sleep while the DMA transfer is running and is waken up by an interrupt request from the DMA engine in the FPGA.

Test program

pp_sp_test is a small test program that can be used to exercise the driver and perform DMA transfers with the DMA engine in the FPGA. The program uses the Avalon-ST Checker and Avalon-ST Generator IPs to verify the content of the transfer.

$ ./pp_sp_test --help                                                               
Usage: ./pp_sp_test --dev DEV [--write] [--read]

Perform DMA transfers using the DMA engine in PP/SP FPGA

options:
  --help      print these help and exit
  --dev       char device (e.g. /dev/pp_sp_pcie...)
  --write     card to host (DMA write) transfer
  --read      host to card (DMA read) transfer
  --nr_bytes  number of bytes to transfer (default 256)
  --count     count of loops to perform the reads and/or writes (default 1)
  --msleep    sleep (in milliseconds) during each loop (default 0)

Here is an example where 8 KiB of data was transferred in both directions. First the data was transferred from the card to the host (c2h), and then from the host to the card (h2c). In the end, the result of the check (in the FPGA) is printed.

$ ./pp_sp_test --dev /dev/pp_sp_pcie_0000:42:00.0 --write --read --nr_bytes 8192
Arguments:
  dev = /dev/pp_sp_pcie_0000:42:00.0
  c2h = 1, h2c = 1
  nr_bytes = 8192, count = 1
===================================
[loop] i = 0
[gen] id reg = a51579e2
[gen] state = 1
[gen] nr samp = 64
[c2h] DMA duration 0.024146 ms
[c2h] ioctl took 0.021000 ms
[gen] state = 0
[gen] nr samp = 8192
[check] id reg = a5157c8c
[h2c] DMA duration 0.021397 ms
[h2c] ioctl took 0.019000 ms
[check] samp = 8192 / 8192

TUI

A more elaborate program to control the DMA engine and the corresponding example application is provided in the form of a Terminal User Interface in the tui directory. With this program a user can select the size and the direction of the transfer for both interfaces. The interface displays the result of the throughput measurement in kernel space and the statistics (minimum, average, and maximum value) for the last 100 measurements.

Some screenshots of TUI are shown in the next chapters.

Test setups

I have used Storey Peak board (in PCIe form-factor) with two systems to test the PCI express:

FUJITSU D3642-B1 motherboard with Intel i5-9600K
Dell PowerEdge R720 with two Intel E5-2680

R720 has an x16 PCIe slot but does not support bifurcation, while the x16 PCIe slot on the Fujitsu motherboard can be split into the x8x8 configuration with a BIOS setting.

The system with the Fujitsu motherboard ran Ubuntu 18.04 LTS with Linux kernel 4.15.0-161 while the R720 ran Ubuntu 20.04 LTS with Linux kernel 5.11.22 (custom build with CMA enabled but not used in this experiment).

I have prepared three different FPGA designs:

one with only lower 8 lanes connected and the Hard IP on the right side of the device,
one with only upper 8 lanes connected and the Hard IP on the left side of the device, and
one with all 16 lanes connected (in x8x8 configuration) and with both Hard IPs

The Quartus project for the last one is available in the commit 3cb9c50 Change subsystem device id for the second PCIe

only PCIE lanes 0 to 7	only PCIe lanes 8 to 15	PCIe lanes 0 to 7 and 8 to 15

As an aside, it can be noted that the JTAG pins are located in the lower-left corner, and it is interesting to see how the fitter decided to place some of the logic halfway between the PCIe core on the right side and the JTAG pins on the left side.

Measurements

Fujitsu motherboard with right side PCIe link (lanes 0 to 7)

$ sudo lspci -d 1172: -vv
01:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
[...]
    Region 0: Memory at a1000000 (32-bit, non-prefetchable) [size=4M]
    Region 2: Memory at a1400000 (32-bit, non-prefetchable) [size=256K]
    Capabilities: [50] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Address: 00000000fee003f8  Data: 0000
[...]
        LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
[...]
    Kernel driver in use: pp_sp_pcie
[...]

Here we see something interesting - the Avalon-ST Checker reports that some of the samples do not match the reference values. I investigate this issue in the following chapter.

Fujitsu motherboard with left side PCIe link (lanes 8 to 15)

$ sudo lspci -d 1172: -vv
02:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
[...]
    Region 0: Memory at a1000000 (32-bit, non-prefetchable) [size=4M]
    Region 2: Memory at a1400000 (32-bit, non-prefetchable) [size=256K]
    Capabilities: [50] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Address: 00000000fee00438  Data: 0000
[...]
        LnkCap: Port #2, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
[...]
    Kernel driver in use: pp_sp_pcie
[...]

Similarly, also here the the Avalon-ST Checker reports corrupted samples.

Fujitsu motherboard with both PCIe links (lanes 0 to 7 and 8 to 15)

pp_sp_example > pcie status
pcie 0:
  id = 2c1e57a7
  version = 10000
  status.cur speed   = 3
  status.LTTSM state = f
  status.lane act    = 8
  status.DL up       = 1
pcie 1:
  id = 2c1e57a7
  version = 10000
  status.cur speed   = 3
  status.LTTSM state = f
  status.lane act    = 8
  status.DL up       = 1

$ sudo lspci -d 1172: -vv | egrep 'Altera|LnkSta|Subsystem'
01:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
        LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
02:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
        LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+

And not surprisingly, when using both interfaces to read the Avalon-ST Checker reports some corrupted samples as well.

Dell PowerEdge R720

$ sudo lspci -d 1172: -vv | egrep 'Altera|LnkSta|Subsys'
42:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
        LnkSta: Speed 8GT/s (ok), Width x8 (ok)
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+

Issues

Handling of out-of-order CplD packets

As we have seen above, when running in combination with Intel i5-9600K there is a small percentage of the samples that Avalon-ST Checker reports as corrupted.

To observe what is going on, I have placed a SignalTap on the important signals of the Avalon-ST Checker, and let it trigger on invalid packets.

We can note that for a couple of cycles the data_valid_p signal is high, but data_ok_p is low, indicating that the received data does not match the reference data. After a couple of clock cycles the data stream seems to recover.

If we store the data from this SignalTap capture into a .csv file and highlight the important fields in the header (blue is Length, magenta is Tag, black is the payload data), we can observe that two of the packets are returned out of order. We can observe that first a packet with the tag 0xb is returned (in the sample at time -11), but only the first 0x30 dwords - there are still 0x10 dwords outstanding to fulfill the 0x40 dwords (256 bytes) long request. Instead of continuing with the remaining part of the packet with the tag 0xb, the next packet contains the tag 0xc (in the sample at time -4). Only after this packet, the transaction with the tag 0xb is completed (in the sample at time -1).

PCI Express devices are allowed to return completion packets out-of-order:

Completions with different Transaction IDs are permitted to pass each other.

My PCIe endpoint IP currently does not support out-of-order CplD packets, but adding support for this case should be relatively straightforward: the receiving side needs to keep track of the number of dwords remaining per each tag, and queue the packets when the current tag is not yet complete.

Summary

In this blog post I have summarized my current experience with PCIe on Storey Peak board. With a small hack one is able to use both PCIe Hard IPs available in this device and use the PCIe interface in the full Gen3 x8x8 mode.

I have developed a PCIe endpoint IP that connects to the PCIe Hard IP and provides register access and DMA capabilities. The DMA is able to achieve 6.5 GB/s in both read and write direction on one interface. With both interfaces active it achieves 11 GB/s in write direction and 12 GB/s in read direction. I would consider this to be a good result.

We can now declare the PCIe part "solved", and this concludes the last major effort in reverse engineering of this board. The last remaining part is providing an example for the use of QSFP connectors, and some general clean-up of the reference design.

Appendix

lspci on Fujitsu motherboard

$ sudo lspci -d 1172: -vv
01:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 136
    Region 0: Memory at a1000000 (32-bit, non-prefetchable) [size=4M]
    Region 2: Memory at a1400000 (32-bit, non-prefetchable) [size=256K]
    Capabilities: [50] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Address: 00000000fee003f8  Data: 0000
    Capabilities: [78] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [80] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
            Status: NegoPending- InProgress-
    Capabilities: [200 v1] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
    Capabilities: [300 v1] #19
    Capabilities: [800 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
    Kernel driver in use: pp_sp_pcie
    Kernel modules: altera_cvp

$ sudo lspci -d 1172: -vv
02:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 137
    Region 0: Memory at a1000000 (32-bit, non-prefetchable) [size=4M]
    Region 2: Memory at a1400000 (32-bit, non-prefetchable) [size=256K]
    Capabilities: [50] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Address: 00000000fee00438  Data: 0000
    Capabilities: [78] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [80] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap: Port #2, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
    Capabilities: [100 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
            Status: NegoPending- InProgress-
    Capabilities: [200 v1] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
    Capabilities: [300 v1] #19
    Capabilities: [800 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
    Kernel driver in use: pp_sp_pcie
    Kernel modules: altera_cvp

lspci on Dell R720

$ sudo lspci -d 1172: -vv
42:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 167
    NUMA node: 1
    Region 0: Memory at d5000000 (32-bit, non-prefetchable) [size=4M]
    Region 2: Memory at d57c0000 (32-bit, non-prefetchable) [size=256K]
    Capabilities: [50] MSI: Enable+ Count=1/4 Maskable- 64bit+
        Address: 00000000fee00918  Data: 0000
    Capabilities: [78] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [80] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
        DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM not supported
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 8GT/s (ok), Width x8 (ok)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR-
             10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS-, TPHComp-, ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
    Capabilities: [100 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
            Status: NegoPending- InProgress-
    Capabilities: [200 v1] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
    Capabilities: [300 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
        LaneErrStat: 0
    Capabilities: [800 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Kernel driver in use: pp_sp_pcie
    Kernel modules: altera_cvp

Intel, the Intel logo, Altera, Nios, Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.

PCI Express® and PCIe® are registered trademarks of PCI-SIG.

Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.

All trademarks and registered trademarks are the property of their respective owners.

Notes from Xilinx® Adapt 2021

2021-09-07T21:00:00+02:00

Xilinx Adapt 2021

Day 1 (2021-09-07)

Adaptive Computing: Innovation Accelerated [Ivo Bolsens (Xilinx)]

the hardware is adapted, rather the other way around
DSA
growing gap between the moore's law and AI requirements
requirements: 6G (100 Gbps, 0.1 ms latency)
Xilinx wants to be a platform company
Vivado/FPGA -> Vitis/MPSoC -> Vitis-AI/ACAP (Peer Processor)
"hardware developers are still the key audience for Xilinx"
Cloud and Edge
some examples: SmartSSD (partnership with Samsung), SmartNIC
guest speaker: Alveo U250 in Azure
- Quantum optimization (10x gains a CPU)
- Synapse - SQL acceleration (e.g. CSV parsing)
- external use cases: Financial Services, Bio-Informatics
AI engine (Scalar ALU, Vector ALU)
AIE Array (non-blocking interconnect, local memory, ISA-based Vector Engine)
guest speaker: Samsung
Bfloat, INT4
guest speaker: HPC at Pacific Northwest National Lab
- computational chemistry application
- outlook: integration between physical science and data science
- heterogenous testbed: AMD Epyc, AMD Instinct GPU, Alveo
Xilinx devices can handle the entire applications (e.g. ADAS)
Kria SOM (most page views ever), comes with predefined bitstreams
"accessible to software developers, domain experts"

The Future of Adaptive Computing [Ivo Bolsens, Vamsi Boppana (Xilinx)]

higher level -> IP Integrator, software
2021.1 -> Machine Learning in Vivado
- "Inteligent Design runs"
- delay estimation, resource usage predictions
AI Engines also for liner algebra applications (not only Machine Learning)
512-bit-wide vector machine
chiplets (e.g. larger devices)
rapidly evolving standards -> adaptable hardware
DFX (e.g. in automotive, two different algorithms)
Vitis AI: CNN, RNN, LSTM
Hennessy & Patterson - Domain Specific Architecture
open-source (community, de-facto standards)
Xilinx App Store, Ubuntu software store
Kria (ready-built application, downloadable from an on-line app store)
roadmap (up to 1 nm), only for a selected products (cost-sensitive apps)

Design Rationale of Two Generations of AI Engines [Kees Vissers (Xilinx)]

again Hennessey and Patterson
processor designed from a ground up
comparison between traditional multi-core and AI Engine Array
interconnect + DMA (no caches)
better latency, efficiency than GPU and CPU
1GHz+, 400 AI Engines per device
AI Engine = conventional AI processor
- VLIW
- 32-bit RISC
- 512-bit SIMD (fixed point, floating point)
- Fixed-Point Vector Unit (similar to DSP48)
- FPMPY
- Multi-precision support (also Complex - for RF)
- memory:
  - double buffering
  - dataflow
  - streaming communication (DMA between memories)
  - multicast support
- integration into PL: AXI-Stream
Vitis libraries (vision, finance, linear algebra, ...)
XAPP1351 (multi-rate filter), XAPP1352 (beamforming), XAPP1356 (FFT)
Vitis AI (PyTorch/TensorFlow/Caffe to FPGA/AI engines)
second gen AI Engine
- use case: ADAS, robotics, media
- in Versal AI Edge
- bfloat16
- matrix * matrix multiply [(A0 x B0) + (A1 x B1) + ... ]
- data multicast (e.g. weights for NN)
- PCIe gen 5
high level AIE API (independent of underlying AIE)

Day 2

What's New in Vitis AI 1.4 and Vitis 2021.1 [George Wang (Xilinx)]

Vitis

DPU
libraries for AI engines: FIR, FFT, GEMM, vision
GZIP, ZSTD library
FIFO allocation with AI Engine
x86 simulator for AIE
Device Tree generator -> ZOCL node
Vitis HLS: Flow Navigator

Vitis AI 1.4

support for 2 Versals and Kria
108 AI models in total in AI Model Zoo
lidar, radar applications
quantization-aware training, automatic network pruning

Introduction to Kria System on Module [Karan Kantharia (Xilinx)]

Xilinx idea: use SoM in final products
vision market (becoming more fragmented): security camera, obj classifications, medial, AR/VR, emotion

KRIA K26 SOM

Zynq (ARM + FPGA)
several interfaces: LVDS, USB, MIPI, Ethernet, HDMI, DisplayPort, ...
Xilinx idea: no more RTL/HW design -> up to 9 months faster Time to Market
"no FPGA experience required"
three options:
- for AI Developer: use AI model
- for SW devloper: use Vitis
- for HW developer: use Vivado ML
Yocto and Ubuntu supported
FCC, ... certified

Workshop: Vitis AI 101: End-to-End Model Deployment with Vitis AI [Fan Zhang (Xilinx)]

https://github.com/fanz-xlnx/Adapt_Workshop_VAI101

ubuntu@ip-***:~$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
00:1e.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)

ubuntu@ip-***:~$ nvidia-smi
Wed Sep  8 17:20:50 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:00:1E.0 Off |                    0 |
| N/A   55C    P0   121W / 149W |   4188MiB / 11441MiB |     94%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     19274      C   python                           4185MiB |
+-----------------------------------------------------------------------------+

IoU = Intersection over Union = Area of Overlap / Area of Union
XIR format
KV260 (KRIA)

xcompiler -t DPUCZDX8G_ISA0_B4096_MAX_BG2 -i quantize_result/ENet_int.xmodel -o compilation_results/KV260/ENet_cityscapes_pt/ENet_cityscapes_pt.xmodel

The compiled xmodel's md5sum is 4bf46f368e9ff2d51fea136a19270c75

Day 3

Expert Panel: Tips and Tricks

improvements in silicon
- "PL is what Xilinx is famous for"
- "end of Moore's law, end of Amdahl's law, end of Dennard scaling" --> more hardened IP (more like ASIC), e.g. DSP58
getting started guide for DFX ([xilinx.com/vivado/dfx])
HDL (VHDL support)
- simulation: working on the support for the VHDL-2008 (it is 2021), based on feature requests
- simulation: "current focus on SystemVerilog"
- synthesis: "at the advanced level"
open-source tools (providing the bitstream information)
- "secret sauce"
- "hacking protection"
- open-source front-end for Vivado and Vitis
- "leave bitstream generation to experts"
ML in Vivado
congestion when 70% CLBs utilized
revision control
RTL workflow
- a couple of features require IPI (e.g. CIPS, NoC)
porting software to AI Engines
- start from the C models
RapidWright
QEMU

Team-Based Collaborative Features in IP Integrator

UG994
Block Design Container
- top-down workflow:
  1. create hierarchy
  2. validate
  3. was distracted
- e.g. debug vs no-debug version
- DFX flow
- Inter-NOC input

Vitis HLS for High-Performance Kernels

v++
optimizations
- pipeline (II)
- SIMD
- dataflow (task parallelism, handshaking)
data types:
- arrays: AXI4 Memory Mapped
- scalar: AXI4-Lite
- stream: AXI4-Stream

#pragma HLS UNROLL factor=N

__attribute__((vector_size(64)))

Cppcon 2019: Faster Code Through Parallelism on CPU and GPU
RAM 1WnR
#pragma HLS BIND STORAGE
function call viewer

Versal Architecture Solutions for PCIe and Cache Coherent Interconnect [Eric Crabill (Xilinx)]

in Versal
- CPM4 and CPM5 (gen 4 and gen 5)
- PL PCIE4 and PL PCIE5
- SRIOV
- integrated DMAs (QDMA and XDMA in hard IP)
- CCIX support
- connection to NoC
- CCIX to CHI bridge

CPM vs PL PCIE

CPM - feature rich
PL PCIE - migration from previous architectures

QDMA vs XDMA

CCIX and CXL

hetergeneous computing
- CPU + GPU
- CPU + ACAP
- CPU + Smart NIC
- ...
classic PCIe = moving the data with DMA (SW-controlled)
Cache cohherence = "move the data without using a driver"
CCIX = symmetrical (CPU and accelerators are peers)
CXL = CPU is the owner, multiple protocols: cxl.io, cxl.mem, cxl.cache

Documentation

PG347 (for CPM)

Day 6

The Xilinx SN1000: Accelerate Your Cloud Data Centers for Scalability and Performance

evolution (according to Xilinx)
1. "traditional NIC"
2. "offload NIC"
3. "Programmable SmartNIC"
specific requirements, needs to adapt to changing workloads
Alveo SN1000 (PCIe gen3 x16, up to 16 A72)
Vitis Networking Stack (P4), HLS (C, C++), RTL
Architecture: plugins
Virtio, vhost-vdpa, VirtIO NET PF
vDPA - virtual Data-Path Acceleration (control plane managed by host)
example: NVME-over-fabric, Ceph, virtio-blk

Azure Quantum Optimizing on FPGAs

"Scaled Quantum Computing"
Q#, Python SDK
problem formulalation
PUBO [0, 1], Ising [-1, 1]
provides: Honeywell, IonQ, 1QBit
example
- scheduling problem
- CPU runtime - 3 min 12 sec
- python from azure.quantum.optimization import SimulatedAnealing
- ..., platform=HardwarePlatform.FPGA)
- FPGA runtime - 23 sec
- "10x speedup"
aka.ms/qsharp-blog

FPGA-Accelerated Structured Query Language (FAStQL) for Azure Synapse Analytics

Apache Spark, Data Source v2 (DSv2) interface
FPGA handles: parsing, filter, projection
support for Decimal
profiling: 80% on parsing, 20% on query
plans for the future: compression/decompression, hash join, ...
architecture: row scheduler, N row parsers, row combiners
parsing: 6 - 7 GB/s
filtering: stack-based processor (same arch: scheduler, N proc, output)

Improving Spark Storage Efficiency with NoLoad Transparent Compression

"Data Tsunami"
computational storage
CSP (Computational Storage Processor): computation, no persistant storage
CSD (Compuational Storage Drive): computation + peristent data storage (FPGA + SSD)
NVMe computation - in the process of standardization, expected in 2022
NoLoad (r)
- NVMe-compilant front-end (looks like an NVMe device to the OS)
- certified by UNH-IOL
- accelerators: compression, decompression
- compute: analytics, ML, AI
NVMe-oF, Peer-to-Peer
Apache Spark (data size up to PB), NoLoadFS, ZLIB compression offload
- Cisco UCS

Breaking the Bonds of CPU-Centric AI Inferencing with NeuReality

"Server-on-a-Chip"
cost, complexity
current state
- training pods (NVIDIA, GRAPHCORE, SambaNova, Cerebras)
- Inferecne Servers: tenstorrent, untether.ai, nvidia, groq, Qualcomm
issue with current systems (according to NeuReality): moving the data between NIC --> CPU --> DLA (Deep Learning Accelerator)
Versal ACAP + unique IP
Kubernetes-managed

Xilinx, Inc. Xilinx, the Xilinx logo, Alveo, Vivado, Vitis, Versal, Zynq are trademarks of Xilinx in the United States and other countries.

All trademarks and registered trademarks are the property of their respective owners.

USB-to-UART cable from an old ISDN modem

2021-07-31T09:00:00+02:00

A short intermezzo from all FPGA-related stuff, this time we deal with an 8051-based USB device. This blog post describes how I converted an ISDN modem (with a USB connection) to a USB-to-UART cable.

During my vacation at my parents' house, I wanted to access the UART on the Ultra96 board to investigate the Linux boot procedure. Surprisingly, I did not manage to find a Raspberry Pi or anything else which can talk UART, but I found a box with old electronics. Among old phones, computer motherboards, GPUs, and other relics of the past I found a PCB with a USB and an RJ45 connector. My sixth sense for electronics made me think that this is a good starting point for a USB-to-UART cable; the device already has a USB, and there will likely be a UART port somewhere.

Note: no half-sane person would go this way to implement a simple USB-to-UART bridge. If talking UART from your computer is your principal objective, just buy a cable, or use a Raspberry Pi.

Initial inspection

USB

The first thing I did is plugging the device into a computer, mainly to verify that it is still somehow alive. The following was printed out in the dmesg output:

[11816.274833] usb 1-4.1: new full-speed USB device number 7 using xhci_hcd
[11818.795110] usb 1-4.1: New USB device found, idVendor=071d, idProduct=1000, bcdDevice= 0.3c
[11818.795119] usb 1-4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[11818.795122] usb 1-4.1: Product: Eicon DIVA USB
[11818.795125] usb 1-4.1: Manufacturer: Eicon Technology
[11818.795128] usb 1-4.1: SerialNumber: 0000001000

We see that the USB works, and we also got the product name (Eicon DIVA USB). A quick search on the internet revealed that this device is an ISDN modem, which matches the observation, an RJ45 socket and the date code of June 1999.

There is even a web page dedicated to various ISDN cards, where the main characteristics of this device are listed: ISDN Cards Central: Eicon Diva USB

PCB overview

There are several easily-identifiable components on the PCB:

Cypress Semiconductor AN2135SC "EZ-USB™" = an 8051-based microcontroller with a dedicated USB engine
Renesas IDTQS32XL384 - 20-bit bus switch and level translator
Siemens PSB2115 - ISDN PC Adapter Circuit (this is where the ISDN magic happens)
VAC 5054x005 ISDN transformer
RJ45 socket
Atmel AT24C32N 32kbit I2C EEPROM (presumably to store manufacturer ID and maybe the program for the EZ-USB)
some power-section related ICs

EZ-USB™

The main microcontroller on this board (AN2135SC) was designed specifically to simplify the development of USB-based devices. EZ-USB Technical Reference Manual is from May 2000, while the USB 1.1 standard was released in August 1998. The TRM goes to a great length to explain the advantages of the USB ("Plug and Play") and also serves as an introduction of the protocol itself.

Resources

EZ-USB on Linux

http://www.linux-usb.org/ezusb/

At this writing, all that firmware is statically linked into the appropriate mini-driver.

Linux drivers

https://github.com/torvalds/linux/blob/v5.11/drivers/usb/misc/ezusb.c

https://github.com/torvalds/linux/commit/8d733e26c076f47e7774c0e5baa74c9b1c01199a

fxload

$ apt info fxload
Package: fxload
Version: 0.0.20081013-1ubuntu2
<...>
Description: Firmware download to EZ-USB devices
 This program is conveniently able to download firmware into FX and FX2
 ez-usb devices. It is intended to be invoked by hotplug scripts when
 the unprogrammed device appears on the bus.

Reverse-engineering the board

Since there are no BGA components on this board, and the main microcontroller has only 44 pins, one can easily use a multimeter to reverse engineer the connections between the most important components.

I gathered the obtained knowledge in the schematics below:

Modifications

From the schematics it is clear that one can easily reach the UART port with some small modifications to the board:

remove U103 MOSFET
connect LED to PC6 (rotate R210 90deg to disconnect one pad, connect the flying pad on the R210 to the uC)
connect UART RX cable to PC0 (R208 pad)
connect UART TX cable to PC1 (R210 pad)
connect ground cable to GND

Firmware

With hardware in place, it was time to write some firmware for the microcontroller. I managed to find a project on GitHub, titled ezusb-firmware, which can serve as a starting point for custom developments. It is licensed under GPLv2 or later and uses sdcc compiler.

Programming the microcontroller

After modifying the code to match the pinout on the (modified) board, and compiling the code with sdcc a HEX file is produces. This gets then easily downloaded to the board with the aforementioned fxload utility:

$ sudo fxload -D /dev/bus/usb/001/008 -s /usr/share/usb/a3load.hex -I firmware.hex -t an21 -v
microcontroller type: an21
1st stage:  load 2nd stage loader
open RAM hexfile image /usr/share/usb/a3load.hex
stop CPU
write on-chip, addr 0x0357 len   23 (0x0017)
<...>
write on-chip, addr 0x036e len   12 (0x000c)
... WROTE: 775 bytes, 10 segments, avg 77
reset CPU
open RAM hexfile image firmware.hex
2nd stage:  write external memory
write external, addr 0x1b00 len   88 (0x0058)
stop CPU
2nd stage:  write on-chip memory
write on-chip, addr 0x0000 len    4 (0x0004)
write on-chip, addr 0x000b len    1 (0x0001)
<...>
write on-chip, addr 0x0862 len   32 (0x0020)
... WROTE: 2378 bytes, 33 segments, avg 72
reset CPU

With the proverbial LED blinking, I was able to confirm that this method of programming the microcontroller works and that the firmware project works correctly.

I have continued the development by adding a UART-related code and later implemented a method to retrieve the UART buffer over the USB and to transmit the data over the UART from the USB. The code can be found in ezusb-uart/firmware.

Software

As the last step, I hacked together an example from the libusb library and some code to manage the receive and transmit buffers. Ideally the terminal management (using the termios API) would be more elaborated, ideally one would also use a dedicated library for such a task. But since the goal was just to capture some data from the Ultra96, the current implementation is sufficient.

The code is available in ezusb-uart/software.

Shown in the screenshot below is the output of ezuart utility when connected to the Ultra96 board:

Summary

Without too much effort, I was able to convert this ISDN modem to a USB-to-UART cable. Several factors contributed to this:

the microcontroller on this board was specifically designed to facilitate the development of USB devices
the firmware for this device is downloaded over the USB, allowing easy modifications and development of custom firmware
the PCB was relatively simple (no BGAs)
a fantastic firmware project by Martin Schmoelzer and Johann Glaser as a skeleton for my development

There are a couple of limitations, the most jarring one is that this microcontroller does not support 115200 baud operation, making it not really suitable as a general-purpose UART cable.

Nevertheless, I would classify this project as a success - I have managed to convert an old piece of junk to a device, which can provide some insight into the boot procedure of the Zynq on the Ultra96 board.

EZ-USB® is a registered trademark of Cypress Semiconductor Corp. All other trademarks and registered trademarks are the property of their respective owners.

Notes from Chisel Community Conference China 2021

2021-06-26T04:00:00+02:00

Here are my notes from the Chisel Community Conference China 2021. The conference took place on June 26th 2021, and was organized as a hybrid conference, open to both on-site and remote participants (over Zoom). The organizers have promised that the recorded talks will be made available online in the next couple of weeks.

Since the conference was located in China it started in the early morning hours for the participants from Europe. I decided that 4:00 is a good compromise between my love for Chisel and my need for sleep; I have only missed the two invited talks at the beginning. In general the conference was quite interesting, and it was clear that Chisel is particularly suited for highly configurable designs, e.g. processors and data-processing pipelines.

Written in italics are my comments.

[Invited Talk] Chisel breakdown 2

...

cloneType

override def cloneType --> autoclonetype2
why have I never needed this?
autoclonetype1 - deprecated, was based on reflection
generated by the compiler plugin

aspect phase

insert additional hardware, layout, verification, ...

Top 10 Common Misconception about Chisel

Institute of Computing Technology at CAS
in Chinese, I understood "Chisel", "DSL", and "okay"

RocketChip

RocketChip is complicated, several additional ... (config, the bus framework, register gen)
"if you are new to Chisel, DO NOT read the source code of RocketChip"
note to self: go read the source code of RocketChip

Verilog is more expressive than Chisel

the presenter argues that logic is fundamentally:
- modules
- combinatorial logic
- registers
technically this is all supported in Chisel
I am not sure if I would agree - I think there is still place for Verilog for low-level stuff, just like some parts of the code (e.g. in Linux kernel) are still written in assembly

Chisel compile errors

types of errors:

Scala compile errors
Scala run-time error
Chisel build error
FIRRTL transform error
Circuit simulation error
distinction between fault (uninit variable), error (returning a garbage value) and failure (segmentation fault)
how are these different?
the presenter argues that Chisel has a stricter type system than Verilog
shouldn't Chisel be compared to SystemVerilog for a fair comparison? (also default_nettype none)
"hidden fault" -> "observable failure"

`Blackbox` for (System)Verilog

Testing

Chisel Tester, Chisel Tester 2, UVM from Verilog
"Agent Faker": TL-C UVM above Chisel Tester 2, open source

equality between Chisel and generated Verilog code

aka "the Chisel compiler is not formally verified"
very complex task and unnecessary, one can run tests also on the generated Verilog
known-good --> successful Chisel projects: RocketChip, BOOM, lowRISC, NutShell, Labeled RISC-V, XiangShan

Quality of Results for Chisel

misconception: Java slower than C --> Chisel hardware slower than Verilog hardware
OK, I experienced this first hand - my colleague was asking me what the fmax is for a typical Chisel-generated logic
will he also mention that Chisel is not an HLS?

Chisel is not HLS

I predict the future
advanced features in Chisel/Scala (i managed to understand "map", "mapper")
PPA: Power-Performance-Area

generated core readability

comments in verilog
EmbeddedTLB
wasn't there a patch to improve readbility of the generated code - check previous CCC

high-performance circuits in Chisel

https://github.com/OpenXiangShan/XiangShan, https://openxiangshan.github.io/
looks impressive

Introducing Decoder Generation API to Chisel

example

7 segment LED
input: b0 - b4, output: a - g, k for cathode
non-valid states = (default, don't care)

theory

AND plane, OR plane (this looks like PAL/GAL, right?, is this really relevant for modern LUT-based FPGAs and ASICs?)
now a full example of the 7-segment decoder implemented in PLA
logic optimization (definition from Wikipedia)
- Quine-McClusky
- Espresso
sparser PLA

Chisel utils

experimental.decode.TruthTable, DecodeTableAnnotation, decoder, QMCMinimizer, EspressoMinimizer
how does this affect later stages (e.g. optimization during the synthesis)
nice Scala feature - "list unpacking" - val a :: b :: [...] = x.toBools()

Practice of High-performance Chip Agile Development with Chisel

XiangShan CPU
agile development (iterative)
SystemVerilog interfaces, Chisel Bundles
processor parameters
Ultra - apparently the highest-end impl of the processor
Chisel solution - Vec in a Bundle as an I/O
FIRRTL transform for printf
configurability
"Chisel = syntactic sugar for Verilog"
link: wallace tree multiplier, CCCC 2021
recursion is allowed in Chisel, the generated Verilog code does not include recursion
https://github.com/OpenXiangShan/XiangShan/pull/812
distinction between Chisel, Scala
Wire, Reg - straightforward to understand
advanced Scala features: object, abstract class, trait
advice: start with Chisel, learn Scala later, i would not agree, learn Scala first
co-simulation
Scala-based modules in Chisel Test2 cannot be used in other tools
behavioral models to replace actual Chisel modules
assertion generation in Chisel
generated names can change between Chisel versions, and can cause problems for physical design

summary

chisel is an advanced HDL, not HLS
parametrization
does not affect PPA (vs Verilog)

Revisiting Diplomacy

not an HLS, HCL = HW Construction Language
configuration
RocketChip
an example of a configuration with Verilog: https://github.com/riscv-mcu/e203_hbirdv2/blob/master/rtl/e203/core/config.v

Verilog defines = "that is a piece of garbage" :D

defines are handled by a preprocessor
no easy way to provide a validation
diplomacy: parameter negotiation framework
CDE (Content Dependent Values)
Scala implicits
Scala type inference and type checking

API

user API (extend Config)
design API (trait)

parameters

hierarchy
topology
global defintion = bad
Parameter, LazyModule, passed implicitly
topology parameters
interfaces as DAG (Directed Acyclic Graph)
acylic: how does DMA look like?
interfaces: AXI, TileLink, ...
2-phase elaboration

Summary

Diplomacy refactor -> stand-alone library
(plans for) TileLink, AXI, ACE, CHI, WishBone
RocketChip newbies
TileLink implementation

Using partial swarm optimization to reduce verification time

prerecorded intro, issues with audio
Partial Swarm Optimization
- v_i - particle velocity
- C - learning factor
- global best

CDMA - some kind of a DMA, apparently
MCIF - presumably memory controller interface

1 channel with weights
3 data channels: IMG, WG, DC

data word: 78*3 bits

process:

generate random solution
calculate the fitness
update the particle speed and position
end a high-quality stimulus was found, else goto 1

A general method of generating stimulus based on SVM

verification consumes a lot of resources
using SVM to predict coverage

SVM - support vector machine (binary classification)

using Matlab -> different types, different functions
Convolution Pipeline:
CDMA, CBUF, CSC, CMAC, ...

CSC - convolution sequence controller = gets the data from DMA, loads/schedules it into MAC

nvdla-csc

SG - sequence generator
DL - data loader
WL - weight loader

training

data, labels = coverage data
process:
training
1. random stimuli
2. get the coverage from VCS
3. use this to create training set for SVM
4. train SVM

an arbitrary value is chosen as a threshold between the labels

couldn't this be done better with Reinforcement Learning? and is the SVM the right tool to use?

Q: relationship between the project and Chisel?
A: answer in Chinese

Summary of Problems and Experiences during the Processor Development based on Chisel

short intro (chisel sheatsheet, Programming in Scala, software thinking, hardware thinking)
val vs def (this could be nasty to debug, def creates a new instance on each call)
Cat vs Map, hardware thinking vs software thinking --> Cat starts at MSB, Map starts at LSB
width - in Cat it should be explicitly defined
careful with DontCare

Syntactic Salt - I like this very much

competitive assignment (loop index)

developer perspective

"Chisel = excelent HDL, free devs from dirty work"
"be familiar with Scala before using Chisel"
check the generated verilog

I liked this talk, the examples were relevant

Q: a question about the competitive assignment (in Chinese)

Q: i understood TPU, SiFive and xie xie

Stimuli Generation by Constrained Markov Chain Monte Carlo Simulation for Chisel-based Deep Learning Accelerator Verification Platform

was this title also generated with a Markov Chain?

nvdla.org

Direction Convolution

input: CSC, output CACC
each MAC cell: 64 multipliers
pipelined status

Constrained Random Testing

Markov-Chain Monte-Carlo (MCMC)
Monte-Carlo: draw independent samples from the distribution
Markov-Chain: the current value is probabilistically dependent on the previous value
Metropolis-Hastings Algorithm (proposal, an acceptance of the proposal)

implementation

use a pool of states to generate a low-correlation stimulus
MCMC-based fault location

very precise and academic presentation, unclear how it relates to Chisel and how this method can be used in practice

Implementation of a Highly Configurable Wallace Tree Multiplier with Chisel

recursion for Wallace tree compression
only 120 lines of Chisel code
configurable pipelining

the algorithm

booth-4 encoding
n*n mul <-> n/2 partial products
sign-extend (i match [...])
tree-compression - columns represented as Array[Seq[Bool]]
Seq vs Array?
what is the benefit of using Array (from Java) vs Vec (from Chisel)
compress the whole tree (+ register insertion)
summary: highly configurable, better scalability, easier to read

Q: something about latency on the slide about pipelining

Agile IC Design Team Working in Chisel, Empowered by Diplomacy and Config

https://www.streamcomputing.com/en/

RocketChip/BOOM
Qs when reading the code: implicit, :*= and :=*, ...

Diplomacy

Scala framework
negotation/castint/modifying parameters
LazymoduleImp

Config

Scala framework
definition of parameters globaly
Chisel =/= Diplomacy and Config
Diplomacy and Config = pure Scala
pros:
- suitable for SoCs
- availability of open-source IP
cons:
- hard to fully understand

building a Chisel Team

firm commitment to Chisel
no modifications of the generated Verilog
divide an conquer: dedicated experts for Diplomacy and Config
stages:
1. Chisel basis (Bundle, Reg, WIre,, ...)
2. Diplomacy and Config (case class, high-order function, other advanced features)
minor tricks: wrap important signals in modules, use CamelCase in Chisel

Experience Sharing: Develop NutShell using Chisel

https://github.com/OSCPU/NutShell

NutShell (5 undergraduates in 4 months)
SDRAM, SPI and UART,
boots Linux (Debian/Fedora)
single-issue, in-order core
RV64IMAC, Zifence, Zicsr
runs at 60 MHz on Zynq 7000, 200 MHz on Zynq US+, 350 MHz in 110 nm SMIC

example:

MaskedRegMap abstract class - address, read and write side effects
apply method, generate method

peripheral devices

MMIO - AXI4 or AXI4Lite
AXI4SlaveModule abstract class

suggestion

avoid Verilog-like Chisel
software-like Chisel - careful with var
find the balance

Chisel Implementation Tutorial - in a lightweight ...

the title in the program was "Light Weight Chisel3 KnitKit"

Data -> Bits -> Clock/Wire/... (outdated diagram)
RegInit with different types of resets

running through slides

io connection with withClockAndReset

https://github.com/colin4124/Chisel-Implementation-Tutorial

created 6 hours ago

Q: something about AutoBundle

Use Firrtl Transform to Control the Effective Range of 'printf' in Large Scale Circuits

printf in Chisel can be translated in fwrite in Verilog
extensive printing will slow down the simulation
typically only as small part of the code is inspected
using FIRRTL transform

Implementation

4 types of annotation:

EnablePrintfAnnotation, Disable.., DisableAll.., Remove Assert...

execute(c: CircuitState)

ancestor (hierarchy)
firrtl.analyses.CircuitGraph

Reinforcement Learning based Stimulus Generation for Chisel Module Verification

Constrained Random Test

test generator
CACC: convolution accumulator
assembly SRAM group, delivery SRAM group (banks of 64Bx32 SRAMs)
assembly = accumulation with saturation

Reinforcement Learning

Agent
Action
Environment
Reward
State

Q-learning (Q = quality), Bellman equation,

states = coverage, actions = stimulus gamma = reward discount

agent is exploring the environment, Q-table is updated, Q is maximized

approach

extremely large input vector: 2 ** 131
multiplexer: stimulus = 128 digits, select = 3 bits

controller written in Python! (at a Chisel workshop) :)

Genetic Algorithm based Stimulus Generation for Chisel Module Verification

Single Point Data Processor = post-processor of NVDLA
inputs: i32 x 16 from CACC

Genetic Algorithm

population
fitness calculation
mating pool
parents selection
mating (crossover and mutation)
offsprings
back to population

approach

20 binary stimulus vectors
fitness calculation - coverage from VCS
matting: exchange bits in vectors between stimuli vectors, mutation: bit flips

again no mention of Chisel, no result presented

Recurring Neural Networks based Stimuli Generation for Chisel Module Verification

constrained random verification (PRG for randomization, constraints for stimuli)

Recurring Neural Network

neuron: x * w - offset + threshold
Hopfield Network (each neurons output is connected to all other neurons but not to itself)

PDP (Planar Data Processing)

results of the pooling process
outputs: max, min, average
traverses width, height, channel
maximum pooling (pipelined)

approach

inputs: control, status from the previous module, data payload
output: status, and data
Python based controller, simulation with VCS

Quasar: SweRV-EL2 implemented in CHISEL

Convert SweRV-EL2 from SystemVerilog to Chisel
comparison of SystemVerilog to Chisel

Quasar

4-stage, mostly in-order, RV32IMC, runs at 600 MHz at 16 nm
development procedure: first unit test, then comparison between SweRV-EL2
LEC (*is this a formal) on the generated Verilog

Pros/Cons

parametreizable & scalable code
no linting problems
people are reluctant about adopting Chisel
"Chisel makes the verification more tedious"

Analysis of the results

fMAX, area and power almost the same (2-3% difference)
Chisel: 12 kLOC, SystemVerilog: 19 kLOC

Roadmap

F-extension (will be open-sourced in the near future)
vector extension

https://github.com/Lampro-Mellon/Chisel-Training

ChiselVerify: A Verification Framework for Chisel

verification = testing before tape-out
validation = testing after tape-out

Current solutions

ChiselTest (not many functions), ScalaTest
SystemVerilog, UVM (verbose, multiple languages)

ChiselVerify

an extension to ChiselTest
4 parts
functional coverage
constraint random verification
bus functional model
timed assertions

functional coverage

statement coverage / functional coverage
verification plan <- cover groups <- cover points] (Range, Conditions, Cross, Timed)
coverage database
coverage reporter

val cr = new CoverageReporter(dut)
  cr.register(
      CoverPoints(...),
      ...
  )
...
cr.printReport()

constraint random verification

constraint programmable language
JaCoP as an SMT solver
custom distribution

bus functional models

AXI4 interface
Transactions
software abstraction

timed assertions

types of delays:
Exactly
Eventually
Always
Never

summary

"test Chisel designs in Scala"

https://github.com/chiselverify/chiselverify

Teaching Digital Design with Chisel

Q at CCC 2020: "Is Chisel ready for class?"
A: yes

two courses: Digital Electronics 1 & 2
VHDL until 2019, DE1 still uses VHDL, DE2 uses Chisel
"VHDL is dying a little bit"
IntelliJ, sbt (sbt run, sbt test)
Digital Design with Chisel (https://www.imm.dtu.dk/~masca/chisel-book.html)
LOC(VHDL)/LOC(Chisel) ~ 2
Simulation + GUI in Swing

Towards Agile Networking Hardware - Chisel at OVHcloud

OVHcloud = 1st European cloud provider
22 Tbps bandwidth, DDoS
anti-DDoS (scrubbing with FPGAs)
attackers are agile, to respond
HLS - reduces performance and agility
does HCL improve agility without affecting performance

de-risking

counter store (hash table with a cuckoo filter)
https://hal.archives-ouvertes.fr/hal-03157426
reset, async reset added recently
with no reset the P&R results are better
Pull Request for Preset

flow

limitation: parameters are embedded in names
solution: a wrapper in Verilog for the Chisel-generated Verilog
successful SV/Chisel cohabitation
sv2chisel ("low level Chisel"), challenges: clock and reset retrieval, choosing types
https://github.com/ovh/sv2chisel

Summary

CocoTB instead of Chisel-testers
Pipeline abstraction (PhD thesis, a DSL on top of Chisel)

Performance counters in Cache Coherent Interconnect in Zynq MPSoC

2021-06-12T20:00:00+02:00

In this blog post I describe my tinkering with the interconnect architecture of the Xilinx Zynq MPSoC. I specifically focus on the Performance Monitoring Unit (PMU) integrated into the Cache Coherence Interconnect (CCI).

I believe that most of the readers of my blog are already familiar with Xilinx Zynq® UltraScale+™ MPSoC, but for the sake of completeness let's do a quick introduction. Zynq MPSoC is a programmable device, combining a quad-core ARM Cortex-A53 (called Application Processing Unit (APU) in Xilinx-speak) and a relatively large FPGA (called Programmable Logic (PL) in Xilinx-speak) in one package. Sitting in between the two parts is an interconnect, more precisely ARM® CoreLink™ CCI-400 Cache Coherent Interconnect. The majority of the connections between the APU and PL go through this interconnect, which makes it one of the more important parts of the Processing System (PS).

In simple use cases the interconnect is mostly transparent for the users; from the APU side, the memory transactions (i.e. reads and writes) come to the interconnect and get routed to the appropriate output port (e.g. DDR controller or PL manager ports - M_AXI_HPMx_FPD).

The overview of the CCI is shown in the excerpt from the UG1085 below - the CCI is shown prominently in its central location.

Linux driver for CCI PMU

Already provided in Linux is a driver for the Performance Monitoring Unit (PMU) in the CCI. This driver is enabled with the CONFIG_ARM_CCI_PMU variable, and for Zynq MPSoC this option is by default already turned on.

The driver prints a short message in the dmesg to indicate that it was successfully loaded:

root@u96v2-sbc:~# dmesg | grep CCI
[    3.218405] ARM CCI_400_r1 PMU driver probed

We can then use perf command to list all performance counters available in the system, and among those there are also listed those from the CCI-400:

root@u96v2-sbc:~# perf list

List of pre-defined events (to be used in -e):

[...]
  CCI_400_r1/cycles/                                 [Kernel PMU event]
[...]
  CCI_400_r1/si_rrq_hs_any,source=?/                 [Kernel PMU event]
[...]
  CCI_400_r1/si_wrq_hs_any,source=?/                 [Kernel PMU event]
[...]
  CCI_400_r1/si_wrq_hs_write_unique,source=?/        [Kernel PMU event]
  CCI_400_r1/si_wrq_stall_tt_full,source=?/          [Kernel PMU event]

Although at this stage the performance counters are already accessible from the system, except for the cycles counter no other counter is actually counting.

It turns out that the CCI-400 IP has some external signals which can disable the performance counters, even if the registers are properly configured.

Presented below is a table from the CCI-400 Technical Reference Manual (archive.today link) which shows that NIDEN must be high for Event counters to be enabled.

Configuration

Looking at the register map for Zynq MPSoC in the UG1087 one can note that there are two modules associated with Cache Coherent Interconnect.

In my opinion, the documentation from Xilinx is a little bit vague, but I presume that the CCI_REG acts as a GPIO which then drives the debug inputs on the CCI module. Shown in the figure below is the CCI module together with this auxiliary module. This block diagram is based on my current understanding.

Accessing the configuration registers

With this in mind, one would be tempted to quickly change the value of NIDEN input directly from user space with devmem utility.

root@u96v2-sbc:~# devmem 0xFD5E0000
Bus error
root@u96v2-sbc:~# devmem 0xFD5E0040
Bus error

Here we encounter another issue, namely the CCI_REG is protected by the Xilinx Memory Protection Unit (XMPU) and is only accessible from the secure environment.

Exception Levels

There are 4 exception levels defined in ARMv8 architecture.

The user space runs in EL0
The kernel space runs in EL1
Hypervisors run in EL2
Firmware runs in EL3

We can get the current Exception Level by reading the CurrentEL register.

In user space this instruction throws Illegal instruction - this is expected
In kernel space the reported level is 1: [ 1091.821735] jan-level: EL = 1
In FSBL the reported level is 3: EL = 3

Patch for the First Stage BootLoader (FSBL)

Since we now know that FSBL runs in EL3, and we need EL3 to access the CCI_REG module, we can patch the FSBL to configure the appropriate registers before continuing with the boot. In this way, the NIDEN and also SPIDEN signals will be already set high before the Linux boots.

I wrote the following patch and included it in the Bitbake recipe for the FSBL:

From 08450fd4c18d11fedf196c65c22f8abf83a6cc2a Mon Sep 17 00:00:00 2001
From: Jan Marjanovic <jan.marjanovic@outlook.com>
Date: Thu, 10 Jun 2021 19:20:25 +0200
Subject: [PATCH] Enable CCI debug (NIDEN and SPINDEN on CCI-400)

---
 lib/sw_apps/zynqmp_fsbl/src/xfsbl_hooks.c | 39 +++++++++++++++++++++--
 1 file changed, 36 insertions(+), 3 deletions(-)

diff --git a/lib/sw_apps/zynqmp_fsbl/src/xfsbl_hooks.c b/lib/sw_apps/zynqmp_fsbl/src/xfsbl_hooks.c
index 80a1314203..b0030a1d67 100644
--- a/lib/sw_apps/zynqmp_fsbl/src/xfsbl_hooks.c
+++ b/lib/sw_apps/zynqmp_fsbl/src/xfsbl_hooks.c
@@ -64,13 +64,46 @@ u32 XFsbl_HookAfterBSDownload(void )
 }
 #endif

+static void print_el(void) {
+   register uint64_t x0 __asm__ ("x0");
+   __asm__ ("mrs x0, CurrentEL;" : : : "%x0");
+   XFsbl_Printf(DEBUG_PRINT_ALWAYS, "EL = %x\r\n", x0 >> 2);
+}
+
+static void cci_reg_dump(void) {
+   // offsets from UG1087
+   uint64_t offsets[] = {0, 0x10, 0x14, 0x18, 0x1c, 0x40};
+
+   XFsbl_Printf(DEBUG_PRINT_ALWAYS, "CCI_REG: register dump\r\n");
+
+   for (int i = 0; i < sizeof(offsets)/sizeof(*offsets); i++) {
+       uint64_t offs = offsets[i];
+       u32 val = XFsbl_In32(XPAR_PSU_CCI_REG_S_AXI_BASEADDR + offs);
+       XFsbl_Printf(DEBUG_PRINT_ALWAYS, "  offset %x = %x\r\n",
+               offs, val);
+   }
+}
+
+static void cci_reg_debug_enable(void) {
+   const uint64_t OFFS_CCI_MISC_CTRL = 0x40;
+
+   const uint32_t CCI_MISC_CTRL_NIDEN_MASK = 0x2;
+   const uint32_t CCI_MISC_CTRL_SPIDEN_MASK = 0x1;
+
+   XFsbl_Printf(DEBUG_PRINT_ALWAYS, "CCI_REG: debug enable\r\n");
+
+   XFsbl_Out32(XPAR_PSU_CCI_REG_S_AXI_BASEADDR + OFFS_CCI_MISC_CTRL,
+           CCI_MISC_CTRL_NIDEN_MASK | CCI_MISC_CTRL_SPIDEN_MASK);
+}
+
 u32 XFsbl_HookBeforeHandoff(u32 EarlyHandoff)
 {
    u32 Status = XFSBL_SUCCESS;

-   /**
-    * Add the code here
-    */
+   print_el();
+   cci_reg_dump();
+   cci_reg_debug_enable();
+   cci_reg_dump();

    return Status;
 }
--
2.25.1

The following is then the output of the FSBL with the patch:

Xilinx Zynq MP First Stage Boot Loader
Release 2020.2   Jun 10 2021  -  19:49:38
Reset Mode      :       System Reset
Platform: Silicon (4.0), Running on A53-0 (64-bit) Processor, Device Name: XCZU3EG
SD0 Boot Mode
PMU Firmware 2020.2     Jun  3 2021   19:28:36
PMU_ROM Version: xpbr-v8.1.0-0
Protection configuration applied
EL = 3
CCI_REG: register dump
  offset 0 = 0
  offset 10 = 0
  offset 14 = 8000003F
  offset 18 = 0
  offset 1C = 0
  offset 40 = 0
CCI_REG: debug enable
CCI_REG: register dump
  offset 0 = 0
  offset 10 = 0
  offset 14 = 8000003F
  offset 18 = 0
  offset 1C = 0
  offset 40 = 3
Exit from FSBL

We can note that the register at the offset 0x40 has changed from 0 to 3, as we have requested.

Lamport's bakery algorithm

To provide a good test case for the cache coherent interconnect I have implemented distributed counting (i.e. multiple workers share one counter), and the synchronization is provided with Lamport's bakery algorithm.

AXI protocol itself provides a possibility (AxLOCK signals for exclusive access) to perform atomic operations on memory and devices, but Lamport's algorithm does not require any special locking primitives, only atomic reads and writes.

In the Vivado design I have connected two LamportsBakeryAlgo IPs to both HPC ports. Each IP can be configured to run for a defined number of loops, and in each loop the counter will be incremented by 1. For each loop we expect to see 5 writes in total:

1 write for setting the entering flag
1 write for setting the number
1 write for clearing the entering flag
1 write to do the actual work (increment the counter in this example)
1 write to clear the number (i.e. release the lock)

Similarly, we can expect the following to be the lower limit of the number of reads - if the flag is set the algorithm continues to poll it until it is cleared. In this highly-contended example we expect the numbers of reads to be much higher.

N reads for the maximum() function
N reads in the inner loop to check the entering flag
N reads in the inner loop to check the number variable

Performance counters usage example

With the Performance Memory Unit enabled we can now start monitoring the values of the counters with the perf tool.

I ran the two LamportsBakeryAlgo IP, each programmed to perform 100000 loops. In addition, there is a third instance of Lamport's bakery algorithm, this one running in the software and accessing the same memory locations as the two FPGA implementations.

In parallel, I ran perf stat and selected the following events: read requests (rrq) handshakes (hs) and write request (wr) handshakes on subordinate ports 0 and 3. From the UG1085 we can see that the APU is connected to port 3 on the CCI-400 and the HPC ports are connected to port 0 on the interconnect.

root@u96v2-sbc:~# perf stat -e \
> CCI_400_r1/cycles/,\
> CCI_400_r1/si_rrq_hs_any,source=0/,\
> CCI_400_r1/si_wrq_hs_any,source=0/,\
> CCI_400_r1/si_rrq_hs_any,source=3/,\
> CCI_400_r1/si_wrq_hs_any,source=3/

^C
 Performance counter stats for 'system wide':

        1103648324      CCI_400_r1/cycles/
           3207971      CCI_400_r1/si_rrq_hs_any,source=0/
           1000000      CCI_400_r1/si_wrq_hs_any,source=0/
           3158786      CCI_400_r1/si_rrq_hs_any,source=3/
            636417      CCI_400_r1/si_wrq_hs_any,source=3/

       4.138963536 seconds time elapsed

We see that expected 1000000 writes from the HPC ports (= 2 (instances) * 5 (per loop) * 100000 (number of loops)) and we also see the number of reads on the HPS port matches our expectation to be equal or greater than 1800000 (= 2 (total instances) * 3 * 3 (total instances = N) * 100000 (number of loops)).

Port 3 (APU port) is also used to load the program from the main memory or the SD card which slightly obscures the number of transactions performed by the algorithm itself, but the numbers match our expectations.

Limitations

It seems that the PMU in the CCI-400 does not provide all facilities needed for perf record to work properly, it reports PMU Hardware doesn't support sampling/overflow-interrupts. Try 'perf stat'.

Conclusion

In this post we have seen how tools commonly used in performance engineering can also be used to observe the behavior and performance of the FPGA. With the proliferation of heterogeneous architectures, having good tools to observe (and debug) the interfaces between individual components provides additional insight into the system.

It is yet to be explored if shouting at the Zynq MPSoC has any impact on the performance in terms of latency.

Xilinx, Inc. Xilinx, the Xilinx logo, Vivado, Zynq are trademarks of Xilinx in the United States and other countries.

All trademarks and registered trademarks are the property of their respective owners.

Stratix V accelerator card from eBay, part 7

2021-05-15T13:30:00+02:00

The reverse engineering of the FPGA accelerator card from eBay is progressing well and documenting the DDR3 interface was a huge step forward. However, there is still one important part of the board which is not fully uncovered, the PCI Express® interface.

Hardware

Pikes Peak

Pikes Peak is a Stratix® V-based FPGA accelerator conforming to Open CloudServer OCS Tray Mezzanine Specification. It connects to the motherboard with a 160-pin connector which provides power, management, and a 16-lane-wide PCIe interface. With the resistors on the PCIE_CFG_ID, Pikes Peak indicates 2 x8 bifurcation.

I have explored the various details of this board in previous posts on my blog. Important for today's discussion is the part 2, where using a custom HW adapter I was able to establish the connection between the PC and the FPGA using the factory image on the board (stored in on-board Flash).

The part number of the device (5SGSD5) indicates that it contains one PCIe hard IP block.

At that time I have assumed that one link of 8 PCIe lanes is connected to the PCIe hard IP block and that another x8 link is connected to 8 transceivers on the other side of the device, maybe with the PCIe IP implemented in logic. Later we will explore why this assumption is not correct.

Storey Peak

Storey Peak contains (roughly) the same components as Pikes Peak but in a "normal" PCIe card form factor. For me, as a hobbyist who is not very keen on designing custom HW and prefers to stay in the FPGA domain, this form factor is much more convenient.

I have presented the main components of this board in a series of tweets a couple of weeks ago:

Tweet of janmarjanovic/1383876145821589505

An attentive reader will note the difference between the PCIe devices IDs on Storey Peak and Pikes Peak: Store Peak reports 0xb101 while Pikes Peak reports 0xb100.

First test

I have instantiated the Avalon-MM Stratix V Hard IP for PCI Express and connected it to a couple of peripherals (GPIO, System ID) to provide a minimal example for access from the computer. I have also connected the Hard IP status interface to the Nios II processor, which allows me to query the status of the PCIe link over the JTAG. There is only 1 location where the PCIe lanes can be connected in a normal 5SGSD5 (annotated in light pink in the image below), so I connected the PCIe lanes there.

First test with Pikes Peak

Once everything was set up I have connected the Pikes Peak board through the x1 extender to one of the slots in my DELL PowerEdge R720.

The "OTMA adapter" does not properly connect the reset pin to the Pikes Peak connector, and I had to manually issue a reset command after power-up. Below is the output from the CLI running on the Nios II processor in FPGA:

init done

pp_sp_example > pcie status
pcie:
  id = 2c1e57a7
  version = 10000
  status.cur speed   = 0
  status.LTTSM state = 0
  status.lane act    = 0
  status.DL up       = 0

pp_sp_example > clks
Clock counter: ident = 0xc10cc272, version = 0x10000
Clock frequency [0] =  124.999 MHz
Clock frequency [1] =  644.531 MHz
Clock frequency [2] =   99.998 MHz
Clock frequency [3] =   99.998 MHz
Clock frequency [4] =    0.000 MHz
Clock frequency [5] =    0.000 MHz
Clock frequency [6] =    0.000 MHz
Clock frequency [7] =    0.000 MHz

pp_sp_example > pcie reset
pcie: asserting reset...
pcie: deasserted reset...
pcie: reset deasserted

pp_sp_example > pcie status
pcie:
  id = 2c1e57a7
  version = 10000
  status.cur speed   = 2
  status.LTTSM state = f
  status.lane act    = 1
  status.DL up       = 0

We see that clocks 2 and 3 are both 100 MHz, which indicates that the PCIe clock distribution works correctly, and after the reset the link reports:

current speed: 5 GT/s - maximum supported by the IP in this configuration
LTSSM state: L0 - this is the state in which the state machine should be
lane active: 1 - as expected, the USB cable contains only one high-speed differential lane

This is all looks very promising; once the PC boots I am also able to get the same information from the lspci utility:

$ sudo lspci -s 03:00 -vv
03:00.0 Non-VGA unclassified device: Altera Corporation Stratix V (rev 01)
    Subsystem: Device 01a2:0001
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 15
    NUMA node: 0
    Region 0: Memory at df800000 (32-bit, non-prefetchable) [size=2M]
<...>
    Capabilities: [80] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
        DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #1, Speed 5GT/s, Width x8, ASPM not supported
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s (ok), Width x1 (downgraded)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
<...>

And finally, I can use pcimem to read and write to the registers in the FPGA, e.g. read system ID and timestamp and write to a GPIO pin:

$ sudo ./pcimem /sys/bus/pci/devices/0000:03:00.0/resource0 0x100000 w
/sys/bus/pci/devices/0000:03:00.0/resource0 opened.
Target offset is 0x100000, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x100000)
PCI Memory mapped to address 0x7f3ecc8ba000.
0x100000: 0x01A20001

$ sudo ./pcimem /sys/bus/pci/devices/0000:03:00.0/resource0 0x100004 w
/sys/bus/pci/devices/0000:03:00.0/resource0 opened.
Target offset is 0x100004, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x100004)
PCI Memory mapped to address 0x7f65b24b3000.
0x100004: 0x6097B8A9

$ sudo ./pcimem /sys/bus/pci/devices/0000:03:00.0/resource0 0x0 w 1
/sys/bus/pci/devices/0000:03:00.0/resource0 opened.
Target offset is 0x0, page size is 4096
mmap(0, 4096, 0x3, 0x1, 3, 0x0)
PCI Memory mapped to address 0x7fec6c0cc000.
0x0000: 0x00000000
Written 0x0001; readback 0x   1

The proverbial "blinky" is actually blinking and so far everything seems to be working correctly.

First test with Storey Peak

Encouraged by the success with the Pikes Peak board and the adapter PCB, I have installed the Storey Peak board instead. Compared to the Pikes Peak board and the cable salad there, this one fits much nicer - both from purely esthetic as well as thermal point-of-view.

And here I encountered an unpleasant surprise. Although the clocks are there, the PCIe link can never reach the L0 state, i.e. the link never finishes training.

pp_sp_example > pcie status
pcie:
  id = 2c1e57a7
  version = 10000
  status.cur speed   = 1
  status.LTTSM state = 1a
  status.lane act    = 8
  status.DL up       = 0

Intel® provides good troubleshooting resources for the PCIe link, and I have decided to have a look at the PIPE interface to see the data exchange between the FPGA and the CPU.

We can decode the data received by the first two transceivers and obtain the information from the TS1 ordered sets:

field	xcvr 0	xcvr 1
COM	0xBC (K28.5)	0xBC (K28.5)
Link #	0x00	0x00
Lane #	0x08	0x09
N_FTS	0x1E	0x1E
Rate ID	0x0E	0x0E

We see that the first transceiver receives lane number 8 and the second transceiver receives lane number 9. At this point one can look again at the board, and realize that the upper 8 lanes are connected to the PCIe Hard IP in the FPGA.

This explains why we see lane 8 on the first transceiver, and also explains why the link does not get established.

Storey Peak in a computer which supports bifurcation

Just for a test, I have plugged the Storey Peak card in a computer that supports 2 x8 bifurcation on a x16 slot.

We can see that in this configuration there are two PCIe endpoints presented:

$ lspci
<...>
86:00.0 Unassigned class [ff00]: Microsoft Corporation Device b100 (rev 01)
87:00.0 Unassigned class [ff00]: Microsoft Corporation Device b101 (rev 01)
<...>

And each of them supports a x8 link. It is theoretically possible that one of the endpoints is implemented in FPGA logic, but we will later see that this is not the case.

$ sudo lspci -s 86:00 -vv
86:00.0 Unassigned class [ff00]: Microsoft Corporation Device b100 (rev 01)
<...>
        LnkCap:    Port #1, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us
<...>
        LnkSta:    Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
<...>

$ sudo lspci -s 87:00 -vv
87:00.0 Unassigned class [ff00]: Microsoft Corporation Device b101 (rev 01)
<...>
        LnkCap:    Port #1, Speed 8GT/s, Width x8, ASPM not supported, Exit Latency L0s <4us, L1 <1us
<...>
        LnkSta:    Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
<...>

Unfortunately the DELL R720 which I use in my homelab does not support PCIe bifurcation.

Investigation

With some confusion regarding the assignment of PCIe lanes and the number of hard IP blocks in this device, it is time to carefully check the presentation on Heterogeneous Computing @ Microsoft

Shown on slide 14 is the screenshot from Quartus® Chip Planner. Comparing this screenshot with the Chip Planner view for a "normal" device, we can note that in the presentation there is an additional PCIe block, which is not present in a normal device.

Web search

There were a couple of speculation on Twitter by @rombik_su and reddit on what this part might be, and we can see that the last part of the code (reserved for Special order devices) contains AC.

Tweet of rombik_su/1257741366144180226

Searching for the full code (5SGSKF40I3LNAC) also finds a PDN from Intel with a comment that the device is available "Upon Request, Please Contact Sales".

There is also a thread on the Intel forum by somebody from Quanta who has a problem with a post-production check on this device.

Bitstream analysis

At this point it is pretty clear that this device is a custom part with 2 PCIe Hard IPs. I was curious if I am able to find the evince of the second IP in the bitstream.

I have used the same approach as for the DDR3 controller: by changing one variable at the time I was able to isolate the address in the JIC file which corresponds to the individual PCIe IP parameter. By repeating this procedure I was able to extract addresses for several parameters, especially interesting are the ones that can be observed from the outside, such as vendor ID, device ID, BAR type and size, ...

After I have managed to find the location of several parameters, I have extracted the values from the original JIC. As expected, one can easily see Microsoft® vendor ID (0x1414) and the device ID (0xb100):

I have then taken the bits which correspond to the PCIe block on the left side and performed a weighted autocorrelation on the bitstream. The procedure and the result can be found in a Jupyter notebook.

One can see that there is a strong peak very close to the original configuration: the second configuration differs by just 1 bit. This is exactly what one can expect since we know that the two endpoints have different device IDs (0xb100 and 0xb101).

With this we can confirm that there is a configuration for the second Hard IP stored in the factory image in the on-board Flash.

Conclusion

It is now clear that this board contains a custom part with two PCIe hard IP blocks. Unfortunately, the block which is available in a normal device is connected to the high lanes (lanes 8-15), and this requires at minimum some support from the motherboard to be useful.

There are several possibilities on how to proceed here:

Playing stupid method

I tried playing stupid and just instantiated two hard IPs in my design. Quartus was (as expected), not so easily fooled:

Riser cable

One could take a x16 riser cable and resolder the wires to swap the two x8 connections on the connector. This will only provide 8 lanes, as the second hard IP cannot be used (yet).

Upgrading my homelab server

Is somebody interested in buying a DELL PowerEdge R720? :)

Soft PCIe IP

One could think about developing a soft IP and connect it directly to the transceivers. The development effort here would be enormous since Physical, Data, and Transaction Layers need to be handled in the logic. On the other hand, this could also be a great learning experience to get familiar with the lower layers of protocol, which are usually handled by a dedicated PCIe block. Additionally, one would have to use few advanced features of the transceiver itself (rate change, 8b/10b and 128b/130b decoder, receiver detection, lane bonding, ...).

Manipulating the bitstream

Since we have the configuration stored in the factory bitstream, one could consider "pasting" this configuration on top of a normal bitstream. This method does not seem user-friendly, and the main obstacle would be connecting the user logic to the second (unknown) hard IP.

Unlocking the custom part in Quartus

The database for this custom part is either already integrated into Quartus or it is delivered to the customer as a separate package (e.g. a DLL or a shared object).

If one greps for 5SGSMD5 in the Quartus directory there is a library that contains a couple of interesting strings:

$ strings libddb_dstr.so | egrep '5SGSMD5'
<...>
5SGSMD5H3F35I3LNAA
<...>
5SGSMD5K3F40I3YY
<...>
5SGSMD5_MS

So far hard work and determination have paid off. It seems that continuing to poke around Quartus libraries would be the right way ahead.

Intel, the Intel logo, Altera, Nios, Quartus and Stratix words and logos are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.

Microsoft® is a registered trademark of Microsoft Corporation in the United States and/or other countries.

PCI Express® and PCIe® are registered trademarks of PCI-SIG.

All trademarks and registered trademarks are the property of their respective owners.

Stratix V accelerator card from eBay, part 6

2021-02-28T13:00:00+01:00

In my last blog post I have presented a method to extract the DDR3 pinout from the bitstream, obtained from the on-board Flash. In this post I will use the information obtained from the bitstream to instantiate a DDR3 memory controller and measure its performance.

DDR3 SDRAM Controller with UniPHY

Once the pinout was obtained from the bitstream and the DDR3 part number was identified from the markings on the board, instantiating the DDR3 controller was relatively straightforward. To speed up the development, I first started with just only 1 DDR3 chip (8-bit width).

The first sign of the success were the LEDs indicating that the DDR3 initialization controller has finished its job and that the calibration (of the delay chains) was successfully completed.

EMIF Debug Toolkit

As mentioned in part 3 of this series, I have spent quite some time making the JTAG work. OpenOCD was able to detect the device in the JTAG chain, but I wanted to have a native integration with Intel tools.

The efforts have now paid off, as I was able to use External Memory InterFace Debug Toolkit to inspect the calibration results and obtain other internal data from the DDR3 memory controller.

Single bank

Presented in the following two figures are the so-called margin reports for the read and write cycles. At the start-up, the controller performs a series of reads and writes, and measures the data valid window. This information can be then retrieved over the JTAG (with EMIF Debug Toolkit).

Full interface

Once it was clear that the interface works correctly (and that the reverse-engineering procedure described in my previous blog post works), I extended the interface to the full 72-bits and enabled the ECC.

The margin report from the EMIF Debug Toolkit unfortunately do not convey information on whether the results are as expected for the DDR3 interface. The data valid window is also not presented in a misleading way, with the Unit Interval (UI) being indicated to be 1400 ps, while in reality it is only 625 ps (for the operation at 1600 MT/s).

In a presentation of a Cyclone 10 board with a DDR3 memory controller, Intel shows a similar result to the one achieved on the Stratix V board. From this I would assume that the DDR3 interface is configured correctly.

Memory Checker IP

At this point we know that all data and control pins are working correctly, but we have not yet really tested if the address decoding works properly.

For this task I have developed a small IP core, called Memory Checker. It has an Avalon-MM master interface that can be connected to the DDR3 memory controller. The IP can be instructed to either populate the memory with one of the 8 patterns ("all 0s", "all 1s", "walking 1", "walking 0", "alternate", "8-bit counter", "32-bit counter", "128-bit counter") or to read the content back and check the content of the memory against the expected value. The results of the checker are made available to software through the control interface.

Initially I wanted to use the AXI interface on this core (to make it compatible) with both Intel and Xilinx tools, but unfortunately I could not make the bursts work in the Intel Platform Designer (previously known as Qsys) with the AXI-to-Avalon adapter.

Memory Checker IP in its natural habitat, connected to the DDR3 memory controller:

Driver

The IP also provides a driver, which gets automatically included in the BSP. The function mem_check() performs the memory check and outputs the information on the stdout.

Output

The output from the memory test procedure is presented below. The main function first confirms that it is talking to the right IP (by comparing the expected and the real value of the ID register), then retrieves the configuration of the IP (Avalon-MM interface width, burst length) and then continues with the test procedure for all eight test modes. After each test is complete, the result (PASS or FAIL) is printed, and the throughput is presented.

To test the entire 4GB of RAM, the entire procedure takes only several seconds, which is significantly faster than the SW-based memory test, provided as a part of Nios example design.

[mem check] =================================================
[mem check] IP id = 0x3e3c8ec8, version = 10003
[mem check] Avalon width = 64 bytes, burst len = 128
[mem check] mode = all 0s
[mem check]   results = PASS (67108860 / 67108860)
[mem check]   write throughput = 7816 MB/s
[mem check]   read throughput = 9662 MB/s
[mem check] mode = all 1s
[mem check]   results = PASS (67108860 / 67108860)
[mem check]   write throughput = 7816 MB/s
[mem check]   read throughput = 9662 MB/s
[mem check] mode = walking 1
[mem check]   results = PASS (67108860 / 67108860)
[mem check]   write throughput = 7816 MB/s
[mem check]   read throughput = 9662 MB/s
[mem check] mode = walking 0
[mem check]   results = PASS (67108860 / 67108860)
[mem check]   write throughput = 7816 MB/s
[mem check]   read throughput = 9662 MB/s
[mem check] mode = alternate
[mem check]   results = PASS (67108860 / 67108860)
[mem check]   write throughput = 7816 MB/s
[mem check]   read throughput = 9662 MB/s
[mem check] mode = 8-bit counter
[mem check]   results = PASS (67108860 / 67108860)
[mem check]   write throughput = 7816 MB/s
[mem check]   read throughput = 9662 MB/s
[mem check] mode = 32-bit counter
[mem check]   results = PASS (67108860 / 67108860)
[mem check]   write throughput = 7816 MB/s
[mem check]   read throughput = 9662 MB/s
[mem check] mode = 128-bit counter
[mem check]   results = PASS (67108860 / 67108860)
[mem check]   write throughput = 7816 MB/s
[mem check]   read throughput = 9662 MB/s

Performance

Although the performance of the HW-accelerated memory test is several times better than the SW-based memory test, I still wanted to see the throughput which can be achieved and how it compares to the full bus throughput (i.e. theoretical maximum).

From my previous experience with Xilinx DDR4 controller I expected to achieve the DDR4 bus utilization of around 80%.

Show in the figure below is the read and write throughput for different burst lengths.

It can be noted that the read throughput reaches a reasonable level at the large bursts. Quite surprisingly, the write level does not reach a reasonable level, it only reaches 60% of the full bus throughput, and it is also not affected by the burst length.

To investigate this further, I have taken two captures of the Avalon interface, once for the burst length of 4 and once for the burst length of 128.

Burst length 4:

Burst length 128:

With these captures I can calculate what is the interface utilization, i.e. for what percentage of cycles was waitrequest signal also being asserted when the write signal was being asserted. This number is roughly 60% for both cases, thus confirming the write throughput measurements from the Memory Checker IP.

Conclusion

In this blog post I have used the knowledge obtained from reverse-engineering the bitstream and instantiated the DDR3 controller. To verify the functionality of the controller and the board itself (it was purchased of eBay for 40 USD after all), I have used Intel tools (EMIF Debug Toolkit) and developed a Memory Checker IP. The test successfully ran overnight and did not detect any errors.

All trademarks and registered trademarks are the property of their respective owners.

Stratix V accelerator card from eBay, part 5

2021-01-10T10:00:00+01:00

The last piece of the puzzle on the Stratix V board is the DDR3 memory pinout. Once this is figured out, I can finally start using the board for my developments. It is obvious that this is not the most simple way to get a cheap FPGA development board, but I generally enjoy the challenges and also took this opportunity to learn something new. Being able to use a board without proper documentation is a valuable skill.

Introduction

Looking at the board from the bottom, we note that the board has 4 ICs on the bottom side and presumably 5 ICs on the top side. The markings on the chips indicate that this is H5TC4G83BFR-PBA - a 4 Gb (512M x 8) DDR3 IC. With a total of 9 ICs, the total interface width is 72-bit, which can be used as a 64-bit interface with ECC.

Looking at the presentation "Heterogeneous Computing @ Microsoft" from A. Putnam and K. Ovtcharov we can see the board without the heatsink which confirms that there are 5 DDR3 ICs on the top side. Another important detail that can be derived from this picture is the orientation of the FPGA; the memory interface is next to the upper right corner, which would correspond to the banks 8A - 8D.

From other sources we also know that there is a 125 MHz clock being feed into the pin M23 (in the IO bank 8D), and a dedicated high-quality clock is a must for a fast DDR3 interface.

Strategy

The DDR3 memory uses Dynamic On-Die Termination (ODT) for the data (DQ) and data strobe (DQS) pins but uses external resistors for termination on the address and control lines. On the Stratix V board, we can observe these resistors under the 5th DDR3 chip on the bottom side of the board.

Since tracks and vias are also somehow visible, figuring out the pinout between the FPGA and external resistors would already give us a starting point.

The data, data strobe, and data mask (DM) pins do not have any external termination resistors, and since all but 1 IC are placed in the clamshell topology, it would be also impossible to probe it with an oscilloscope (without physically modifying the board).

There are a couple of vias for DQ and DQS visible under the 5th IC which can be probed with an oscilloscope, but this is only a small portion of the interface. A different approach is needed.

Configuration Flash (EPCS/EPCQ)

The board contains an on-board Flash which is used to configure the FPGA upon boot. We already know that the board configures itself and exposes the PCIe endpoint, so we know that the Flash was not erased as a part of the disposal process. It is very likely that the DDR3 controller is also a part of the bitstream.

Altera provides a method to encrypt the bitstream with a 256-bit AES algorithm, with the key which is either stored in the fuses inside the device or in a volatile memory, backed by a battery.

Since the boards were never meant to be distributed to customers and were originally only used inside the data center, and since the management of the keys and setting up the provisioning workflow are non-trivial tasks, I would assume/hope that the bitstream stored in the Flash memory is not encrypted.

If the bitstream is not encrypted, then it should be possible to extract the pinout from the bitstream. Since the data pins in a DDR3 memory interface can be swapped around freely (one only needs to observe that the grouping between the data strobe and data), it is most likely enough to classify the I/O standard of the pins in question.

For DQ pins we can expect bidir SSTL, for DM SSTL output, for DQS differential bidir SSTL, ...

HW inspection

We start our exploration by inspecting the hardware itself. One can find some notes from VirtLab where a couple of pins are already mapped; this offers a good start.

I prepared a small FPGA project where pins are transmitting their location as an UART message - e.g. pin K21 will continuously transmit "K21 ". This allows quick determination of which pins are connected to which resistors with an oscilloscope.

Annotated on the picture below are the FPGA pins which can be detected on the termination resistors and vias. I have used color-coding to indicate the voltage level; the signals annotated in cyan have driven the output from rail to rail, which means that there is no external termination while with the signals in red the effect of the external termination can be observed in the signal level.

The annotation can be then transferred to the DDR3 chip pinout - we can see that we have managed to determine all address and control pins and a couple of data and data strobe pins. This will be useful as a validation for our future steps.

Reverse engineering the bitstream

At this point we have obtained every piece of information from the HW we can (without more intrusive procedures such as desoldering the DDR3 chips away). Now it is time to have look at the bitstream for Startix V.

The collection of scripts and other resources is available on my GitHub in a repository otma-pin-re.

JIC format

JIC stands for JTAG Indirect Configuration file which contains both the data to configure the EPCS/EPSQ device itself and the actual FPGA bitstream. Using Quartus Programmer one can dump the content from the EPCS or EPCQ into the JIC format.

Wirebond on GitHub already did this and stored the result here. I have used this file for my analysis.

EPCQ content

As one of the first steps I have plotted the bit density, i.e. how many bits are set in a 1024-bit block. 1024 is an arbitrary number, chosen to make plotting simple/fast.

Shown in the figure below are four different JIC files. On the first subplot we have a JIC file which I have generated from a DDR3 example design. On the second subplot there is a JIC file from the on-board EPCQ and on the third subplot are two JIC files (one with an offset for the FPGA bitstream) where the compression was enabled.

We can see that the bitstream from the EPCQ (factory.jic) contains two compressed FPGA images - the first one immediately after the header and the second one at the half of the memory. Presumably, the first image is a recovery image (since it is smaller in size) and the second one is an application image.

Checksum

Zooming in at the beginning of the bitstream we can note an increase in bit density every 1188 bits. This is presumably a checksum.

Explored in the notebook bitstream_analysis/02_crc_checksum.ipynb is the checksum calculation. It can be confirmed that the checksum is calculated with CRC16 with Modbus polynomial, as it is mentioned in P. Swierczynski: Security Analysis of the Bitstream Encryption Scheme of Altera FPGAs.

Being able to calculate a checksum on the data can serve as a confirmation that one is correctly able to interpret the raw data. This will become especially important once we try to decompress the bitstream.

Compression

To save the space in the Flash memory, the bitstream in the EPCQ is compressed - this can be observed from the increased density in the plots in the previous section.

The algorithm for compression (and decompression) is described in the US patent 6525678.

To give an example, here are two excerpts, the first one from the uncompressed bitstream and the second one from the compressed.

227550 | 00 00 b6 f9 81 8b 00 00 2f e8 00 00 00 00 00 00
227560 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
2279f0 | 00 00 00 00 00 00 b6 f9 81 8b 00 00 2f e8 00 00
227a00 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
227e90 | 00 00 00 00 00 00 00 00 00 00 b6 f9 81 8b 00 00
227ea0 | 2f e8 00 00 00 00 00 00 00 00 00 00 00 00 00 00

225900 | 00 00 6f 9b ff 81 8b f0 2f e8 00 00 00 00 00 00
225910 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
225a30 | 00 6f 9b ff 81 8b f0 2f e8 00 00 00 00 00 00 00
225a40 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
225b60 | 6f 9b ff 81 8b f0 2f e8 00 00 00 00 00 00 00 00
225b70 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

We see that the non-compressed bitstream contains a sequence b6, f9, ... after a long string of zeros. In the compressed bitstream the long string of zeros is interrupted by an f (which indicates that the next 4 elements should be copied from the compressed bitstream) and followed by b6, f9 (as in the original bitstream). The checksum is calculated after the decompression.

With this out of the way, we have a full bitstream where we can start identifying individual features.

Weak pull-up bit

One of the most simple changes regarding an IO pin is turning the weak pull-up resistor on and off.

I have explored the effect of changing the value of the weak pull-up resistor in the notebook extract_pin_addr/test_01-extract_pin_addr.ipynb. In one of the graphs, we can determine the address of the bit responsible for the weak pull-up, and the corresponding checksum bits.

With the confirmation that a pull-up is controlled by a single bit, I have decided to use this bit as a reference address for the rest of the configuration bits. For the classification I use bits at addresses relative to the pull-up bit, i.e. the pull-up bit has an address 0.

In the notebook extract_pin_addr/01_get_pu_addr.ipynb I have then determined the address of the weak pull-up bit for all pins in the banks 8A-8D. Those addresses are then (manually) copied to knowledge.py, which is then used in subsequent notebooks.

Pin configuration

The next step is to determine which bits are relevant for the I/O standard on the pin. I have prepared a lot of bitstreams with pins in different configurations, and then in the notebook extract_pin_addr/03_extract_io_std.ipynb extracted all the bits which are relevant for the IO standard.

Classifier

Once I have managed to get a vector of values that are relevant for the I/O pin standard, I have started to build a classifier. In the first attempt I let the classifier itself figure out which bits are relevant for which features, but I had better success with a hand-crafted classifier, presented in the notebook extract_pin_addr/05_classification_manual.ipynb.

Validation

I have prepared a small project with an 8-bit DDR3 interface and let the classifier classify the pins. I have purposely places also some address and control pins in the same bank, and left some of the pins disabled. The classifier correctly classifies all the pins, the results are presented in the notebook extract_pin_addr/06_validation.ipynb

Classification

At this point we can run the classifier on the decompressed image from the on-board Flash. Since there are two images (a recovery image and an application image) available, I have decompressed both and called them factory_decompress0.jic and factory_decompress1.jic.

I then ran the classifier on the second image and was amazed by the results. The classifier managed to find:

exactly all 72 DQ pins
17 (out of 18) DQS and DQSn pins,
10 DM pins (9 + something which is counted as a DM but it is something else),
25 address and control pins (out of 25), and
5 unknown pins

At this point one could investigate why, for example, one of the DQS pins is not detected and improve the classifier to correctly detect all pins, but I have decided to go ahead and clean-up the results manually.

Clean-up

The clean-up was quite straightforward, since one can base the corrections on the pinout which was determined by physically probing vias and resistors in one of the previous paragraphs.

Out of 5 unknown pins, two were CK and CKn, one was the missing DQSn pin, one was the 125 MHz clock and one was the RZQ pin. One of the pins which was incorrectly classified as DM pin was actually a reset pin (L20).

The full table with the final pinout (DQ pins are omitted for brevity, check the last notebook) is presented below.

pin	dbg	class	addr	control	data	misc
J27	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A0
J21	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A1
J29	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A2
L28	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A3
P26	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A4
M26	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A5
N25	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A6
P25	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A7
N22	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A8
N26	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A9
K27	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A10
L27	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A11
N27	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A12
M27	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A13
N21	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A14
K28	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	A15
J28	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	BA0
K21	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	BA1
L26	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr	BA2
L24	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr		CASn
J23	inp=1, out=1, pu=0, io_std='Scls1', term=0, diff=1	?		CK
J24	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=1	?		CKn
K24	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr		CKE
N23	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr		CSn
M21	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr		ODT
L21	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr		RASn
L20	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM		reset
P23	inp=0, out=1, pu=0, io_std='Scls1', term=0, diff=0	addr		WEn
N33	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM			DM0
T31	inp=1, out=1, pu=0, io_std='S', term=1, diff=0	DQ			DQ0
P34	inp=1, out=1, pu=0, io_std='S', term=1, diff=0	DQ			DQ1
N34	inp=1, out=1, pu=0, io_std='S', term=1, diff=0	DQ			DQ4
M33	inp=1, out=1, pu=0, io_std='S', term=1, diff=0	DQ			DQ5
L34	inp=1, out=1, pu=0, io_std='S', term=1, diff=0	DQ			DQ7
N32	inp=1, out=1, pu=0, io_std='S', term=1, diff=1	DQS			DQS0
M32	inp=1, out=1, pu=0, io_std='S', term=1, diff=1	DQS			DQS0n
M23	inp=1, out=0, pu=0, io_std='2V5', term=None, diff=0	?				125 MHz clk
E23	inp=0, out=1, pu=0, io_std='S', term=1, diff=1	?				DQS2n
B34	inp=1, out=0, pu=0, io_std='2V5', term=None, diff=0	?				RZQ
D21	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM				1
D22	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM				2
D25	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM				3
E27	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM				4
D28	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM				5
E30	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM				6
C34	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM				7
H34	inp=0, out=1, pu=0, io_std='S', term=1, diff=0	DM				8

Summary

In this blog post I have presented the procedure to obtain the DDR3 pinout from the Stratix V board. I have not yet tested the pinout on the real hardware, but I am pretty confident that it should work - I will leave this for the next post.

In general, the procedure was relatively straightforward, but had some obstacles on the way, for example compression, the not-completely-regular structure of the bitstream, and the checksums. I was also quite lucky that the bitstream itself was not encrypted, although it is understandable why this was not the case.

In the next blog post I plan to verify that the pinout presented here is accurate and perform some memory tests.

All trademarks and registered trademarks are the property of their respective owners.

Stratix V accelerator card from eBay, part 4

2020-10-11T16:00:00+02:00

In the part 4 of the Odyssey with the Startix V board from the eBay, I explore the QSFP+ port and establish a 40 Gigabit Ethernet connection to a computer with an Intel X710 40GbE network adapter.

HW overview

Since this board was designed to run in a datacenter, the connectivity is of major importance. The board can connect to the host CPU over PCIe x16 connection, and can communicate with other boards and external world over 2 QSFP+ connectors.

The transceivers in the Stratix V GS can operate at 14.1 Gbps, which means that the board can run 40 Gigabit Ethernet (4x10.3125 Gbps), InfiniBand FDR (56 Gbps = 4x14.0 Gbps) and other fast protocols.

QSFP+ management interface

As described in SFF-8436: Specification for QSFP+ 4X 10 Gb/s Pluggable Transceiver, the QSFP module contains a relatively simple management interface, accessible over an I2C bus.

First I implemented a mini equivalent to i2cdetect to check if I am probing a correct I2C bus. As expected, an I2C device with address 0x50 replies to our request:

 0000 |          -- -- -- -- -- -- -- -- -- -- -- -- --
 0010 | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
 0020 | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
 0030 | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
 0040 | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
 0050 | 50 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
 0060 | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
 0070 | -- -- -- -- -- -- --

We can then continue by dumping the content of the module EEPROM:

rc = 0 | 0d 02 06 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 0d 00 23 08 00 00 00 00 00 00 00 05 8d 00 00 00
rc = 0 | 00 00 02 a0 4d 65 6c 6c 61 6e 6f 78 20 20 20 20
rc = 0 | 20 20 20 20 0f 00 02 c9 36 37 30 37 35 39 2d 42
rc = 0 | 32 33 20 20 20 20 20 20 41 31 04 06 09 00 46 27
rc = 0 | 00 00 00 00 36 43 32 32 31 32 30 37 36 58 20 20
rc = 0 | 20 20 20 20 31 32 30 33 32 37 20 20 00 00 00 64
rc = 0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
rc = 0 | 00 00 00 00 00 00 00 00 00 00 02 00 00 30 00 00

The cable is correctly identified as Mellanox 670759-B23 cable.

40G Ethernet MAC

To implement the physical layer of the Ethernet protocol I used 40- and 100-Gbps Ethernet MAC and PHY MegaCore. Without an appropriate license this IP can be evaluated for an hour, which is not a lot, but enough to test the hardware capabilities of the FPGA board.

To clock the transceiver part I have used the 644.53125 MHz on-board oscillator which I have explored in detail in my previous post. The application part is clocked at 312.5 MHz, generated from the same 644 MHz oscillator.

In the Qsys I have added a Transceiver Reconfiguration Controller and connected it to a JTAG to Avalon Master Bridge, so that I can perform Eye Scan measurements and change other transceiver configurations.

With everything in place, I have programmed the bitstream to the FPGA, and the dmesg output in Linux on the other side of the link reported some good news, the link is up:

[  615.469842] i40e 0000:01:00.0 enp1s0: NIC Link is Up, 40 Gbps Full Duplex, Flow Control: None

Eye scan

Once the link is established, we can use Transceiver Toolkit EyeQ to verify the link quality on the receiver. EyeQ circuitry uses an additional data sampler to sample the receiving data at a time and voltage offset, and compares those to the one received from the center of the data eye. With this method, it can determine BER (Bit Error Rate) for each point in the 2D eye diagram.

Shown in figures below are the results from eye scan on all 4 lanes.

We can see that the eyes are not open very wide, but on the other hand this is an expected result with a relatively high data rate and a passive coper cable. Anyway, the eyes seems to be open wide enough to provide reliable transmission of the data.

Logic Analyzer

At this point we can use SystemTap II Logic Analyzer to capture the received packets from the 40G Ethernet MAC interface.

Shown in figure below is the capture with the SignalTap, with the data from an ARP packet, send from the PC when we want to ping an address.

An experienced reader will recognize a broadcast MAC address (0xFFFFFFFFFF), ARP Ether Type (0x0806), and some other elements in the ARP packet.

Summary

In this short blog post I have explored the 40 Gigabit Ethernet on the Stratix V board. I have managed to establish a link to a computer, and verify the signal integrity of the received signals.

For the next post I will prepare a minimal UDP/IPv4 core, and transfer some UDP packets between the computer and the FPGA.

All trademarks and registered trademarks are the property of their respective owners.

Stratix V accelerator card from eBay, part 3

2020-09-06T16:00:00+02:00

As mentioned in my previous blog post, the next step would be to get the JTAG running in Quaruts. In this blog post I describe how I managed to develop a library as an interface between the FPGA board and Quartus, and demonstrate the developed interface to download the bitstream and to debug the Nios II soft-core processor.

Introduction

HW overview

The Stratix V board contains a 4-pin USB port (with a non-standard connector), to which an FT232H is attached, containing Multi-Protocol Synchronous Serial Engine (MPSSE) which can be used to implement the JTAG protocol.

I am a little bit surprised that the board includes the full JTAG programmer and not only the 10-pin JTAG header, which can be then used by the developers on their development setups. On the other hand, the cost of an FT232H is 2.70 EUR on Mouser for a reel, which is negligible compared to the cost of the board. I can also understand that having a JTAG debugger available on each board is valuable when monitoring the operating conditions (e.g. transceiver link quality with the EyeQ) and to debug and investigate the synchronization issues between the servers in the deployment.

I was not able to find more information about this particular USB/JTAG connection on the Open Compute Project website.

Quartus software suite

Intel® Quartus® provides several programs that are extremely useful for development for Intel/Altera FPGAs.

Quartus Programmer which, as the name suggests, allows programming/configuring the FPGA devices over the JTAG chain, and it can also program the non-volatile memories attached to the FPGAs.
System Console provides "visibility into your system" - one can use JTAG to Avalon MM bridge to read and write the registers in the IPs, and Transceiver Toolkit and External Memory Interface Toolkits both greatly simplify the bringup and debug of transceivers and external memories
SignalTap is an embedded logic analyzer, useful for debugging the logic in hardware, with the input from real devices
Nios II Debugger provides access to soft-core Nios II processor, and it can be used to download the programs, to debug them through GDB, and to obtain the output from the program over the JTAG UART IP

To my knowledge OpenOCD does not provide all these features and Quartus also does not seem to be able to interface to OpenOCD.

OpenOCD

Nonetheless, OpenOCD can be used to scan the JTAG chain, and confirm that the FTDI really is connected to the JTAG port of the FPGA:

$ openocd -f interface/ftdi/um232h.cfg -c "adapter_khz 100; transport select jtag; jtag newtap auto0 tap -irlen 10 -expected-id 0x029070dd";
Open On-Chip Debugger 0.10.0
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
adapter speed: 100 kHz
Info : clock speed 100 kHz
Info : JTAG tap: auto0.tap tap/device found: 0x029070dd (mfg: 0x06e (Altera), part: 0x2907, ver: 0x0)
Warn : gdb services need one or more targets defined

This gives hope that at least on the HW side, integrating the board with Quartus will be easy.

JTAG library

Some information on how to add a custom cable can be found on Intel forums. At the startup, Quartus scans the linux64 folder (if running on 64-bit Linux) and searches for the shared object files which start with the libjtag_hw. The shared object is then loaded, and a function called get_supported_hardware is called. This function returns a structure, containing function pointers for various operations that the programmer and other utilities can perform.

To understand this (undocumented) interface a little bit better, I first implemented a library with a dummy JTAG TAP controller. This library can be copied or linked in the linux64 folder, and then used in Quartus Programmer. The implementation of the JTAG pretends that it is a Stratix V device (by having the same IDCODE) and then discards all the bits which are downloaded to the device. If somebody has too much time, one can easily extend this to create a very complicated .sof to .rbf converter. Finally, the dummy device fakes the status register to communicate that the CONF_DONE is high at the end of the programming. For debugging purposes, the library prints extensive debug information over a UNIX socket.

Once this part in place, it is easy to imagine that combining the library of the dummy JTAG and the HW-related part of the OpenOCD is not so complicated. Since the OpenOCD is licensed under GPL v2, I have decided to re-use the code for MPSSE and FT232H and to also license my library under the same license.

Downloading the bitstream

Here is the documentation of the first victory in this convoluted JTAG bring-up process, the bitstream is successfully downloaded into the FPGA, and an LED on the board is blinking, indicating a total success.

JTAG to Avalon

To check if all functions of the JTAG cable are working, I have prepared a small IP (discussed below) and connected it to JTAG to Avalon Master Bridge. This IP provides access to the Avalon interconnect over System Console.

Shown here is a read of 12 words from a certain address in the Avalon MM memory space:

% set jtag_master [lindex [get_service_paths master] 0]
/devices/5SGSMD5H(1|2|3)|5SGSMD5K1|..@1#bus-instance#OTMA FT232H/(link)/JTAG/(110:132 v1 #0)/phy_0/master
% open_service master $jtag_master

% master_read_32 $jtag_master 0x1000 12
0xc10cc272 0x00010000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x266ac4b1 0x00000000 0x00000000
% master_read_32 $jtag_master 0x1000 12
0xc10cc272 0x00010000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x266ac4ac 0x00000000 0x00000000
% master_read_32 $jtag_master 0x1000 12
0xc10cc272 0x00010000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x266ac4a8 0x00000000 0x00000000
% master_read_32 $jtag_master 0x1000 12
0xc10cc272 0x00010000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x266ac4a3 0x00000000 0x00000000

I will explain later what exactly are we seeing here, for now let's just accept that JTAG to Avalon bridge works, and it reads a magic number register (0xc10cc272), a version register (0x00010000), and the meas_clk[5] register reports a value between 644531368 Hz and 644531377 Hz.

Nios II

For a final test, I wanted to see if I can download a program into the Nios II instruction memory, run the program and observe the output over the JTAG UART interface.

Also here there were no obstacles with the home-made JTAG driver, and I could successfully perform all tasks necessary to download and debug the Nios II core, as also presented on the screenshot below.

Clock

The board contains an IDT8N4Q001 programmable clock oscillator, which is most likely used to generate the clocks needed for 40 Gigabit Ethernet (e.g. 156.25 MHz) and maybe for other communication protocols on the QSFP slots. Since none of the currently available resources (wirebond/catapult_v2_pikes_peak and Microsoft's Catapult v2 (Pikes Peak)) mentions where the IDT is connected to FPGA, I had to find out this myself.

The oscillator on my board has a code 2059, which according to the document from IDT produces 644.53125 MHz for all values of FSEL in the default configuration. This matches the previously measured frequency at the input CLK_R_REFCLK5 (pins T7 and T6).

Clock counter

I have written a small IP to measure the frequency of several clocks from a known clock frequency. As a known frequency I have used the 125 MHz on-board oscillator.

The clock-counter IP generates a strobe signal with a frequency of 0.5 Hz (i.e. pulse width of 1 s). Each of the measured clocks counts with an independent counter. When a transition of the strobe signal is detected, the counter value is stored in a register (accessible on Avalon MM interface) and the counter value is reset to 0. The counter then continuos counting, until the next transition is detected and the same procedure is repeated.

Since the counter is active for exactly a second (within a certain ppm range) before it gets stored in a register, the value stored in the register is the frequency of the measured clock, in Hertz.

The clock-domain crossing for the registers is non-existent, the registers are loaded from one clock (from the measured clock) and read from the Avalon interface clocks. Since the register is updated only once per second (and thus the possibility that we read during an update is quite low) and since this is only used for diagnostics, not implementing a proper CDC can be tolerated.

IDT driver

I prepared a driver for the IDT oscillator, which provides:

a function to decode the bytes into a proper structure (can be used to inspect the current configuration),
a function to encode the structure in bytes (can be used to generate the bytes to be written into the device),
a function to configure all relevant fields (per channel) to obtain the desired frequency

and some other functions.

The main program configures 4 different frequencies, selects one of the four, and then goes into a loop where it prints the measured frequency once per second.

Here is the output of the program:

Clock counter: ident = Oxcl0cc272, version = 0x00010000
IDT8NXQOO1 config:
  MINT     : 25 25 24 25
  MFRAC    : 000000 000000 000000 00000
  N        : 20 16 12 08
  P        : 00 00 00 00
  DSM_ENA  : 0 0 0 0
  LF       : 1 1 1 1
  CP       : 3 3 3 3
  FSEL     : 1
  nPLL_BYP : 1
  ADC_ENA  : 0
Finished configuring IDT oscillator, entering while(1) loop...
Clock frequency [IDT osc]: 6250002 MHz
Clock frequency [IDT osc]: 103430663 MHz
Clock frequency [IDT osc]: 156250082 MHz
Clock frequency [IDT osc]: 156250082 MHz
Clock frequency [IDT osc]: 156250085 MHz
Clock frequency [IDT osc]: 156250086 MHz
Clock frequency [IDT osc]: 156250086 MHz
Clock frequency [IDT osc]: 156250085 MHz
Clock frequency [IDT osc]: 156250086 MHz
Clock frequency [IDT osc]: 156250084 MHz
Clock frequency [IDT osc]: 156250083 MHz
Clock frequency [IDT osc]: 156250082 MHz
Clock frequency [IDT osc]: 156250082 MHz
Clock frequency [IDT osc]: 156250082 MHz
Clock frequency [IDT osc]: 156250081 MHz
Clock frequency [IDT osc]: 156250084 MHz
[...]

We see that after the measurement has stabilized, we are receiving the clock which we have configured, and we see that both the reference clock (125 MHz to generate the strobe signal) and the measured clock have a low wander and a low offset.

Conclusion and future plans

I have managed to prepare a setup that will allow me to develop and debug the Stratix V FPGA directly from Quartus. I am satisfied that I could develop a software solution and use the on-board FTDI chip, and I did not have to solder wires for the JTAG to the board.

With the access to useful tools in Quartus (e.g. SignalTap, Console, Nios II debugger, Transceiver toolkit, ...) I believe bringing up the rest of the board will be much easier.

Finally, to validate that the library for the JTAG cable runs reliably, I have used it to develop a small program that can configure the on-board oscillator to a desired frequency, in my case 156.25 MHz. I plan to use this for a test of the transceivers connected to the QSFP slots.

All trademarks and registered trademarks are the property of their respective owners.

Stratix V accelerator card from eBay, part 2

2020-06-07T16:00:00+02:00

In my previous blog post I have explored the FPGA card I have purchased on eBay, and in this post I will present an adapter card which I have developed. The adapter provides PCI Express connection between a normal card-edge slot and the FPGA card, as well as the access to the I2C bus and some additional signals.

To summarize my previous blog post, the FPGA card has a 160-pin Samtec connector, providing power (12V), I2C bus for management, and a total of 16 lanes for PCI Express.

The adapter

To keep the cost down, I have tried to make the adapter as small as possible, and at this point to only develop a proof-of-concept. On this first variant of the adapter only connects one PCIe lane, and I plan a second variant where all 8 or 16 lanes will be connected to the PCIe edge connector.

The KiCad project for the adapter card is available on my GitHub.

Shown in the image below are the relevant parts of the adapter card.

To connect to the PCIe slot I have purchased one of those Bitcoin mining riser cards, which provide connections between PCIe slot and a USB3 cable. Since there are a total of 4 PCIe reference clock inputs per OCS specification, I have distributed the clock to all inputs using a dedicated IC. I have connected the management I2C bus (MEZZ_SDA and MEZZ_SCL) to a header, which allows me to explore the bus with a Raspberry Pi for example, as shown in the picture below.

I2C

Having a Raspberry Pi connected to the I2C bus, we can first explore the present devices.

Using i2cdetect we can find all devices which have acknowledged their I2C address:

pi@raspberrypi:~ $ i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- 4c -- -- -- 
50: -- 51 -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- 77

We see the EEPROM at the address 0x51, and another two devices at addresses 0x4C and 0x77 - I would assume that these are some kind of sensors and/or regulators.

IPMI FRU

We can now use the i2cdump to dump the content of the EEPROM. As I have expected, the EEPROM is used by the Baseboard Management Controller (or something similar) and the content is compliant with IPMI Platform Management FRU Information Storage Definition.

The following information are stored in the EEPROM:

ChassisArea:
  version: 1
  length: 32 bytes
  chassis type: Rack Mount Chassis
  part nr: X907370-001
  serial nr: (len 0)
  checksum: 146 (OK)
BoardArea:
  version: 1
  length: 64 bytes
  lang: 25
  mfg date: 2016-01-06 03:26
  mfgr: Microsoft
  prod_name: PPFPGA
  serial: OLJ60100194
  part: X900563-001
  file id: FRU 1.0
  checksum: 108 (OK)
ProductArea:
  version: 1
  length: 72 bytes
  mfgr: Microsoft
  prod_name: PPFPGA
  part_nr: X900563-001
  part_ver: 1.0
  part_sn: OLJ60100194
  asset_tag:
  file id: 1.0
  checksum: 241 (OK)

From my understanding of the IPMI standard, the Chassis information should not be present on this card:

A system can have multiple FRU Information Devices within a chassis, but only one device should provide the Chassis Info Area. Thus, this area will typically be absent from most FRU Information Devices.

PCI Express

And finally, the most interesting and the most challenging part, the PCI Express connection.

I have plugged in the card, as shown on the image below and turned on the computer.

And it works! Using lspci command to list all the devices visible to the CPU, one can also note the Microsoft card:

$ lspci
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.2 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family IDE-r Controller (rev 04)
00:16.3 Serial controller: Intel Corporation 6 Series/C200 Series Chipset Family KT Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b4)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b4)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation Q67 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 04)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 04)
03:00.0 Unassigned class [ff00]: Microsoft Corporation Device b100 (rev 01)
04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)
05:03.0 FireWire (IEEE 1394): LSI Corporation FW322/323 [TrueFire] 1394a Controller (rev 70)

Using lspci -vv to get more information, we see that the link is established at 2.5 GT/s (probably too many connectors in series to go faster) and at x1 width (as expected, since we only pass one lane through the USB cable):

$ sudo lspci -s 03:00 -vv
03:00.0 Unassigned class [ff00]: Microsoft Corporation Device b100 (rev 01)
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 11
    Region 0: Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
    Region 2: Memory at fb000000 (32-bit, non-prefetchable) [size=1K]
    Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [78] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [80] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
        DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 128 bytes
        DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM not supported
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR-
             10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS-, TPHComp-, ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
             AtomicOpsCtl: ReqEn-
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [100 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
            Status: NegoPending- InProgress-
    Capabilities: [200 v1] Vendor Specific Information: ID=0000 Rev=0 Len=044 <?>
    Capabilities: [300 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
        LaneErrStat: 0
    Capabilities: [800 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000

Summary and outlook

The adapter card did its job and provided access to the I2C bus and a connection to the PCI Express. I have managed to parse the content of the EEPROM (and realize that there is nothing interesting there) and establish the PCIe connection to the FPGA, as a first step of getting the hardware ready for custom developments.

Eventually I plan to develop a card with a wider PCIe link and get rid of the USB cable setup.

There were some mistakes on the adapter board (rotated Samtec connector, swapped TX and RX on the USB connector) which I could work around, and serve as a lesson to be more careful next time and double-check everything.

As the next step, I would like to understand better how the JTAG chip works. I know that it can be used with OpenOCD, but I would imagine that one can somehow also make it talk to Quartus directly.

All trademarks and registered trademarks are the property of their respective owners.

Stratix V accelerator card from eBay

2020-05-03T14:30:00+02:00

A couple of weeks ago @rombik_su in a Twitter thread pointed out a very cheap FPGA accelerator card on eBay. The board contains a proprietary Samtec board-to-board connector, most likely carrying power, PCI Express, and auxiliary signals (JTAG, IPMI to BMC, ...), two QSFP cages, DDR3 and a large FPGA, hidden under the heatsink.

Being passionate about everything FPGA-related, and with Coronavirus lockdown limiting the number of fun things to do, I decided to purchase the board. An evaluation kit of this kind can easily cost thousands, and 40 USD is a real bargain.

Initial research

While waiting for the board to arrive, I did some initial investigation. The description on eBay is quite cryptic. The title of the listing included all the text from the labels on the board, including the label "AIRFLOW" indicating the direction of the forced air through the board.

One of the most fruitful clues was the label "Microsoft" on the board. It was well-publicized a couple of years ago that Microsoft is using FPGAs to accelerate Bing searches, and this might be one of the boards used in the servers.

Remembering that Microsoft went with Altera (now Intel PSG) and that this was some years back, it is most likely that the card contains a Stratix V FPGA.

The first link on Google (or should I have used Bing? would the FPGAs be aware that I am looking for information about them?) for "microsoft catapult stratix v" presented some conceptually similar cards, but not exactly the same.

The second link, however, presented the exact card which I have purchased:

(from A. Caulfield et al: A Cloud-Scale Acceleration Architecture)

The paper also contains a block diagram highlighting the main components of the board:

(from A. Caulfield et al: A Cloud-Scale Acceleration Architecture)

The paper also mentions "widely-used OpenCompute server" which could give out some information about the pinout on the Samtec connector. Since there are numerous formats for mezzanine cards standardized by the OpenCompute project, and finding the matching one is not trivial, I decided to leave this task for later.

Further digging in the search results, I was also able to find another photo of the board with the heatsink removed.

The marking on the FPGA are removed, but we already know that it is a Stratix V, and some things were expected from previous documents, e.g. 5 DDR3 chips for a combined of 9 chips for 72-bit DDR3 data width).

What I found interesting in this picture is a Flash memory (Micron 25Q256) in the top right corner, which means that the image for the FPGA is most likely stored on the board and is not downloaded through the connector at the startup. It is also possible that Configuration via Protocol is used, and only the basic image is stored in the Flash.

Board overview

Then one day, the board finally arrives. The board matches the description in the previously-mentioned article.

A quick look at the board identifies the following components:

160 pin Samtec SEARAY™ connector
USB (FT232H, used as a JTAG interface)
power entry (FDMS0310AS MOSFET, 0.01 Ohm resistor, unidentified IC with the markings "L536FCD")
power converters (Enpirion® EN2342QI DC-DC converter)
programmable oscillator (IDT8N4Q001)
Flash memory for FPGA (Micron N25Q256A)
I2C EEPROM (ST M24128-BW)
DDR3 memory (SK hynix H5TC4G83BFR)

USB/JTAG

At this point, it was time to start experimenting with the board. The FT232H seems to be powered from the USB BUS voltage: pin 1 of J3 is connected to pin 40 (VREGIN) of FT232H. This is why I decided to first start with the USB. With the datasheet for FT232H it was trivial to determine the pin assignments on the connector. I did not want to solder directly on the connector pins, as I would like to make a proper cable in the future.

After plugging the cable in the computer, the FT232H is recognized as "USB Serial Converter":

Connector, 1st look

At this point I could start trying to reverse engineer the connector pinout. I managed to figure out the following connections:

Ground
Input power: there are a couple of pins connected to the Drain of the large MOSFET. I also assume that the level is, as it is common for PCIe cards, 12V.
There are 16 pairs of AC-coupling capacitors near the connector. PCIe standard mandates capacitors on the TX side, so I assume this is PCIE_TX. Not knowing the exact lane numbering, I decided to enumerate them with letters instead of numbers.
PCIe RX is TBD, but looking at unassigned pins a clear pattern is visible
Some of the pins are connected to the circuit above the connector - I have annotated these pins, but right now I do not have a clear idea what is the purpose. I annotated those pins as AUX.

Power

I have connected the 12V on the Drain side of the Q12 (FDMS0310AS), which has also energized the PU12 and +12V pins on the Samtec connector. However, the power consumption was only 3 mA, and the MOSFET was left closed.

I could not find any information about PU12 (L536FCD). I assume it is some kind of a current-limit protection, measuring the current through the 0.01 Ohm shunt resistor and controlling the Gate pin of Q12.

To literally bypass this problem, I have decided to also connect 12 V on the other side of the Q12. This yielded some results; the power consumption raised to 695 mA, which is what one would expect from such board, and the LEDs turned on.

On various points in the circuit I could also measure all the voltages one would expect to find in such a circuit:

1.5V on the Enpirion (PU16) output
1.35 V on C406 (DDR3 voltage)
0.674 V on C1005 (DDR3 termination voltage)
2.5 V on C442 (periphery)
3.3 V on QSFP capacitors
0.9 V on C354 (FPGA core voltage)

Memories

FPGA configuration memory

The heatsink covers the large part of the top side of the board including the Flash memory for the FPGA configuration. One can, however, still reach the pins 8 and 9 (DQ1 and DQ2, respectively) with an oscilloscope probe.

After the power is applied to the board, we can observe that the FPGA gets programmed in roughly a second. This is above the 100 ms/200 ms limit required by the PCIe standard, but in this custom form factor the value might be different.

DQ2 pin on N25Q256A:

LEDs on the board are also driven by the FPGA and remain lit for a second until the FPGA is not programmed, another indication that the FPGA gets configured from the memory.

EEPROM

Another memory on the board is a small 128Kbit EEPROM, which probably stores MAC addresses, serial numbers, and other similar information.

Quite interestingly, SDA and SCL lines remain stuck low after some time. Maybe the EEPROM is not used at all, or maybe there is some other part of the circuit is keeping the EEPROM interface state machine in a reset.

SDA pin on M24128-BW, SCL is very similar:

Connector, 2nd look

With the board powered on, I could measure the voltage on the connector pins. As expected, on 16 differential pairs I can sense a bias voltage of the PCIe receivers, around 0.7 V.

Some of the AUX pins have a slight bias, but this is probably caused by pull-up resistors and other components. It would require more investigation to fully understand the purpose of these pins.

Components

QSFP

Plugging in a QSFP cable only marginally increases the power consumption. It is clear that the high-speed circuit is disabled, most likely because of an internal register configuration and less likely because the cable is "not compatible", i.e. the board parses the EEPROM and it does not enable the high-speed circuit.

Consumption from the 12V input:

without the QSFP module inserted: 696.7 mA
with the QSFP module inserted: 697.5 mA

IDT oscillator

IDT oscillator is producing 645 MHz clock.

DDR3

I have probed what I presume are the DDR3 termination resistors (on the bottom side, on the other side of the 5th DDR3 component on the top) and observed no switching activity. It seems that the DDR3 controller is kept in reset.

Outlook

The first bring-up session was quite successful; I have managed to turn on the board without damaging it and figure out the basic pinout of the connector.

For the next step, I plan to investigate the OpenCompute website if there exists a document that would describe the pinout of the connector. Still left to be determined are the reference clock and an enable signal from the connector.

Eventually I plan to produce a small PCB that would allow plugging this board in a normal PCIe card slot.

All trademarks and registered trademarks are the property of their respective owners.

Books read: Synchrotron Radiation Sources

2019-11-02T12:30:00+01:00

Here are my notes on the book "Synchrotron Radiation Sources: A Primer", edited by H. Winick. Although a little bit dated (the book is from 1995) it gives a nice overview of all components of a modern synchrotron and helps with better understanding of how all subsystem interconnect together. It also serves as a good introduction in the field of machine physics, which was for me (I have a diploma degree in electronics) quite effective to better understand the challenges faced in synchrotrons of the 4th generation (diffraction-limited storage rings).

Chapter 1

page 2: "electron (or positrons)" --> how would a a SLS with protons look like? larger insertions devices? different SR wavelengths?

page 4: betatron - early machines, vertical magnetic field (spatially const, time varying)

page 7: non top-up mode --> from what I heard it took several hours to start the machine

page 9: $\gamma = \frac{mc^2}{E}$

page 12: time structure - ESRF hybrid mode: "pulsed experiments in us, ns and ps time scale" (M. Wulff et al., Time-resolved structures of macromolecules at the ESRF)

page 13: are the main advantage of FELs short pulses or higher brightness?

page 15: TESLA CDR published in 1988, the book is from 1994.

Chapter 2

page 34: theory on strong focusing --> very interesting, study E. D. Curant et al., Theory of the alternating-gradient synchrotron.

page 35: are betatron oscillations period or is there a phase advance?

page 39: (slightly) rotated quadrupoles introduce x-y coupling

page 41: correctors at the beggining and at the end of the insertion device --> modern take presented in G. Rehm et al.: First projects at Diamond Light Source involving MTCA (https://indico.desy.de/indico/event/20703/session/0/contribution/77/material/slides/1.pdf, page 9)

page 41: what is a "tune shift"?

page 43: planar design --> prevents orbit "growth" in y (apart from coupling from quadrupoles)

page 44: dispersion-free region

page 46: dynamic aperture: max betatron osc that can be sustained

page 48: for how much do insertion devices reduce the energy? is this important for the orbit?

page 49: vertical/horizontal emittance ratio for most machines: 0.01 to 0.03

Chapter 3

page 60: septum magnets: what is the purpose, how do they work

page 62: off axis <-/-> on axis

page 68: max booster boost ratio of 50, in reality a little less. example 1: DESY II (injection at 450 MeV, ejection at 6 GeV - factor 13). example 2: SPS (max energy 450 GeV) to LHC (max energy 6500 GeV), factor 14

page 65: RF photo cathode gun now more popular

page 73: "brute force" used at MedAustron

page 75: 3 kA, 13 kV!!!

page 78: high-Z = high atomic number (e.g. W, Pb)

page 81: harmonic nr = number of buckets

Chapter 4

page 95: first mention of time-resolved measurement

page 91: synchrotron osc = longitudinal, betatron osc = transversal

page 96: cavity impedance scales lineraly with the number of cells

page 105: what about traveling-wave cavity? is this used in SLS?

page 111: How a Klystron amplifier works (https://www.youtube.com/watch?v=Fvud81pYGOg)

page 117: protection for kylstron: Klystron Lifetime Management (http://accelconf.web.cern.ch/AccelConf/ICALEPCS2013/talks/tucoca09_talk.pdf)

Chapter 5

page 122: photon BPMs to global orbit feedback also possible - Diamond, ELETTRA (here only local feedback is described)

page 132: "DIAMOND at Deresbury"

page 145: "the designer is relying very much on the good will of the stell company" --> the reality we work in

Chapter 6

page 159: digital feedback have taken over since the book was written

Chapter 7

page 163: book is from 1994, more modern methods could be used -> PXI or MTCA crate with motor controller and fast ADC

page 194: corrector magnets are not mentioned, but it would be convenient (and interesting) to measure the resp with high freq (e.g. at 1 kHz)

Chapter 8

page 197: "the beam would propagate only a few meters in atmosphere" --> more than I would expect

page 199: desorption <-/-> absorption (photon- and electron-stimulated desorbtion)

page 203: beam stop - photones after diploe --> more than 10 kW of power

page 211: 1e-11 Torr = still 1e11 molecules per L (= 2.5e22 molecules/L of air * 1e-11 Tor in atm)

Chapter 9

page 218: AI: last wave of AI, Lisp-based, very advanced but still very limited (=specific)

page 219: interesting from historical point of view - only EPICS is mentioned

page 220: communication protocols from the past Bitbus (http://accelconf.web.cern.ch/accelconf/p91/PDF/PAC1991_1496.PDF) and Multibus

page 220: reflective memory techniques (for FOFB)

page 225: drift and negative drift sounds very hackish; wouldn't it be easier to take the position and angle of the insertion device (2 step simulation)

page 226: check ref 27: "Computer Codes for Particle Accelerator Design and Analysis: A Compendium"

page 227: phase-space 6D - x, x', y, y' dp/p, ds - each coordinate relative to ideal orbit

page 228: R matrix sometimes 2x2, should is be 6x6 in "normal" case? find some examples for individual elements ...

page 228: beta-function is a solution for single particle motion

page 229: Twiss parameters <-> beta func and beta' (alfa, beta, gamma)

page 239: read again G. Strang: Linear Algebra and Its Applications

Chapter 10

page 245: time-resolved spectroscopy - learn more on this

page 251: "Signal Processing" chapter was written before DSP became mainstream

page 262: section on BPM is rather short --> study ref 61: K. Wittenburg "Beam Loss Detection"

page 271: wire scan is not mentioned?

Chapter 11

page 297: successive alignment steps: non-converging, circling around 0 in N-dim space

page 301: "Cultural noise at DESY" :D

Chapter 12

page 306: "resonate for a long time" - wake field decay --> check at FLASH

page 308: slightly of topic: we are dealing with 18 orders of magnitude

page 312: wake functions = causal functions; here i do not understand enough physics, isn't the field also present in front of the bunch? or is this valid only for ultra-relativistic bunches?

Chapter 13

page 346: three types of motion, three different time scales: longitudinal osc, transversal osc and closed orbit errors

page 349: Fig 13.2 (SSRL) --> feedback too slow to suppres 60 Hz and harmonics

page 351: phBPM: gap few times RMS of the beam

page 353: a mention of feedback simulation, no references given

page 358: Z transform: http://techteach.no/publications/discretetime_signals_systems/discrete.pdf

page 358: "beyond the Nyquist freq" --> not entirely true, undersampling is possible

page 362: MIMO, check:

ref 19
ref 20
ref 21
ref 22

Chapter 14 and 15

no relevant notes

Chapter 16

page 432: "safety is a part of doing things"

page 435: general observation: unreliable safety feature (i.e. interlock) will increase the danger

page 440: "The OPCOs have [...] the authority to stop any activity where safety [...] is in question" - everybody has (or should have) the Stop Work Authority

page 448: tungsten --> impossible to melt with beam

page 456: interlock testing: each input --> response

page 457: for PLCs the standard for functional safety (IEC 61508) should be mentioned. The standard was first published in 1998, while the book is from 1994.

New Ubuntu, old problems with ModelSim

2019-04-20T08:40:00+02:00

A couple of days ago a new release of Ubuntu, Ubuntu 19.04 Disco Dingo was released. On my personal laptop I follow the non-LTS line, which brings me cool features and updated programs (e.g. Python 3.7, GCC 8.3, ...) out-of-the-box.

Unfortunately, because the libraries are updated, the update process causes some programs to become broken. Mentor Graphics ModelSim is for example one of the tools which required some tweaks to make it work on Ubuntu 19.04.

Described here are the steps which made ModelSim to work on Ubuntu 19.04.

Please note: according to Intel® FPGA Software Installation and Licensing, ModelSim - Intel FPGA Edition officially supports RHEL 5, 6 or 7 and Windows. Ubuntu is officially not supported.

Initial attempt

I have started with a fresh installation of ModelSim*-Intel® FPGA Starter Edition Software from Quartus 19.1 package.

When running vsim from intelFPGA/19.1/modelsim_ase/bin I get the following error message:

$ ./vsim
Error: cannot find "./../linux_rh60/vsim"
$

From the path it is clear that ModelSim thinks it is running on RHEL 6. As described in an extensive Wiki entry on Altera software on Arch Linux Wiki, one needs to modify vco file and downgrade freetype library.

Once this is settles (by the way, this used to be enough to make ModelSim work on Ubuntu 18.10) we get the following error:

$ ./vsim
Reading pref.tcl
./../linuxaloem/vish: symbol lookup error: /usr/lib/i386-linux-gnu/libfontconfig.so.1: undefined symbol: FT_Done_MM_Var
** Fatal: Read failure in vlm process (0,0)

This message is new and it required me to do some investigation to get it fixed.

Downgrading fontconfig

From the error message it is clear that libfontconfig.so tries to use function called FT_Done_MM_Var and is unable to find it.

To investigate further I cloned fontconfig source code from: https://gitlab.freedesktop.org/fontconfig/fontconfig.git

A quick grep finds the following instances of the symbol in question:

$ grep -rn FT_Done_MM_Var .
./README:119:      Use FT_Done_MM_Var if available
./src/fcfreetype.c:2261:    FT_Done_MM_Var (ftLibrary, mm_var);
./configure.ac:321:AC_CHECK_FUNCS(FT_Get_BDF_Property FT_Get_PS_Font_Info FT_Has_PS_Glyph_Names FT_Get_X11_Font_Format FT_Done_MM_Var)

One git blame after we find the following commit which introduced this function:

commit 94683a1255c065a7f8e7fadee9de605f3eaf9ac7
Author: Behdad Esfahbod <behdad@behdad.org>
Date:   Mon Jan 8 09:55:41 2018 +0000

    Use FT_Done_MM_Var if available

And then we can find out that release 2.12.92 is the last one which does not contain this change.

I checked out the code from release 2.12.92:

git checkout -b 2.12.92 2.12.92

Installed the libraries needed:

sudo apt install libxml2-dev:i386 uuid-dev:i386

And used the following commands to compile and install an older version of fontconfig library:

aclocal -I m4
autoconf
libtoolize
./autogen.sh
CFLAGS=-m32 LDFLAGS=-L/home/jan/local/packages/freetype-2.4.7-32bit/lib ./configure --prefix=/home/jan/local/packages/fontconfig-2.12.92-32bit --enable-libxml2
make
make install

Changes to vco

Finally, I needed to change vco in intelFPGA/19.1/modelsim_ase/bin folder to load the freshly-recompiled libraries:

$ diff vco vco.orig
11,13d10
< # added for Ubuntu 19.04: recompiled libraries
< export LD_LIBRARY_PATH=/home/jan/local/packages/freetype-2.4.7-32bit/lib:/home/jan/local/packages/fontconfig-2.12.92-32bit/lib:$LD_LIBRARY_PATH
< 
16,17c13
< # changed for Ubuntu 19.04: force 32-bit mode
< mode=${MTI_VCO_MODE:-"32"}
---
> mode=${MTI_VCO_MODE:-""}
213,214d208
<           # added for Ubuntu 19.04: if kernel version 5.x then use linuxaloem
<           5.[0-9]*)         vco="linuxaloem" ;;

Conclusion

Ta-da, ModelSim now works on Ubuntu 19.04. The font styles are a little bit broken, but being humble is a good characteristic, and we won't ask too much.

Notes from FOSDEM 2019

2019-02-03T20:10:00+01:00

I attended FOSDEM 2019 in Brussels:

and these are my notes from Quantum Computers, CAD and Open Hardware and Python tracks:

Quantum Computing

Delivering Practical Quantum Computing on the D-Wave System

Intro

marketing slide for D-Wave Leap
practical = Adiabatic Quantum computer
Riggeti uses gate model instead
physical impl: 3 m high box + 3 normal racks
QPU (Quantum Processor Unit): 16x16 grid

theory

language
- qubit
- coupler (either both in same direction or opposite)
- weights (? initial state)
- strength (for couplers)
- objective (function which gets minimized)
BQM (?)
5 us annealing
problems: noise
noise can bring you in more than one solution (might be useful for some problems)

Markets

portfolio
internet ad
high-energy physics
image recognition

Q & A

clock speed -> no clock speed
total computation time -> 5 us to 1 s
Pegasus: proposed arch for a new machine
new Hamiltonioan in Pegasus
error corrections: from 2048 bits, no error corrections because of what they calculate
classical solutions vs quantum solutions: wall-clock time is the benchmark
hello world:
D-Wave: not a threat for a normal crypto (factoring a number is not a good problem)

D-Wave's Software Development Kit

tools and utilities for QC development

motivation

solution space is "smooth": good solutions are "grouped together"
step 1: problem as polynomial

equation

b-terms: linear bias
a-terms: quadratic bias

Quantum Machine Instr: biases in certain range: because of physical limitaiton? * Chimera graph and Pegasus graph

Ocean software

Python front-end, C++ for high performance
mapping methods
samplers
compute resources

dimod

API for samplers
BQM

cloud client

minorminer

dwavebinaryscp

constraint satisfaction

dwave-networkx

graph theory problem, same API as networkx

Steps

translate to binary
define BQM function
BQM to matrix form
BQM through sampler
post-processing and interpretation

Conclusion

pip install dwave-ocean-sdk

Q & A

resolution: 0.01 resolution
Fujitsu and Hitachi: another providers
8 couplers per qbit

D-Wave Hybrid Framework

decomposition -> split the problem to fit into BQM

github.com/dwavesystems/dwave-hybrid

solver/sampler framework
uses both quantum and classical resources
dataflow paradigm

What is IBMQ

quantum algorithms

current algorithms --> quantum algorithms
supra-polynomial speed-up
Schor's algorithm: polynomial time for factoring of the numbers
simulating quantum mechanics (Hamiltonian equations); for chemistry
factoring 1024-bit number: hours with QC
how to program a QC: mapping interference pattern on qbits
entanglement -> consistent quantum system -> colapse
every quantum program: circuit (no feedback!)
result is non deterministic (robust algorithms provide good results)
quantum volume (metric used at IBM)
coherence time: 100s of us
technology is quantum ready

quantum technologies

superconductive Josephson junction
entaglion (only hypothetical?)
QASM (quantum assembly language)
5 GHz, 240 mK, low noise
all QC look similar: only way to do it
current state: oscilloscopes, signal generators, ...
pizza box for controlling the QC in the future

quantum advantage

IBM provides a lot of stuff on their GitHub
Jupyter notebooks, vscode plugin
Quiskit Acua: example problem: bonding energy for a molecule
Quiskit Aer: ...
publicly available QC
gate error and readout error are publicly available (~1e-3 for the example)
IBM has several QC architectures (Tokyo, Melburne)

Q & A

noise problems for QC, do qbits produce noise -> engineering will find a solution
quantum volue of Tokyo -> "the best, it is not just the number of qbits"
Quiskit Aqua: chemistry API

CAD and Open Hardware

gnucap

intro

mixed-signal simulator
prototype for Verilog-AMS (~10 years ago)
analog circuits and digital circuits simulators are different (transient vs event-based)

analog circuit simulation

node equations --> matrix form
often: differential equations, Newton iteration

digital circuit simulation

event based, with evaluation queue
can be used

Gnucap

Gnucap decompose the circuit matrix into L and U
Gnucap keeps track of the changes to the matrix, schedules an update to the circuit matrix
bypass = not computing something
Gnucap uses all the tricks to calculate inverse of the matrix (pivoting)

architecture of Gnucap

the concept (matrix solving + event-based updates) is tighly integrated in the codebase
plugin infrasctructure: modeling languages (VHDL, Verilog-AMS, SystemC considered)
shared library for basic s
compoenents are plugins (dlopen)
components
commands
algorithms

plugins

Turing complete
examples of plugins
import module (python)
insmod module (linux)
gnucap-python, e.g. Jupyter, user can access internal data, use Scipy, ...
Verilog-A in QUCS/gnucsator

license for models

Gnucap supports models from other sources
two types:
distributed as source code: --> just log it into the Gnucap (no issues)
distributed as binary: --> wrapper + blob

summary

mixed-mode is faster
more front-end work needed

ngspice

talk is not about the details, but about the framework, user interface and future

intro

input: standard SPICE text inpu
output: transient simulator
successor of spice3f5 from Berkley
three flavors:
standard executable: CLI, file and graphics output, control language
shared library for tcl/tk (not used so much)
C shared library (so/dll): input and output via callbacks

scripting language

its own library (developer don't like python)
94 commands, math functions, loops, ...

device models

hard-coded models (BJS, MOS, JFET, xFET, trans lines, Verilog A interface via adms)
B source with build-in function
XSPICE shared library (written in C, both analog and digital)

application areas

PCB design
mix of ICs and discrete components
requires a comfortable user interface (offered by 3rd parties - e.g. KiCAD)
PSPICE and LTSPICE model requirements
IC design
models from the foundries (very reliable but complex)
supports HSPICE
MOS models, large circuits, certain speed
integration with other tools ongoing

mixed-signal capabilities

from XSPICE
digital: event based, signal strengths and delays
analog: C coded models, time and freq domain
simple example: digital is 50x faster than analog

experimental developments

KLU solver: 2x, 3x faster
CUDA for GPU: development on-going
Cider: 1D and 2D TCAD: device structure, solve physics equations

licenses

core: BSD, LGPL, ... no issues
Verilog A models: more complicated
vendor devices: can be used, but not distributed
IC model data: PDKs are under NDA (also encryption)

future

unicode
some commands (pz, ...)
integration with other tools and flows

openEMS - An Introduction and Overview

intro

FOSS solver for electromagnetics fields
simulate and evaluate RF and optical devices
uses FDTD (finite differences in time domain)
co-ordinate systems: cylindrial and cartesian
lumped elements available
human body models
dispersive models
support for remote simulation (cluster)

show cases

notch filter, very nice demo
examples: helical antena, antenna array, MRI antenna design (loop coils)
small size PCB antenna

interfacing

nice to have: interface to PCB editors
problem: link (between EMS and PCB editors) is very week
some examples:
hyp2mat
pcb-rnd
pcbmodelgen (KiCAD to openEMS)
ultimate goal: Circuit simulation <--> PCB design <--> RF simulation

status

openEMS is a mature EM simulation package
TODO list: improve the documentation, interface to tools, ...

Project Trellis and nextpnr

ECP5

85k logic cells (4 LUTS, FF, carry), block RAM, 18x18 DSPs, SERDES (up to 5 GTs)
split into tiles, tiles split into slices
fixed wires
arcs and pip
all arcs and wires are undirectional - mux topology
dedicated clock network
programmable interconnect: pass gates (cascade of 2 mux)

status

bit and routing done
missing: DSP
timing documentation for fabric, logic cells, RAM, ...

text configuration format

tools to convert from and to bitstream
intermediate format for place & route

timing

not enought vendor support
delays for the cells extracted from SDF files
routing delay obtained using least-squares from reports for entire net

workflow

yosys

support ECP5, iCE40, Xilinx, ...
uses Berkley ABC for logic optimization
formal equivalence checking, assertions
....

nextpnr

replacement for arachnepnr
developments from May 2018
timing-driven
architecture implements an API: useful for different architectures
each arch has its own binary: a lot of optimization possible
7-series is VERY experimental (more work planned)
first implementation:
SA placer
A*+ripup router
future
analyty placer
SAT-based placer,
...
nice graphical interface

Design Automation in Wonderland

intro

motivation and goals
reuse common functionality
easy to integrate, easy to adapt libraries
a set of modular libraries
based on Berkley ABC
use C++14 or C++17
header-only
well documented, well tested

libraries

lorina: parsing library

can pasrse very simple Verilog
parser reads Verilog and provides data to mockturtle

mockturtle: logic network library

network interface API
logic synth, opt, technology mapping
impelementations: and-inverted, kLUT, ...
performance tweeks
cut the combinatorial network into LUTs, based on cost function (speed/area)

kitty: truth table

manipulation of truth table

percy: exact synthesis library

re-synthesis

conclusion

exctract the logic function
optimization
mapping to tech

github.com/lsils

Open source virtual prototyping for faster hardware and software co-design

virtual prototyping

current development

idea
SW and HW developed in parallel
integration

virtual prototype

idea
virtual prototype --> SW can be developed in parallel

virtual prototyping: SW environment simulating the HW

example

model entire SoC (RPi: quad core, peripheral)
issues:
models (of the IP) are hard to find
too much components --> needs to be done: shared effort

?

"interoperability is the key" --> take advantage of the community

toolchain: marketplace for components, GUI, ...

Q & A

interface with SystemC and TLM: support for TLM is there
modules: how to verify the model: ?

Lesson learned from Retro-uC and search for ideal HDL for open source silicon

intro

idea: open source microcontroller (Z80, MOS 65 and 68000, 3 uC in one chip)
- a development board
"VHDL and Verilog are not the right tools for the job"

RTL faults

clock is logic signal
if --> mux or flip-flop
synth vs non-synth
Verilog: block and non-blocking
FPGA vs ASIC
RTFLRM

improvements

signed
process(all)
generate
...

"putting a lipstick on a pig"

new developments

TL-Verilog
SystemC/TLM
"good tools are proprierty" (Vivado HLS, Catapult)
Panda Bamboo
GAUT (gaut.fr)
MyHDL
Chisel/SpinalHDL --> "going in the right direction" --> first need to learn Scala
Migen/MiSoc/nmigen --> prefered by the speaker

Python

CPython Memory Management

motivation

what user needs to know
learn how to control (gc, sys.getrefcount)
memory leaks

allocation of memory

CPython has PyObject for everything
size: obj size < 512 bytes --> small, ele big
big objects: system allocator
small object: 3 levels, pools and arena
8-byte alignment --> size idx: size / 8 - 1

pools

4k size, objects of same size
blocks

arenas

encapsulate pools
containts 64 pools

object specificts

string interning (simple string)
small integers (-5 to 256)

garbage collection

reference counting
easy to find unused obj
no marking
memory overhead
no cyclical references

tools in python

two modules: gc and tracemalloc
plus: sys._debugmallocstats()

That's all folks, 'till next year!

Chisel tester with overridden step() method

2018-10-14T21:00:00+02:00

Introduction

Chisel is a modern take on Hardware Description Languages, such as (System)Verilog and VHDL. Both Verilog and VHDL were conceived in 80s, and are currently still the main two options when it comes to describing hardware. From the Developer Experience point-of-view, I would say that both languages are kind of OK once one gets used to them.

Short comparison to VHDL and Verilog

Obviously there are still areas where this two languages could be improved. That is why I have started to experiment with Chisel in my free time. The modules written in Chisel are shorter and thus more readable.

Verbosity

Having one implicit clock domain is (in most cases) great, and everything is then clocked from this clock. This saves a lot of typing compared to the Verilog:

always @(posedge clk) begin: proc_smth
  if reset begin
    // reset logic
   end else begin
    // here comes the real useful stuff
  end
end

and VHDL:

proc_smth: process (clk)
begin
  if rising_edge(clk) then
    if reset = '1' then
      -- reset logic
    else
      -- here comes the real useful stuff
    end if;
  end if;
end process;

I would argue that half of the lines in a typical VHDL modules are not needed, as demonstrated in previous example. A typical module for me would be some DSP or protocol processing module, operating in a single clock domain. For special cases, where precise control of clocks is needed, such as in ADC interface with ISERDES, one can still write the "sensitive" parts in classic HDL.

Development tools

Other advantage of Chisel is: one can use IntelliJ IDEA Community Edition to write code. Compared even to the best VHDL/Verilog IDEs, e.g. Sigasi, IntelliJ is light-years ahead when it comes to refactoring, autocompletion, integration with Git and countless little helpers.

Testing

Chisel is based on Scala, and for hardware generation and testing this is a significant advantage. Chisel provides ChiselFlatSpec which is based on FlatSpec and allows declaring specifications (in style of "Module" should "do something") which are then evaluated.

One area where Chisel is seriously lacking compared to VHDL and Verilog are the implementation of the testbenches (or testers in Chisel-speak). In Verilog and VHDL one can write testbench in a same language with the same constructs as "Device" Under Test. In Chisel, synthesizable logic is written in Chisel, while testbenches are written in Scala.

Better testbenches

If we cannot write the testbenches in same language as logic, let's explore other options. Chisel itself provides multiple testers, such as PeekPokeTester, SteppedHWIOTester and OrderedDecoupledHWIOTester. In my opinion, OrderedDecoupledHWIOTester and SteppedHWIOTester are only suitable for very small modules, and do not provide enough features to sufficiently test a DSP module with AXI4-Stream input, AXI4-Stream output and AXI4-Lite slave port for configuration.

PeekPokeTester allows poke-ing the inputs to DUT and peek-ing the outputs from DUT. It also provides a step() method to advance simulation time by one or more clock period.

In the previously-described case with a DUT with three ports (two AXI4-Stream and one AXI4-Lite) one would ideally need three separate Bus Functional Models (BFMs) which get executed (read their inputs and update their outputs) every clock cycle. This can be achieved by overriding the step() method of the PeekPokeTester.

Overriding `step()` method

The code for this example is available on my GitHub, in chisel-stuff/example-1.

In this simplified (stripped to the minimum) example, we have a DUT with two interfaces. Each interface consist only of data and valid signals, neither DUT nor monitor BFM are unable to backpressure the stream of data. The testbench will consist of three logical units: DUT, Driver BFM and Monitor BFM. Both BFMs are updated every clock cycles, so that driver BFM can drive the input port of the DUT and monitor BFM can in parallel monitor the output port of the DUT.

The core of this examples are the following couple of lines (from OverrideStepExampleTester.scala:126):

  //==========================================================================
  // step

  val rm = runtimeMirror(getClass.getClassLoader)
  val im = rm.reflect(this)
  val members = im.symbol.typeSignature.members
  def bfms = members.filter(_.typeSignature <:< typeOf[ChiselBfm])

  def stepSingle(): Unit = {
    for (bfm <- bfms) {
      im.reflectField(bfm.asTerm).get.asInstanceOf[ChiselBfm].update(t)
    }
    super.step(1)
  }

  override def step(n: Int): Unit = {
    for(_ <- 0 until n) {
      stepSingle
    }
  }

Through Scala's Reflection API we are able to find all instances of classes which have a trait of ChiselBfm, and then call their update() methods. This allows both BFMs to read and write to the ports as they desire, independent from each other.

The instantiations of both BFMs is a little clunky, we need to manually provide them all the methods from PeekPokeTester which are needed during the operation of the BFMs.

Running sbt test in example-1-override-step, we obtain the following result:

[info] [0.002] SEED 1539634207505
[info] [0.023]     0 Test starting...
[info] [0.271]     5 Driver: sent 0
[info] [0.274]     6 Driver: sent 1
[info] [0.278]     7 Monitor: received 1
[info] [0.278]     7 Driver: sent 2
[info] [0.289]     8 Monitor: received 2
[info] [0.290]     8 Driver: sent 10
[info] [0.310]     9 Monitor: received 3
[info] [0.310]     9 Driver: sent 99
[info] [0.314]    10 Monitor: received 11
[info] [0.315]    10 Driver: sent 100
[info] [0.324]    11 Monitor: received 100
[info] [0.326]    11 Driver: sent 65534
[info] [0.340]    12 Monitor: received 101
[info] [0.340]    12 Driver: sent 65535
[info] [0.344]    13 Monitor: received 65535
[info] [0.348]    14 Monitor: received 0
[info] [0.401]    23 Test finished.
Enabling waves..
Exit Code: 0
[info] [0.409] RAN 23 CYCLES PASSED
[info] OverrideStepExampleTest:
[info] pipeline tester
[info] - should compare expected and obtained response

And this is the display of the waveforms from GTKWave:

It can be noted that both Driver and Monitor are able to perform their tasks in parallel.

Conclusion

Shown here is a convenient method to enhance the Chisel PeekPokeTester. In this particular case (when DUT has only one input and one output port), one could also use OrderedDecoupledHWIOTester, but it should be obvious that the method presented here provides more control and flexibility in more complex cases.

Blog restart

2018-09-15T19:00:00+02:00

This blog was left dormant for quite some time now. From the last blog post in February 2017 a lot has happened. I have moved to Hamburg, Germany, to start a new position as FPGA Developer at MicroTCA Technology Lab at DESY.

This is me in European XFEL tunnel:

There are a couple of topics which I would like to explore as a hobby, and having a blog is a nice way to organize your thoughts and outputs. Writing a blog post at the end of the project requires someone to gather his thoughts and to write down the conclusion.

That is all for now, expect more interesting blog posts (probably focusing on FPGA, mid-level languages as Chisel and high-level synthesis) in the future.

FOSDEM 2017

2017-02-03T18:00:00+01:00

I am writing this post from Brussels where I am attending FOSDEM 2017 conference.

There are a lot of interesting talks, and sometimes it is quite hard to decide on which one to go. Luckily there the talks are recorded and one can later check also the missed ones.

Here is my list of the talks I plan to attend:

Saturday

Time	Title	Location
9:00	Welcome to FOSDEM 2017	Janson
11:00	Let's talk about hardware: The POWER of open	H.2215 (Ferrer)
12:00	LoRaWAN for exploring the Internet of Things	K.1.105 (La Fontaine)
13:00	lunch
14:00	Groking the Linux SPI Subsystem	UD2.120 (Chavanne)
14:00 (alternative)	FPGAs in SDR -- Why, when, and how to use them (with RFNoC)	AW1.120
15:00	Everything You Always Wanted to Know About "Hello, World"*	Janson
16:30	Scientific MicroPython for Microcontrollers and IoT	AW1.126
17:00	Libreboot	K.1.105 (La Fontaine)
18:05	Understanding The Complexity of Copyleft Defense	Janson

Sunday

On Sunday I plan to spend most of the time in Electronic Design Automation (EDA) devroom.

Here is a selfie of me with the famous Manneken Pis status:

Books read: E. Stavinov: 100 Power Tips for FPGA Designers

2016-10-18T23:00:00+02:00

I recently found a great book explaining in details FPGA workflow for Xilinx tools, titled 100 Power Tips for FPGA Designers. Evgeni Stavinov is an experienced FPGA designer who previously worked for Xilinx. It is not evident from the title, but this book focuses almost entirely on the Xilinx, while Altera, Lattice and Microsemi are mentioned just briefly in an FPGA vendor list and every once in a while. Due to a fast-paced development of the FPGAs and corresponding tools, it is clear that a book from 2011 would be slightly outdated in 2016. Most notable change in the previous years was a new software suite from Xilinx, called Vivado and the slow introduction of C-to-FPGA tools, such as Vivado HLS.

Nonetheless, this book is ideal for somebody who already has some (formal) education about the FPGA but lacks the real world experience. The author manages to touch every aspect of the FPGA design, from device selection, simulation, coding, debugging, communication protocols, FPGA board bring-up and all small details one should know about FPGAs.

The book (as the title suggest is organized into 100 tips). Here are my notes and comments to some tips this book provides.

Tip 9

The FPGA field has seen some new tools emerge in the past few years, while some other tools ceased to exist or were integrated in other software suites. Under Lint tools should definitely be added Sigasi Editor, an Eclipse based editor for VHDL and Verilog. Verilator is a cycle based simulator, since it can be used as a linter (--lint-only) it should be also added on this list.

Another interesting tool worth mentioning is Doxygen which can create the documentation from the comments in the code and other Markdown documents. The original program does not support Verilog, but there is a fork Doxverilog which also adds a support for Verilog.

Tip 15

This tip states that initial block are ignored by FPGA synthesis tools. This probably a feature which was added after the release of the book, but both XST in ISE 14.7 (http://www.xilinx.com/support/documentation/sw_manuals/xilinx14_7/xst.pdf) and Vivado Synthesis (http://www.xilinx.com/support/documentation/sw_manuals/xilinx2016_2/ug901-vivado-synthesis.pdf) now support initialization of the register from initial block.

Vivado Synthesis Guidelines go even further and suggest using inital instead of reset:

Avoid operational set/reset logic whenever possible. There may be other,
less expensive, ways to achieve the desired effect, such as taking
advantage of the circuit global reset by defining an initial content.

Tip #18

A small typo on page 81, in SystemVerilog logic is a 4-state type which should replace reg, especcialy in cases in which reg keyword may cause a confusion (e.g. always_comb block).

Tip #19

While mentioning code editors for Verilog and VHDL it should be worth mentioning that the one integrated in Xilinx and Altera tools are complete garbage. Vivado did not even had a auto-complete until 2016!

The list of code editors could also be extended with Sublime Text and Atom Editor.

Tip #22

This tip discusses meta-stability and data-coherency on clock-domain crossing logic. It would probably be a good idea to also mention how to proceed when a state machine transitions are controlled with asynchronous signals. This is similar problem to data coherency, all input signals should be re-sampled to the state machine clock domain before they are connected to state-transition logic. Otherwise it is possible for state machine to enter illegal states due to different delays from IO pins to registers.

Tip #26

A small typo on page 124, the last line of code example for shift registers should be:

initial shift4 = init2 >>> 2;   // result is 8'hE0

Note that result after shift operation should be 8'hE0 (-32) instead of 8'h0E.

Tip #27

I was always wondering if a state machine which has a transition from default case defined is equivalent to state machine generated with -safe_implementation switch.

Tip #29

I enjoyed the discussion about various reset mechanism. When I stared developing with FPGAs I always started writing process blocks the same way (if !reset_n init_regs_to_something else my_logic_here) not realizing that often a reset is not needed. This is especially true in data-processing pipelines.

Tip #34

When initializing Block RAM I would suggest using readmemb and readmemh system calls instead of proposed Xilinx custom format, since readmemb and readmemh work also for simulation.

Tips #45-#55

These tips discuss ASIC prototyping with FPGA, which is not my area of interest.

Tip #61

Here it could also be mentioned that Altera offers a free version of ModelSim, called ModelSim-Altera Starter Edition. Compared to Xilinx ISIM and Vivado Simulator, the ModelSim-ASE is stripped-down version of a full ModelSim. Therefore there is a possibility to easily migrate from free to paid version if the need for additional features (such as code coverage) arises.

Tip #62

The figure with the basic testbench components it is a good starting point even for the testbenches which do not use any verification methodology, such as UVM. Several points of what I consider a good testbench (especially for non-UVM, handcrafted testbenches):

There is a main procedure which performs the setup, runs the driver and monitor and at the end runs the checker.
The setup should be done exactly as software would do it, e.g. if there is an AXI4-Lite port with configuration and status registers AXI4-Lite Master BFM should load the configurations settings on that port. Additionally it is also good to try reading back the configuration values and check them against the values written.
The driver and the monitor should operate completely asynchronously one from another. When simulating a module which operates on data stream (e.g. a DSP module with AXI-Stream slave port for input and AXI-Stream master port for output) I like to additionally throttle the output port, to observe how the module behaves when the upstream module is not able to temporarily keep up with data flow.
The checker (as the name suggest) checks the data received on the monitor with the reference implementation. When there is a mismatch between the received and expected value, the checker should clearly show the received and expected value (SystemVerilog assertions are a nice way to do this).

Tip #63

The example which shows the delta cycle delays is fantastic and it also demonstrates that Verilog is very powerful, but also very dangerous language. The example is also based on two different registers being clocked by two different clock. If not absolutely necessary I would advise agains using different clock in the same modules, and to use the FPGA vendor provided FIFOs for synchronization between clock domains.

At the end of this tip there is a Verilog code which stores the state name string in a separated variable. This is one possible source of errors if the names of states are changed or if new states are added. Much better solution would be to use SystemVerilog enums, which add a little bit of type strictness to this otherwise type non-strict language.

Tip #67

I would agree with the observation that the IP, TCP and UDP protocols were not designed to be implemented in hardware. Most problematic is the position of the checksum word in packet header, which does not help neither transmitting neither receiving side. However, https://www.ietf.org/rfc/rfc768.txt foresees sending the packets without the checksum calculation, all bits in the checksum field must be 0. In some cases this may offer an improvement in link latency (if there is some other method to check the data correctness).

Tip #70

This tip describes various FPGA interconnect buses. Due to the Xilinx shift from PowerPC to ARM and with introduction of Vivado, the bus of choice for an FPGA designer should be one of 3 version of AXI buses, either AXI4, AXI4-Lite for configuration registers and AXI4-Stream for streaming data.

Missing on this list is Avalon bus, which is widely used with Altera QSys. There are two versions (memory-mapped and streaming) and provide a very convenient way to interface registers to CPU. Only needed signals need to be specified, while others are automatically added by QSys during the "compilation".

Tip #76

With the new FPGA family, UltraScale, Xilinx provides a new PCIe DMA controller (https://www.youtube.com/watch?v=TzzzM97L4HI). This saves a lot of work to FPGA designers or significantly reduces the price of IPs. By providing various AXI interfaces the PCIe DMA controller enables easy integration with Vivado Block Diagrams. The interfaces on PCIe DMA controller are also similar to the one on embeeded ARM in Zynq FPGAs. Two different form-factors of same product can be easily develop by using either PCIe DMA or Zynq ARM core. A tabletop instrument can use a Zynq to run Linux and provide an interface to the world by touch-screen display or TCP/UDP server. On the other hand, a mezzanine card based solution (e.g. MicroTCA) can use PCIe DMA to provide a link to the main CPU in crate.

Altera also provides similar modules, such as Cyclone V Avalon-MM Interface (https://www.altera.com/literature/ug/ug_c5_pcie_avmm.pdf).

Tip #86

When talking about ChipScope the signal attribute keep should be mentioned, since it often provides a way to find the post-synthesis nets. The keep attribute could also be combined with debug attribute for easier identification of signals.

Verilog example:

(* keep = "true", mark_debug = "true" *) wire [19:0] signal_to_be_dbged;

Tip #90

UCF constants are completely out of date, and were replaced by an equivalent of Synopsys Design Contraints in Vivado.

Tip #93

There is a parameter called cost table between the options for placer, but the description would suggest that this may be better called seed, since it provides a starting point for randomized algorithm (placement).

Tip #97

Jenkins is a more-popular fork of Hudson CI. With a good support for tcl, Vivado offers a reasonable easy way to automate the compilation of the entire project directly from the source code in the repository.

Another tools which could be added to this list is hdlmake which is meant to be an equivalent of Make for FPGA projects. Currently it is not able to tackle more complex compilation procedures, such as re-compiling vendor IP cores or handling Vivado Block Diagrams.

Debugging Linux start-up on Altera Cyclone V SoC with OpenOCD

2016-07-12T22:00:00+02:00

This blog post will show you how one can use the OpenOCD debugger with Altera Cyclone V SoC. Altera Cyclone V SoC is a very interesting integrated circuit, combining dual-core ARM processor and a decent FPGA, allowing a wide variety of possibilities to partition the application between the two.

Xilinx offers Xilinx SDK as the tool to program and debug their MicroBlaze soft-core and ARM cores in their Zynq FPGAs. Altera on the other hand has two different tools to program and debug their portfolio of processors. There is Nios II EDS which provides support for Nios soft-core processor and there is ARM DS-5 Development Studio, which provides support for ARM cores in Altera SoCs. While I believe DS-5 can be useful tool, unfortunately the free-as-a-beer DS-5 Community Edition only allows debugging Linux user-space applications. In order to use it, the Linux should be up and running in order to run gdbserver on processor.

When doing initial bring-up or experimenting this may not be the case. If there is something wrong with a kernel, a device tree or the drivers, one can easily find himself with a non-responsive system. Even in this case the JTAG debugger offers a side-door access to the system. This greatly simplifies determining the cause which lead to the system halt.

Installation of OpenOCD

OpenOCD

OpenOCD is a free and open-source on-chip debugger. It provides a link between hardware components and a command line interface, which can be used to control and monitor the hardware over JTAG interface. It can also be interfaced with GDB (GNU Debugger) integrated with Eclipse, to provide a graphical way to debug programs. If you want to know more, at the bottom of OpenOCD Documentation page is a link to the presentation on FOSDEM 2006.

First we need to install all needed tools:

sudo apt-get install libtool autotools-dev automake libusb-1.0 libhidapi-dev pkg-config git

Then we get the source code from OpenOCD SourceForge repository. I have used the latest available commit in master, which was the one with a git tree-ish value of 12ff09f:

git clone git://git.code.sf.net/p/openocd/code openocd-code

root@eee2f562f8f2:/openocd-code2# git show HEAD
commit 12ff09f7f27a707fe42226262f55b8ce8351cbf9
Author: Esben Haabendal <esben@haabendal.dk>
Date:   Fri Nov 27 09:13:36 2015 +0100
cfi: Add support for strangely endianness broken SoC implementations

Perform all needed steps to compile the code (have a look in INSTALL for detailed instructions):

cd openocd-code
aclocal
./bootstrap

At the end of configure step make sure that support for Altera USB-Blaster II and CMSIS-DAP Debugger are configured. The output should look something like this:

./configure

...

OpenOCD configuration summary
--------------------------------------------------
MPSSE mode of FTDI based devices        yes (auto)
Segger J-Link JTAG Programmer           yes (auto)
ST-Link JTAG Programmer                 yes (auto)
TI ICDI JTAG Programmer                 yes (auto)
Keil ULINK JTAG Programmer              yes (auto)
Altera USB-Blaster II Compatible        yes (auto)
Versaloon-Link JTAG Programmer          yes (auto)
OSBDM (JTAG only) Programmer            yes (auto)
eStick/opendous JTAG Programmer         yes (auto)
Andes JTAG Programmer                   yes (auto)
USBProg JTAG Programmer                 no
Raisonance RLink JTAG Programmer        no
Olimex ARM-JTAG-EW Programmer           no
CMSIS-DAP Compliant Debugger            yes (auto)

Then we just need to compile everything and install the openocd binary.

make

sudo make install

`udev` rules for USB-Blaster

After the OpenOCD is installed, we must take care to set the correct udev rules (access permisions for USB device). As a workaround I have been chmod-ing the /dev/bus/usb/002/ folder to 0666 and that gave me correct permission to use the USB-Blaster from Altera Quartus software.

The more elegant solution is described in the comment section of ALTERA USB-BLASTER WITH UBUNTU 14.04. The USB-Blaster has multiple personalities (one for FPGA JTAG and one for ARM JTAG), the udev rule therefore needs to specify both 6010 and 6810 as the targeted devices.

Create /etc/udev/rules.d/51-usbblaster.rules with the following content:

# For Altera USB-Blaster on SoCkit
SUBSYSTEM=="usb",\
ENV{DEVTYPE}=="usb_device",\
ATTR{idVendor}=="09fb",\
ATTR{idProduct}=="6010|6810",\
MODE="0666",\
NAME="bus/usb/$env{BUSNUM}/$env{DEVNUM}",\
RUN+="/bin/chmod 0666 %c"

udev rules should be reloaded with the following command to take effect immediately:

sudo udevadm control --reload

First checks

Now we can try running Altera jtagconfig program to check if the permissions are OK. When the SoCkit board is attached, the output should look something like this:

➜  ~ ~/altera/16.0/quartus/bin/jtagconfig                                                                                            
1) CV SoCKit [1-1.1]                          
  02D020DD   5CSEBA6(.|ES)/5CSEMA6/..
  4BA00477   SOCVHPS

Now we can also try running OpenOCD:

➜  ~ openocd -f interface/altera-usb-blaster2.cfg -f target/altera_fpgasoc.cfg
Open On-Chip Debugger 0.10.0-dev-00324-g12ff09f (2016-06-26-19:19)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
Warn : Adapter driver 'usb_blaster' did not declare which transports it allows; assuming legacy JTAG-only
Info : only one transport option; autoselect 'jtag'
adapter speed: 1000 kHz
cycv_dbginit
Info : Altera USB-Blaster II (uninitialized) found
Info : Loading firmware...
Info : Waiting for renumerate...
Info : Waiting for renumerate...
Info : Altera USB-Blaster II found (Firm. rev. = 1.36)
Info : This adapter doesn't support configurable speed
Info : JTAG tap: fpgasoc.dap tap/device found: 0x4ba00477 (mfg: 0x23b (ARM Ltd.), part: 0xba00, ver: 0x4)
Info : JTAG tap: fpgasoc.fpga.tap tap/device found: 0x02d020dd (mfg: 0x06e (Altera), part: 0x2d02, ver: 0x0)
Info : DAP transaction stalled (WAIT) - slowing down
Info : DAP transaction stalled (WAIT) - slowing down
Info : DAP transaction stalled (WAIT) - slowing down
Info : fpgasoc.cpu.0: hardware has 6 breakpoints, 4 watchpoints

This gives us information that ARM core was recognized, this is therefore the correct command to use it later with the GDB debugger.

Eclipse plug-in

Now we just have to set-up the Eclipse with the OpenOCD plug-in. I have used the newest version of Eclipse available at the moment, Eclipse Neon.

This page describes how to install the OpenOCD plug-in for Eclipse, the easiest way is to drag and drop install icon into a running instance of Eclipse.

Once this is set, we can create a new project. For simpler (bare-metal) project it would probably make sense to go with "Makefile Project with Existing Code". For a Linux kernel debugging the plain Project will be enough.

Now let's find the original Linux binary. When building kernel with Yocto, the created file in deploy directory is zimage file. This is a compressed image with all debugging symbols stripped out. It is optimized to be used in embedded environment. We need to find the original vmlinux image before the debugging symbols were stripped out:

➜  tmp git:(jethro) find . -name vmlinux                                       
./work/cyclone5-poky-linux-gnueabi/linux-altera/4.3+gitAUTOINC+5938523338-r0/
linux-cyclone5-standard-build/arch/arm/boot/vmlinux
./work/cyclone5-poky-linux-gnueabi/linux-altera/4.3+gitAUTOINC+5938523338-r0/
linux-cyclone5-standard-build/arch/arm/boot/compressed/vmlinux
./work/cyclone5-poky-linux-gnueabi/linux-altera/4.3+gitAUTOINC+5938523338-r0/
linux-cyclone5-standard-build/vmlinux
./work/cyclone5-poky-linux-gnueabi/linux-altera/4.4+gitAUTOINC+969478b841-r0/
linux-cyclone5-standard-build/arch/arm/boot/vmlinux
./work/cyclone5-poky-linux-gnueabi/linux-altera/4.4+gitAUTOINC+969478b841-r0/
linux-cyclone5-standard-build/arch/arm/boot/compressed/vmlinux
./work/cyclone5-poky-linux-gnueabi/linux-altera/4.4+gitAUTOINC+969478b841-r0/
linux-cyclone5-standard-build/vmlinux

There are various images from various different runs (and different versions of the kernel).

In Eclipse we then select: Run -> Debug Configurations...

And then we just need to setup the correct parameters for debug. Under "Main tab" we need to select the right Linux binary image:

Under "Debugger tab" we need to set-up the OpenOCD setting and use the correct gdb (the one produced by Yocto):

If we want to attach to a running kernel, we should un-check "Inital Reset" and "Load executable" fields under "Startup tab".

And we are good to go. After pressing the "Debug" button, the debugging perspective will show up. Now we can access __log_buf buffer to determine what is stopping the kernel boot.

Books read: 97 Things Every Programmer Should Know

2016-06-05T22:00:00+02:00

I have seen this book referenced on several occasions when the software engineering was discussed, but I never had the time to read from first page to the last. I bought the dead-tree version, which gives you a better motivation to read the whole book. Here are my comments and thoughts about each contribution in this book.

(1) Act with Prudence by Seb Rose

Technical debt is (similarly to a cash debt) a good way to kickstart the development, but I agree with the author that is should be kept under control and paid back as soon as possible. I much enjoy when a sub-optimal solution gets fixed and the worries about possible problems with it vanish away.

(2) Apply Functional Programming Principles by Edward Garson

Yes, I am already excited about this book. The functional programming principles (even in OO code) greatly improve testability of smaller components. It also gives a programmer a different insight on how algorithms work, which its a little more detached from how a classical computer works. Programming in C feels like explaining complicated science to a 7 year old ("lets have a number i, which is is set to 0. each time we will check if this number is less than some other number n..."), while programming in Haskell feels like a discussion with a philosopher ("there is this function fact for which the value for 1 is 1. - oh, i see. - and for every other positive number, the value is the argument itself multiplied by the value of function for an argument minus one. - well, this is everything i needed to know about this function, i think i know how to calculate it."

(3) Ask, "What Would the User Do?" (You Are Not the User) by Giles Colborne

This contribution is quite similar to Joel Spolsky comment on how to test the use interface: "One good way to evaluate the usability of a program or dialog you've never seen before is to act a little stupid". The important reminder to take away from this is to try being conservative with the user interface, to minimize the learning how to use your program or application.

(4) Automate Your Coding Standard by Filip van Laenen

Nicely formatted code demonstrates that the person who wrote it has put in a little effort to make it nicer and that the same person is somebody who like to keep things organized. On the contrary, when you see a messy code with sections of the code commented out, you know that there is almost surely something wrong with its behavior as well. Clearly defined language-prescribed formating standard (such as PEP8) and the tools to support it (such as pep8 tool and pylint) are much welcomed. Meanwhile, the C++ coding guidelines currently looks more like a brainstorming session (why is there std::endl if you are not supposed to use it?)

(5) Beauty Is in Simplicity by Jørn Ølmheim

Simple solution are often the ones who work better, not only in software but also in other fields, such as rock music. Trying to write the most legendary rock riff of all time? You only need 4 notes (Yngwie does not agree with this: More is more ). As an example lets look at one of the biggest code bases ever made: Linux kernel. The style itself (tabs are 8 spaces, line is 80 characters) prevent you from writing too complicated solution. The style guide for Linux kernel is a little extreme, but since the kernel should be as lean as possible it produces indisputably good results.

(6) Before You Refactor by Rajith Attapattu

This contribution is again very similar to one of Joel Spolsky's blog posts. Since reading code is harder than writing it everybody assumes that a complete rewrite will create better code. All the ugly patches in the code are actual bugfixes and the software (although with ugly code) works.

(7) Beware the Share by Udi Dahan

Using already written libraries is a nice practice, and this contribution talks about creating unwanted dependencies in the code base. The discussion here regards the internally developed libraries. The case described in this contribution can be also analyzed in terms of the time spent maintaining reusable code, something which is described in The Mythical Man-Month. Use of public (external) stable libraries should be encouraged, although I would in the controversial case of leftPad function prefer to have the leftPad function embedded directly in code.

(8) The Boy Scout Rule by Robert C. Martin

This is one of the most famous contribution in this book, I don't think there is much to add to it. It is always nice when you look back at the clean campground (I sometimes like to admire the cleanup in side-by-side diff).

(9) Check Your Code First Before Looking to Blame Others by Allan Kelly

I would also like to add that a really good understanding of the language (read the standard) and the compiler (read the manual) can help you resolve the problem without blaming the compiler. Personally I know about some miner differences between Verilog compilers from Xilinx and Altera, which could let someone think that the compiler is buggy.

(10) Choose Your Tools with Care by Giovanni Asproni

The author of this contribution mentions that he likes to isolate external tools from his own business logic. I see this approach sometimes in open source FPGA development when a wrapper around some commonly used IP cores is build (e.g. wrapper around block RAM for both Altera and Xilinx which gets chosen based on a parameter or a macro).

(11) Code in the Language of the Domain by Dan North

This contribution could be summarized as a call to use sensible variable/functions/object/methods names. This could be as well extended to writing informative comments, and not have comments like this:

if (maxLim < 0) // checks if max limit is negative

(12) Code Is Design by Ryan Brush

The abstract process of making virtual artifacts (software) from nothing cannot be compared with construction process. The possibility to experiment with the real product gives direct feedback, which can be easily incorporated into design process. I often see everyday physical objects which weren't optimized to perfection, probably because of high iteration cost. The location of power button on my HTC Desire is one of this design flaws, it cannot be reached without changing the grip on the phone.

(13) Code Layout Matters by Steve Freeman

Indentation of the code is important, it provides the overview of the complexity on the first glance. I look forward to GCC6 new warning on wrongly indented C/C++ code.

(14) Code Reviews by Mattias Karlsson

Code reviews are important because they introduce all-seeing-eye in the code writing process, and this makes programmers care more about code quality. One cannot just make a ugly patch and hide in somewhere in the code base. The ultimate code review is the release of the code to the public or to the clients. In this case one can be sure that the company will be judged by the quality of the released code. The same is true for the examples which demonstrates the use of someone own tools.

To be continued in the next blog post...

HDL data type for Python parser implementations

2015-11-15T22:00:00+01:00

Recently I had to implement a parser for the PCIe protocol. The data was captured with Xilinx ChipScope and saved as TSV (tab-separated value) text file. I wanted to implement a parser in Python, my favorite language for this kind of tasks. I have stumbled to a problem when I needed an elegant way to represent the vector of bits of arbitrary length. I have found several libraries but none of them satisfied my needs, so I put together a small class, which mimics SystemVerilog vectors.

Let's have a look at other alternatives which were available but did not completely suit my needs. I wanted a vector slicing syntax which is similar to the one in SystemVerilog and it allows to catch the typos quickly.

bitstring

From their site: bitstring is a pure Python module designed to help make the creation and analysis of binary data as simple and natural as possible.

It quick test finds two things which I would did not like: taking slice wider than vector length pads the resulting vector with zeros and the slice indexes are inverted compared to more used [higher_limit:lower_limit] notation in HDLs. The output of the slicing is a closed interval, which is the behavior I would expect.

BitArray

The first thing which comes in mind is that there is not an easy way to create a bitarray and initialize it from int in a single step (using constructor). The only way to initialize BitArray is to use binary-formated string. This requires a call of bin() function and dropping first two characters if your data is stored as an int. At this point one can already start thinking of implementing it's own class. The slicing has the same behavior as bitstring, which I did not like for the application I need.

A simple solution on Stack Overflow

There is a similar solution already posted on Stack Overflow, however it lacks an equality operator.

MyHDL

Since MyHDL is a way to write HDL with Python it comes as obvious choice to use it in a simple Python parser. MyHDL has a intbv data type which is very similar to vectors in Verilog and VHDL. However, there are some minor things which discouraged me from using it in my parser.

Lets have a look at a modified version of the VerilogBits unit test:

import unittest
import myhdl

class Testmyhdlintbv(unittest.TestCase):
    def test_equality(self):
        self.assertEqual(myhdl.intbv(0xAB), myhdl.intbv(0xAB))
        self.assertEqual(myhdl.intbv(0xAB), myhdl.intbv(0x0AB))
        self.assertNotEqual(myhdl.intbv(0xAB), myhdl.intbv(0xCD))

    def test_slicing(self):
        ab = myhdl.intbv(0xAB)
        self.assertEqual(ab[7:4], myhdl.intbv(0xA))
        self.assertEqual(ab[3:0], myhdl.intbv(0xB))

    def test_unpack(self):
        abcd = myhdl.intbv(0xABCD)
        a, b, c, d = abcd[15:12], abcd[11:8], abcd[7:4], abcd[3:0]
        self.assertEqual(a, myhdl.intbv(0xA))
        self.assertEqual(b, myhdl.intbv(0xB))
        self.assertEqual(c, myhdl.intbv(0xC))
        self.assertEqual(d, myhdl.intbv(0xD))

    def test_slice_up_vect(self):
        with self.assertRaises(ValueError):
            dummy = myhdl.intbv(0xAB)[0:7]

    def test_invalid_slice(self):
        with self.assertRaises(IndexError):
            dummy = myhdl.intbv(0xAB)[9:0]


if __name__ == '__main__':
    unittest.main()

This results are 3 failing tests: test_invalid_slice, test_slicing, and test_unpack. test_invalid_slice fails because taking a slice wider than a vector width produces fills the missing bits with zero. This is similar to SystemVerilog vector of bits, which is a 2-level data type (it can be only 0 or 1). I prefer more rigorous behavior when slicing vectors, since errors like that can be quite hard to catch. The VerilogBits throws an exception when an invalid slice is requested.

If the zero padding problem with MyHDL is something I could live with, the other two failing test are much more discouraging for someone who sometimes dreams (System)Verilog. The bit slicing in MyHDL is half- open as is expected in Python and not a closed interval as expected from HDLs (e.g. to get the LSB one should write [8:0] instead of [7:0]). Again, this is just a convention and the software world is using the half-open interval for decades (E.W.Dijkstra: Why numbering should start at zero). But if your parser in Python is there to find bugs in your SystemVerilog code, it makes much more sense to use the same notation in both languages.

SystemVerilog

The SystemVerilog provide all the necessary tools to effectively manipulate bits (duh), but the Python with the generators, list comprehensions and dictionaries (well, SystemVerilog does have associative array) is much more elegant language. The ability to test commands on-the-fly in the interpreter is also much welcomed.

Compilation of Linux kernel for Raspberry Pi

2015-03-01T19:00:00+01:00

Yesterday I got my Raspberry Pi 2, the evolution of the legendary Raspberry Pi. The evolution is the right word to describe what has changed compared to the previous version. The processor it is now a quad-core, it runs faster, it has got a newer instruction set (ARMv7) and the board now incorporates 1GB of RAM and 4 USB connector. The RCA jack is no longer present, the additional place is being occupied by a larger extension header instead. A welcomed update of already great hardware, in my opinion (for people complaining that it can't play 4K video, run desktop version of the Windows, ..., well, consider buying proper computer, RPi was meant to be cheep enough to tinker with without fear of breaking something).

I am the kind of the person which thinks that compiling Linux kernel would be great a way to spend the Sunday. Since one of my projects with RPi includes custom driver for communication with FPGA, I needed Linux headers and kbuild system for building kernel modules (those who are not familiar with the procedure, imagine that you are building a program linked with libraries; in this case a program is a kernel module and kernel and its modules represent the libraries).

On the Ubuntu (and I think on other Debian systems but I am not 100% sure) you can get kernel headers and source directly from the Debian repository, something like:

apt-get install linux-headers-$(uname -r)

The above command will install the headers the headers for the currently running kernel. It copies the header files to /usr/src and creates link in /lib/modules/, this gives the user tools needed to compile kernel modules.

The hard way

Before you continue reading I would just like to inform you that the procedure described in this section did not yield desired result. Although I managed to get a kernel image, the kbuild system did not work. In Thomas Alva Edison style I want to present a method which does not work, but at least we know it does not work. The working (and easier) procedure is described in the next section.

Initially I wanted to get the headers in the same way as I would on Ubuntu, by using apt (Advanced Packaging Tool). Searching for available packages did not show the package needed (the Raspbian was running the 3.18.7-v7+ kernel).

pi@raspberrypi ~ $ sudo aptitude search linux-headers
v   linux-headers                                   -                                                           
p   linux-headers-3.10-3-all                        - All header files for Linux 3.10 (meta-package)            
p   linux-headers-3.10-3-all-armhf                  - All header files for Linux 3.10 (meta-package)            
p   linux-headers-3.10-3-common                     - Common header files for Linux 3.10-3                      
p   linux-headers-3.10-3-rpi                        - Header files for Linux 3.10-3-rpi                         
p   linux-headers-3.12-1-all                        - All header files for Linux 3.12 (meta-package)            
p   linux-headers-3.12-1-all-armhf                  - All header files for Linux 3.12 (meta-package)            
p   linux-headers-3.12-1-common                     - Common header files for Linux 3.12-1                      
p   linux-headers-3.12-1-rpi                        - Header files for Linux 3.12-1-rpi                         
p   linux-headers-3.18.0-trunk-all                  - All header files for Linux 3.18 (meta-package)            
p   linux-headers-3.18.0-trunk-all-armhf            - All header files for Linux 3.18 (meta-package)            
p   linux-headers-3.18.0-trunk-common               - Common header files for Linux 3.18.0-trunk                
p   linux-headers-3.18.0-trunk-rpi                  - Header files for Linux 3.18.0-trunk-rpi                   
p   linux-headers-3.18.0-trunk-rpi2                 - Header files for Linux 3.18.0-trunk-rpi2                  
p   linux-headers-3.2.0-4-all                       - All header files for Linux 3.2 (meta-package)             
p   linux-headers-3.2.0-4-all-armhf                 - All header files for Linux 3.2 (meta-package)             
p   linux-headers-3.2.0-4-common                    - Common header files for Linux 3.2.0-4                     
p   linux-headers-3.2.0-4-rpi                       - Header files for Linux 3.2.0-4-rpi                        
p   linux-headers-3.6-trunk-all                     - All header files for Linux 3.6 (meta-package)             
p   linux-headers-3.6-trunk-all-armhf               - All header files for Linux 3.6 (meta-package)             
p   linux-headers-3.6-trunk-common                  - Common header files for Linux 3.6-trunk                   
p   linux-headers-3.6-trunk-rpi                     - Header files for Linux 3.6-trunk-rpi                      
p   linux-headers-rpi                               - Header files for Linux rpi configuration (meta-package)   
p   linux-headers-rpi-rpfv                          - This metapackage will pull in the headers for the raspbian
p   linux-headers-rpi2-rpfv                         - This metapackage will pull in the headers for the raspbian

I tried my luck with linux-headers-rpi2-rpfv which installed headers for kernel 3.18.0. That might sound close enough, but Linux refuses modules which have not been compiled with exactly the same same kernel.

So I decided to recompile the kernel on my laptop running Ubuntu. The procedure of compiling on one type of machine (in this case Intel x86_64 processor) for another (in this case ARM processor) is called cross-compilation.

The process of cross-compilation is described here: Raspberry Pi Kernel Compilation

Everything went fine, I got the kernel image and /lib/modules directory. The only thing that the guide does not mention is that make modules_install creates symbolic links to the original directory. I copied the entire kernel directory to /usr/src and recreated symbolic links, so everything should be fine.

At this point if everything went OK we would have a working kernel and its kbuild system which gives us the tools to compile the kernel modules out of the tree (in a separate directory). The RPi boots up, which is a good sign, meaning that there is nothing wrong with the kernel image.

However, when I try to compile the kernel module, I get the error saying that there is a syntax error in script in kernel module (the same problem as described here. The answer by Joe C states:

Anyway, if you cross compile you don't get a /usr/src/linux-header-x.x.x/scripts dir that's usable on your target system.

Since I previously downloaded another kernel headers from Debian repository, I tired copying the scripts directory from there:

pi@raspberrypi /usr/src/linux-headers-3.18.0-trunk-rpi2 $ sudo cp -rv scripts/ ../linux-sources-3.18.8-v7+/

This fixed the compilation problem just to cause complete disaster when I tried inserting the module. The following kernel message (dmesg command) shows nothing promising:

[ 4246.164155] Unable to handle kernel paging request at virtual address fe220b30
[ 4246.176292] pgd = b5dd8000
[ 4246.183812] [fe220b30] *pgd=00000000
[ 4246.192196] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[ 4246.202324] Modules linked in: pi64_dev(O+) snd_bcm2835 snd_soc_pcm512x_i2c snd_soc_wm8804 snd_soc_pcm512x snd_soc_tas5713 regmap_spi regmap_i2c snd_soc_bcm2708_i2s regmap_mmio snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm snd_seq snd_seq_device snd_timer r8712u(C) snd spi_bcm2708 i2c_bcm2708
[ 4246.239611] CPU: 3 PID: 3468 Comm: insmod Tainted: G         C O   3.18.8-v7+ #1
[ 4246.252306] task: b87aa840 ti: b760a000 task.ti: b760a000
[ 4246.262985] PC is at load_module+0x1a64/0x1f0c
[ 4246.272699] LR is at load_module+0x1a50/0x1f0c
[ 4246.282373] pc : [<800940d4>]    lr : [<800940c0>]    psr: 90000013
[ 4246.282373] sp : b760be88  ip : 7f110770  fp : b760bf44
[ 4246.304401] r10: 00000000  r9 : 7f110600  r8 : 80536d44
[ 4246.314883] r7 : b86dc8c4  r6 : fe220b1c  r5 : 7f11060c  r4 : b760bf48
[ 4246.326685] r3 : 00000000  r2 : b760be70  r1 : b7555280  r0 : 807f2a3c
[ 4246.338495] Flags: NzcV  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[ 4246.350916] Control: 10c5387d  Table: 35dd806a  DAC: 00000015
[ 4246.361932] Process insmod (pid: 3468, stack limit = 0xb760a238)
[ 4246.373206] Stack: (0xb760be88 to 0xb760c000)
[ 4246.382808] be80:                   7f11060c 00007fff 8009168c ffffffff b760bee4 bd949000
[ 4246.396352] bea0: 00000000 7f11060c 00000000 b760bedc 807e4d78 7f110648 b760a000 7f110770
[ 4246.409938] bec0: 00001f4b 806c9bdc ba7b961c b760a000 b9330000 808ab8fc 76fd3000 00000000
[ 4246.423538] bee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4246.437137] bf00: 00000000 00000000 00000000 00000000 00000000 00000000 00000080 00001f4b
[ 4246.450707] bf20: 76fd3000 76f91948 00000080 8000f0a4 b760a000 00000000 b760bfa4 b760bf48
[ 4246.464292] bf40: 80094664 8009267c bd949000 00001f4b bd949f3c bd949de3 bd94acc4 00000850
[ 4246.477932] bf60: 00000940 00000000 00000000 00000000 0000001f 00000020 00000017 00000014
[ 4246.491605] bf80: 00000010 00000000 00000000 7eb7861c 00000000 7768b038 00000000 b760bfa8
[ 4246.505297] bfa0: 8000ee20 80094588 7eb7861c 00000000 76fd3000 00001f4b 76f91948 76fd3000
[ 4246.519052] bfc0: 7eb7861c 00000000 7768b038 00000080 7768af80 00001f4b 76f91948 00000000
[ 4246.532810] bfe0: 00000000 7eb785c4 76f88fb4 76ef3ab4 60000010 76fd3000 d466662e d5f7abd0
[ 4246.546635] [<800940d4>] (load_module) from [<80094664>] (SyS_init_module+0xe8/0xfc)
[ 4246.560087] [<80094664>] (SyS_init_module) from [<8000ee20>] (ret_fast_syscall+0x0/0x48)
[ 4246.573935] Code: e51bc088 e15c0006 e2466008 0a000009 (e5963014) 
[ 4246.604593] ---[ end trace 8e4bd0982f9f1fd0 ]---

It basically says that there was an error trying to access the wrong part of the memory from the insmod process.

At this point I decided to stop experimenting with this approach. I was getting frustrated with constant fighting with all kind of errors. When I initially started writing this blog post I wanted to describe the method to cross-compile Linux kernel image and kbuild build system. Unfortunately, although it may sound appealing to speed up the process by cross compiling on x86 machine, the complication with various stuff make it more time consuming that compiling directly on Raspberry Pi. I know myself good enough that I know I wont quit this easy. I will probably return to this problem on some other occasion and tried again, be assured that I will write a blog post If I succeed.

The easy way

At the end I dropped the cross-compilation idea and resorted to compiling kernel on the RPi itself. This can also serve as a performance test of the new RPi. I overclocked it to 1000MHz (using raspi-config I selected the RPi2 setting).

The compilation of kernel took something less than 2 hours, quite decent result compared to 17 minutes it takes on my laptop with not-so-new i3 and 6GB of RAM.

The following figure shows the temperature during the compilation. The temperature was captured every minute with a Cron job which read from /sys/class/thermal/thermal_zone0/temp

Finally, using freshly compiled kernel on RPi, I managed to get compile my module and to load it in kernel (debug message from kernel):

[   38.324756] ============================================
[   38.324786]        Pi64 driver by Jan Marjanovic        
[   38.324796] 
[   38.324808]   built: Mar  1 2015 11:38:10
[   38.324818] ============================================

Lattice iCEcube2 on Ubuntu 14.04

2015-03-01T19:00:00+01:00

/home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/synplify_pro: 186: [: unexpected operator /home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/synplify_pro: 200: [: !=: argument expected /home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/c_hdl: 186: [: unexpected operator /home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/c_hdl: 200: [: !=: argument expected /home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/syn_nfilter: 186: [: unexpected operator /home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/syn_nfilter: 200: [: !=: argument expected /home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/m_generic: 186: [: unexpected operator /home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/m_generic: 200: [: !=: argument expected /home/jan/opt/lscc/iCEcube2.2014.12/synpbase/bin/m_generic: 186: [: unexpected operator

sed -i 's/\/bin\/sh/\/bin\/bash/g' *

jan@jan-ThinkPad-T510 ~/opt/lscc/iCEcube2.2014.12/synpbase/bin % for file in $(ls); do sed -i 's/\/bin\/sh/\/bin\/bash/g' $file; done 19:33:27 on 2015-03-07 sed: couldn't edit config: not a regular file jan@jan-ThinkPad-T510 ~/opt/lscc/iCEcube2.2014.12/synpbase/bin %

for file in $(ls -la | grep -E '^[^d]' | awk -F "/" '{print $NF}' ); do echo $file; done

for file in $(ls -la | tail -n +2 | grep -E '^[^d]' | awk '{print $NF}' ); do sed -i 's/\/bin\/sh/\/bin\/bash/g' $file; done

Lattice iCE40 configuration using Raspberry Pi

2015-01-18T22:00:00+01:00

As I mentioned in previous post, I started playing around with Lattice iCE40 FPGA. In the last post I did a quick overview of the iCE40 tools. The iCEcube2 cannot compete with Xilinx ISE and Altera Quartus II, not to mention the Vivado, but since this is a low-cost FPGA the current tool offers all you need to do this kind of simple projects (I would definitely recommend the beginners to stay away from Lattice as it is not as user friendly as vendor X or vendor A, you need to have some experience to master the work flow).

Last time I found out that iCEcube2 Programmer runs only on Windows, on GNU/Linux you need to find other solutions. How the Programmer works is another interesting thing. One would expect that is uses JTAG port on FPGA to configure it, but that is not the case. The Programmer communicates with Atmel microcontroller which programs Serial NOR Flash memory. Then it reset the FPGA which boots in SPI Master mode, and it reads configuration from Flash.

A quick look at TN1248: iCE40 Programming and Configuration shows that it can be programmed also by writing from another device (e.g. microprocessor to the SPI port). This is called SPI Slave programming mode and it is enabled by holding the line CS_n low at the reset of the FPGA.

So I tried programming it using Raspberry Pi. I connected:

grounds together
SI on iCE40 to MOSI on RPi
SO on iCE40 to MISO on RPi (this one is actually not needed)
SCK on iCE40 to CLK on RPi
pin GPIO25 on RPi to SS on iCE40 (this one is needed to enter the slave mode)

I wrote the following script and used one of the .bin files from one of the project in iCEcube2.

This is the output (well, the real output it is the configured board which blinks the LEDs while the DONE led is lit):

pi@raspberrypi ~/Jan/ice40/test1 $ sudo bash conf_FPGA.sh proto1_top_bitmap.bin
GPIO 25 not exported, trying to export...

spidev does not exist
SPI driver not loaded, try to load it...
OK: SPI driver loaded

Changing direction to out
out
Setting output to low
1

Please power cycle the iCE40 FPGA board
Press any key...

Continuing with configuration procedure
63+1 records in
63+1 records out
32300 bytes (32 kB) copied, 0.606931 s, 53.2 kB/s
Setting output to high
1

I find this type of configuration very useful when the FPGA is not the main chip in the system (when there is as in the previous example an RPi). The configuration file can be stored on Raspberry Pi SD card and at each start-up the FPGA gets programmed. The image update can be done very easily and there is no way a user can brick the FPGA (which can easily happen if the FPGA writes the configuration image and boots from its Flash, in this case two images (factory and user) are recommended).

My first encounter with Lattice Semiconductor

2014-12-24T20:00:00+01:00

The FPGA market is one of those classical markets where there are two players who have nearly 100% market share, e.g. PC processors (Intel and AMD), graphic cards (NVIDIA and ATI), ... The two mayor players on the FPGA market are Xilinx and Altera. Both of this two companies follows their rival very closely. When one company announces the next big thing, support for new technology or improvement to their current products, the other will shortly follow with similar announcement.

This tempo of development has brought the two main programs of these two rivals at very high level. There is a tool (Altera Qsys and Xilinx Vivado) which lets you build a system from standardized building blocks with standardized interfaces (Altera Avalon and ARM AXI). They both offer a large palette of IP ready to be used with their program. The debugging of the designs is simplified with good integration with simulators (Altera by default uses Mentor Graphic ModelSim, Xilinx has its own XSim, but it can also use ModelSim) and integrated logic analysers (Altera SignalTap and Xilinx ILA) are a great help when simulation works but real world FPGA does not.

I have an idea for my next hobby project which needs an FPGA (actually, CPLD will do just fine) for translating one communication protocol to another. The board will be an expansion board for Raspberry Pi, so the cost should be really low (I have in mind something bellow 5$). Altera and Xilinx do not offer anything in this price range, so I recalled Lattice ads. The iCE40 series are small FPGAs with few K logic cells and low prices. Another advantage for my project is possibility to configure it by SPI, something that most CPLDs does not support.

So I ordered iCEblink40HX1K Evaluation Kit.

First I tried installing the iCEcube2 software on Centos 7 which is my usual working environment. The installation went smoothly. The problem started to occur when the licence manager was unable to find network adapter MAC address:

Error: License checkout failed.


Invalid host.
 The hostid of this system does not match the hostid
 specified in the license file.
Feature:       LSC_ICECUBE2_A
Hostid:        xxxxxxxxxxxx
License path:  /home/jan/opt/lscc/license.dat:
FLEXnet Licensing error:-9,57.  System Error: 19 "(null)"
For further information, refer to the FLEXnet Licensing End User Guide,
available at "www.macrovision.com".

The problem is that the FLEXnet software searches for the MAC address of the eth* interface, while Centos uses completely different naming. Being familiar with the problem, I used a trick to create a new interface named eth0 (you should replace the x's with desired MAC address):

modprobe dummy
ip link set name eth0 dev dummy0
ifconfig eth0 hw ether xx:xx:xx:xx:xx:xx

After setting everything up, a view on the iCEcube2 showed before me.

It is a little Spartan but it has all basic functions the FPGA developer needs. The text editor lacks code coloring, auto-complete but there is a chance to use a 3rd party text editor. The text in the output window is light violet on white background and it should be changed to something more visible.

The iCE family of FPGAs lacks JTAG port. Instead, user programs SPI Flash memory through an USB to SPI converter and then the FPGA boots from SPI memory. That means that debugging is done or by using 4 LEDs on the side or with oscilloscope, not a very user friendly approach.

Under Linux, there is additional problem:

Note : The Integrated Aldec Active HDL simulation software and iCEcube2 programming 
tools are only available in windows platform.

Well this is it for now, i need to switch to Windows. Merry Christmas to everybody!

All trademarks and registered trademarks are the property of their respective owners.

Theremin First Demo

2014-11-18T23:00:00+01:00

During the weekend I was able to take some time to do first test of the theremin. Here are two recordings of my friend Luka playing.

There are some more articles explaining how digital theremin works coming, remember to check out my blog.

I just got a Nintendo 64

2014-11-16T18:00:00+01:00

I just got a Nintendo 64, my friend Rok was kind enough to lend it.

I have an interesting project in mind, I will keep you updated. Rok will for sure be the first one to get the alpha version.

Theremin Antenna Measurements

2014-11-09T09:00:00+01:00

Last week I briefly explained how theremin works. I also presented my idea to develop a digital version, using FPGA as a detector of distance between hand and antenna.

You have probably already heard a joke about theory and practice.

Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. Here we combine the two, nothing works and no one knows why.

Jokes aside, today we will try to measure the time constant of the antenna. In this implementation of theremin the measured time constant will be a control for tone pitch. Having a good measurement of time constant is therefore very important. A good instrument will produce stable tone when hand is hold still. Also, the quality of measurement of time constant determines a difference between two consecutive tones and fine control of the pitch is also desired.

Here you can see this highly advanced test - a ruler strapped to the antenna. This will allow us to measure the relationship between time constant and distance of the hand.

Since the ruler is made out of plastic (we should say dielectric, when studying electric fields) it won't affect the antenna field.

The FPGA is producing a square wave signal at 10kHz, which is then sent to antenna through a 2.2 MOhm resistor. The resistor is on the other side connected to antenna, thus creating an RC circuit. The voltage on antenna is feed back to FPGA through a Schmitt Trigger to improve the measured value. The module in FPGA measures how much time did it take for voltage on antenna to reach certain value (determined by Schmitt Trigger). This time is directly correlated to time constant and therefore to capacitance of the antenna.

Here we see what the FPGA measured when I placed hand on different distance from antenna.

The blue dots denote mean value, the green lines denote one standard deviation and the black lines denote minimum and maximum value.

Right now, the measurements do not look very promising, we can see some increase of time constant as hand approaches antenna, but the noise is extremely high.

If we have a look at the frequency spectrum of the measurement, we see the reason for the noise.

The antenna is 45 cm long copper rod and acts not only as a capacitor for theremin but also as radio antenna. All the frequency components above Nyquist frequency (which is 5000 kHz in our case) are being aliased to lower frequencies in our frequency range. A 50Hz signal from mains is also being picked up.

We need a filter! A filter will reject all the undesired frequencies and that will greatly improve.

I will write another article on filtering in FPGA, meanwhile let's enjoy much improved results:

Theremin Basics

2014-11-06T22:00:00+01:00

Theremin, a first electronic instrument. Leon Theremin invented it in 1928. Try to imagine people seeing somebody waving hand in the middle or the air and producing an extraterrestrial sounds. Theremin, should be considered a true pioneer of electronic music.

The operational principle is quite simple, however an good implementation is not so trivial. I recall playing on it in Deutsches Museum in Munich and that particular model had a sphere as an antenna. The tone frequency is dependant on antenna capacitance. To achive maximum control, the relationship between capacitance and distance between hand and antenna should be linear. However, the model in Munich had reciprocal relationship and that made hitting right notes quite hard.

I am going to make a new version of theremin, a reinterpretation for 21st century. I will still use antenna to control the pitch, but the "back-end" will be completely different.

The antenna acts as one plate of the capacitor and the hand is the other. I did a quick simulation in Python to be able to show some numbers. This is meant to be a demonstration of operating principle, so it lacks few not-so-minor details. The calculations were done in 2D, where antenna in real world exist in 3 dimensions.

Code

Now we know that antenna acts as a capacitor, so next challenge is to produce a tone with the pitch related to capacitance. The original Theremin uses antenna as a capacitor of the LC resonator. Here is the first big difference between my implementation and original one, I will be measuring time constant of the RC circuit with an FPGA.

The FPGA will generate a square wave and feed it to antenna through a resistance. This will create a current which will charge capacitor (antenna). The other circuit (module) in FPGA will measure time needed for voltage on capacitor to reach certain level. A bigger capacitance (smaller distance between hand and antenna) will result in longer time and a smaller capacitance (longer distance between hand and antenna) will result in shorter time. This difference will generate a different pitch.

This is all for this part, stay tuned for more.

Hello world

2014-11-02T13:00:00+01:00

Hi and welcome to my blog.

This is a "hello world" style post, I am curently testing pelican static blog generator. Since I don't know how much time I will have to manage this blog I prefer that everything is static. If I some day stop writing this blog, at least there won't be any security threats due to out-of-date web page.

This blog will be about one of my biggest passions, electronics and technology in general. Since I was a kid I was curious about electronic devices and to this day I always enjoy taking thing appart (and sometimes also together, that is what I am being paid for).

This is all for now, let's see if it "compiles"...