For a project (building a low-power LoRaWAN gateway to be solar powered) I am looking at simple and low-power linux boards. One board that I came across is the Milk-V Duo, which looks very promising. I have been playing around with it for just a few hours, and I like the board (and its SoC) very much already - for its size, price and open approach to documentation.

The board itself is a small (21x51mm) board in Raspberry Pi Pico form factor. It is very simple - essentially only a SoC and regulators. The SoC is the CV1800B by Sophgo, (a vendor I had never heard of until now, seems they were called CVITEK before). It is based on the RISC-V architecture, which is nice. It contains two RISC-V cores (one at 1Ghz and one at 700Mhz), as well as a small 8051 core for low-level tasks. The SoC has 64MB of integrated RAM.

The SoC supports the usual things - SPI, I²C, UART. There is also a CSI (camera) connector and some AI accelaration block, it seems the chip is targeted at the AI computer vision market (but I am ignoring that for my usecase). The SoC also has an ethernet controller and PHY integrated (but no ethernet connector, so you still need an external magjack to use it). My board has an SD card slot for booting, the specs suggest that there might also be a version that has on-board NAND flash instead of SD (cannot combine both, since they use the same pins).

There are two other variants - the Duo 256M with more RAM (board is identical except for 1 extra power supply, just uses a different SoC with more RAM) and the Duo S (in what looks like Rock Pi S form factor) which adds an ethernet and USB host port. I have not tested either of these and they use a different SoC series (SG200x) of the same chip vendor, so things I write might or might not be applicable to them (but the chips might actually be very similar internally, the CVx to SGx change seems to be related to the company merger, not necessarily technical differences).

The biggest (or at least most distinguishing) selling point, to me, is that both the chip and board manufacturers seem to be going for a very open approach. In particular:

• Full datasheets for the SoC are available (the datasheets could be a bit more detailed, but I am under the impression that this is still a bit of a work-in-progress, not that there is a more detailed datasheet under NDA).

• The tables (e.g. pinout tables) are not in the datasheet PDF, but separately distributed as a spreadsheet, which is super convenient.

• For the the SG200x chips, the datasheet is created using reStructuredText (a text format a bit like markdown but more expressive), and the rst source files are available on github under a BSD license. This is really awesome: it makes contributions to the documentation very easy, and (if they structure things properly when more chips are added) could make it very easy to see what peripherals are the same or different between different chips.

• Sophgo seems to be actively trying to get their chips supported by mainline linux (maybe by contributing code directly, or at least by supporting the community to do so), which should make it a lot easier to work with their chips in the future.

  Other vendors often just drop a heavily customized and usually older kernel or BSP out there, sometimes updating it to newer versions but not always, and relying on other people to do the work of cleaning up the code and submitting it to linux.

• The second core can be used as a microcontroller and Milk-V supports running FreeRTOS on it it, and provides an Arduino core for it (have not looked if it is any good yet). It seems the first core then remains active running Linux, providing a way to flash the secondary core through the primary core.

All this is based on fairly quick observations, so maybe things are not as open and nice and they seem to be at first glance, but it looks like something cool is going on here at least.

Other things I like about the board:

• It is small and simple and fits in a breadboard.
• It is power-efficient (first measurement with the default buildroot-based linux idles at 200mW-250mW - around 45mA at 5V, which is a lot less than other SBC's I have tried so far. This was just a quick measurement, I still have to do more (detailed) measurements with load and under other circumstances.
• It is cheap: Only $5 for the 64M Duo.
• The packaging is very neat: A very tightly fitting cardboard box (and antistatic bag of course). Edit: I have since ordered a Duo 256 which came in a bigger plastic box, which is less nice (though the box is solid enough to be reusable, which is good). This might be because I ordered the version with presoldered pins, so next time I will try the bare PCB again.
• The default image presents itself as an USB ethernet device, allowing direct connectivity via USB.

There are also some (potential) downsides that might complicate matters: - Only 64MB RAM is very limited. In practice, some RAM is used for peripherals (I think) too, the default buildroot image has something like half of the RAM available to Linux. Other images configure this differently so full RAM is available to the kernel (leaving 55M for userspace). See this forum topic for more details.

Low memory limits options - people have reported apt needs around 50M to work, which means it ends up using swap and is super slow.

• The official Linux distribution from milk-v is a buildroot-built image, which means all software is built into the image directly, no package manager to install extra software afterwards.

  The buildroot files are available, so it should be easy too build your own image with extra software, though I think this does mean compiling everything from source.

  There does seem to be a lively community of people that are working on making other distributions work on these boards. In most cases this means building a custom kernel for this board (with milk-v/sophgo patches, often using buildroot) and combining it with existing RISC-V packages or a rootfs from these distributions. Sometimes with instructions or a script, sometimes someone just hand-edited an image to make it work.

  Hopefully proper support can be added into the actual distributions as well, though a lot of distributions do not really have the machinery to create bootable images for specific boards (i.e. they only support building images for generic BIOS or EFI booting). One distribution that does have this is Armbian, but that is Debian/apt-based so probably needs more than 64MB RAM.

  I have briefly tried the Alpine and Arch linux images that are available. Alpine is really tiny, but like the official buildroot image uses musl libc. This is nice and tiny, but not all software plays well with it (and in all cases I think software must be compiled specifically against musl). The main application I needed (The basicstation LoRaWAN forwarder) did not like being compiled against musl (and I did not feel like fixing that, especially since I am doubtful such changes would be merged upstream).

  So I am hoping I can use the Arch image, which does use glibc and seems to run basicstation (at least it starts, I have not had the time to reallly set it up yet). Or maybe a Debian/Ubuntu/Armbian image after all - I have also ordered the 256M version (which was not in stock initially).

  For an overview of various images created by the community, see this excellent overview page.

• It is not entirely clear to me what bootloader is used and how the devicetree is managed. On most single-board linux devices I know, there is u-boot with a boot script, which can be configured to load different devicetrees and overlays to allow configuring the hardware (e.g. remapping pins as SPI or I²C pins). On the buildroot-image for the Duo, I could find no evidence of any of this in /boot, but I did see u-boot being mentioned in some places, so maybe it is just configured differently.

• Even though the documentation is very open, some of it is a bit hard to find and spread around. Here's some places I found:

  • spreadsheets and PDF datasheets for CV1800B chip and schematics for the Duo board: https://github.com/milkv-duo/duo-files
  • Datasheets (rst source and PDF) for the SG200x chips: https://github.com/sophgo/sophgo-doc/
  • Documentation about the Milk-v boards: https://milkv.io/docs/duo/overview
  • Links to Cvitek docs (which I could not find linked from anywhere else): https://milkv.io/docs/duo/resources/mmf
  • The milk-v forum about the Duo (mostly English, partly chinese): https://community.milkv.io/c/duo/5
  • The English part of the Sophgo forum about the Duo: https://forum.sophgo.com/c/duo-180-english-forum/12
  • The Milk-V chip pages have a bunch of PDF documents for each chip: https://milkv.io/chips

• The USB datapins are available externally, but only on two pads that need pogopins or something like that to connect to them. Would have been more convenient if these had a proper pin header.

• Some of the hardware setup is done with shell scripts that run on startup (for example the USB networking), some of which actually do raw register writes. This is probably something that will be fixed when kernel support improves, but can be fragile until then.

• Sales are still quite limited - most of the suppliers linked from the manufacturer pages seem to be China-only, and I have not found the boards at any European webshop yet. I have now ordered from Arace Tech, a chinese webshop that ships internationally and worked well for me (except for to-be-expected long shipping times of a couple of weeks).

So, that was my first impression and thoughts. If I manage to get things running and use this board as part of my LoRaWAN gateway design, I'll probably post a followup with some more experiences. If you have used this board and have things to share, I'm happy to hear them!

On my server, I use LVM for managing partitions. I have one big "data" partition that is stored on an HDD, but for a bit more speed, I have an LVM cache volume linked to it, so commonly used data is cached on an SSD for faster read access.

Today, I wanted to resize the data volume:

# lvresize -L 300G tika/data
Unable to resize logical volumes of cache type.

Bummer. Googling for the error message showed me some helpful posts here and here that told me you have to remove the cache from the data volume, resize the data volume and then set up the cache again.

For this, they used lvconvert --uncache, which detaches and deletes the cache volume or cache pool completely, so you then have to recreate the entire cache (and thus figure out how you created it in the first place).

Trying to understand my own work from long ago, I looked through documentation and found the lvconvert --splitcache in lvmcache(7), which detached a cache volume or cache pool, but does not delete it. This means you can resize and just reattached the cache again, which is a lot less work (and less error prone).

For an example, here is how the relevant volumes look:

# lvs -a
LV                VG   Attr       LSize   Pool              Origin       Data%  Meta%  Move Log Cpy%Sync Convert
data              tika Cwi-aoC--- 300.00g [data-cache_cvol] [data_corig] 2.77   13.11           0.00
[data-cache_cvol] tika Cwi-aoC---  20.00g
[data_corig]      tika owi-aoC--- 300.00g

Here, data is a "cache" type LV that ties together the big data_corig LV that contains the bulk data and small data-cache_cvol that contains the cached data.

After detaching the cache with --splitcache, this changes to:

# lvs -a
LV               VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
data             tika -wi-ao---- 300.00g
data-cache       tika -wi-------  20.00g

I think the previous data cache LV was removed, data_corig was renamed to data and data-cache_cvol was renamed to data-cache again.

The actual resize

Armed with this knowledge, here's how the ful resize works:

lvconvert --splitcache tika/data
lvresize -L 300G tika/data @hdd
lvconvert --type cache --cachevol tika/data-cache tika/data --cachemode writethrough

The last command might need some additional parameters depending on how you set up the cache in the first place. You can view current cache parameters with e.g. lvs -a -o +cache_mode,cache_settings,cache_policy.

Cache pools

Note that all of this assumes using a cache volume an not a cache pool. I was originally using a cache pool setup, but it seems that a cache pool (which splits cache data and cache metadata into different volumes) is mostly useful if you want to split data and metadata over different PV's, which is not at all useful for me. So I switched to the cache volume approach, which needs fewer commands and volumes to set up.

I killed my cache pool setup with --uncache before I found out about --splitcache, so I did not actually try --splitcache with a cache pool, but I think the procedure is actually pretty much identical as described above, except that you need to replace --cachevol with --cachepool in the last command.

For reference, here's what my volumes looked like when I was still using a cache pool:

# lvs -a
LV                 VG   Attr       LSize   Pool         Origin       Data%  Meta%  Move Log Cpy%Sync Convert
data               tika Cwi-aoC--- 260.00g [data-cache] [data_corig] 99.99  19.39           0.00
[data-cache]       tika Cwi---C---  20.00g                           99.99  19.39           0.00
[data-cache_cdata] tika Cwi-ao----  20.00g
[data-cache_cmeta] tika ewi-ao----  20.00m
[data_corig]       tika owi-aoC--- 260.00g

This is a data volume of type cache, that ties together the big data_corig LV that contains the bulk data and a data-cache LV of type cache-pool that ties together the data-cache_cdata LV with the actual cache data and data-cache_cmeta with the cache metadata.

A few months ago, I put up an old Atom-powered Supermicro server (SYS-5015A-PHF) again, to serve at De War to collect and display various sensor and energy data about our building.

The server turned out to have an annoying habit: every now and then it would start beeping (one continuous annoying beep), that would continue until the machine was rebooted. It happened sporadically, but kept coming back. When I used this machine before, it was located in a datacenter where nobody would care about a beep more or less (so maybe it has been beeping for years on end before I replaced the server), but now it was in a server cabinet inside our local Fablab, where there are plenty of people to become annoyed by a beeping server...

I eventually traced this back to faulty sensor readings and fixed this by disabling the faulty sensors completely in the server's IPMI unit, which will hopefully prevent the annoying beep. In this post, I'll share my steps, in case anyone needs to do the same.

Shorting fan pins

At first, I noticed that there was an alarm displayed in the IPMI webinterface for one of the fans. Of course it makes sense to be notified of a faulty fan, except that the system did not have any fans connected... It did show the fan speed as 0RPM (or -2560RPM depending on where you looked) as expected, so I suspected it would start up realizing there was no fan but then sporadically seeing a bit of electrical noise on the fan speed pin, causing it to mark the fan as present and immediately as not running, triggering the alarm. I tried to fix this by shorting the fan speed detection pins to the GND pins to make it more noise-resilient.

Temperatures also wonky

However, a couple of weeks later, the server started beeping again. This time I looked a bit more closely, and found that the problem was caused by too high temperature this time. The IPMI system event log (queried using ipmi-sel) showed:

43   | Feb-17-2023 | 09:18:58 | CPU Temp         | Temperature       | Upper Non-critical - going high ; Sensor Reading = 125.00 C ; Threshold = 85.00 C
44   | Feb-17-2023 | 09:18:58 | CPU Temp         | Temperature       | Upper Critical - going high ; Sensor Reading = 125.00 C ; Threshold = 90.00 C
45   | Feb-17-2023 | 09:18:58 | CPU Temp         | Temperature       | Upper Non-recoverable - going high ; Sensor Reading = 125.00 C ; Threshold = 95.00 C
46   | Feb-17-2023 | 16:26:16 | CPU Temp         | Temperature       | Upper Non-recoverable - going high ; Sensor Reading = 41.00 C ; Threshold = 95.00 C
47   | Feb-17-2023 | 16:26:16 | CPU Temp         | Temperature       | Upper Critical - going high ; Sensor Reading = 41.00 C ; Threshold = 90.00 C
48   | Feb-17-2023 | 16:26:16 | CPU Temp         | Temperature       | Upper Non-critical - going high ; Sensor Reading = 41.00 C ; Threshold = 85.00 C

This is abit opaque, but the events at 9:18 show the temperature was read as 125°C - clearly indicating a faulty sensor. These are (I presume) the "asserted" events for each of the thresholds that this sensor has. Then at 16:26, the server was rebooted and the sensor read 41°C again (which I believe is still higher than realistic) and each of the thresholds emits a "deasserted" event.

Looking back, I noticed that the log showed events for both fans and both temperature sensors, so it seemed all of these sensors were really wonky. I could also see the incorrect temperatures clearly in the sensor data I had been collecting from the server (using telegraf, collected using lm-sensors from within the linux system itself, but clearly reading from the same sensor as IPMI):

Note that the graph above shows two sensors, while IPMI only reads two, so I am not sure what the third one is. The alarm from the IPMI log is shown clearly as a sudden jump of the temp2 purple line (jumping back down when the server was rebooted). But also note an unexplained second jump down a few hours later, and note that the next day temp1 dives down to -53°C for some reason, which also matches what IPMI reads:

$ sudo ipmitool sensor                                                
System Temp      | -53.000    | degrees C  | nr    | -9.000    | -7.000    | -5.000    | 75.000    | 77.000    | 79.000    
CPU Temp         | 27.000     | degrees C  | ok    | -11.000   | -8.000    | -5.000    | 85.000    | 90.000    | 95.000    
CPU FAN          | -2560.000  | RPM        | nr    | 400.000   | 585.000   | 770.000   | 29260.000 | 29815.000 | 30370.000 
SYS FAN          | -2560.000  | RPM        | nr    | 400.000   | 585.000   | 770.000   | 29260.000 | 29815.000 | 30370.000 
CPU Vcore        | 1.160      | Volts      | ok    | 0.640     | 0.664     | 0.688     | 1.344     | 1.408     | 1.472     
Vichcore         | 1.040      | Volts      | ok    | 0.808     | 0.824     | 0.840     | 1.160     | 1.176     | 1.192     
+3.3VCC          | 3.280      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712     
VDIMM            | 1.824      | Volts      | ok    | 1.448     | 1.480     | 1.512     | 1.960     | 1.992     | 2.024     
+5 V             | 5.056      | Volts      | ok    | 4.096     | 4.320     | 4.576     | 5.344     | 5.600     | 5.632     
+12 V            | 11.904     | Volts      | ok    | 10.368    | 10.496    | 10.752    | 12.928    | 13.056    | 13.312    
+3.3VSB          | 3.296      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712     
VBAT             | 2.912      | Volts      | ok    | 2.560     | 2.624     | 2.688     | 3.328     | 3.392     | 3.456     
Chassis Intru    | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na        
PS Status        | 0x1        | discrete   | 0x01ff| na        | na        | na        | na        | na        | na

Note that the voltage sensors show readings that do make sense, and looking at the history, they show no sudden jumps, so those are probably still reliably (even though they are read from the same sensor chip according to lm-sensors).

Disabling the sensors

It seems you can disable generation of events when a threshold is crossed, can even disable reading the sensor entirely. Hopefully this will also prevent the BMC from beeping on weird sensor values.

To disable things, I used ipmi-sensor-config (from the freeipmi-tools Debian package):

1. First I queried the current sensor configuration:

    sudo ipmi-sensors-config --checkout > ipmi-sensors-config.txt
   

2. Then I edited the generated file, setting Enable_All_Event_Messages and Enable_Scanning_On_This_Sensor to No. I also had to set the hysteresis values for the fans to None, since the -2375 value generated by --checkout was refused when writing back the values in the next step.

3. Commited the changes with:

   sudo ipmi-sensors-config --commit --filename ipmi-sensors-config.txt
   

I suspect that modifying Enable_All_Event_Messages allows the sensor to be read, but prevents the threshold from being checked and generating events (especially since this setting seems to just clear the corresponding setting for each available threshold, so it seems you can also use this to disable some of the thresholds and keep some others). However, it is not entirely clear to me if this would just prevent these events from showing up in the event log, or if it would actually prevent the system from beeping (when does the system beep? On any event? Specific events? This is not clear to me).

For good measure, I decided to also modify Enable_Scanning_On_This_Sensor, which I believe prevents the sensor from being read at all by the BMC, so that should really prevent alarms. This also causes ipmitool sensor to display value and status as na for these sensors. The sensors command (from the lm-sensors package) can still read the sensor without issues, though the values are not very useful anyway...).

Note that apparently these settings are not always persistent across reboots and powercycles, so make sure you test that. For this particular server, the settings survive across a reboot, I have not tested a hard power cycle yet.

I cannot yet tell for sure if this has fixed the problem (only applied the changes today), but I'm pretty confident that this will indeed keep the people in our Fablab happy (and if not - I'll just solder off the beeper from the motherboard, but let's hope I will not have to resort to such extreme measures...).

When sorting out some stuff today I came across an "Ecobutton". When you attach it through USB to your computer and press the button, your computer goes to sleep (at least that is the intention).

The idea is that it makes things more sustainable because you can more easily put your computer to sleep when you walk away from it. As this tweakers poster (Dutch) eloquently argues, having more plastic and electronics produced in China, shipped to Europe and sold here for €18 or so probably does not have a net positive effect on the environment or your wallet, but well, given this button found its way to me, I might as well see if I can make it do something useful.

I had previously started a project to make a "Next" button for spotify that you could carry around and would (wirelessly - with an ESP8266 inside) skip to the next song using the Spotify API whenever you pressed it. I had a basic prototype working, but then the project got stalled on figuring out an enclosure and finding sufficiently low-power addressable RGB LEDs (documentation about this is lacking, so I resorted to testing two dozen different types of LEDs and creating a website to collect specs and test results for adressable LEDs, which then ended up with the big collection of other Yak-shaving projects waiting for this magical moment where I suddenly have a lot of free time).

In any case, it seemed interesting to see if this Ecobutton could be used as poor-man's spotify next button. Not super useful, but at least now I can keep the button around knowing I can actually use it for something in the future. I also produced some useful (and not readily available) documentation about remapping keys with hwdb in the process, so it was at least not a complete waste of time... Anyway, into the technical details...

How does it work?

I expected this to be an USB device that advertises itself as a keyboard, and then whenever you press the button, sends the "sleep" key to put the computer to sleep.

Turns out the first part was correct, but instead of sending a sleep keypress, it sends Meta-R, ecobutton (it types each of these letters after each other), enter. Seems you need to install a tool on your pc that is executed by using the windows-key+R shortcut. Pragmatic, but quite ugly, especially given a sleep key exists... But maybe Windows does not implement the key (or maybe this tool also changes some settings for deeper sleep, at least that's what was suggested in the tweakers post linked above).

Replacing the firmware?

I considered I could maybe replace the firmware to make the device send whatever keystroke I wanted, but writing firmware from scratch for existing hardware is not the easiest project (even for a simple device like this). After opening the device I decided this was not a feasible route.

The (I presume) microcontroller in there is hidden in a blob, so no indication as to its type, pin connections, programming ports (if it actually has flash and is not ROM only).

I did notice some solder jumpers that I figured could influence behavior (maybe the same PCB is used for differently branded buttons), but shorting S1 or S5 did not seem to change behavior (maybe I should have unsoldered S3, but well...).

Remap keys, then?

The next alternative is to remap keys on the computer. Running Linux, this should certainly be possible in a couple of dozen ways. This does need to be device-specific remapping, so my normal keyboard still works as normal, but if I can unmap all keys except for the meta key that it presses first, and map that to someting like the KEY_NEXTSONG (which is handled by spotify and/or Gnome already), that might work.

I first saw some remapping solutions for X, but those probably will not work - I'm running wayland and I prefer something more low-level. I also found cool remapping daemons (like keyd) that grab events from a keyboard and then synthesise new events on a virtual keyboard, allowing cool things like multiple layers or tapping shift and then another key to get uppercase instead of having to hold shift and the key together, but that is way more complicated than what I need here.

Then I found that udev has a thing called "hwdb", which allows putting files in /etc/udev/hwdb.d that match specific input devices and can specify arbitrary scancode-to-keycode mappings for them. Exactly what I needed - works out of the box, just drop a file into /etc.

Understanding and taming udev and hwdb

The challenge turned out to be to figure out how to match against my specific keyboard identifier, what scancodes and keycodes to use, and in general figure out how the ecosystem around this works (In short: when plugging in a device, udev rules consults the hwdb for extra device properties, which a udev builtin keyboard command then uses to apply key remappings in the kernel using an ioctl on the /dev/input/eventxx device). In case you're wondering - this means you do not need to use hwdb, you can also apply this from udev rules directly, but then you need a bit more care.

I've written down everything I figured about hwdb in a post on Unix stackexchange, so I'l not repeat everything here.

Using what I had learnt, getting the button to play nice was a matter of creating /etc/udev/hwdb.d/99-ecobutton.hwdb containing:

evdev:input:b????v3412p7856e*
  KEYBOARD_KEY_700e3=nextsong # LEFTMETA
  KEYBOARD_KEY_70015=reserved # R
  KEYBOARD_KEY_70008=reserved # E
  KEYBOARD_KEY_70006=reserved # C
  KEYBOARD_KEY_70012=reserved # O
  KEYBOARD_KEY_70005=reserved # B
  KEYBOARD_KEY_70018=reserved # U
  KEYBOARD_KEY_70017=reserved # T
  KEYBOARD_KEY_70011=reserved # N
  KEYBOARD_KEY_70028=reserved # ENTER

This matches the keyboard based on its usb vendor and product id (3412:7856) and then disables all keys that are used by the button, except for the first, and remaps that to KEY_NEXTSONG.

To apply this new file, run sudo systemd-hwdb update to recompile the database and then replug the button to apply it (you can also re-apply with udevadm trigger, but it seems then Gnome does not pick up the change, I suspect because the gnome-settings media-keys module checks only once whether a keyboard supports media keys at all and ignores it otherwise).

With that done, it now produces KEY_NEXTSONG events as expected:

$ sudo evtest --grab /dev/input/by-id/usb-3412_7856-event-if00
Input driver version is 1.0.1
Input device ID: bus 0x3 vendor 0x3412 product 0x7856 version 0x100
Input device name: "HID 3412:7856"

[ ... Snip more output ...]

Event: time 1675514344.256255, type 4 (EV_MSC), code 4 (MSC_SCAN), value 700e3
Event: time 1675514344.256255, type 1 (EV_KEY), code 163 (KEY_NEXTSONG), value 1
Event: time 1675514344.256255, -------------- SYN_REPORT ------------
Event: time 1675514344.264251, type 4 (EV_MSC), code 4 (MSC_SCAN), value 700e3
Event: time 1675514344.264251, type 1 (EV_KEY), code 163 (KEY_NEXTSONG), value 0
Event: time 1675514344.264251, -------------- SYN_REPORT ------------

More importantly, I can now skip annoying songs (or duplicate songs - spotify really messes this up) with a quick butttonpress!

After I recently ordered a new laptop, I have been looking for a USB-C-connected dock to be used with my new laptop. This turned out to be quite complex, given there are really a lot of different bits of technology that can be involved, with various (continuously changing, yes I'm looking at you, USB!) marketing names to further confuse things.

As I'm prone to do, rather than just picking something and seeing if it works, I dug in to figure out how things really work and interact. I learned a ton of stuff in a short time, so I really needed to write this stuff down, both for my own sanity and future self, as well as for others to benefit.

I originally posted my notes on the Framework community forum, but it seemed more appropriate to publish them on my own blog eventually (also because there's no 32,000 character limit here :-p).

There are still quite a few assumptions or unknowns below, so if you have any confirmations, corrections or additions, please let me know in a reply (either here, or in the Framework community forum topic).

Parts of this post are based on info and suggestions provided by others on the Framework community forum, so many thanks to them!

Getting started

First off, I can recommend this article with a bit of overview and history of the involved USB and Thunderbolt technolgies.

Then, if you're looking for a dock, like I was, the Framework community forum has a good list of docks (focused on Framework operability), and Dan S. Charlton published an overview of Thunderbolt 4 docks and an overview of USB-C DP-altmode docks (both posts with important specs summarized, and occasional updates too).

Then, into the details...

Lines, lanes, channels, duplex, encoding and bandwidth

• The transmission of high-speed signals almost exclusively used "differential pairs" for encoding. This means two wires/pins are used to transmit a single signal in a single direction (at the same time). In this post, I will call one such pair a "line".
• A single line can either be simplex (unidirectional, transmitting in the same direction all the time) or half-duplex (alternating between directions).
• Two lines (one for each direction) can be combined to form a full-duplex channel, where data can be transmitted in both directions simultaneously. Sometimes (e.g. in USB, PCIe) such a pair of lines (so 4 wires in total) is referred to as a "(full duplex) lane", but since the term "lane" is also used to refer to a single unidirectional wire pair (e.g. in DisplayPort), I'll use "full duplex channel" for this instead.
• I'll still sometimes use "lane" when referring to specific protocols, but I'll try to qualify what I mean exactly then.
• A line will be operated at a specific bitrate, which is the same as the available bandwidth for that line. Multiple lines can be combined, leading to higher total bandwidth (but the same bitrate).
• To really compare different technologies, it is often useful to look at the bitrate used on the individual lines, for which I'll use Gbps-per-line below. I'll use "total bandwidth" refer to the sum of all the lines used in either direction, and "full duplex bandwidth" means the bandwidth that can be used in both directions at the same time.
• Most of the numbers shown here are for bits on the wire. However, that does not represent the amount of actual data being sent. For example, SATA, PCIe 2.0, USB 3.0, HDMI, and DisplayPort 1.4 transmit 10 bits on the wire for every 8 bits of data, otherwise known as 8b/10b encoding. This means USB 3.0 is more like 4 Gbps instead of 5 Gbps. This data includes protocol overhead, such as framing, headers, error correction, etc. so the effective data transfer rates (i.e. data written to disk) is lower still.

USB1/2/3

• USB1/2: single half-duplex pair up to 480Mbs half-duplex.
• USB1/2 link setup happens using pullups/pulldowns on the datalines, always starting up at USB1 speeds (low-speed/full-speed) and then using USB data messages to coordinate the upgrade to USB2 speeds (hi-speed).
• When a device using USB1 speeds ("low speed" or "full speed") is connected to a USB2 hub, the difference in speed is translated using a "Transaction Translator". Most hubs have a single TT for the entire hub, meaning that when multiple such devices are connected, only one of them can be transmitting at the same time, and they have to share the 12Mbps (full speed) bandwidth amongst them. In practice, this can sometimes lead to issues, especially with devices that need guaranteed bandwidth, like USB audio devices. Note that this applies even to modern devices that support USB2, but only implement the slower transmission speed (like often audio devices, mice, keyboard, etc.). Some hubs have multiple TTs, meaning each downstream port has the full 12Mbps available, making multi-TT hubs a better choice (but unfortunately, manufacturers usually do not specify this, you can use lsusb -v to tell).
• USB3.0: Adds two extra lines at 5Gbps-per-line (== 1 full-duplex channel, 10Gbps total bandwidth) using a new "USB3" connector with 4 extra data wires.
• USB3 is a separate protocol from USB1/2 and works exclusively over the newly added lines. This means that a USB3 hub is essentially two fully separate devices in one (USB1/2 hub and USB3 hub) and USB3 traffic and USB1/2 traffic can happen at the same time over the same cable independently.
• USB3.0 link setup uses "LFPS link training" pulses, where both sides just start to transmit on their TX line and listen on their RX line). [source].
• USB3.0 over a USB-C connector only uses 2 of the 4 high-speed lines available in the connector and cable.
• USB3.1 increases single line speed to 10Gbps (with appropriate cabling), for 20Gbps total bandwidth.
• USB3.2 allows use of all 4 high-speed lines in a USB-C connection at the same 10Gbps-per-line speed of 3.1, for 40Gbps total bandwidth.

DisplayPort

[source]

• Displayport is essentially a high-speed 4 line unidirectional connection (main link) for display and audio data, plus one lower speed half-duplex auxilliary channel.
• The auxilliary channel is used for things like EDID (detecting supported resolution) and CEC (controlling playback and power state of devices). Deciding the link speed to use on the high-speed lines is done using "link training" on those lines themselves, not using the aux channel. [source]
• Unlike VGA/DVI/HDMI, Displayport is based on data packets (similar to ethernet), rather than a continuous (clocked) stream of data on specific pins.
• Displayport speed evolved over time: 4x1.62=6.48Gbps for DP1/RBR, 4x2.7=10.8Gbps for DP1/HBR, 4x5.4=21.6Gbps for DP1.2/HBR2, 4x8.1=32.4Gbps for DP1.3/HBR3, 4x10=40Gbps for DP2.0/UHBR10, 4x13.5=54Gbps for DP2.0/UHBR13.5, 4x20=80Gbps for DP2.0/UHBR20. All versions use the same 4-line connection, just driving more data over the same wires (needing better quality cables). [source]
• Displayport 1.2 introduced optional Display Stream Compression (DSC), which is a "visually lossless" compression that can reduce bandwidth usage by a factor 3. Starting with DisplayPort 2.0, supporting DSC is mandatory.
• Displayport 1.2 introduced Multi-Stream Transport (MST), which allow sending data for multiple displays (up to 63 in theory) over a single DP link (multiplexed inside the datastream, so without having to dedicate lines to displays). Such an MST signal can then be split by an MST hub (often a 2-port hub inside a display to allow daisy-chaining displays). MST is also possible for DP inside an USB-C or thunderbolt connection. It can also use DSC (compression) on the input and uncompressed on the output, to drive displays that do not support compression. [source]
• Displayport dualmode (aka DP++) allows sending out HDMI/DVI signals from a displayport connector. This essentially just repurposes pins on the DP connector to transfer HDMI/DVI signals, so can be used with a passive adapter or cable (though the adapter does need to do voltage level shifting). This only works to connect a DP source to a HDMI/DVI sink (display), not the other way around. This also repurposes two additional CONFIG1/2 pins from being grounded to carry HDMI/DVI signals, so dual-mode is not available on an USB-C DP alt mode connection. [source]
• Some Displayport devices support Adaptive Sync, where the display refresh rate is synchronized with the GPU framerate. This seems to still work when DisplayPort is tunneled through Thunderbolt or USB4, but often breaks when going through an MST hub (though some explicitly document support for it and probably do work). [source]

Thunderbolt 1/2/3

[source]

• Thunderbolt 1/2 is an Apple/Intel specific protocol that reuses the mini-displayport connector to transfer both PCIe (for e.g. fast storage or external GPUs) and DP signals using four 10Gbps high-speed lines (40Gbps total bandwidth) configured as two 10Gbps full-duplex channels (TB1) or a single 20Gbps full-duplex channel (TB2). [source].
• It is a bit unclear to me how these signals are multiplexed. Apparently TB2 combines all four lines into a single full-duplex channel, suggesting that on TB1 there are two separate channels, but does that mean that one channel carries PCIe and one carries DP on TB1? Or each device connected is assigned to (a part of) either channel?
• TB1/2 are backwards compatible with DP, so you can plug in a non-TB DP display into a TB1/2 port. This uses a multiplexer to connect the port to either the TB controller or the GPU. This mux is controlled by snooping the signal lines and detecting if the signals look like DP or TB. In early Light Ridge controllers, this was external circuitry, from Cactus Ridge all this (including the mux) was integrated in the TB controller. [source: comments below]
• TB1/2/3 ports can be daisychained with up to 6 devices in the chain. Hubs (i.e. a split in the connection) are not supported until TB4. The last device in the chain can be a non-TB DP display as well. [source]
• USB (1/2/3) over thunderbolt is not supported directly, but is implemented by putting a full USB-host controller in a thunderbolt hub/dock that is then controlled by the host through PCIe-over-thunderbolt.
• Thunderbolt 3 is an upgrade of TB2 that uses an USB-C connector instead of mini-displayport. It uses 4 high-speed lines at 20Gbps-per-line (double of TB2), to get 40Gbps full-duplex, or 80Gbps total bandwidth (and maybe also the sidechannel SBU line for link management or so, but I could not find this). [source]
• Thunderbolt 4 seems to be essentially just USB4 with all optional items implemented (but USB4 is more like TB3 than USB3...).
• TB1/2/3 are only implemented by Intel chips, and TB1/2 is used almost exclusively by Apple. TB4/USB4 can be implemented by other chip makers too, but this has not happened yet I believe.

USB-C

[source]

• USB-C is a specification of a connector, intended to use for USB by default, but also for other protocols (usually in addition to USB) and not tied to any particular USB version.
• USB-C has a huge number of pins (24 pins), consisting of:
  • 4 VBUS and 4 GND pins to allow delivering more current,
  • 4 high-speed lines (8 pins), typically used as two full-duplex channels,
  • USB2 D+/D- pins (duplicated, to allow reverse insertion without additional circuitry)
  • Two CC/VCONN pins for low-speed connection setup, USB-PD negotiation, altmode setup, querying the cable, etc.
  • Two SBU sidechannel pins as an additional lower speed line.
• USB-C cables come in different flavors: [source]
  • The simplest cables have just power and USB2 pins, and do not support anything other than USB1/2. Cables that do connect all pins are apparently called "full-featured", though these might still not support the higher (10/20Gbps-per-line) speeds.
  • Cables can be passive (direct copper connections), or active (with electronics in the cable to repeat/amplify a signal to allow higher speeds and longer cables).
  • Active cables typically seem to be geared towards (and/or limited to) a particular signalling scheme and/or direction. This means that they are usually not usable for protocols they are not explicitly designed for. This partially comes down to the bitrates used, but also the exact low-level signalling protocol (e.g. USB3 Gen2, USB4 Gen2 and DP UHBR 10 all use 10Gbps-per-line, but use different encodings, error correction and/or direction, meaning active cables might support one, but not the other).
  • Passive cables can generally be used more flexibly, mainly limited to a certain bitrate based on cable quality and length.
  • Cables can also contain a "e-marker" chip that can be queried using USB-PD messages on the CC pin, containing info on the cable's spec. Such a chip is required for 5A operation (all other cables are expected to allow up to 3A) and for all USB4 (even the lower-speed 10Gbps-per-line USB4) operation.
  • Cables with "e-marker" chip but no signal conditioning are still considered passive cables.
  • USB4/TB4 requires an e-marker chip in the cable (falling back to USB3 or TB3 without it). Passive cables (including passive TB3 cables) can always be used for USB4 (with the exact speed depending on the cable rating), while active cables can only be used when they are rated for USB4 (so active USB3 cables cannot be used for USB4, but active USB4 cables can be used for USB3). [source]
• USB-C uses the CC pins for detecting a connection and deciding how to set it up. A host (or dowstream facing port or sourcing port) has pullups on both CC pins (value depens on available current), a device (or upstream facing port or sinking port) has pulldowns on both pins (value is fixed). Together this forms a voltage divider that allows detecting a connection and maximum current. A cable connects only one of the CC pins (leaving the other floating, or pulling it down with a different resistor value) which allows detecting the orientation of the cable. [source]
• After initial connection, the CC pin is also used for other management communication, such as USB-PD (including setting up alt modes and USB4), or vendor-specific extensions.
• The other CC pin (pulled down inside the cable) is repurposed as VCONN and used to power any active circuitry (including the e-marker) inside the cable.
• USB-C can provide up to 15W (5V×3A) on the VBUS pins with passive negotiation using resistors on the CC pins. USB-PD uses active negotiation using data messages on the CC pin and allows up to 100W (20V×5A) of power on the VBUS pins and allows reversing the power direction as well (i.e. allow a laptop USB-C port to receive power instead of supplying it).

USB-C alt modes

• USB-C alt modes allow using the pins (in particular the 4 high-speed lines and the lower speed SBU line) for other purposes.
• Alt modes are negotiated actively through the CC pin using USB-PD VDM messages. Once negotiated, this allows using these pins different physical signalling, including different directions. This means that e.g. a USB-C-to-DP adapter using DP alt mode can be mostly a passive device, though it does need some logic for the USB-PD negotiation and might need some switches to disconnect the wires until the alt mode is enabled.
• Alt modes are single-hop only (hubs might connect upstream and downstream alt mode channels, but there is no protocol to configure this AFAICT).
• Alt modes are technically separate from the USB protocol (version) supported (i.e. they can be configured without having to do USB communication), but the bandwidth capabilities does introduce some correlation (i.e. DP alt mode 2.0 supports DP 2.0 and might need up to 20Gbps-per-line, so effectively needs a USB4-capable controller, cable and hub/dock).
• DP alt mode (and probably others) can use 1, 2, or all 4 of the high-speed lines (plus the SBU line for the aux channel). When using only 1 or 2 lines, the 2 other lines can still be used for USB3 traffic (just using 1 line for USB is not possible, USB3/4 always needs a full-duplex channel, so 2 or 4 lines). In all cases, the USB2 line can still be used for USB1/2.

• Because in DP alt mode all four lines can be used unidirectionally (unlike USB, which is always full-duplex), this means the effective bandwidth for driving a display can be twice as much in alt mode than when tunneling DP over USB4 or using DisplayLink over USB3.

  In practice though, DP-altmode on devices usually supports only DP1.4 (HBR3 = 8.1Gbps-per-line) for 4x8.1 = 32.4Gbps of total and unidirectional bandwidth, which is less than TB3/4 with its 20Gbps-per-line for 2x20 = 40Gbps of full-duplex bandwidth. This will change once devices start support DP-altmode 2.0 (UHBR20 = 20Gbps-per-line) for 4x20=80Gbps of unidirectional bandwidth.

• It seems that DP alt mode can only be combined with USB3, not TB3 or USB4 (USB4 and DP alt mode both need the SBU pins). Since DP can be tunneled over TB/USB4, there is not much need for DP alt mode when using TB/USB4, except that it could potentially have allowed more bandwidth (2x20Gbps unidirectional DP2.0 plus 1x20Gbps full-duplex TB/USB4).
• HDMI alt mode exists, but is apparently not actually implemented by anyone. USB-C-to-HDMI adapters typically seem to use DP alt mode combined with an active DP-to-HDMI converter. I'm assuming this also holds for the Framework HDMI expansion cards. [source]
• Thunderbolt 3 is also implemented as an alt mode.
• Audio Adapter Accessory Mode allows sending analog audio data over an USB-C connector (using the USB2 and sideband pins). This looks like another alt mode, but instead of active negotiation over the CC pin, this mode is detected by just grounding both CC pins so adapters for this mode can be completely passive. [source]
• Ethernet or PCIe are sometimes mentioned as alt modes, but apparently do not actually exist. [source]

USB4 / Thunderbolt 4

[source] and [source] and [source]

• USB4 is again really different from USB3 and looks more like Thunderbolt 3 than USB3.
• USB4 does not specify any device types by itself (like USB1/2/3), but is a generic bulk data transport that allows tunneling of other protocols, like PCIe, DisplayPort and USB3.
• USB4 uses the 4 USB-C high-speed lines at up to 20Gbps-per-line for 40Gbps full-duplex and 80Gbps total bandwidth.
• USB4 link setup happens using USB-PD messages over the CC pins, falling back to USB3 setup if no USB4 link is detected within a second or so (so also in that sense USB4 is also more like Thunderbolt 3 alt mode than USB3). [source]
• USB4 also uses two additional pins (sideband channel) of the USB-C connector for further link initialization and management, so it needs USB-C and cannot be used over USB3-capable USB-A/USB-B/USB-micro connectors and cables. [source]
• Like with TB3, this tunneling happens on a data level: The physical layer signalling is defined by USB4 and different tunneled protocols are mixed in the same bitstream (unlike USB-C alt mode, where individual lines are completely allocated to some alt mode, which also determines the physical signalling used). This also means that DP alt mode and USB4-tunneled-displayport are two different ways to transfer DP over a USB4-capable connection.
• Unlike with TB3, USB2 is not tunneled but still ran in parallel over the USB2 pins.
• Unlike with TB3, USB3 is tunneled directly over USB4 and uses the host USB3 controller with only USB3 hubs in docks (though I guess docks could still choose to include PCIe connected USB controllers). In some cases, this might result in lower total USB bandwidth (10 or 20Gbps shared by all devices) compared to the TB3 approach of (sometimes multiple) USB3 controllers in the dock that are accessed through tunneled PCIe.
• USB4 end devices have most features optional. USB4 hosts require 10Gbps-per-line, tunneled 10Gbps USB3, tunneled DP, and DP altmode (unsure if this requires DP altmode 2.0), leaving 20Gbps-per-line, tunneled 20Gbps USB3, tunneled PCIe, TB3 and other altmodes optional. USB4 hubs require everything, except 20Gbps USB3 and other altmodes. [source]
• Thunderbolt 4 is (unlike thunderbolt 3) not really a protocol by itself, but is essentially just USB4 with all or most optional features enabled, and with additional certification of compatibility and performance. In particular, TB4 ports must support 20Gbps-per-line, computer-to-computer networking, PCIe, charging through at least one port, wake-up from sleep, 2x4K@60 or 1x8K@60 display, up to 2m passive cables. I have not been able to find the actual specification to see what these things really mean technically (i.e. are these display resolutions required over USB4, or altmode (2.0), must they be uncompressed, from which states does this wakeup wake up, etc.). [source] and [source]
• TB1/2/3 were propietary protocols, implemented only on Intel chips. USB4 (and also TB4) are (somewhat) more open and can be implemented by multiple vendors (though I could find the USB4 spec easily, but not the TB4 spec, or the TB3 spec which is required for compatibility, or the DP alt mode spec, which is required for USB4 ports).
• USB4 hubs must also support the TB3 for compatibility, controllers and devices may support TB3. [source]
• Different USB3/4 transfer modes have been renamed repeatedly (i.e. USB3.0 became USB3.1 gen1 and then USB3.2 gen1x1). Roughly, gen1 (aka SuperSpeed aka SuperSpeed USB) means 5Gbps-per-line, gen2 (aka SuperSpeed+ aka SuperSpeed USB 10Gbps, or SuperSpeed USB 20Gbps aka USB4 20Gbps for gen2x2) means 10Gbps-per-line, gen3 (aka USB4 40Gbps) means 20Gbps per line. No suffix or x1 means using only one pair of lines, x2 means two pairs of lines (so e.g. gen2x2 uses all four lines, at 10Gbps-per-line, for 20Gbps full-duplex and 40Gbps of total bandwidth). Also, you can mostly ignore the USB versions (i.e. USB3.1 gen1 and USB3.2 gen1x1 are the same transfer mode), except for USB4 gen2, which is also 10Gbps-per-line, but uses different encoding. [source]
• A USB4 hub now effectively contains 3 separate USB hubs: A USB1/2 hub, a USB3 hub, and a USB4 hub (aka USB4 router). As with USB3 hubs, the USB1/2 hub is completely distinct, always connected to the USB-C connectors and its communication runs in parallel with the USB3/4 communication. The USB3 (superspeed) hub can either connect directly to the USB-C connectors directly (when an USB3 device or host is connected), or it can connect through the USB4 router to receive / send USB4-tunneled-USB3 traffic. USB3 traffic seems to be always tunneled over just one hop in the connection, so a host -> USB4 hub -> USB4 hub -> USB3 device connection is tunneled in USB4 from the host to the first hub, is then unpacked into the USB3 hub, then tunneled again for the trip to the second hub, unpacked again into the USB3 hub inside the second hub, and then to the USB3 device directly over the USB-C (or USB-A) connector without further tunneling. [source]
• PCIe tunneling works similar to USB3 tunneling, where the tunneling is done over a single hop, unpacked into a PCIe switch, and then tunneled again for further hops is needed. [source]
• For DP tunneling, this works differently, there a single tunnel can stretch over multiple hops, all the way from the DP source in the USB4 host towards the DP output in the last USB4 hub (or USB4-capable display). [source]

Thunderbolt / USB4 hardware

• Thunderbolt 1/2/3 is exclusively implemented by Intel controllers. USB4/TB4 (including TB3 compatibility) is open for others to implement too (but currently Intel controllers are still used mostly).
• Intel Thunderbolt and USB4 controller families are all named "Something Ridge". Initially there were quite a few families implementing TB1, then Falcon Ridge implemented TB2, Alpine and Titan ridge for TB3, Maple and Goshen ridge for TB4 [source].
• Most of these controllers can be used in hosts, devices and hubs, but some of them are exclusively for devices/hubs (such as Goshen Ridge) or hosts.
• Originally, thunderbolt was implemented using discrete thunderbolt controller chips, which were connected to the CPU using high-speed PCIe and DP traces on the mainboard, making thunderbolt support a complex and expensive thing. With Intel Ice Lake and Tiger Lake, the thunderbolt controller is integrated in the processor, making the implementation cheaper and more power efficient. [source] and [source]

Understanding tunneling and bandwidth limitations

[source]

• Looking at how Thunderbolt/USB4 hardware is constructed, you typically have a controller chip (router in USB4 terminology) that handles the TB/USB4 traffic (and usually also TB3/USB3 backward compatibility). Both the host and the dock (or real TB device) have such a controller (often the same controller can be used in hosts, docks/hubs and devices).
• These controllers have one or more TB outputs, connected to TB (USB-C) connectors, forming the TB link between then.
• These controllers also have other interfaces (adapters in USB4 terminology), such as DP inputs or outputs, PCIe upstream/downstream interfaces, or USB upstream/downstream interfaces.
  • In the host these interfaces are usually connected to the CPU/GPU. USB links are usually internally connected to a USB controller integrated into the controller.
  • In a dock/hub DP interface(s) are usually connected to output connectors (DP, HDMI, TB or USB-C). These are often multiplexed, so there might be 2 DP interfaces available that can be routed to any two of the available outputs, or they can be connected through an internal MST hub.
  • In a dock/hub PCIe interface(s) are usually internally connected to devices (e.g. PCIe ethernet) or exposed (e.g. PCIe NVMe slot). If multiple PCIe lanes are available, they can be connected to the same device, or split among multiple.
  • In a dock/hub the USB interface(s) are usually internally connected to USB2/3 hubs (often integrated into the TB/USB4 controller).
  • Thunderbolt ports can usually be internally connected to the TB interface, a DP interface (for DP alt mode) or an USB3 controller (for non-thunderbolt USB compatibility).
• Often these interfaces provide a transparent connection, where e.g. the GPU negotiates a certain DP bitrate with an attached display through the tunnel, just as if the display was directly connected.
• On Linux, to get some basic info on present TB controllers and devices and the used link speed, you can use boltctl (e.g. boltctl list -a).
• On Linux, to see what interfaces are present on e.g. a TB dock, run echo "module thunderbolt +p" > /sys/kernel/debug/dynamic_debug/control, plug in your dock and check dmesg (which will call the USB4 controller/router in the the dock a "USB4 switch" and its interfaces "Ports").
• When looking at what limits / determines the bandwidth available when a tunneled (i.e. TB3 or TB4) link is involved, there are mainly two kinds of limit:
  1. The total bandwidth of (each hop of) the link itself. This bandwidth is shared between all tunneled connections and depends on the protocol and cable used (e.g 20/40Gbps for TB3/TB4/USB4 depending on cable and controller capabilities, TB3/TB4 controllers always support 40Gbps).
  2. The bandwidth (and number) of individual tunnels. This is mostly a limit are limited by the hardware interfaces present on the controllers/routers used (e.g. dual 4xHBR2 interfaces on Alpine Ridge, dual 4xHBR3 interfaces on Titan Ridge). These limitations are mostly linked to the hardware used, not so much limited by the protocol (though the protocols do seem to typically specify minimum bandwidth/features for each port).
• The second limit is the most complex to figure out, because it involves limitations on both ends of the connection (and for USB3/PCIe also any intermediate steps).
• The link bandwidth is shared between all tunnels on a single port, but the tunnel bandwidth is often shared between all ports on the same controller.
• When multiple tunnels run over the same link, only the actual bandwidth usage counts (except DP does limit links based on max link bandwidth, see below). For example, an USB3 10Gbps tunnel with only a thumb drive connected will only use very little bandwidth, leaving the rest available for other tunnels. Only when the combined bandwidth usage of all tunnels exceeds the link bandwidth will bandwidth of each tunnel be limited.
• The number of tunnels over a single link seems to be practically unlimited by the protocols, but only limited by the number of interfaces on the used controllers (so we could in theory see future controllers that support three or four DP interfaces, though that probably does not make too much sense given MST exists and total bandwidth is still limited). Maybe TB1/2/3 do have protocol limitations, but since it is unlikely that new controllers will be made for these, it won't matter anyway.

• For DP:

  • TB1 controllers support one 4xHBR interface (DP1.0), TB2 controllers support one 4xHBR2 interface (DP1.2), Alpine Ridge (TB3) supports two 4xHBR2 interfaces (2xDP1.2), Titan Ridge (TB3), Goshen/Maple Ridge (TB4) and Tiger Lake (TB4) support two 4xHBR3 interfaces (2xDP1.4).
  • Each DP interface consists of four lanes, which cannot be split.
  • Each of the interfaces (four lanes each) on the host controller can be routed to a single dock or display (where they can still be split into multiple DP or DP-alt-mode outputs using MST, but they cannot be split into multiple TB-tunneled-DP-links).
  • Goshen Ridge seems to have no dedicated DP output pins, but it can still internally multiplex its two DP interfaces to all three of its downstream TB ports for DP altmode. Docks that have a dedicated DP connector usually sacrifice one of these TB ports (leaving only two downstream TB ports) which is then tied to the DP connector (and probably permanently configured in DP "altmode" in firmware).
  • DP links can be fully transparent (a single DP link between GPU and display), or the tunnel can be a "DP LTTPR" (like an active cable) resulting in two DP links (GPU to host TB controller, dock TB controller to display). [source, section 2.2.10.2]
  • Note that the full bandwidth on the newest controllers (2x4xHBR3 == 51.84Gbps after encoding) cannot be sent over a single TB3/TB4 port (40Gbps after encoding). But when using two TB ports on the same controller, both can transmit a full 4xHBR3 signal, allowing to use the full DP bandwidth of the controller.

  • Bandwidth allocation for DP links happens based on the maximum bandwidth for the negotiated link rate (e.g. HBR3) and seems to happen on a first-come first-served basis. For example, if the first display negotiates 4xHBR3, this takes up 25.92Gbs (after encoding) of bandwidth, leaving only 2xHBR3 or 4xHBR1 for a second display connected.

    This means that the order of connecting displays can be relevant to the supported resolutions on each (on multi-output hubs with displays already connected, it seems the hub typically initializes displays in a fixed order).

    If the actual bandwidth for the resolution used is less than the allocated bandwidth, the extra bandwidth is not lost, but can still be used for other traffic (like bulk USB traffic, which does not need allocated / reserved bandwidth). [source]

  • In some cases, it seems bandwidth can still be over-allocated if the OS knows that the full bandwidth will not be used (e.g. allocate 2x4xHDR3, knowing that with the resolutions in use, the total bandwidth will fit in the available 40Gbps). [source]
  • MST hubs often seem to allocate maximum supported bandwidth whenever the first display is connected (to prevent the need for renegotiation when another display is added), which might significantly over-allocate. [source]
• For USB3:
  • When using USB4/TB4, the USB3 controller is in the host, with hubs in downstream devices. This means the total USB3 bandwidth is 5, 10 or 20Gbps, limited by the hosts's USB3 controller (often integrated into the TB/USB4 controller), as well as any hubs along the way. This bandwidth is probably also shared between all (both) ports connected to the same TB/USB4 controller.
  • USB3 tunnels are single-hop, so on a USB4/TB4 dock, the TB4/USB4 controller has an upstream USB interface (tunneled to the upstream host or hub) and one downstream interface for each downstream TB4 port (tunneled to the downstream devices or hubs) and an (integrated) USB3 hub that connects all these USB3 interfaces.
  • When using TB3, the USB3 controller (and usually also one or more hubs) is in the (each) dock, connected to the host through tunneled PCIe. This means that the USB3 bandwidth is limited by the controllers in the docks (each controller having its own chunk of 5/10/20Gbps). When there are multiple USB3 controllers involved (inside the same dock, in multiple daisy-chained docks, or connected to multiple ports using the same host TB controller), the total bandwidth can be higher but is still limited by the available PCIe bandwidth.

Bandwidths & Encoding

[source] and [source] and [source]

As mentioned, different protocols use different bitrates and different encodings. Here's an overview of these, with the effective transfer rate (additional protocol overhead like packet headers, error correction, etc. still needs to be accounted for, so i.e. effective transfer rate for a mass-storage device will be lower still). Again, these are single-line bandwidths, so total bandwidth is x4 (unidirectional) for DP and x2 (full-duplex) for TB/USB3.2/USB4.

• PCIe 3.0 8 Gbps using 128b/130b = 7.877 Gbps
• PCIe 6.0 64 Gbps using 242b/256b = 60.5 Gbps
• DP 1.0/1.1 RBR 1.62Gbps using 8b/10b = 1.296Gbps
• DP 1.0/1.1 HBR 2.70Gbps using 8b/10b = 2.16Gbps
• DP 1.2 HBR2 5.40Gbps using 8b/10b = 4.32Gbps
• DP 1.3/1.4 HBR3 8.10Gbps using 8b/10b = 6.48Gbps
• DP 2.0 UHBR10 10Gbps using 128b/132b = 9.697Gbps
• DP 2.0 UHBR13.5 13.5Gbps using 128/132b = 13.091Gbps
• DP 2.0 UHBR20 20Gbps using 128/132b = 19.394Gbps
• USB 3.x 5 Gbps (gen 1) using 8b/10b = 4 Gbps
• USB 3.x 10 Gbps (gen 2) using 128b/132b = 9.697 Gbps
• USB4 10 Gbps (gen 2) uses 64b/66b = 9.697 Gbps
• USB4 20 Gbps (gen 3) uses 128b/132b = 19.39 Gbps
• Thunderbolt 3 10.3125Gbps uses 64b/66b = 10Gbps
• Thunderbolt 3 20.625Gbps uses 64b/66b = 20Gbps

Note that for TB3, the rounded 10/20Gbps speeds are after encoding, so the raw bitrate is actually slightly higher. Thunderbolt 4 is not listed as (AFAICT) this just uses USB4 transfer modes.

Displaylink

[source]

• Displaylink is a technology (made by the company with the same name) that allows sending display data over USB2/3.
• Some DisplayLink chips also support ethernet and analog audio, presumably by just presenting these as parts of a composite device.
• Displaylink is used in docks as well as inside monitors directly, to allow sending video without needing any specific hardware support other than USB2/3. A driver for these devices is needed in the OS though.
• It seems at least some DisplayLink devices are supported in Linux, though mostly using a closed-source binary blob. There is also basic support in open-source drivers, but these are apparently reverse-engineered, so I suspect DisplayLink is not forthcoming with documentation or code themselves. [source]
• DisplayLink performance is limited, mainly because even though the laptop GPU can apparently be used for some acceleration, actually transferring the resulting display data is largely a software/driver affair, involving compression and wrapping data in USB packets. Performance can also be affected by other activity on the bus. DisplayLink seems to be mostly intended for office work, less for video and gaming (unlike other solutions, like DP alt mode or tunneling DP over Thunderbolt or USB4, where the GPU takes care of generating the DisplayPort data stream, which is delivered directly to the USB/Thunderbolt controller without involving software, and which allocates dedicated bus bandwidth for that stream). [source]

Docks / hubs

• Docks come roughly in two flavors:
  • "USB-C docks", without thunderbolt, that connect to the host using USB2, USB3 and/or DP-altmode. These are typically cheaper, but also run at lower bandwidth.
  • "Thunderbolt 3/4" docks, which tunnel everything (including USB3) over Thunderbolt/USB4 (for TB3 even USB2.0 is tunneled, for TB4 it runs over its own wires). These typically have higher upstream bandwidth and more features.
  • USB4 (non-Thunderbolt) docks could technically exist too, but since TB4 is really just USB4 with most optional features made mandatory (and some additional certification), and most of these optional features are already mandatory for USB4 hubs (including even TB3 compatibility), it seems likely that such hubs will just be made TB4-compatible anyway.
• Thunderbolt docks often also support a fallback mode where they work like a USB-C dock using USB2/3 and/or DP-alt-mode to connect to the host, at reduced bandwidth and features (i.e. PCIe connected devices are not supported).
• USB-A ports on docks are fairly uncomplicated and will just support USB devices (using either USB2 or USB3), though note the caveat about single-TT vs multi-TT above.
• Downstream USB-C ports are more tricky: I expect these will always support USB2 and/or 3, but it seems that often only a selection also supports TB3, USB4/TB4, or DP alt mode.
• I suspect that if a dock is TB4-certified, it must probably support all TB4 features on all of its downstream USB4 ports too (which are probably going to be labeled "TB4" ports anyway). USB4 downstream ports are at least required to support DP alt mode.
• Video on docks is complex, see below.
• Other things like ethernet, SD-card readers and audio are typically just implemented using USB devices inside the dock, connected to an integrated USB2/3 hub.
• Some docks have their ethernet (especially 2.5G/10G) connected through PCIe instead of USB. These often allow better performance (because these chips often offload more processing, and this moves bandwidth usage from the USB3 tunnel to the PCIe tunnel, which can be helpful depending on what else is connected), but does require specific drivers for the ethernet chip used (often shipped with OS's, though). See the source for a list of docks and the ethernet chips they use. [source]

Displays on docks / hubs

• Here the situation becomes more complicated. There seem to be at least 6 ways that display data can be transferred over USB-C to a dock: DP alt mode 1.0, DP alt mode 2.0, tunneled through USB4, tunneled through TB3, using a PCIe-connected eGPU (where PCIe can run over TB3 or USB4), or using DisplayLink or a similar USB2/3 connected device.
• AFAICS, DisplayLink has significant downsides (so I would avoid it), a PCIe-connected eGPU is quite expensive (so only used when you really need that extra GPU power), and the other options (altmode and tunneling DisplayPort) are largely equivalent (except for available bandwidth and thus resolution).
• It seems DP-over-USB4 and DP-over-Thunderbolt (also TB3) can always be sent through multiple hops (i.e. when using multiple hubs or docks, or a single hub or dock with a thunderbolt or USB4 capable display connected using USB-C to the dock), provided the hubs or docks have downstream ports supporting Thunderbolt or USB4.
• For DP altmode, multiple hops can be supported in theory, but this depends on how the dock is constructed. I'm not sure if docks actually implement this (and if they do, if they properly document how it works, on which downstream ports this works, etc.).
• HDMI (or DP++) outputs on docks typically just take a DP signal and add an active DP-to-HDMI converter. These converters have been seen to cause problems with some displays, so it might be better to avoid HDMI and use DP exclusively. [source]
• How video is routed inside docks can apparently also be complex. For example the HP Thunderbolt dock G2 has a TB3 input, which contains two tunneled DP signals, one of which is split using MST into three outputs, two available on DP++ connectors and one available either as DP alt mode on a downstream USB-C connector, or converted into a VGA connector. The second DP signal is forwarded entirely to the downstream TB3 port. [source]
• Since MacOS does not support MST, any docks that document support for > 2 display outputs on MacOS likely use DisplayLink, while docks that document support for > 2 display outputs on Windows/PC, but only 2 on MacOS likely use MST.
• I also recall reading something (on this forum?) about a dock that uses DisplayLink for the first two monitor outputs, and DP alt mode only when three outputs are used, but I cannot seem to find the posting again (also, it seems like the decision of how to send display data to the dock is made by the OS, the dock can only decide or limit how it outputs the signals sent, I guess?).
• Displays with a USB-C connector are usually internally similar to a USB-C or Thunderbolt dock supporting video input through DP-altmode and/or tunneling over Thunderbolt combined with USB over the remaining wires or also tunneled. Displays with integrated USB ports can sometimes be configured in 2-line USB3.0 + 2-line DP-alt-mode, or USB2.0-only mode, leaving all 4 lines available for a full-bandwidth 4-lane DP-alt-mode connection.

Framework laptop

• The framework laptop supports USB4, and is intended to be TB4 certified, so I'm assuming that it already has the right hardware and software to support the entire TB4 featureset (though there might still be problems in either hardware or software, of course).
• The Framework laptop GPU supports driving 4 simultaneous display outputs, including the builtin display (which can be disabled by closing the lid to allow driving 4 external monitors). [source]
• All four USB-C ports can be used to output video, and multiple displays can be driven through one USB-C port with a DisplayPort MST hub in the dock or by separate DP-over-TB/USB4 tunnels. [source]
• The USB-C ports are divided into two pairs, each pair sharing a single USB4 router / TB controller, and with two DP 4xHBR3 interfaces on each controller. This means at most two DP streams divided between the two ports in the pair (which includes both TB/USB4 tunnels and DP-alt-mode). [source, section 10, Framework uses UP3]
• Using MST, multiple displays can be driven from a single DP stream, so this probably allows driving more than two displays from a single port (or pair of ports), but probably still limited to four displays in total (it is not entirely clear where in the Tiger Lake chip the MST-encoding happens).
• The DisplayPort expansion card (and also the Tiger Lake TB controller) supports DisplayPort 1.4 (so not 2.0), I assume it is mostly a passive unit with DP altmode 1.0 then. [source]
• Looking at the maximum uncompressed resolution supported by the framework DP expansion card (5120x3200 60Hz), according to the wikipedia resolution list that needs 22.18Gbps total bandwidth, so HBR3 (4x8.1Gbps-per-line), so it seems the expansion card supports all DP 1.4 bitrates (HBR3 is the highest available).
• The expansion card also supports "8192x4320 60Hz uncompressed dual-port", which needs 49.65Gbps total bandwidth, which is indeed too much for HBR3, so I guess this means you'd need two DP expansion cards and two DP cables to a single monitor. [source] and [source]

That's it for now. Kudos if you made it this far! As always, feel free to leave a comment if this has been helpful to you, or if you have any suggestions or questions!

Update 2022-03-29: Added "Understanding tunneling and bandwidth limitations" section and some other small updates.

Update 2022-06-10: Mention boltctl as a tool for getting device info.

For a while, I've been considering replacing Grubby, my trusty workhorse laptop, a Thinkpad X201 that I've been using for the past 11 years. These thinkpads are known to last, and indeed mine still worked nicely, but over the years lost Bluetooth functionality, speaker output, one of its USB ports (I literally lost part of the connector), some small bits of the casing (dropped something heavy on it), the fan sometimes made a grinding noise, and it was getting a little slow at times (but still fine for work). I had been postponing getting a replacement, though, since having to figure out what to get, comparing models, reading reviews is always a hassle (especially for me...).

Then, when I first saw the Framework laptop last year, I was immediately sold. It's a laptop that aims to be modular, in the sense that it can be easily repaired and upgraded. To be honest, this did not seem all that special to me at first, but apparently in the 11 years since I last bought a laptop, manufacturers have been more using glue rather than screws, and solder rather than sockets, which is a trend that Framework hopes to turn.

In addition to the modularity, I like the fact they make repairability and upgradability an explicit goal, in attempt to make the electronics ecosystem more sustainable (they remind me of Fairphone in that sense). On top of that, it seems that this is also a really well made laptop, with a lot of attention to details, explicit support for Linux, open-source where possible (e.g. code for the embedded controller is open, ), flexible expansion ports using replacable modules, encouraging third parties to build and market their own expansion cards and addons (with open-source reference designs available), a mainboard that can be used standalone too (makes for a nice SBC after a mainboard upgrade), decent keyboard, etc.

The only things that I'm less enthusiastic about are the reflective screen (I had that on my previous laptop and I remember liking the switch to a matte screen, but I guess I'll get used to that), having just four expansion ports (the only fixed port is an audio jack, everything else - USB, displays, card reader - has to go through expansion modules, so we'll see if I can get by with four ports) and the lack of an ethernet port (apparently there is an ethernet expansion module in the works, but I'll probably have to get a USB-to-ethernet module in the meanwhile).

Unfortunately, when I found the Framework laptop a few months ago, they were not actually being sold yet, though they expected to open up pre-orders in December. I really hoped Grubby would last long enough so I could get a Framework laptop. Then pre-orders opened only for US and Canada, with shipping to the EU announced for Q1 this year. Then they opened up orders for Germany, France and the UK, and I still had to wait...

So when they opened up pre-orders in the Netherlands last month, I immediately placed my order. They are using a batched shipping system and my batch is supposed to ship "in March" (part of the batch has already been shipped), so I'm hoping to get the new laptop somewhere it the coming weeks.

I suspect that Grubby took notice, because last friday, with a small sputter, he powered off unexpectedly and has refused to power back on. I've tried some CPR, but no luck so far, so I'm afraid it's the end for Grubby. I'm happy that I already got my Framework order in, since now I just borrowed another laptop as a temporary solution rather than having to panic and buy something else instead.

So, I'm eager for my Framework laptop to be delivered. Now, I just need to pick a new name, and figure out which Thunderbolt dock I want... (I had an old-skool docking station for my Thinkpad, which worked great, but with USB-C and Thunderbolt's single cable for power, display, usb and ethernet, there is now a lot more choice in docks, but more on that in my next post...).

Recently, I've been working with STM32 chips for a few different projects and customers. These chips are quite flexible in their pin assignments, usually most peripherals (i.e. an SPI or UART block) can be mapped onto two or often even more pins. This gives great flexibility (both during board design for single-purpose boards and later for a more general purpose board), but also makes it harder to decide and document the pinout of a design.

ST offers STM32CubeMX, a software tool that helps designing around an STM32 MCU, including deciding on pinouts, and generating relevant code for the system as well. It is probably a powerful tool, but it is a bit heavy to install and AFAICS does not really support general purpose boards (where you would choose between different supported pinouts at runtime or compiletime) well.

So in the past, I've used a trusted tool to support this process: A spreadsheet that lists all pins and all their supported functions, where you can easily annotate each pin with all the data you want and use colors and formatting to mark functions as needed to create some structure in the complexity.

However, generating such a pinout spreadsheet wasn't particularly easy. The tables from the datasheet cannot be easily copy-pasted (and the datasheet has the alternate and additional functions in two separate tables), and the STM32CubeMX software can only seem to export a pinout table with alternate functions, not additional functions. So we previously ended up using the CubeMX-generated table and then adding the additional functions manually, which is annoying and error-prone.

So I dug around in the CubeMX data files a bit, and found that it has an XML file for each STM32 chip that lists all pins with all their functions (both alternate and additional). So I wrote a quick Python script that parses such an XML file and generates a CSV script. The script just needs Python3 and has no additional dependencies.

To run this script, you will need the XML file for the MCU you are interested in from inside the CubeMX installation. Currently, these only seem to be distributed by ST as part of CubeMX. I did find one third-party github repo with the same data, but that wasn't updated in nearly two years). However, once you generate the pin listing and publish it (e.g. in a spreadsheet), others can of course work with it without needing CubeMX or this script anymore.

For example, you can run this script as follows:

$ ./stm32pinout.py /usr/local/cubemx/db/mcu/STM32F103CBUx.xml
name,pin,type
VBAT,1,Power
PC13-TAMPER-RTC,2,I/O,GPIO,EXTI,EVENTOUT,RTC_OUT,RTC_TAMPER
PC14-OSC32_IN,3,I/O,GPIO,EXTI,EVENTOUT,RCC_OSC32_IN
PC15-OSC32_OUT,4,I/O,GPIO,EXTI,ADC1_EXTI15,ADC2_EXTI15,EVENTOUT,RCC_OSC32_OUT
PD0-OSC_IN,5,I/O,GPIO,EXTI,RCC_OSC_IN
(... more output truncated ...)

The script is not perfect yet (it does not tell you which functions correspond to which AF numbers and the ordering of functions could be improved, see TODO comments in the code), but it gets the basic job done well.

You can find the script in my "scripts" repository on github.

Update: It seems the XML files are now also available separately on github: https://github.com/STMicroelectronics/STM32_open_pin_data, and some of the TODOs in my script might be solvable.

For this blog, I wanted to include some nicely-formatted formulas. An easy way to do so, is to use MathJax, a javascript-based math processor where you can write formulas using (among others) the often-used Tex math syntax.

However, I use Markdown to write my blogposts and including formulas directly in the text can be problematic because Markdown might interpret part of my math expressions as Markdown and transform them before MathJax has had a chance to look at them. In this post, I present a customized MathJax configuration that solves this problem in a reasonable elegant way.

An obvious solution is to put the match expression in Markdown code blocks (or inline code using backticks), but by default MathJax does not process these. MathJax can be reconfigured to also typeset the contents of <code> and/or <pre> elements, but since actual code will likely contain parts that look like math expressions, this will likely cause your code to be messed up.

This problem was described in more detail by Yihui Xie in a blogpost, along with a solution that preprocesses the DOM to look for <code> tags that start and end with an math expression start and end marker, and if so strip away the <code> tag so that MathJax will process the expression later. Additionally, he translates any expression contained in single dollar signs (which is the traditional Tex way to specify inline math) to an expression wrapped in \( and \), which is the only way to specify inline math in MathJax (single dollars are disabled since they would be too likely to cause false positives).

Improved solution

I considered using his solution, but it explicitly excludes code blocks (which are rendered as a <pre> tag containing a <code> tag in Markdown), and I wanted to use code blocks for centered math expressions (since that looks better without the backticks in my Markdown source). Also, I did not really like that the script modifies the DOM and has a bunch of regexes that hardcode what a math formula looks like.

So I made an alternative implementation that configures MathJax to behave as intended. This is done by overriding the normal automatic typesetting in the pageReady function and instead explicitly typesetting all code tags that contain exactly one math expression. Unlike the solution by Yihui Xie, this:

• Lets MathJax decide what is and is not a math expression. This means that it will also work for other MathJax input plugins, or with non-standard tex input configuration.
• Only typesets string-based input types (e.g. TeX but not MathML), since I did not try to figure out how the node-based inputs work.
• Does not typeset anything except for these selected <code> elements (e.g. no formulas in normal text), because the default typesetting is replaced.
• Also typesets formulas in <code> elements inside <pre> elements (but this can be easily changed using the parent tag check from Yihui Xie's code).
• Enables typesetting of single-dollar inline math expressions by changing MathJax config instead of modifying the delimeters in the DOM. This will not produce false positive matches in regular text, since typesetting is only done on selected code tags anyway.
• Runs from the MathJax pageReady event, so the script does not have to be at the end of the HTML page.

You can find the MathJax configuration for this inline at the end of this post. To use it, just put the script tag in your HTML before the MathJax script tag (or see the MathJax docs for other ways).

Examples

To use it, just use the normal tex math syntax (using single or double $ signs) inside a code block (using backticks or an indented block) in any combination. Typically, you would use single $ delimeters together with backticks for inline math. You'll have to make sure that the code block contains exactly a single MathJax expression (and maybe some whitespace), but nothing else. E.g. this Markdown:

Formulas *can* be inline: `$z = x + y$`.

Renders as: Formulas can be inline: $z = x + y$.

The double $$ delimeter produces a centered math expression. This works within backticks (like Yihui shows) but I think it looks better in the Markdown if you use an indented block (which Yihui's code does not support). So for example this Markdown (note the indent):

    $$a^2 + b^2 = c^2$$

Renders as:

$$a^2 + b^2 = c^2$$

Then you can also use more complex, multiline expressions. This indented block of Markdown:

    $$
    \begin{vmatrix}
      a & b\\
      c & d
    \end{vmatrix}
    =ad-bc
    $$

Renders as:

$$
\begin{vmatrix}
  a & b\\
  c & d
\end{vmatrix}
=ad-bc
$$

Note that to get Markdown to display the above example blocks, i.e. code blocks that start and with $$, without having MathJax process them, I used some literal HTML in my Markdown source. For example, in my blog's markdown source, the first block above literall looks like this:

<pre><code><span></span>    $$a^2 + b^2 = c^2$$</code></pre>

Markdown leaves the HTML tags alone, and the empty span ensures that the script below does not process the contents of the code block (since it only processes code blocks where the full contents of the block are valid MathJax code).

The code

So, here is the script that I am now using on this blog:

<script type="text/javascript">
MathJax = {
  options: {
    // Remove <code> tags from the blacklist. Even though we pass an
    // explicit list of elements to process, this blacklist is still
    // applied.
    skipHtmlTags: { '[-]': ['code'] },
  },
  tex: {
    // By default, only \( is enabled for inline math, to prevent false
    // positives. Since we already only process code blocks that contain
    // exactly one math expression and nothing else, it is also fine to
    // use the nicer $...$ construct for inline math.
    inlineMath: { '[+]': [['$', '$']] },
  },
  startup: {
    // This is called on page ready and replaces the default MathJax
    // "typeset entire document" code.
    pageReady: function() {
      var codes = document.getElementsByTagName('code');
      var to_typeset = [];
      for (var i = 0; i < codes.length; i++) {
        var code = codes[i];
        // Only allow code elements that just contain text, no subelements
        if (code.childElementCount === 0) {
          var text = code.textContent.trim();
          inputs = MathJax.startup.getInputJax();
          // For each of the configured input processors, see if the
          // text contains a single math expression that encompasses the
          // entire text. If so, typeset it.
          for (var j = 0; j < inputs.length; j++) {
            // Only use string input processors (e.g. tex, as opposed to
            // node processors e.g. mml that are more tricky to use).
            if (inputs[j].processStrings) {
              matches = inputs[j].findMath([text]);
              if (matches.length == 1 && matches[0].start.n == 0 && matches[0].end.n == text.length) {
                // Trim off any trailing newline, which otherwise stays around, adding empty visual space below
                code.textContent = text;
                to_typeset.push(code);
                code.classList.add("math");
                if (code.parentNode.tagName == "PRE")
                  code.parentNode.classList.add("math");
                break;
              }
            }
          }
        }
      }
      // Code blocks to replace are collected and then typeset in one go, asynchronously in the background
      MathJax.typesetPromise(to_typeset);
    },
  },
};
</script>

Update 2020-08-05: Script updated to run typesetting only once, and use typesetPromise to run it asynchronously, as suggested by Raymond Zhao in the comments below.

Update 2020-08-20: Added some Markdown examples (the same ones Yihui Xie used), as suggested by Troy.

Update 2021-09-03: Clarified how the script decides which code blocks to process and which to leave alone.

Or: Forcing Linux to use the USB HID driver for a non-standards-compliant USB keyboard.

For an interactive art installation by the Spullenmannen, a friend asked me to have a look at an old paint mixing terminal that he wanted to use. The terminal is essentially a small computer, in a nice industrial-looking sealed casing, with a (touch?) screen, keyboard and touchpad. It was by "Lacour" and I think has been used to control paint mixing machines.

They had already gotten Linux running on the system, but could not get the keyboard to work and asked me if I could have a look.

The keyboard did work in the BIOS and grub (which also uses the BIOS), so we know it worked. Also, the BIOS seemed pretty standard, so it was unlikely that it used some very standard protocol or driver and I guessed that this was a matter of telling Linux which driver to use and/or where to find the device.

Inside the machine, it seemed the keyboard and touchpad were separate devices, controlled by some off-the-shelf microcontroller chip (probably with some custom software inside). These devices were connected to the main motherboard using a standard 10-pin expansion header intended for external USB ports, so it seemed likely that these devices were USB ports.

Closer look at the USB devices

And indeed, looking through lsusb output I noticed two unkown devices in the list:

# lsusb
Bus 002 Device 003: ID ffff:0001
Bus 002 Device 002: ID 0000:0003
(...)

These have USB vendor ids of 0x0000 and 0xffff, which I'm pretty sure are not official USB-consortium-assigned identifiers (probably invalid or reserved even), so perhaps that's why Linux is not using these properly?

Running lsusb with the --tree option allows seeing the physical port structure, but also shows which drivers are bound to which interfaces:

# lsusb --tree
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
    |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 2: Dev 3, If 0, Class=Human Interface Device, Driver=, 12M
(...)

This shows that the keyboard (Dev 3) indeed has no driver, but the touchpad (Dev 2) is already bound to usbhid. And indeed, runnig cat /dev/input/mice and then moving over the touchpad shows that some output is being generated, so the touchpad was already working.

Looking at the detailed USB descriptors for these devices, shows that they are both advertised as supporting the HID interface (Human Interface Device), which is the default protocol for keyboards and mice nowadays:

# lsusb -d ffff:0001 -v

Bus 002 Device 003: ID ffff:0001
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          255 Vendor Specific Class
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         8
  idVendor           0xffff 
  idProduct          0x0001 
  bcdDevice            0.01
  iManufacturer           1 Lacour Electronique
  iProduct                2 ColorKeyboard
  iSerial                 3 SE.010.H
(...)
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      1 Keyboard
(...)

# lsusb -d 0000:00003 -v
Bus 002 Device 002: ID 0000:0003
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x0000 
  idProduct          0x0003 
  bcdDevice            0.00
  iManufacturer           1 Lacour Electronique
  iProduct                2 Touchpad
  iSerial                 3 V2.0
(...)
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      1 Boot Interface Subclass
      bInterfaceProtocol      2 Mouse
(...)

So, that should make it easy to get the keyboard working: Just make sure the usbhid driver is bound to it and that driver will be able to figure out what to do based on these descriptors. However, apparently something is preventing this binding from happening by default.

Looking back at the USB descriptors above, one interesting difference is that the keyboard has bDeviceClass set to "Vendor specific", whereas the touchpad has it set to 0, which means "Look at interface descriptors. So that seems the most likely reason why the keyboard is not working, since "Vendor Specific" essentially means that the device might not adhere to any of the standard USB protocols and the kernel will probably not start using this device unless it knows what kind of device it is based on the USB vendor and product id (but since those are invalid, these are unlikely to be listed in the kernel).

Binding to usbhid

So, we need to bind the keyboard to the usbhid driver. I know of two ways to do so, both through sysfs.

You can assign extra USB vid/pid pairs to a driver through the new_id sysfs file. In this case, this did not work somehow:

# echo ffff:0001 > /sys/bus/usb/drivers/usbhid/new_id
bash: echo: write error: Invalid argument

At this point, I should have stopped and looked up the right syntax used for new_id, since this was actually the right approach, but I was using the wrong syntax (see below). Instead, I tried some other stuff first.

The second way to bind a driver is to specify a specific device, identified by its sysfs identifier:

# echo 2-2:1.0 > /sys/bus/usb/drivers/usbhid/bind
bash: echo: write error: No such device

The device identifier used here (2-2:1.0) is directory name below /sys/bus/usb/devices and is, I think, built like <bus>-<port>:1.<interface> (where 1 might the configuration?). You can find this info in the lsusb --tree output:

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
|__ Port 2: Dev 3, If 0, Class=Human Interface Device, Driver=, 12M

I knew that the syntax I used for the device id was correct, since I could use it to unbind and rebind the usbhid module from the touchpad. I suspect that there is some probe mechanism in the usbhid driver that runs after you bind the driver which tests the device to see if it is compatible, and that mechanism rejects it.

How does the kernel handle this?

As I usually do when I cannot get something to work, I dive into the source code. I knew that Linux device/driver association usually works with a driver-specific matching table (that tells the underlying subsystem, such as the usb subsystem in this case, which devices can be handled by a driver) or probe function (which is a bit of driver-specific code that can be called by the kernel to probe whether a device is compatible with a driver). There is also configuration based on Device Tree, but AFAIK this is only used in embedded platforms, not on x86.

Looking at the usbhid_probe() and usb_kbd_probe() functions, I did not see any conditions that would not be fulfilled by this particular USB device.

The match table for usbhid also only matches the interface class and not the device class. The same goes for the module.alias file, which I read might also be involved (though I am not sure how):

# cat /lib/modules/*/modules.alias|grep usbhid
alias usb:v*p*d*dc*dsc*dp*ic03isc*ip*in* usbhid

So, the failing check must be at a lower level, probably in the usb subsystem.

Digging a bit further, I found the usb_match_one_id_intf() function, which is the core of matching USB drivers to USB device interfaces. And indeed, it says:

/* The interface class, subclass, protocol and number should never be
 * checked for a match if the device class is Vendor Specific,
 * unless the match record specifies the Vendor ID. */

So, the entry in the usbhid table is being ignored since it matches only the interface, while the device class is "Vendor Specific". But how to fix this?

A little but upwards in the call stack, is a bit of code that matches a driver to an usb device or interface. This has two sources: The static table from the driver source code, and a dynamic table that can be filled with (hey, we know this part!) the new_id file in sysfs. So that suggests that if we can get an entry into this dynamic table, that matches the vendor id, it should work even with a "Vendor Specific" device class.

Back to new_id

Looking further at how this dynamic table is filled, I found the code that handles writes to new_id, and it parses it input like this:

fields = sscanf(buf, "%x %x %x %x %x", &idVendor, &idProduct, &bInterfaceClass, &refVendor, &refProduct);

In other words, it expects space separated values, rather than just a colon separated vidpid pair. Reading on in the code shows that only the first two (vid/pid) are required, the rest is optional. Trying that actually works right away:

# echo ffff 0001 > /sys/bus/usb/drivers/usbhid/new_id
# dmesg
(...)
[ 5011.088134] input: Lacour Electronique ColorKeyboard as /devices/pci0000:00/0000:00:1d.1/usb2/2-2/2-2:1.0/0003:FFFF:0001.0006/input/input16
[ 5011.150265] hid-generic 0003:FFFF:0001.0006: input,hidraw3: USB HID v1.11 Keyboard [Lacour Electronique ColorKeyboard] on usb-0000:00:1d.1-2/input0

After this, I found I can now use the unbind file to unbind the usbhid driver again, and bind to rebind it. So it seems that using bind indeed still goes through the probe/match code, which previously failed but with the entry in the dynamic table, works.

Making this persistent

So nice that it works, but this dynamic table will be lost on a reboot. How to make it persistent? I can just drop this particular line into the /etc/rc.local startup script, but that does not feel so elegant (it will probably work, since it only needs the usbhid module to be loaded and should work even when the USB device is not known/enumerated yet).

However, as suggested by this post, you can also use udev to run this command at the moment the USB devices is "added" (i.e. enumerated by the kernel). To do so, simply drop a file in /etc/udev/rules.d:

$ cat /etc/udev/rules.d/99-keyboard.rules
# Integrated USB keyboard has invalid USB VIDPID and also has bDeviceClass=255,
# causing the hid driver to ignore it. This writes to sysfs to let the usbhid
# driver match the device on USB VIDPID, which overrides the bDeviceClass ignore.
# See also:
# https://unix.stackexchange.com/a/165845
# https://github.com/torvalds/linux/blob/bf3bd966dfd7d9582f50e9bd08b15922197cd277/drivers/usb/core/driver.c#L647-L656
# https://github.com/torvalds/linux/blob/3039fadf2bfdc104dc963820c305778c7c1a6229/drivers/hid/usbhid/hid-core.c#L1619-L1623
ACTION=="add", ATTRS{idVendor}=="ffff", ATTRS{idProduct}=="0001", RUN+="/bin/sh -c 'echo ffff 0001 > /sys/bus/usb/drivers/usbhid/new_id'"

And with that, the keyboard works automatically at startup. Nice :-)

For a customer, I've been looking at RS-485 and MODBUS, two related protocols for transmitting data over longer distances, and the various Arduino libraries that exist to work with them.

They have been working on a project consisting of multiple Arduino boards that have to talk to each other to synchronize their state. Until now, they have been using I²C, but found that this protocol is quite susceptible to noise when used over longer distances (1-2m here). Combined with some limitations in the AVR hardware and a lack of error handling in the Arduino library that can cause the software to lock up in the face of noise (see also this issue report), makes I²C a bad choice in such environments.

So, I needed something more reliable. This should be a solved problem, right?

RS-485

A commonly used alternative, also in many industrial settings, are RS-485 connections. This is essentially an asynchronous serial connection (e.g. like an UART or RS-232 serial port), except that it uses differential signalling and is a multipoint bus. Differential signalling means two inverted copies of the same signal are sent over two impedance-balanced wires, which allows the receiver to cleverly subtract both signals to cancel out noise (this is also what ethernet and professional audio signal does). Multipoint means that there can be more than two devices on the same pair of wires, provided that they do not transmit at the same time. When combined with shielded and twisted wire, this should produce a very reliable connection over long lengths (up to 1000m should be possible).

However, RS-485 by itself is not everything: It just specifies the physical layer (the electrical connections, or how to send data), but does not specify any format for the data, nor any way to prevent multiple devices from talking at the same time. For this, you need a data link or arbitration protocol running on top of RS-485.

MODBUS

A quick look around shows that MODBUS is very commonly used protocol on top of RS-485 (but also TCP/IP or other links) that handles the data link layer (how to send data and when to send). This part is simple: There is a single master that initiates all communication, and multiple slaves that only reply when asked something. Each slave has an address (that must be configured manually beforehand), the master needs no address.

MODBUS also specifies a simple protocol that can be used to read and write addressed bits ("Coils" and "Inputs") and addressed registers, which would be pretty perfect for the usecase I'm looking at now.

Finding an Arduino library

So, I have some RS-485 transceivers (which translate regular UART to RS-485) and just need some Arduino library to handle the MODBUS protocol for me. A quick Google search shows there are quite a few of them (never a good sign). A closer look shows that none of them are really good...

There are some more detailed notes per library below, but overall I see the following problems:

• Most libraries are very limited in what serial ports they can use. Some are hardcoded to a single serial port, some support running on arbitrary HardwareSerial instances (and sometimes also SoftwareSerial instances, but only two librares actually supports running on arbitrary Stream instances (while this is pretty much the usecase that Stream was introduced for).
• All libraries handle writes and reads to coils and registers automatically by updating the relevant memory locations, which is nice. However, none of them actually support notifying the sketch of such reads and writes (one has a return value that indicates that something was read or written, but no details), which means that the sketch should continuously check all register values and update them library. It also means that the number of registers/coils is limited by the available RAM, you cannot have virtual registers (e.g. writes and reads that are handled by a function rather than a bit of RAM).
• A lot of them are either always blocking in the master, or require manually parsing replies (or both).

Writing an Arduino library?

Ideally, I would like to see a library: - That can be configured using a Stream instance and an (optional) tx enable pin. - Has a separation between the MODBUS application protocol and the RS-485-specific datalink protocol, so it can be extended to other transports (e.g. TCP/IP) as well. - Where the master has both synchronous (blocking) and asynchronous request methods.

The xbee-arduino library, which also does serial request-response handling would probably serve as a good example of how to combine these in a powerful API. - Where the slave can have multiple areas defined (e.g. a block of 16 registers starting at address 0x10). Each area can have some memory allocated that will be read or written directly, or a callback function to do the reading or writing. In both cases, a callback that can be called after something was read or writen (passing the area pointer and address or something) can be configured too. Areas should probably be allowed to overlap, which also allows having a "fallback" (virtual) area that covers all other addresses.

These areas should be modeled as objects that are directly accessible to the sketch, so the sketch can read and write the data without having to do linked-list lookups and without needing to know the area-to-adress mapping. - That supports sending and receiving raw messages as well (to support custom function codes). - That does not do any heap allocation (or at least allows running with static allocations only). This can typically be done using static (global) variables allocated by the sketch that are connected as a linked list in the library.

I suspect that given my requirements, this would mean starting a new library from scratch (using an existing library as a starting point would always mean significant redesigning, which is probably more work than its worth). Maybe some parts (e.g. specific things like packet formatting and parsing) can be reused, though.

Of course, I do not really have time for such an endeavor and the customer for which I started looking at this certainly has no budget in this project for such an investment. This means I will probably end up improvising with the MCCI library, or use some completely different or custom protocol instead of MODBUS (though the Arduino library offerings in this area also seem limited...). Maybe CANBus?

However, if you also find yourself in the same situation, maybe my above suggestions can serve as inspiration (and if you need this library and have some budget to get it written, feel free to contact me).

Existing libraries

So, here's the list of libraries I found.

https://github.com/arduino-libraries/ArduinoModbus

• Official library from Arduino.
• Master and slave.
• Uses the RS485 library to communicate, but does not offer any way to pass a custom RS485 instance, so it is effectively hardcoded to a specific serial port.
• Offers only single value reads and writes.
• Slave stores value internally and reads/writes directly from those, without any callback or way to detect that communication has happened.

https://github.com/4-20ma/ModbusMaster

• Master-only library.
• Latest commit in 2016.
• Supports any serial port through Stream objects.
• Supports idle/pre/post-transmission callbacks (no parameters), used to enable/disable the transceiver.
• Supports single and multiple read/writes.
• Replies are returned in a (somewhat preprocessed) buffer, to be further processed by the caller.

https://github.com/andresarmento/modbus-arduino

• Slave-only library.
• Last commit in 2015.
• Supports single and multiple read/writes.
• Split into generic ModBus library along with extra transport-specific libraries (TCP, serial, etc.).
• Supports passing HardwareSerial pointers and (with a macro modification to the library) SoftwareSerial pointers (but uses a Stream pointer internally already).
• Slave stores values in a linked list (heap-allocated), values are written through write methods (linked list elements are not exposed directly, which is a pity).
• Slave reads/writes directly from internal linked list, without any callback or way to detect that communication has happened.
• https://github.com/vermut/arduino-ModbusSerial is a fork that has some Due-specific fixes.

https://github.com/lucasso/ModbusRTUSlaveArduino

• Fork of https://github.com/Geabong/ModbusRTUSlaveArduino (6 additional commits).
• Slave-only-library.
• Last commit in 2018.
• Supports passing HardwareSerial pointers.
• Slave stores values external to the library in user-allocate arrays. These arrays are passed to the library as "areas" with arbitrary starting addresses, which are kept in the library in a linked list (heap-allocated).
• Slave reads/writes directly from internal linked list, without any callback or way to detect that communication has happened.

https://github.com/mcci-catena/Modbus-for-Arduino

• Master and slave.
• Last commit in 2019.
• Fork of old (2016) version of https://github.com/smarmengol/Modbus-Master-Slave-for-Arduino with significant additional development.
• Supports passing arbitrary serial (or similar) objects using a templated class.
• Slave stores values external to the library in a single array (so all requests index the same data, either word or bit-indexed), which is passed to the poll() function.
• On Master, sketch must create requests somewhat manually (into a struct, which is encoded to a byte buffer automatically), and replies returns raw data buffer on requests. Requests and replies are non-blocking, so polling for replies is somewhat manual.

https://github.com/angeloc/simplemodbusng

• Master and slave.
• Last commit in 2019.
• Hardcodes Serial object, supports SoftwareSerial in slave through duplicated library.
• Supports single and multiple read/writes of holding registers only (no coils or input registers).
• Slave stores values external to the library in a single array, which is passed to the update function.

https://github.com/smarmengol/Modbus-Master-Slave-for-Arduino

• Master and slave.
• Last commit in 2020.
• Supports any serial port through Stream objects.
• Slave stores values external to the library in a single array (so all requests index the same data, either word or bit-indexed), which is passed to the poll() function.
• On Master, sketch must create requests somewhat manually (into a struct, which is encoded to a byte buffer automatically), and replies returns raw data buffer on requests. Requests and replies are non-blocking, so polling for replies is somewhat manual.

https://github.com/asukiaaa/arduino-rs485

• Unlike what the name suggests, this actually implements ModBus
• Master and slave.
• Started very recently (October 2020), so by the time you read this, maybe things have already improved.
• Very simple library, just handles modbus framing, the contents of the modbus packets must be generated and parsed manually.
• Slave only works if you know the type and length of queries that will be received.
• Supports working on HardwareSerial objects.

https://gitlab.com/creator-makerspace/rs485-nodeproto

• This is not a MODBUS library, but a very thin layer on top of RS485 that does collision avoidance and detection that can be used to implement a multi-master system.
• Last commit in 2016, repository archived.
• This one is notable because it gets the Stream-based configuration right and seems well-written. It does not implement MODBUS or a similarly high-level protocol, though.

https://github.com/MichaelJonker/HardwareSerialRS485

• Also not MODBUS, but also a collision avoidance/detection scheme on top of RS485 for multi-master bus.
• Last commit in 2015.
• Replaces HardwareSerial rather than working on top, requiring a customized boards.txt.

https://www.airspayce.com/mikem/arduino/RadioHead/

• This is not a MODBUS library, but a communication library for data communication over radio. It also supports serial connections (and is thus an easy way to get framing, checksumming, retransmissions and routing over serial).
• Seems to only support point-to-point connections, lacking an internal way to disable the RS485 driver when not transmitting (but maybe it can be hacked internally).

Update 2020-06-26: Added smarmengol/Modbus-Master-Slave-for-Arduino to the list
Update 2020-10-07: Added asukiaaa/arduino-rs485 to the list