<PSA> Twitter is going great! Here's my new profile on Mastodon: @gynvael@infosec.exchange. Now back to our show! </PSA>

I woke up today, turned on my PC, went to the kitchen to put on water for a cup of tea and came back to a set of black screens. This is pretty weird as Haven – my main workstation (named after a city from one of my favorite book series) – while being a rather old computer starts pretty fast. But I patently waited and in a few seconds the typical BIOS/UEFI screen appeared followed by the OS booting... with some errors:

Photo of part of an LCD showing two lines with the same error message: usb 3-11: device descriptor read/64, error -110
Greeted by errors

This wasn't the first time it happened – rather than that it had a tendency to come and go. And to add a few minutes to the boot time. Being me I decided it's a great opportunity to wasted an odd hour to trace (and hopefully fix) the issue of the few minute longer boot. And you're invited to join me on this debugging adventure!

After my PC finally booted I was able to check dmesg:

$ sudo dmesg | grep 3-11 [ 3.322054] usb 3-11: new full-speed USB device number 6 using xhci_hcd [ 18.918083] usb 3-11: device descriptor read/64, error -110 [ 34.534084] usb 3-11: device descriptor read/64, error -110 [ 34.770056] usb 3-11: new full-speed USB device number 7 using xhci_hcd [ 50.150082] usb 3-11: device descriptor read/64, error -110 [ 65.766082] usb 3-11: device descriptor read/64, error -110 [ 66.526056] usb 3-11: new full-speed USB device number 8 using xhci_hcd [ 77.366057] usb 3-11: device not accepting address 8, error -62 [ 77.494056] usb 3-11: new full-speed USB device number 9 using xhci_hcd [ 88.118054] usb 3-11: device not accepting address 9, error -62

Seems a handful of devices numbered 6, 7, 8 and 9 connected to usb 3-11 port decided it's time to stop working. So, 3-11, which port is that? Luckily I have all the USB ports mapped out in my PC documentation (surprisingly yes, I really do keep a detailed set of notes – counting 29 pages at this moment – on Haven's configuration):

Screenshot from my notes. The top part is a photo of the back plate of my PC's motherboard with drawn in lables for rows and columns of USB ports. This is followed with a hort list of bus numbers and xhci_hcd identifiers. And this in turn is followed by 3 tables denoting the usb id of every port, one table for each of USB 1.x, USB 2.0 and USB 3.x. There is no usb 3-11 device in any of the tables.
Notes on USB ports

Well. Uhm. There's no usb 3-11 in any of the tables. Which means it's most likely not a device connected to any of these ports. This left me with internal USB headers, a random USB-C connector on the opposite side of the motherboard which I always forget exists, and potentially any USB device soldered directly onto the motherboard.

Only one device is connected to the internal USB headers – a NZXT Kraken Z display on the water pump mounted the CPU (shows CPU temperature – useful). And lsusb showed it as Bus 003 Device 004 (i.e. usb 3-4) so it's not what I was looking for.

That random USB-C connector doesn't have anything connected either, so that left some kind of on-motherboard device. But which one? Given that it's broken and not visible in lsusb's output, checking this might be tricky. Thankfully I remember that linux-hardware.org exists – it's a huge community-driven database of all kinds of hardware related reports gathered from Linux computers – an ideal place to throw in my motherboard type (as shown by dmidecode – "X99A XPOWER GAMING TITANIUM (MS-7A21)") and check the USB related reports. This turned out to be a perfect hit:

[ 2.176749] usb 3-11: new full-speed USB device number 4 using xhci_hcd [ 2.325805] usb 3-11: New USB device found, idVendor=<8087, idProduct=0a2b, bcdDevice= 0.01 [ 2.325806] usb 3-11: New USB device strings: Mfr=0, Product=0, SerialNumber=...

So it's 8087:0a2b, which apparently is an Intel Bluetooth wireless interface – here's its page on Intel's website. The official name is "Intel® Dual Band Wireless-AC 8260" and it's a WiFi+Bluetooth adapter that works both on PCI Express (WiFi) and on USB (Bluetooth):

Screenshot from the official page for this device featuring the Package Specification with three entries. Board Form Factor is apparently M.2 2230 or 1216. Package size is 22mm by 30mm or 12mm by 16mm. And – most importantly – the system interface type is PCI express for WiFi and USB for Bluetooth.
Source: Intel's AC 8260 website

Knowing what it is I decided I don't really need it. Given this, the fastest solution was of course to get rid of it.

Being hopeful I've opened the motherboard manual and looked through the BIOS/UEFI options to disable the AC 8260. No luck. There was always a chance that the manual is outdated, so I rebooted into BIOS and found... nothing. There goes the easy solution.

At this point I devised three plans:

  1. Do nothing. I mean, it adds a few minutes to the boot time, so whatever. And while I maybe could do something with Linux, it's unlikely I could do anything with the POST phase.
  2. Disable it on Linux. There might be some kernel option somewhere to do that. Or maybe the slowdown is happening somewhere late in the boot process, so I could just set some udevd rules or something.
  3. Find the chip on the motherboard and desolder it because YOLO. I admit to be a bit afraid to try this tactic as I actually need Haven to work properly.

Eventually I decided on the first option of doing nothing. However curiosity got the better of me and I decided to at least find the chip on the motherboard. Admittedly I also put some hope into the "Board Form Factor: M.2" remark spotted in the official specification. Of course there is no reason why a motherboard manufacturer wouldn't just get the parts themselves to solder directly onto the motherboard, but maybe, just maybe, they actually went for a detachable M.2 board. While I had hope, it wasn't that much hope – if the motherboard would have a detachable M.2 board somewhere on it, I would have noticed it in the past. On the flip side that motherboard sure has a lot of fancy looking plastic covers and radiators...

A picture of a silver and black motherboard, with several plastic covers that might have easilly hiden a small M.2 board.
Source: MSI's official X99A XPOWER GAMING TITANIUM website

Since the motherboard I had was in use, not to mention populated with a lot of visual obstructions (also known as cables and PCI Express cards), I've started with Google images search looking for high-resolution photos of the motherboard with no cover. After a few minutes I found an in-depth motherboard review by Brendan van Varik for Play3r and this gem of an image:

A photo of a small black box vertically attached to the back of the motherboard labeled MSI AC Intel WiDi Wi-Fi 802.11 a/b/g/n/a... Bluetooth 4.0.
Source: Brendan van Varik, Pl3yer, MSI X99A XPOWER Gaming Titanium Motherboard Review

Given that this is under the large back plastic cover I never noticed it being something else than just two antenna connectors. I proceeded to check how is this box soldered to the motherboard, but found... a screw.

A photo from underneeth of the motherboard shoting a screw and no through-the-hole soldering joints of any kind. Part of the backplate with anntena connectors is also visible right above where the screw is.
The screw

I decided to google for more images of the box itself to see if someone described a way to detach it and I've actually found the opposite. The "MSI Z97 GAMING 9 AC (Intel LGA 1150) Review" by cadaveca from TechPowerUp mentioned a different motherboard where this box would come detached by default and the user could optionally attach it to the motherboard. Here's what cadaveca had to say about it:

MSI's wireless AC card includes Bluetooth and supports Intel's WIDI functionality, so you can broadcast video wirelessly to supporting devices. Both the LAN and Bluetooth ports get their own antenna (which is why there are two in the box), and the card itself mates to the board via a pin header on the card itself. IF you plan on using it, you'll want to take a bit of extra caution while installing MSI's wireless card since its delicate pins can be bent out of place easily. I didn't run into such a problem, however.

There were also a couple of photos of the box from different sides showing one screw hole and a pin header (keyed – one pin was missing).

As soon as I've learned this I turned off the PC, took it out of the server rack, unscrewed several screws and successfully detached the module:

A photo of a detached WiFi/Bluetooth module lying on the motherboard with a small keyed 20-pin header in the background.
Success!

Next the PC went back into the rack, I've hooked up all the cables and turned it on. It booted nice and fast, no issues whatsoever. Success!

After a short chat with D0han on my Discord about the weird PCI Express header I decided to take the box apart to verify my hypothesis of it being just an adapter for M.2 plus the Intel AC 8260 card itself. Turns out I was right (mostly):

A photo the WiFi/Bluetooth module without the front cover. Inside there's indeed just an adapter from the weird PCI express header into a normal M.2 with Intel's WiFi/Bluetooth card connected.
AC 8260 without the cover

The whole WiFi/Bluetooth module disassembled. On the top of the photo there are two parts of the metal cover. On the bottom left side of the image there's the adapter – it has only the connectors and two resistors on it. The M.2 connector is missing – per description on the silk mask – pins from 24 to 31. On the right bottom side of the image there's a set of screws and the WiFi/Bluetooth card itself with two anntena connectors attached. It has an M.2 connector keyed E and A.
AC 8260 disassembled

I've mentioned above I was mostly – but not fully – right. I actually expected an M.2 connector with the M key. Long story short is that the M.2 connector can be used for multiple protocols and pin configurations. Which exact set of protocols is it is signaled by missing pins and a "blocker" in their place (or notches in case of device connectors). So M.2 key M would denote PCI Express x4 (see the table from Wikipedia below). Which is what I expected – a PCI Express device that has a chip on it that's both a WiFi module and pretends to be a USB controller for the purpose of connecting the Bluetooth module. I would know this to be incorrect had I paid any attention to how the USB device identifiers are assigned. USB 3-11 means 11th device on the 3rd bus. If that device would be a USB controller, the Bluetooth module itself wouldn't be 3-11 – since 3 denotes a different USB controller – it would be 9-1 or something. Everything made more sense once I've noticed it's actually the E key, which can both support PCI Express x1 and USB 2.0 connections at the same time. Given this, the card is actually two separate devices (even if potentially on one chip) – one connected to the PCI Express x1 pins and the other to the USB 2.0 pins.

A screenshot of a table of M.2 keys. The A key is missing pins 8 to 15, and supports 2 times PCI Express x1, USB 2.0, I2C and standard 4-lane DisplayPort. The B key is missing pins 12 to 19 and supports PCI Express x2, SATA, USB 2.0 and 3.0, as well as audio, and some other stuff like. Key E is pretty similar to key A in the sense that it also does two PCI Express x1 and USB 2.0, but doesn't do Display Port. Instead there are some other protocols there like SDIO, UART or CNVi. The E key is missing pins 24 to 31. The F key, missing pins 28 to 35, is apperently something called the Future Memory Interface. The G key – that's missing pins 39 to 46 – is reserved for custom use. And there's also the M key used for PCI Express x4 – commonly NVMe and SATA. Other keys are reserved for future use.
Source: M.2 article on Wikipedia (EN)

The only question that remains is whether one can attach a GPU through an adapter to this slot (will be pretty slow, especially that it's PCI Express x1 version 2.0 – since it's attached to the chipset and X99 does only PCIe 2.0). Not that there is any reason to do it. Anyway, I might or might not have ordered a proper cursed adapter from a certain Chinese electronics site, so guess we'll find out.

Comments:

2022-11-12 15:35:44 = Maciek
{
On PC motherboards this is a popular placement of WiFi + Bluetooth module, and actually it was the first thing that came to my mind when I started reading ;)

As far as I know, WiFi and BT devices need to be certified e.g. by FCC, so due to regulatory burden motherboard OEMs prefer to get approved modules instead of bare chips to solder. M.2 2230 is popular and replaceable which is a good thing as WiFi standards evolve quite fast. There is also smaller M.2 LGA 1216 form factor to be soldered e.g. in laptops, but still it has its own PCB and is sold as a complete module.
Intel uses CNVi in which many parts of functional blocks were moved into their PCH (or SoC in general), but even then, the non-standalone RF modules use M.2 2230 or 1216.
}

Add a comment:

Nick:
URL (optional):
Math captcha: 2 ∗ 10 + 3 =