<PSA> Twitter is going great! Here's my new profile on Mastodon: @gynvael@infosec.exchange. Now back to our show! </PSA>
I woke up today, turned on my PC, went to the kitchen to put on water for a cup of tea and came back to a set of black screens. This is pretty weird as Haven – my main workstation (named after a city from one of my favorite book series) – while being a rather old computer starts pretty fast. But I patently waited and in a few seconds the typical BIOS/UEFI screen appeared followed by the OS booting... with some errors:
Greeted by errors
This wasn't the first time it happened – rather than that it had a tendency to come and go. And to add a few minutes to the boot time. Being me I decided it's a great opportunity to wasted an odd hour to trace (and hopefully fix) the issue of the few minute longer boot. And you're invited to join me on this debugging adventure!
After my PC finally booted I was able to check dmesg:
$ sudo dmesg | grep 3-11
[ 3.322054] usb 3-11: new full-speed USB device number 6 using xhci_hcd
[ 18.918083] usb 3-11: device descriptor read/64, error -110
[ 34.534084] usb 3-11: device descriptor read/64, error -110
[ 34.770056] usb 3-11: new full-speed USB device number 7 using xhci_hcd
[ 50.150082] usb 3-11: device descriptor read/64, error -110
[ 65.766082] usb 3-11: device descriptor read/64, error -110
[ 66.526056] usb 3-11: new full-speed USB device number 8 using xhci_hcd
[ 77.366057] usb 3-11: device not accepting address 8, error -62
[ 77.494056] usb 3-11: new full-speed USB device number 9 using xhci_hcd
[ 88.118054] usb 3-11: device not accepting address 9, error -62
Seems a handful of devices numbered 6, 7, 8 and 9 connected to usb 3-11 port decided it's time to stop working. So, 3-11, which port is that? Luckily I have all the USB ports mapped out in my PC documentation (surprisingly yes, I really do keep a detailed set of notes – counting 29 pages at this moment – on Haven's configuration):
Notes on USB ports
Well. Uhm. There's no usb 3-11 in any of the tables. Which means it's most likely not a device connected to any of these ports. This left me with internal USB headers, a random USB-C connector on the opposite side of the motherboard which I always forget exists, and potentially any USB device soldered directly onto the motherboard.
Only one device is connected to the internal USB headers – a NZXT Kraken Z display on the water pump mounted the CPU (shows CPU temperature – useful). And lsusb showed it as Bus 003 Device 004 (i.e. usb 3-4) so it's not what I was looking for.
That random USB-C connector doesn't have anything connected either, so that left some kind of on-motherboard device. But which one? Given that it's broken and not visible in lsusb's output, checking this might be tricky. Thankfully I remember that linux-hardware.org exists – it's a huge community-driven database of all kinds of hardware related reports gathered from Linux computers – an ideal place to throw in my motherboard type (as shown by dmidecode – "X99A XPOWER GAMING TITANIUM (MS-7A21)") and check the USB related reports. This turned out to be a perfect hit:
[ 2.176749] usb 3-11: new full-speed USB device number 4 using xhci_hcd
[ 2.325805] usb 3-11: New USB device found, idVendor=<8087, idProduct=0a2b, bcdDevice= 0.01
[ 2.325806] usb 3-11: New USB device strings: Mfr=0, Product=0, SerialNumber=...
So it's 8087:0a2b, which apparently is an Intel Bluetooth wireless interface – here's its page on Intel's website. The official name is "Intel® Dual Band Wireless-AC 8260" and it's a WiFi+Bluetooth adapter that works both on PCI Express (WiFi) and on USB (Bluetooth):
Source: Intel's AC 8260 website
Knowing what it is I decided I don't really need it. Given this, the fastest solution was of course to get rid of it.
Being hopeful I've opened the motherboard manual and looked through the BIOS/UEFI options to disable the AC 8260. No luck. There was always a chance that the manual is outdated, so I rebooted into BIOS and found... nothing. There goes the easy solution.
At this point I devised three plans:
- Do nothing. I mean, it adds a few minutes to the boot time, so whatever. And while I maybe could do something with Linux, it's unlikely I could do anything with the POST phase.
- Disable it on Linux. There might be some kernel option somewhere to do that. Or maybe the slowdown is happening somewhere late in the boot process, so I could just set some udevd rules or something.
- Find the chip on the motherboard and desolder it because YOLO. I admit to be a bit afraid to try this tactic as I actually need Haven to work properly.
Eventually I decided on the first option of doing nothing. However curiosity got the better of me and I decided to at least find the chip on the motherboard. Admittedly I also put some hope into the "Board Form Factor: M.2" remark spotted in the official specification. Of course there is no reason why a motherboard manufacturer wouldn't just get the parts themselves to solder directly onto the motherboard, but maybe, just maybe, they actually went for a detachable M.2 board. While I had hope, it wasn't that much hope – if the motherboard would have a detachable M.2 board somewhere on it, I would have noticed it in the past. On the flip side that motherboard sure has a lot of fancy looking plastic covers and radiators...
Source: MSI's official X99A XPOWER GAMING TITANIUM website
Since the motherboard I had was in use, not to mention populated with a lot of visual obstructions (also known as cables and PCI Express cards), I've started with Google images search looking for high-resolution photos of the motherboard with no cover. After a few minutes I found an in-depth motherboard review by Brendan van Varik for Play3r and this gem of an image:
Source: Brendan van Varik, Pl3yer, MSI X99A XPOWER Gaming Titanium Motherboard Review
Given that this is under the large back plastic cover I never noticed it being something else than just two antenna connectors. I proceeded to check how is this box soldered to the motherboard, but found... a screw.
The screw
I decided to google for more images of the box itself to see if someone described a way to detach it and I've actually found the opposite. The "MSI Z97 GAMING 9 AC (Intel LGA 1150) Review" by cadaveca from TechPowerUp mentioned a different motherboard where this box would come detached by default and the user could optionally attach it to the motherboard. Here's what cadaveca had to say about it:
MSI's wireless AC card includes Bluetooth and supports Intel's WIDI functionality, so you can broadcast video wirelessly to supporting devices. Both the LAN and Bluetooth ports get their own antenna (which is why there are two in the box), and the card itself mates to the board via a pin header on the card itself. IF you plan on using it, you'll want to take a bit of extra caution while installing MSI's wireless card since its delicate pins can be bent out of place easily. I didn't run into such a problem, however.
There were also a couple of photos of the box from different sides showing one screw hole and a pin header (keyed – one pin was missing).
As soon as I've learned this I turned off the PC, took it out of the server rack, unscrewed several screws and successfully detached the module:
Success!
Next the PC went back into the rack, I've hooked up all the cables and turned it on. It booted nice and fast, no issues whatsoever. Success!
After a short chat with D0han on my Discord about the weird PCI Express header I decided to take the box apart to verify my hypothesis of it being just an adapter for M.2 plus the Intel AC 8260 card itself. Turns out I was right (mostly):
AC 8260 without the cover
AC 8260 disassembled
I've mentioned above I was mostly – but not fully – right. I actually expected an M.2 connector with the M key. Long story short is that the M.2 connector can be used for multiple protocols and pin configurations. Which exact set of protocols is it is signaled by missing pins and a "blocker" in their place (or notches in case of device connectors). So M.2 key M would denote PCI Express x4 (see the table from Wikipedia below). Which is what I expected – a PCI Express device that has a chip on it that's both a WiFi module and pretends to be a USB controller for the purpose of connecting the Bluetooth module. I would know this to be incorrect had I paid any attention to how the USB device identifiers are assigned. USB 3-11 means 11th device on the 3rd bus. If that device would be a USB controller, the Bluetooth module itself wouldn't be 3-11 – since 3 denotes a different USB controller – it would be 9-1 or something. Everything made more sense once I've noticed it's actually the E key, which can both support PCI Express x1 and USB 2.0 connections at the same time. Given this, the card is actually two separate devices (even if potentially on one chip) – one connected to the PCI Express x1 pins and the other to the USB 2.0 pins.
Source: M.2 article on Wikipedia (EN)
The only question that remains is whether one can attach a GPU through an adapter to this slot (will be pretty slow, especially that it's PCI Express x1 version 2.0 – since it's attached to the chipset and X99 does only PCIe 2.0). Not that there is any reason to do it. Anyway, I might or might not have ordered a proper cursed adapter from a certain Chinese electronics site, so guess we'll find out.
Comments:
As far as I know, WiFi and BT devices need to be certified e.g. by FCC, so due to regulatory burden motherboard OEMs prefer to get approved modules instead of bare chips to solder. M.2 2230 is popular and replaceable which is a good thing as WiFi standards evolve quite fast. There is also smaller M.2 LGA 1216 form factor to be soldered e.g. in laptops, but still it has its own PCB and is sold as a complete module.
Intel uses CNVi in which many parts of functional blocks were moved into their PCH (or SoC in general), but even then, the non-standalone RF modules use M.2 2230 or 1216.
Add a comment: