How a kernel developer made my styluses work again on newer kernels!

Published on

🎉 🎉 🎉 Whooohooo! It works again!

Both styluses of my XPPen Artist 24 Pro and XPPen Artist 16 Pro Gen2 styluses are now usable again on newer Linux kernels.

This blog-post is a follow-up post of "How a kernel update broke my stylus... Need help!" published 10 days ago. Please read it if you want to know more about the problem I had with the stylus.

After this first blog post (and thanks to your comments and guidance, especially I was able to email my bug to experts in the area.

Then Jiri Kosina republished my email to the mailing-list. A big thanks to them, to Illia Ostapyshyn for the discussion, and to Benjamin Tissoires for developing a solution.

If you have the same issue with a similar device, you'll have to compile:

This solution is still W.I.P. and I still have some homework to send more data about my tablets after this blog post, but in overall I'm already using a newer kernel (Linux workstation 6.5.10-200.fc38.x86_64) and I don't have the problem with the eraser mode on the top button of my XPPen Artist 24 Pro and XPPen Artist 16 Pro Gen2 styluses. The buttons are also now perfectly customisable via xsetwacom CLI tool. Yay! That's why I wanted to share this blog-post as soon as possible.

On the mailing list Benjamin wrote me a detailed answer about the whole story. It's very interesting and I decided to copy and paste it here. Thanks again Benjamin! 👍

Benjamin's detailed answer:

(Original email here.)

Hi David,

Here is a little bit of history of why you were encountering this bug:

First, HID is a quite old protocol and has the benefit of being "plug and play" [0] [1].

But plug and play often means for a hardware maker: "let's do trial and error until Windows seems to behave in a correct way".

In some other cases, Microsoft puts more restrictions on the HID part (Windows 7 enforced touchscreens to follow a specific pattern, and then Windows 8 did it for touchpads). And they also sometimes provide a test suite that hardware makers have to pass to be "certified". They have to pass the test suite by using the Windows provided generic driver, but Windows also allows them to create a virtual HID device and let a custom driver create that virtual HID device. Which means, we sometimes have devices behaving badly but working perfectly fine on Windows because there are bits missing in the device itself that are fixed by adding an extra software layer. Sigh.

But I digress and we need to go back to the pen bits, and AFAIK, there is no such test suite and certification.

So basically, hardware makers follow the empiric rule of "if Windows is happy, I am too".

To do so, they have to use several events from HID [2] (quoting them):

  • Tip Switch → A switch located at the tip of a stylus, indicating contact of the stylus with a surface. A pen-based system or system extension would use this switch to enable the input of handwriting or gesture data. The system typically maps Tip Switch to a primary button in a non-pen context.
  • In Range → Indicates that a transducer is located within the region where digitizing is possible. In Range is a bit quantity
  • Barrel Switch → A non-tip button located on the barrel of a stylus. Its function is typically mapped to a system secondary button or to a Shift key modifier that changes the Tip Switch function from primary button to secondary button.
  • Secondary Barrel Switch → A second non-tip button located on the barrel of a stylus further from the tip than the Barrel Switch. Its function is typically mapped to a system secondary or tertiary button.
  • Eraser → This control is used for erasing objects. Following the metaphor of a pencil, it is typically located opposite the writing end of a stylus. It may be a bit switch or a pressure quantity.
  • Invert → A bit that indicates that the currently sensed position originates from the end of a stylus opposite the tip.

I'm sure that by reading those, everybody should be able to immediately know how to write a Pen HID device, and how the interactions between the events should be. :) (If you are, please contact me ASAP, we have plenty of kernel work to do).

So for years the state of pen devices in the Linux kernel was 2 fold:

  • Wacom shipped an in-kernel driver for their own devices, that they tested and that defined the de-facto "standard" in Linux
  • the rest was trying to catch up by luck or with the help of projects like DiGiMend, by relying on the generic HID processing of the Linux kernel

However, they were no specification for how the events should come: basically in the hid generic input processing each event was mapped to a single input keycode and we had situations were we would get both BTN_TOOL_PEN and BTN_TOOL_ERASER at the same time (because the In Range and the Eraser bits were sent by the device at the same time). Which means "hey, the user is interacting with a pen with both the tail and the tip at the same time. Good luck with that!"

This led to a complex situation where userspace (libinput mostly) had to do guesses on what is the intent of the user. But the problem is that when you are in userspace, you don't know all of the events of the device at the same time, so it was messy to deal with. Again, Wacom devices were unaffected because they controlled all of the stack: a kernel driver to fix/interpret their devices and a userspace component, xf86-drv-wacom, in the world.

Once, as you mentioned in your blog, Microsoft decided to use the second barrel button as the "rubber" mode. The reason was practical: people liked the rubber capability on the styluses, but they wanted to have a separate button on the tail end of the styluses. And I suppose that at the time, given that no other hardware vendors were capable of having no-battery styluses but Wacom (IP protection and capabilities of the hardware IIRC), you still had to put the battery somewhere. And that tail end is convenient for storing such a battery. But that makes it harder to have an eraser end because you need to link both ends of the pen on the same IC, with a battery in the middle that is roughly the same size as your pen's barrel. So having just 2 wires for the battery allows you to have a separate bluetooth button on one end, and the normal stylus on the other end, and keep the width of the pen reasonable.

So that choice of using the second button as an eraser was made, and the hardware makers followed: on the XP-Pen Artist Pro 24, the device reports Tip Switch, Barrel Switch, Eraser, In Range. Which is "correct" according to the HID Usage Table [2], but which doesn't adhere to the recommendation Microsoft is doing [3]: the device should report an extra Invert when the pen is in range with the intent to erase...

But you can see that XP-Pen tried to adhere to it in some ways because if you look carefully at the events coming from the device with hid-recorder [4], you'll notice that when you are in range of the sensor and press this button, you'll get an extra "In Range = 0" event to notify that you went out of proximity of the sensor.

In kernel 5.18, with commit 87562fcd1342 ("HID: input: remove the need for HID_QUIRK_INVERT"), I tried to remove those multiple tool states to have a straightforward state provided by the kernel that userspace can deal with easily. However, given that there were no regression tests at the time for generic tablets, I wrote some based on Microsoft's recommendation [3] and also tested on a Huion device I have locally. And it was working fine. But I didn't have the devices that were not sending Invert, which explained why it was bogus on those devices.

This was "fixed" in kernel 6.6 with commit 276e14e6c399 ("HID: input: Support devices sending Eraser without Invert"). Putting quotes around "fixed" because I'm still not happy about this patch.

But the point is, from kernel 5.18, the Pen processing in the kernel became a state machine, which means we can not have external actors tampering with it.

Why using the ioctl EVIOCSKEYCODE is bad to remap Eraser to BTN_STYLUS2 (through tools like evmap):

Having the ability to do something doesn't mean it's the right thing to do. And in that case, this is definitely wrong because you have to call the ioctl after the kernel presents the device to userspace. Which means userspace (and the kernel) already made assumptions on the device itself. There is a high chance libinput (or the Wacom driver) opens the device before evmap, and that it is considering that the device doesn't have BTN_STYLUS2. So sending that event would break userspace.

And in our case here, the kernel expects some state between the input layer and its internal HID state, and remapping that HID event to something else confuses it.

There is another side effect of this: usually end users configuring their devices with such tools do not report back their configuration to the kernel community. In some cases this is valid (this is my preference and my choice), but in other cases it's not (there is a bug here and I'm papering over it).

So, what can be done?

Basically 2 options:

  1. write a kernel patch to fix that problem once and for all
  2. use the brand new HID-BPF[5] capability introduced in kernel v6.3 and send me back the BPF program so I can eventually integrate the source in the kernel tree itself and fix that problem once and for all as well

For 1., you need:

  • to be able to dig into the kernel code
  • to be able to write a patch with the correct kernel standard (with a regression test in tools/testing/selftests/hid, please)
  • to be able to compile your own kernel and test it
  • to be able to submit your contribution by email (I can suggest using b4 for that, very nice tool)
  • to be able to take reviews into account, and learn git rebase -i to submit v2, v3, and potentially v10 or more in some complex cases
  • to wait for the patch to be integrated into Linus' tree
  • to wait for Greg to backport your patch into a stable kernel tree
  • to wait for your distribution to pick up the stable tree with your patch

That's relatively easy, no? :)

OTOTH, we have 2.: HID-BPF [5]

Very quickly, eBPF [6] is a state machine inside the kernel that allows user space to include a verified code path in the kernel to tweak its behavior. And I adapted this for HID so you can:

  • change the report descriptor of the device: this disconnects/reconnects the device, meaning the kernel works on the new report descriptor and is consistent with its state
  • change the event flow of the device: to fix the spurious out-of-prox event for example
  • more to come

What is interesting in BPF (or eBPF), is that nowadays, libbpf implements something named CORE (Compile Once Run Everywhere). Which means that if I compile locally an eBPF program on my machine with my development kernel, as long as I only use functions available from kernel v6.3 for instance, the same compilation output (that changes the event flow of your HID device) will work on any kernel from v6.3 unless there are some later API breakages[7].

Which means, anybody can modify the event flow of an HID device, put the file in the filesystem, and have the device still fixed even if they upgrade their kernel.

In the long run, I intend to include those HID-BPF fixes in the kernel tree to centralize them, but also to be able to automatically load them from the kernel when those devices appear.

Which means, for the reporter of such a bug you:

  • can now rely on someone else to write the code, compile it and provide the compilation result [10]
  • just put that result in the filesystem to have the device tested and fixed

Behind the scenes, that other knowledgeable person can take the heavy task of submitting the kernel patch for you, but given that the code has been tested, it's way easier to do (and to eventually re-test).

Currently, the "let's integrate that bpf program in the kernel" is not completely done, so we use udev-hid-bpf[8][9] to give it a jump start.

And that's exactly what happened in your case David. Which is why I'm so happy (also because I fixed the tools from an author I like and already had the books at home :-P):

You want your device to be fixed now, but going through a regular kernel patch means months before it's fixed in your distribution. But with HID-BPF, I fixed it now, and you can safely upgrade the kernel, because even if I do changes in the kernel, the HID-BPF fix will still convert the device into something valid from the HID point of view, and it has a regression test now. When your device will be fixed in the future in the kernel, there is a high chance the probe function of the HID-BPF program will say that it's not the correct device, and so the program will not even load and rely on the fixed kernel only. Transparently for you, without you having to change your filesystem.

On my side, what's left to be done:

  • First, I need to fix the tablets not sending the Invert usage. Commit 276e14e6c399 ("HID: input: Support devices sending Eraser without Invert") is IMO not good enough, and we might as well simply say that if there is no Invert usage, we can convert the Eraser usage into Secondary Barrel Switch
  • then I need to fix the XP-Pen Artist Pro 16 gen 2 from the kernel too, by replacing the Eraser usage with Secondary Barrel Switch. Ideally I would just dump the HID-BPF program in the kernel, but this is not available yet, so I'll probably write a small kernel driver using the same code path as the HID-BPF program.
  • then Peter and I need to write a more generic HID-BPF program to convert "eraser mode buttons" into Secondary Barrel Switch, basically unwinding what the hardware does. This can only happen when libinput will be able to do the opposite transformation so we don't regress. But we can rely on libwacom to tell us if this pen has a tail end eraser or not, and then have userspace choose if they want the button to be a button, or an eraser mode.

I think that's pretty much it.

Thanks for reading through everything :)

Cheers, Benjamin

[7]. but if API breakage happens, all that will happen is that the HID-BPF program will not be loaded. No kernel crash involved.


link David Revoy Author, - Reply

📽️ Painting process of the illustration.
📦 Sources:
:cc: :ccby:

📎 d540758221f09c42.mp4  201 ★

link bbbhltz   - Reply

over my head, but still quite a read

link David Revoy Author, - Reply

@bbbhltz Oh yes, that's a big one for a sunday evening. 🤓
I read it almost 5 times and I'll probably reread it later after following the links. So many concept in it I'm not familiar, but I'm really curious to know more about them.

link Ryuno-Ki   - Reply

@bbbhltz I'm not a kernel hacker myself, but if you consider it helpful I could try and unpack some parts of the email…

link lebout2canap ⏚   - Reply

Very good news, and interesting email from BenjaminTissoires.

link Chris. R. 🎧🎼☕🍍   - Reply

A great example for open-source development. Glad to hear that your issue has been fixed.

link Ryuno-Ki   - Reply

I know this cave!

link David Revoy Author, - Reply

@RyunoKi 😆 😆 😆 Haha, it totally could be!

link Arne Babenhauserheide   - Reply

wow — very cool!

link Arne Babenhauserheide   - Reply

Regarding "(there is a bug here and I'm papering over it)" — that was the case with my USB pre-amp. Once the configuration worked, I gave my thanks to Chaos and went with it until it broke with udev.

Some time later it worked again and I don’t really know why, but I also don’t know which part of the meddling fixed it in the end, so this is super hard to report …

link Arne Babenhauserheide   - Reply

(and it is quite an interesting read — thank you for sharing it in full!)

link Ryuno-Ki   - Reply

@ArneBab What I consider noticeable is that HID is an old protocol. would be affected then as well?

(Used it with Wii remote so far)

link F4GRX Sébastien   - Reply

@RyunoKi @ArneBab why the heck do we need hid in a browser haha

link Ryuno-Ki   - Reply

@f4grx @ArneBab

If only you knew

link David Revoy Author, - Reply

@RyunoKi :blobaww: Wow. It's possible to vibrate the device with code on a webpage?! Big TIL.

@f4grx @ArneBab

5 ★

link F4GRX Sébastien   - Reply

@RyunoKi @ArneBab yeah I also discovered that. I wonder how they do that :D

link Ryuno-Ki   - Reply

@f4grx @ArneBab

If you feel inspired to draw: Project Fugu.

I leave the discoveries to you.

link Arne Babenhauserheide   - Reply

@RyunoKi those always make me afraid. So many possibilities available to untrusted code. Just think about all the vulnerabilities hiding there that could be exploited by any random website … @f4grx

link Ryuno-Ki   - Reply

@ArneBab @f4grx Yep.

Let alone fingerprinting.

Guess who's behind drafting those APIs under Project Fugu umbrella?

link Arne Babenhauserheide   - Reply

@RyunoKi Do they start with G and continue to Oogle? @f4grx

link F4GRX Sébastien   - Reply

@ArneBab @RyunoKi with their two friends starting with M

link Arne Babenhauserheide   - Reply

@f4grx Do you mean the Eta and I-crowsoft?

(just rolling with the words, no insult to crows intended ☺)

link Ryuno-Ki   - Reply

@ArneBab @f4grx Mayhaps

link ȺՀղöɾէհ ⏚, βօӀìçҽ ժմ βօղƓօûէ   - Reply

En plus t’as même fait une chic illustration. C’est super que tu prennes le temps — et le plaisir, j’imagine ! — de faire une illus’ pour un article.
À la place du dev, je crois que je prendrais ça comme un remerciement plus ou moins direct. 😁

link David Revoy Author, - Reply

@Aznorth Merci. Oui, j'aime bien faire ces petits speedpainting cinématiques; j'y mets quelques heures mais ça me permets de prendre le temps de faire le plan dans ma tête de ce que je vais écrire et dans quel ordre je vais présenter les choses (car sinon, j'ai souvent tout le tiercé qui arrive dans le désordre et en même temps, et donc dur dur d'articuler mes idées sous forme de textes.)

link FLOX Advocate   - Reply

"The buttons are also now perfectly customisable via xsetwacom CLI tool."

New *and* improved! Woohoo!

link David Revoy Author, - Reply

@FLOX_advocate Yes, perfect for the future review of the XPPen 16 Artist Pro (Gen2); as it now works at 100% its potential with Free/Libre and Open Source driver.

3 ★

link FLOX Advocate   - Reply

also you now have opening and closing images for your kernel bug illustration book :)

link David Revoy Author, - Reply

@FLOX_advocate 😺

link nunux   - Reply

Encore une oeuvre pleine de talent et d'humour.


link David Revoy Author, - Reply

@nunux 😊 Merci

link Rolf Steinort   - Reply

What for a loving depiction of a kernel hacker!

link David Revoy Author, - Reply

@rstein Thanks!

link Rolf Steinort   - Reply

They should hang a print into their cave/bunker.

link phαη†⊕m :fedora: 🎸 🏳️‍🌈 ⏚   - Reply

Un artwork plus "sombre", magnifique, comme d'hab, qui mériterait bien une bd entière.
Heureux qu'il y ait une issue positive à ton souci.
Et avec f39 ?

link David Revoy Author, - Reply

@Steve12L Merci! Je pense faire le passage à Fedora 39 avant le Capitole du Libre, donc courant de la semaine qui s'annonce. J'ai une deadline pour demain (la nuit va être encore longue) et je voulais pas prendre ce risque avant que les dessins soient livrés. 😺

link phαη†⊕m :fedora: 🎸 🏳️‍🌈 ⏚   - Reply

Je comprends bien, logique. J'ai fait l'upgrade ce matin, aucun souci. J'ai réactivé mes extensions, hormis OpenWeather, tout est là.
Dash to dock et les autres sont là.
Je ne me souviens pas de toutes les specs de f39, faut relire le post de Renault sur le forum, j'espère que la solution sera intégrée en tout cas.
Et encore merci, et bravo, pour tout ce que tu fais. 👍🏼

link Mostafa Hussein Omar   - Reply

I'll always think of kernel developers as cheeky penguins fixing stuff from now on! I received an xppen tablet (different model) recently as a birthday gift. Thanks for this write up, I will go through this post next weekend. I am sure it will be valuable in improving its performance.

link Aral Balkan   - Reply

Yay, happy to hear it, David :)

link Suburban Druid   - Reply

Retooting for the *rad* image of a penguin doing tech support.

link David Revoy Author, - Reply

@Suburban_Druid 😊 Thank you!

link Nic ⏚   - Reply

impressionnant !

link OrbitalMartian :GoToSocial: 🏴󠁧󠁢󠁥󠁮󠁧󠁿   - Reply

I love this, great to see the process too :)

link Arne Babenhauserheide   - Reply

thank you!

It sounds like if you now sent a signed print of this to a specific kernel developer, you might create a treasure that survives the decades ☺

(and I would totally add a 5€ donation to reduce any cost you may have due to that)

link David Revoy Author, - Reply

@ArneBab Good idea.👍 I'll check with Benjamin. And thank you for your proposal to help to reduce the cost of postal transit but I'll manage with the budget I have (that's also in part thanks to your donation, so you already gave for that 😉) .

link penguin42   - Reply

A very fine penguin.

link Ertain   - Reply

Another good picture, David.

link Jan van Acken   - Reply

I see I need to re-draw my penguin that I shared here a while back and equip it with a stylus now ;)

link Reese Polilla :AAAAAA:   - Reply

I love your work David!

link David Revoy Author, - Reply

@Reese_pony Thanks!

link MHunt   - Reply

that's great !

link 🦇Mike Ravenblack🦇   - Reply

🐧 Your penguine developer might look like he's just waddling around, but he's actually got some
major coding skills! 😂

link Soh Kam Yung   - Reply

Not directly related, but this might be a fun thing to try in the future, when available.

"D-POINT is an open-source digital stylus that uses camera tracking and inertial measurements to achieve 6DoF (six degrees of freedom) inputs, with low latency, pressure sensitivity, and sub-millimetre accuracy. The stylus can be used on any flat surface, and works with consumer-grade webcams."

link David Revoy Author, - Reply

@sohkamyung Thanks! I saw it yesterday on homepage of HN. Very very interesting research. I was really suprised by the way pressure of the tip was also recorded with only visual info.

link Brian Swetland   - Reply

I love how this illustration captures the reality of kernel/driver development. (I say, in all seriousness, as an embedded/systems engineer).

link David Revoy Author, - Reply

@swetland 😊 Thanks!

link Jason Bowen   - Reply

Well now I want all kernel patches to be accompanied by artwork :)

link Sophie Tyalie   - Reply

ohh wow. That is such a detailed answer to the question. I love that they took their time to write that out.

link David Revoy Author, - Reply

@tyalie Same, it felt like a gift to receive such a detailed and long reply.

2 ★

link Kathy Murdoch :pastille_lemon:   - Reply

That was an impressively fast turnaround! Kudos to the devs for providing such an in-depth answer, too, even though a lot of it was over my head it's interesting to see how the tablet stuff is implemented!

link Puniko :verifiedsabakan: :disability_flag:   - Reply that was quick

link David Revoy Author, - Reply

@puniko Oh yes, I was prepared for a long exile in the old kernel land; switching distro and all. That's a relief to can do my update. I'll even do the upgrade to F39 this week.

link F4GRX Sébastien   - Reply

we'll call it the pepper patch!

link thedæmon   - Reply

#unix_surrealism :flan_cool:

link Tomáš   - Reply


He's not real, D.

link Kevin Karhan :verified:   - Reply


Glad to hear that loud ranting worked.

I suggest @fuchsiii should do the same re: her hardware and software issues...

link Surya Teja K   - Reply

I tried reading the blog post but the lack of a dark theme is a big problem for me.

The light theme is too bright in my dark room.

Sorry :(

link David Revoy Author, - Reply

@shanmukhateja Hey, thanks for the feedback! Can you link to me a blog post you like to follow with a good combination of color for the font and background? This way I can start to get a reference in how I can put this to my blog post.

Here my full display are color calibrated, and target a limited cd/m² for the light emission, but I know not everybody can do this.

Also, do you need a switch on top of the blog? or should it just follow the theme of the O.S. of the visitor?


2 ★

link Surya Teja K   - Reply


So I can’t seem to find a blog with dark mode right now. I don’t follow any of them, I prefer finding interesting ones here.

Maybe take a look at my blog? It is managed by Hashnode and is by no means perfect but I did my best styling it up.

Yes you are right. I don’t know what cd/m^2 is supposed to be 😅

So, I prefer websites to use my OS enforced color scheme.
A toggle button is a “good to have” but the website should use my OS color scheme.

Hope this helps!

link David Revoy Author, - Reply

@shanmukhateja It does! Thank you. I put it on my TODO 👍

link madmax   - Reply

@shanmukhateja I am just using dark reader browser extension for getting a dark theme and it has an auto mode in which it switches to dark or light theme according to system settings. I haven't checked it on linux but on firefox android its working without any problems so far.

link Surya Teja K   - Reply

@pikachu_sensei Yes I used to use it on my phone before switching to iPhone.

It has been a hit-or-miss in my case because the dark theme is never perfect.

I don’t have extensions on Firefox in iOS (thanks Apple) and so I have to rely on webmasters to setup and enable dark mode on websites.

Side note, “webmasters” makes me feel old. I don’t know why I thought of that word? 🤣

link Badri   - Reply

oh nice! Congratulations to everyone involved who helped make it happen 👏👏👏

(I read your earlier blog post and went through the threads, so I'll do the same in a bit to learn how it got fixed!)

link mafe   - Reply

As awesome as the previous artwork!

link fanf42   - Reply

Thank you for sharing, super interesting. The answer by Benjamin is a pearl 👌

#linux #tablet #driver #krita #MastoArt #xppen #digimend #kernel

Post a reply

The comments are synchronised every 4h with the replies to this post on Mastodon:

How to use this? (click here to unfold)
Open a new Mastodon account on the server of your choice. Then, Copy/Paste the adress above in your Mastodon 'Search' field. The post will appear and you'll be able to fully interact with it. You'll have full control of your posts: edit, remove, etc. After that, your message will appear here.

Just please note that it may take up to 4 hours for your changes to be reflected here.