• Have two Linux machines sitting next to each other?
• Want to transfer some files?
• Not have a USB stick handy?
• Have an ethernet cable?

Then this simple hack guide is for you! You can connect the two computer directly with each other using an ethernet cable and transfer a file using netcat. This will pipe the file in on one end of the cable, and it will come out the other end.

Prerequisites

On both machines, install netcat-openbsd. By default, Debian comes with netcat-traditional, which does not support IPv6.

sudo apt remove netcat-traditional
sudo apt install netcat-openbsd

Connect the computers

Directly connect the two computers using an ethernet cable (possibly using RJ45-to-USB-C adapters).

There is no switch or any other networking equipment inbetween! It is really a direct link. On modern machines this should work with any ethernet cable, because recent network cards support Auto-MDI-X. On older hardware, you need to use a crossover cable (due to the absence of a switch).

Get the link-local IPv6 address

On the target machine, list the IPv6 addresses of the interfaces:

ip -6 a

Look for the link-local address, which starts with fe80:. Note down this address.

If either of your computers has a network manager (which is very likely if it has a desktop environment), you need to change the IPv4 and IPv6 address selection method from “Automatic” to “Link-local” for the wired interface. Otherwise it will look for a DHCP server (to no avail).

On KDE Plasma, the network manager settings look like this:

Check connectivity

Check that the source machine can ping the target machine. Run the following command on the source machine:

ping fe80::1122:3344:5566:7788%eth0

This is the IPv6 address of the target machine followed by the interface name of the source machine.

In IPv6 you need to add the interface name at the end, separated by %. This is because all interfaces share the same link-local subnet (fe80::/10), so the network stack needs to know through which interface to send the traffic.

Your network interface is called something like enp0s10 or eth0. You can list the available interfaces with:

ip link show

This also shows the state of the link (UP/DOWN).

Do the file transfer

On the target machine, start netcat:

nc -vl6 9000 > file.txt

This causes netcat to listen on port 9000 and to write any data that it receives to the named file.

On the source machine, send the data with netcat:

nc -N fe80::1122:3344:5566:7788%eth0 9000 < file.txt

This causes netcat to connect to the given IPv6 address and port, and then send the contents of the file. Again, note the interface name at the end, separated by %.

The netcat instance on the target machine should receive the contents and write them to its file.

Security

This is a very basic transfer mechanism. It’s secure because (and only if) you physically control the entire, single network cable. It’s intended for quick-and-dirty on-desk setups.

Importantly, the netcat approach does not have any cryptographic confidentiality, integrity, or authenticity guarantees! As a basic integrity check, you can compute the sha256sum of the file on both ends.

If you are doing this over a real network, you should use ssh combined with scp or rsync. They require a bit more setup than netcat, but provide security against a network adversary.

The focus of this post is on explaining the high-level ideas, why MTE is important, and how developers can enable MTE in their Android and iOS apps.

Background: memory allocations

Programs store data in memory either on the stack or on the heap. Generally, the stack is used for data that is small and whose size is known when the program is compiled. The heap is used for larger data and for data whose size is only known when the program executes (for example, any data that the user inputs).

Memory on the stack is “automatically managed” in the sense that the compiler inserts instructions to automatically “allocate” and “deallocate” memory on the stack (by moving the stack pointer). This is possible because the compiler already knows the size of the needed memory.

Memory on the heap is dynamically managed by the program when it executes. The program calls malloc(size) to allocate some memory. The memory allocator then reserves the requested memory size and gives the program a pointer to where this memory section is. Once the program no longer needs this memory, it should call free(pointer) to tell the allocator that the memory can be reused.

This manual memory management (malloc+free) is common in C and C++. In other languages, developers don’t need to manually manage heap memory: In Java and Go the garbage collector periodically frees unused memory. In Rust, the compiler automatically inserts free() statements into the compiled code (since Rust’s ownership semantics allow the compiler to figure out when a value will become unused). In Swift and in C++’s smart pointers, Automatic Reference Counting automatically frees memory once it is no longer referenced.

Background: memory vulnerabilities

Manual memory management puts a burden on programmers using C/C++. Programmers need to make sure to only access memory that is allocated, to not forget to free the memory (causing memory leaks), and to not access the memory again once it has been freed. Unfortunately, as the last 30+ years have shown, C/C++ programmers repeatedly make these mistakes. A lot.

These mistakes in memory management are a common source of vulnerabilities:

• Spatial safety (out-of-bounds access, buffer overflow): The program accesses memory outside its allocated region. (This can happen with stack memory, too.)
• Temporal safety (use-after-free, double free): The program accesses memory after it has freed it.

By maliciously crafting inputs, an attacker can exploit such memory issues to take over the program flow and execute custom code.

Defenses against memory safety vulnerabilities

Unfortunately, memory safety vulnerabilities are very common in C and C++, and a LOT of code is written in those languages. Therefore unsurprisingly, people have been looking into defenses for a long time. Examples of such defenses are:

• Hardened memory allocators, for example in iOS and GrapheneOS.
• Pointer authentication (PAC), for example in iOS and Chromium for Android.
• Control flow integrity (CFI) such as PAC, BTI, shadow stacks, and stack canaries.
  • This prevents attackers from redirecting the control flow of a program to an unintended path.
• MTE
  • This prevents attackers from corrupting memory (possibly to redirect the control flow later).

Note that these approaches are complementary, and should be used together.

What these approaches and MTE have in common, is that they don’t require changes to existing code (except that some require re-compiling the program). Additionally, they eliminate entire classes of attacks across the entire program.

Other approaches (like smart pointers) also exist, but they require developers to rewrite their code (and all of their dependencies!) to use modern, “safe” idioms/language features. This is a lot of work – work that is arguably better spent on rewriting the code in a memory-safe language such as Rust. Also, they only eliminate issues in the narrow sections of code where they are applied.

Sidenote: unsafe memory usage hides even in modern C++ idioms!

How MTE works

Memory tagging works by assigning a unique, random tag to every memory location. This tag changes every time a memory location is allocated with malloc() and every time it is free()d.

malloc() returns a pointer to the start of the allocated memory region. The tag for this memory region is embedded/encoded into the pointer.

Memory is accessed via pointers, possibly with an offset (“please read the value stored at the memory location this pointer points to”). An access can be a read or a write. For each memory access, the hardware checks that the tag in the pointer matches the tag of the memory location. If they mismatch, the access is denied. In practice, this causes the app to crash.

This scheme prevents invalid memory accesses, because invalid accesses will have non-matching tags:

• Spatial safety (out-of-bounds access, buffer overflow): The memory location has a different tag than the pointer, because the pointer came from the allocation of a different memory region.
• Temporal safety (use-after-free): The pointer and the memory location had matching tags in the past, but when the memory was freed, the memory tag changed.

To attack MTE, an attacker needs to guess the tag of the memory location they want to access, and then modify the pointer to have the same tag. For example, an attacker might leak the tag via a side-channel (see below).

For a more detailed explanation, see the links at the bottom of this post.

For a visual representation, see the figures below, taken from the slides of this talk by Andrey Konovalov. Tags are represented as colors.

MTE on Android

Among Android devices, Google Pixel 8 (and later) are the only known devices with hardware support for MTE, thanks to Google’s Tensor G3 chip. They were released in October 2023.

No other Android phones seem to have practical MTE. A big reason is the lack of hardware support (see the appendix). However, even if the chip would support it, every phone vendor customizes Android, and may not enable MTE in their operating system.

On Google Pixel’s Android, MTE is NOT enabled for the kernel, but it is enabled for critical userspace system processes such as Bluetooth, NFC, and SecureElement (source). Additionally, Android Advanced Protection Mode enables MTE. (However, I haven’t found an offical source that says for which parts MTE is additionally enabled by Advanced Protection.)

On GrapheneOS (a security-focussed Android-based operating system), MTE is enabled for the kernel, for some of the system apps, and for user-installed apps that don’t have native code (source 1, source 2).

To enable MTE for a given app, the app developer needs to explicitly opt-in by setting android:memtagMode="async" in the AndroidManifest.xml of their app. Additionally, GrapheneOS provides a toggle allowing users to force MTE for specific apps.

MTE on iOS

Apple’s marketing name for MTE is Memory Integrity Enforcement (MIE). MIE is supported on the A19 chip, included in iPhone 17 devices, launched in September 2025. It is unclear if and when MIE is coming to M-series chips (for iPads and Macs). MIE uses a newer version of Arm MTE; the concepts are the same.

On iOS, MIE is enabled in the “the kernel and over 70 userland processes”.

To enable MIE for a given app, the app developer needs to explicitly opt-in by setting the appropriate entitlement.

Why is MTE not more widely deployed?

Why has MTE not been deployed sooner? Why is MTE not enabled for every app and every process on the phone, but only for some?

MTE has two “problems”:

1. Performance: Maintaining and checking the tags introduces a computational and memory overhead.
2. Crashes: Invalid memory accesses cause crashes. Without MTE, buggy apps that make invalid accesses (such as use-after-free) often silently work. With MTE, these accesses are denied and the apps crash (by design). (Note that MTE does not introduce new bugs, but it makes existing bugs very visible to the user.)

Performance is one of the reasons why Google is hesitant to enable MTE for more parts of Android, and why it is keeping MTE disabled for the kernel. Apple has solved the performance problem by deeply optimising the hardware for MIE. Its new A19 chip is much faster, even with MIE enabled.

Sidenote: Product requirements driving chip design is a recurring pattern at Apple. Apple doesn’t build a fast chip and then thinks about what it can do with it. Instead, they start with a requirement (cinematic mode in iPhone 13, MIE in iPhone 17), and then design the chip to support the desired feature (massive GPU performance in A15, MIE performance in A19).

Crashes are the reason why both Apple and Google don’t force-enable MTE for user-installed apps. Instead, app developers need to opt-in to MTE on both Android and iOS.

It is not clear if Apple and Google force-enable MTE for user-installed apps when Lockdown Mode/Advanced Protection are enabled. Doing so would make sense because users with such high security requirements benefit the most from MTE, while also being more likely to accept some app crashes (as the price of security).

To underline the point about app crashes: Apple Music on Android regularly crashes on my Pixel 8a with GrapheneOS. Here is an example of a stack trace:

type: crash
osVersion: google/akita/akita:16/BP2A.250805.005/2025091000:user/release-keys
package: com.apple.android.music:1472, targetSdk 35
sharedUid: com.apple.android
process: com.apple.android.music

signal 11 (SIGSEGV), code 9 (SEGV_MTESERR), fault addr 0x0d00c4378c1aecb8

backtrace:
      #00 pc 0000000000604c5c  /data/app/~~qvCT42FscwNrFow1psc79g==/com.apple.android.music-SUeV4RIatUrpGVV1gt7VDg==/lib/arm64/libandroidappmusic.so (BuildId: b52cdcc1f1daf9a96aaaa911f060d467d7b54552)
      #01 pc 0000000000601ae0  /data/app/~~qvCT42FscwNrFow1psc79g==/com.apple.android.music-SUeV4RIatUrpGVV1gt7VDg==/lib/arm64/libandroidappmusic.so (BuildId: b52cdcc1f1daf9a96aaaa911f060d467d7b54552)
      #02 pc 0000000000600c24  /data/app/~~qvCT42FscwNrFow1psc79g==/com.apple.android.music-SUeV4RIatUrpGVV1gt7VDg==/lib/arm64/libandroidappmusic.so (BuildId: b52cdcc1f1daf9a96aaaa911f060d467d7b54552)
      #03 pc 000000000033ae04  /data/app/~~qvCT42FscwNrFow1psc79g==/com.apple.android.music-SUeV4RIatUrpGVV1gt7VDg==/lib/arm64/libandroidappmusic.so (Java_com_apple_android_music_renderer_javanative_LevelComposer_compose+268) (BuildId: b52cdcc1f1daf9a96aaaa911f060d467d7b54552)
      #04 pc 0000000000049d7c  /system/framework/arm64/boot-core-libart.oat (art_jni_trampoline+140) (BuildId: e221ad2e1579dba8c552c66ad1435db15dd3b38c)
      #05 pc 00000000025b4a64  /data/app/~~qvCT42FscwNrFow1psc79g==/com.apple.android.music-SUeV4RIatUrpGVV1gt7VDg==/oat/arm64/base.odex (com.apple.android.music.playback.player.PlayerAudioFadeControl.setComposerTransition+1092)
      #06 pc 00000000025ae7d0  /data/app/~~qvCT42FscwNrFow1psc79g==/com.apple.android.music-SUeV4RIatUrpGVV1gt7VDg==/oat/arm64/base.odex (com.apple.android.music.playback.player.PlayerAudioFadeControl$computeTransitionJob$1.invokeSuspend+304)
      #07 pc 0000000003541ef8  /data/app/~~qvCT42FscwNrFow1psc79g==/com.apple.android.music-SUeV4RIatUrpGVV1gt7VDg==/oat/arm64/base.odex (Vb.a.resumeWith+152)
      #08 pc 000000000362b070  /data/app/~~qvCT42FscwNrFow1psc79g==/com.apple.android.music-SUeV4RIatUrpGVV1gt7VDg==/oat/arm64/base.odex (yd.T.run+1024)
      #09 pc 0000000000214c54  /system/framework/arm64/boot.oat (java.util.concurrent.ThreadPoolExecutor.runWorker+676) (BuildId: 5e31ba31fc4f779978b4ee5a158a82f8b97f6aad)
      #10 pc 0000000000218bf8  /system/framework/arm64/boot.oat (java.util.concurrent.ThreadPoolExecutor$Worker.run+56) (BuildId: 5e31ba31fc4f779978b4ee5a158a82f8b97f6aad)
      #11 pc 00000000000a94f0  /system/framework/arm64/boot.oat (java.lang.Thread.run+64) (BuildId: 5e31ba31fc4f779978b4ee5a158a82f8b97f6aad)
      #12 pc 00000000002faf94  /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: c3d2bff6e80fd46838179f7753be74fa)
      #13 pc 00000000002e59dc  /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+220) (BuildId: c3d2bff6e80fd46838179f7753be74fa)
      #14 pc 000000000042b08c  /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+940) (BuildId: c3d2bff6e80fd46838179f7753be74fa)
      #15 pc 000000000042accc  /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallbackWithUffdGc(void*)+12) (BuildId: c3d2bff6e80fd46838179f7753be74fa)
      #16 pc 000000000008e254  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+180) (BuildId: e688446b48f00a577c82f6e36cac2c6e)
      #17 pc 000000000007f9f4  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68) (BuildId: e688446b48f00a577c82f6e36cac2c6e)

The good news is that crashes are the exception. I have MTE force-enabled for nearly all apps on my Pixel 8a with GrapheneOS, and I have seen very few crashes in the past year.

How to enable MTE for your Android or iOS app

As a user, there is little you can do. At the minimum, you need to buy a Pixel 8 or iPhone 17 (and later) to have the hardware support, and rely on Google/Apple to enable MTE for the operating system. On Android, you can also enable Advanced Protection or install GrapheneOS. For manually installed apps, you are reliant on app developers enabling MTE for their apps (ask them about it!).

As an app developer, you can opt-in your app into using MTE (by setting the manifest flag or the entitlement, see above). Especially if you don’t have native code (only Java/Kotlin), this should be easy and not cause any issues. If you have custom native code, MTE may even detect some bugs that you were previously not aware of.

Important: before you ship a release with MTE enabled, thoroughly test your app on a supported device (Pixel 8 and iPhone 17), and fix any crashes that you find.

Should you enable MTE for your app?

Yes!

It is easy to do, and should not impact your app much. You likely won’t notice the performance impact (especially on iPhone), and crashes are few (in my experience on GrapheneOS).

MTE crashes only surface existing bugs that you didn’t know about and should fix anyway. So MTE is also a useful bug finding tool.

In Java, when you try to access data outside of an array, you get an ArrayIndexOutOfBoundsException. In C, writing data outside of a buffer may sometimes work, but it silently corrupts memory. This corruption can lead to crashes much later, which are very difficult to debug! With MTE, your C code instead crashes immediately¹, just like an ArrayIndexOutOfBoundsException would crash your app. The crash is not really the fault of MTE, you need to fix your code. So don’t be “scared” of MTE, just test your app properly on MTE-enabled hardware.

Interestingly, even if your Android app is written in Java/Kotlin and has no native bundled into the APK file, at runtime your app’s process may call Android framework functions that use native code, which can have memory vulnerabilities. Therefore, even if you don’t explicitly use native code, MTE can still protect your app. For a real-world example, see this bug in the Signal messenger that was detected by MTE.

Especially apps that handle input that can be controlled by a remote attacker over the network should enable MTE. Among others, this includes web browsers, messenger apps, e-mail clients, social media, file sharing and cloud storage apps, and media streaming (video, music, podcasts). These apps allow attackers to remotely and relatively easily cause your phone to load malicious content (by sending a message, sharing a file, tricking you into opening a website or watching a video). And unfortunately, media parsers are a common source of vulnerabilities, because image/audio/video formats are very complex. For example, see the 2021 FORCEDENTRY and the 2023 BLASTPASS exploits.

No silver bullet

None of the discussed defenses are magical solutions:

• Both Pointer Authentication (PACMAN) and MTE (StickyTags, TikTag) have been vulnerable to side-channel attacks.
• MTE only applies to main memory. GitHub researchers demonstrated an attack that (ab)uses GPU memory.

Nevertheless, these defenses raise the bar and eliminiate entire classes of attacks. This forces attackers to come up with more complicated exploitation techniques. MTE raises the cost of attacks.

Conclusion

This post gave a high-level introduction into Arm MTE. MTE is a strong defense against memory vulnerabilities. Hardware support is still limited to a few devices, but will hopefully grow. MTE already protects parts of the operating system and system apps. To protect user-installed apps, app developers need to manually enable MTE and test their apps with it.

Here is what you can do next:

• If you want to dive deeper into the technical details, continue reading with the resources linked below.
• If you are a security-conscious user, consider buying a device with hardware support for MTE, such as a Google Pixel 8 (and later) or an iPhone 17 (or another Apple device with an A19 chip).
• If you are a mobile app developer, opt-in to MTE now!

Additionally, if you have more information about the unclear-to-me points mentioned in this post, please let me know! In particular:

1. For which parts does Android Advanced Protection enable MTE (additionally to what is already enabled anyway)? The kernel? What else?
2. Do Advanced Protection mode and Lockdown mode force-enable MTE for user-installed apps?
3. What is the state of MTE on other Android phones? Are other OEMs considering it?
4. What is the state of MTE on Apple Silicon M-series chips? Is it coming?

Further reading

For a deeper technical dive into MTE, I recommend the following resources (accessible at the time of writing, 2025-09-16):

• General:
  • https://www.usenix.org/system/files/login/articles/login_summer19_03_serebryany.pdf
  • https://www.youtube.com/watch?v=9wRT2hNwbkA
  • https://www.youtube.com/watch?v=KmFVPyHyfqQ
• Android:
  • https://source.android.com/docs/security/test/memory-safety/arm-mte
  • https://developer.android.com/ndk/guides/arm-mte
  • https://cs.android.com/android/platform/superproject/main/+/main:bionic/docs/mte.md
  • https://googleprojectzero.blogspot.com/2023/08/mte-as-implemented-part-2-mitigation.html
• iOS:
  • https://security.apple.com/blog/memory-integrity-enforcement/
  • https://developer.apple.com/documentation/xcode/enabling-enhanced-security-for-your-app
• Linux:
  • https://www.kernel.org/doc/html/v6.12/arch/arm64/memory-tagging-extension.html
  • https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/mte/mte/
• Arm:
  • https://developer.arm.com/-/media/Arm%20Developer%20Community/PDF/Arm_Memory_Tagging_Extension_Whitepaper.pdf

Appendix: hardware support

In this section, I will try to briefly summarise my understanding of the hardware support for MTE.

Instruction set architecture (ISA)

MTE was first introduced to the instruction set architecture (ISA) in Armv8.5. In Armv8.7 and Armv8.9, MTE was extended with additional features. (Apple’s MIE is based on FEAT_MTE4 in Armv8.9) Armv9 contains no changes to MTE.

Notably, MTE is optional in all of these ISA versions. This means that chip vendors can produce a chip that is Armv9-compliant, but does not contain MTE!

Chips

Arm’s own Cortex CPU design has MTE starting from the first Armv9-compliant Cortex CPUs.

However, Arm does not actually produce these CPUs! Instead, other vendors license the CPU design and build their own system-on-a-chip (SoC) with it. Prominent examples are Google Tensor, Qualcomm Snapdragon, Samsung Exynos, and Apple Silicon. Some of these are based on Cortex (with varying degrees of modification) while others (like Apple Silicon) are independent.

This breadth of different chips is why so far no Android phones except the Google Pixel have support for MTE. According to GrapheneOS, some high-end Samsung Exynos and MediaTek chips have MTE support, and Qualcomm Snapdragon is planning it. On the server side, Ampere has recently added MTE support. See also this overview table maintained by a RedHat engineer.

Therefore, the difference between what features the ISA describes and what the physical chip implements is important.

There is some confusion online about when MTE was “introduced”. For example, the Linux kernel docs say Armv8.5, while the Android NDK docs say Armv9. This confusion comes from the ISA–chip mismatch. Even though Armv8.5 defined MTE, there were no chips that supported it. The first Cortex designs with MTE support happened to already support the Armv9 ISA.

The most reliable way to check if your chip supports MTE is to query it:

adb shell grep mte /proc/cpuinfo

My Google Pixel 8a reports to support mte mte3.

The article: https://www.woz.ch/2525/digitale-oekonomie/die-gatekeeper-umgehen/!M44FC5J9CXJP

I have been using F-Droid for the past ~8 years. Over time, I increasingly contributed to it: from small app metadata fixes, to website edits, to a tool visualizing app download statistics (see the graphs assembled by Tavi).

In the current 1.23 release of the F-Droid Android client, I rewrote the repository details screen in Jetpack Compose. Not only does it look more modern and more integrated with the rest of the app (compare it to the app details screen), but also mirror management has become a lot easier. Previously, the mirror list was a fixed-height nested scroll view…

And as it happens, if you get the WOZ app from F-Droid you will use my new repository details screen! Since the WOZ app is not in the main repo but in a custom repo.

Repository details before and after

Before:

After (collapsed):

After (expanded):

You can find an interactive version here.

The source code is available on Github.

For each speed, only the lowest known base price (at that point in time) is included. Reductions from the base price are not considered (temporary promotions, combining home internet with a mobile subscription, …). There are too many possible combinations, they change too often, and they are too intransparent to track without a lot of effort.

Note that price is only one factor when choosing your internet. E.g., you may want to consider: the physical architecture (P2P/AON vs. P2MP/PON), the backhaul capacity of the POP you are connected to, the connectivity of the ISP towards other ASes (good peering and internet exchange capacity), or the quality of customer support.

Please let me know if there is any base price data that I missed!

Paying for the physical fiber

ISPs don’t build their own fiber to each house. That would be insanely expensive and simply redundant. It would also require a huge amount of capital for a new ISP to enter the market. Fiber should be a public utility, just like water and electricity. Politicians and the resulting regulations don’t always see it that way, though…

ISPs rent fiber access from Swisscom. Swisscom (through their subsidiary Cablex) builds and operates the fiber, often in cooperation with the local municipial utility company (Stadtwerke), e.g. see Zürich. They do this until the building entrypoint. The cabling inside the building is also done by local electricians.

Swisscom’s product for renting fiber access is called Access Line Optical (ALO). The prices for ALO are the following:

• 24 Fr. per month recurring fee
• 107 Fr. one-time fee for setting up a new connection

For a new connection, a Swisscom (!) employee has to physically go to the switchboard building in your town or neighbourhood and plug the fiber coming from your home into the equipment of your ISP.

So even if your ISP is not Swisscom, 24 Fr. of what you pay every month still goes to Swisscom for operating Layer 1.

Cost of running a PoP as an ISP

init7 has a blog post where they break down what it costs them as an ISP to build and operate the equipment for a point-of-presence (PoP).

Even if all of your other software is proprietary, your documentation is a great candidate to be open sourced. For several reasons:

1. Everything is already public

Most docs today are some flavour of Markdown that is compiled with a static site generator to a static website. Unless you wrote a custom static site generator, there is nothing novel about your docs site. The content is public, and the site generator is probably FOSS anyway, so the git repository can be public, too.

2. Docs are easy to contribute to

Fixing a software bug requires writing code. Contributors need to set up a build environment, which is often time-consuming. Personally, I mostly contribute bug fixes to Android projects, because I already have the Android toolchain installed on my laptop. For other software (iOS, desktop, NodeJS, Ruby, etc.) I only open issues.

By contrast, fixing a typo in the docs requires editing a Markdown file. All I need is GitHub/GitLab. I can do everything via the web GUI: fork your repo, create a branch, edit the file, create a pull request. I can even preview the rendered Markdown directly in GitHub/GitLab.

This makes documentation a natural candidate for getting started in open source. For new contributors, but also for your company (if it is new to doing open source).

3. The git-based workflow is more time-efficient (for you and me)

Documentation fixes are often small, from a simple typo to a few lines of text.

For me, it takes roughly the same amount of time to “fork, edit, make PR” as it takes me to “write e-mail to support, explain to them where in the text the issue is, what the correct text should be, and convince them that ’no, I don’t actually have a question, I just want to report an issue’.” Both approaches take the same amount of time, but git is much less stressful for me. It is more precise, less verbose, and less back-and-forth.

For your team, git saves A LOT of time. Reviewing my PR with a few lines of English text in Markdown is trivial. Review, press merge, your CI deploys. 5 minutes of effort. On the other hand, consider this: having your support staff answer my e-mail, forwarding the information to your technical writing team, creating a ticket, having them do the work of making the edits, committing them, creating a PR, moving the ticket to done. That is a lot more work. Work that you can outsource to me save by simply open sourcing your docs.

When not to open source docs

There are some situations where you may not want to open source your docs, at least not directly:

a. Keeping unpublished drafts private

This blog is currently not open source. This is because I often write posts to get them out of my brain, but later decide not to publish them. I like keeping them around and tracked in git for my own reference, but I don’t want the world to read my half-finished thoughts.

In technical writing, too, you can encounter the need to keep documentation private: e.g., while preparing documentation for an upcoming, not-yet-announced feature.

And in software development this also happens: e.g., the fixing of embargoed security issues often happens on a private fork.

Your choice depends on how often you find yourself in such situations: are they one-off or regular? Therefore, do you keep all docs closed, or do you set up a private repo when the need arises?

Conclusion

Note that open source != FOSS. You can publish your docs under whatever license you like, and optionally have contributors sign a CLA.

Additionally, you can keep your existing processes of getting feedback from non-git-savy customers via your support portal and your feedback forms, and manually feed that back into your docs. Handling PRs just complements these methods.

As a developer/sysadmin/engineer/etc, I use publicly available docs every day. Whenever I stumble over an issue, I like to see it fixed, so that the next person doesn’t have to suffer the same confusion. I have made PRs with doc fixes to projects as big as GitLab and Google’s Flutter, but also smaller to ones like OPNsense. This would not have been as smooth and pleasant for both me and their product/dev teams had the docs not been open source.

So please make it easy for drive-by contributors to fix your docs – open source them!

The Internet is a mess

E-Mail is a mess. It has been around since the 80s and has been implemented hundreds, if not thousands, of times by different people, in different ways, with different levels of compatibility.

But: everything in the internet that has to be interoperable and that is older than 1 year is a mess. As humans, we never get a design right on the first try. People want new features. Security issues are found. So a new scheme is designed. But because not everyone will upgrade to the new scheme (because of cost, unawareness, loss of skilled staff, or just not caring), the updated scheme has to be backwards compatible with the old scheme. And what is worse, many did not even implement the old scheme correctly.

Thus over time, as new requirements come in, the whole thing inevitably evolves into a huge pile of patches.

E-Mail is not alone here, you can find this pattern in most, if not all, (internet) protocols. Just look at the Internet Protocol (IP) and NAT: even 25+ years later, many big organisations are still not migrating to IPv6 – not successfully, or not at all. (I have studied and worked at such organisations. In 2025, there are still IPv4-only mobile carriers in Switzerland.) So instead of IPv6, we got NAT. And because of NAT, we got STUN, TURN, and ICE. (To learn more about NAT, I highly recommend Eric Rescorla’s four-part series.)

In fact, we are moving towards a new narrow waist of the OSI model.

NAT is an example where it was made to work: WebRTC works, your Jitsi/Zoom/Teams/Signal/WhatsApp calls work. Even though under the hood, there is a pile of protocol patches trying to find a way through your home network’s NAT sitting behind a CGNAT, because your ISP could not be bothered.

It is easy to say “let’s abandon e-mail and build something better and more modern”. If your new thing is centralised: great, that will work for you. But if your goal is to build something that is interoperable (“federated”) across different entities, then your system will, over time, evolve into a similar mess as all the other protocols before.

Plus-aliases

Back to e-mail: one of my pet peeves is the so-called “plus-alias”.

If my main email address is “john.doe@example.com”, then “john.doe+shopping@example.com” and “john.doe+banking@example.com” are two plus-aliases. You can suffix your username (the local-part) with “+” and some label. These new emails will be delivered to your normal inbox.

This is NOT supported by all email providers. GMail and Proton Mail support this, but your university or work email might not.

This is where internet standards¹ come in:

1. The “+” character is allowed in the local-part. In fact, like it or not, all of these characters are allowed in the local-part:

! # $ % & ' * + - / = ? ^ _ ` { | } ~

So as far as the IETF standards are concerned, john!#$doe%&'=?{shopping+bank}@example.com is a valid e-mail address.

2. The local-part is locally interpreted. “[…] it is simply interpreted on the particular host as a name of a particular mailbox” (RFC 5322). So if GMail decides to interpret the “+” as an alias, and another mailserver interprets it as separate mailboxes, this is completely up to them. It is NOT up to the sender to make any decision about this.

All of this is defined in RFC 5322, which defines the format of Internet Messages. It superceded RFC 2822, which in turn superceded the original RFC 822, written for the ARPANET in 1982.

People ignore standards

Unfortunately – and this part is my pet peeve – there is a lot of software that thinks that “+” is not a valid character in the local-part.

I have seen this fail in so many wonderful ways:

• In the best case, the input form (wrongly) claims “this is not a valid e-mail”.
• Some sites will let me register an account with a plus-alias e-mail, but then won’t let me log in with it.
• Some sites silently strip the “+” (“john.doeshopping@example.com”). This is obviously a different address, so the mailhost will interpret it differently. I had to register a new e-mail account once in order to regain access to my domain registrar after their internal migration stripped my “+”.
• Putting the email address in an URL, but not URL-encoding the address, (+ would need to be encoded as %2B). This often happens in links such as “click here to confirm your e-mail address for this account”. And it breaks those links.

Companies have lost business over not supporting plus-aliases. I have turned to alternatives or have lost interest completely when a site didn’t let me register with a plus-alias.

Why bother?

The easy way would be to just stop using plus-aliasses. However, while it is annoying to recover from these bugs, I find it a very useful first indicator of how well (or badly) written the software is.

When they implemented e-mail handling, the people writing those softwares:

• Either never read the standards about e-mail. After all, “+” has been an allowed character ever since e-mail was designed in 1982.
• Are not e-mail power users, otherwise they would have been aware that GMail supports plus-aliases since at least 2008, i.e., 17 years ago.
• Or they knew, but had some technical reason or constraint to deny “+” anyway.

While I see some reason to ignore some characters (like / ` { | }), plus is not one of them. Plus-aliases are widely used:

• By “power users” – GMail has supported them for 17 years.
• By mailing list software. Look for headers like List-Unsubscribe: <mailto:listname+unsubscribe@example.org>
• By ticketing software. Look for headers like Reply-To: Acme Corp <support+id123456@acmecorp.zendesk.com>

So your software, too, ought to allow “+” in e-mail addresses. If only for compatibility with, you know, GMail.

Conclusion

Interoperable things, like e-mail, will grow to a bunch of patches over time. You can contribute to ameliorate the situation by reading the existing standards and actually implementing them. If you do that, you will make your customers happier. If everyone did that, it would make the work of the people who have to put patches on top of the existing pile a tiny bit easier. Thank you!

Often, maintainers upload precompiled assets for each release. Depending on the project, this can be *.msi, *.deb, *.rpm, or *.apk files. Sometimes maintainers also upload *.sha256 or even *.gpg files to accompany these assets. In addition, GitHub automatically adds both a *.zip and a *.tar.gz archive of the source code at the corresponding git tag to each releases’ assets.

Today, I am interested in getting the download count of these release assets. GitHub does not show this in the UI, but they do expose it in their API:

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/OWNER/REPO/releases/tags/TAG

This way, we can e.g. learn that at the time of writing, ~4100 people have downloaded the *.apk for Signal’s Android app v7.40.0.

Of course, not all of these downloads will be real people (and the data is easy to maliciously increase, if that’s what you want to do). But mostly, the order of magnitude is correct.

If you have too much time, you can pipe this data into jq for further processing, build a pretty web page with graphs around it, and whatever else you may do with this data. For my curiousity – which is getting a quick insight into the scale of things – a | grep download_count also does the job.

And if you don’t have curl handy (don’t you have a Linux VM on your phone?), you can get the same data via a shields.io badge, directly from your browser: https://img.shields.io/github/downloads/signalapp/signal-android/v7.40.0/total

<rant>

Fiber7 costs 65 CHF/month, which is a medium market price in Switzerland. 10G and 25G have the same monthly price, only the setup fee differs, because the hardware cost differs. 65 CHF is not too expensive (there are more expensive offerings), but also not the cheapest (there are cheaper offerings, but at worse service quality).

Especially when you look beyond the Swiss border and into Germany, and you see Deutsche Telekom offering Magenta 2000: 2G down, 1G up for 140 €/month. Twice the price for 1/10th of the speed.

What is more, this speed smells a lot like GPON. But on the announcement Telekom says that they have XGSPON equipment on their end and that you need a XGSPON-capable router. But XGSPON can carry symmetric 10G! So is Telekom throttling XGSPON to GPON? Maybe. Ein Schelm, wer Böses denkt. Telekom is clearly leaving itself room to give customers an artifical speed upgrade in the future (or even to sell the upgrade at a premium).

</rant>

All of this is to say: I am very happy init7 exists and provides this great service. Now I just need to get a router.

A number of other people have also written about their home router builds for their init7 connection:

• Michael Stapelberg
• Scott Dier
• James Kavakopoulos

Check them out! I got inspired by their blog posts, too. This post is just yet another home router build.

A small note on hardware before we start: There are two form factors for network cables. RJ45 (the classic) and SFP (and its variants). At 10G, people tend to use both SFP+ (data center) and RJ45 (home lab) connectors. At 25G, however, SFP28 is the go-to option (but Cat8 would work, too). Wikipedia has a full table showing all the different SFP types. Here is an extract:

Initial reconnaissance

My original plan was to simply go for 10G and buy an off-the-shelf appliance as a router: the DEC740 from Deciso, the makers of OPNsense, the router and firewall I wanted to use. It is fanless, quiet, and has a low power consumption of ~15W, and costs 750 €. The simple router boxes other ISPs give their customers are also in the 10-15W range.

But I needed to buy a network card anyway to connect my existing NAS to the router. So I started shopping. Initially, I was searching for 10G cards. Most 10G cards cost around 200–250 CHF. SFP28 cards tend to cost 300–350 CHF. (As of December 2023, YMMV.)

Eventually, I came across the Mellanox ConnectX-4 Lx on Digitec:

Or rather, I took this screenshot too late. I actually came across it a few days earlier when it was available not for 145 CHF but for 115 CHF! (This likely was some temporary deal from supplier jacob.de.)

The Mellanox ConnectX-4 Lx has two SFP28 ports, not SFP+ ports. I was hooked. If this 25G card is cheaper than the 10G cards (115 < 200), then let’s buy this one! And if my NAS has a 25G networking card, it would be a shame if my router only had 10G, right? Sure, SFP28 is backwards compatible with SFP+, but still.

Thus, I gave up on my “buy an off-the-shelf appliance and be done with it” plans and started researching a custom router build. Spoiler: at the end, it was way more expensive than the DEC740. But I learnt a lot about the hardware, and it was enormous fun doing the shopping research for parts and finally the assembly!

Settling on a NUC

My first idea was to build a tower-sized computer. Michael Stapelberg’s blog post drew my attention to the AMD Ryzen 7 Pro 5750GE, with only 35W TDP! Unfortunately, it was out-of-stock wherever I looked.

Later, Scott Diers’s build with the Intel NUC 9 Pro inspired me to take a look at NUCs instead of a full tower. I liked the smaller form factor. Before reading Scott’s post, I simply wasn’t aware that there exist NUCs that have PCIe slots (and you need a PCIe slot for the network card).

The reason that these NUCs have a PCIe slot is that they are designed for gamers. We can (ab)use that: instead of putting in a graphics card, we can put in a networking card!

Sadly, the “Pro” series that Scott used does not come with a PCIe slot anymore (for the 11, 12, 13 versions). Therefore, I had to go for the more expensive “Extreme” series. The Pros cost around 500 CHF while the Extremes cost around 1000 CHF…

So let’s compare the two most relevant NUC Extremes. The i9 variant of the NUC 12 Extreme and the NUC 13 Extreme compare as follows (an i7 variant also exists):

In all other regards (that are relevant to me) they are similar: 1 PCIe x16 Gen5 slot, 2 Thunderbolt 4 ports, lots of USB-A ports, up to 64 GB RAM (though 12 is DDR4, 13 is DDR5), 3 M.2 slots, two RJ45 ports (2.5 Gbit/s and 10 Gbit/s).

The only difference between the i9 variant and the i7 variant (other than the CPU) is the 10 Gbit/s port, which is only present on the i9 variant. I chose the i9 variant because I wanted the additional RJ45 port.

In the end, I chose the NUC 12 Extreme simply because the NUC 13 Extreme was not available in a reasonable timeframe from any online store. Note that the TDP is lower, but the TDP doesn’t say much about the idle power consumption.

Having a 16-core home router is a waste. Let’s use it for other computing tasks as well! I decided to not build a router anymore, but instead build a server that happens to be a router. In practice, this means that the bare metal is running Proxmox as a hypervisor, and one of the VMs is running OPNsense as the router and firewall.

In total the server will feature: 16 cores, 64 GB RAM, 8 TB SSDs (but as RAID-1, so only 4 TB usable). A compute-heavy complement to my existing storage-heavy NAS (which I built a few years ago).

(Granted, it is still a desktop CPU not a server CPU. It has “only” 8 performance cores, the other 8 are energy-efficient cores. An Intel Xeon or AMD Epyc CPU is a very different beast.)

The final build

Fast forward, I ended up with the following parts for the router server:

To connect the router to the optical termination outlet (OTO) of my flat, and to upgrade my NAS to 25G and connect it to the router, I also bought the following parts:

I bought all of the above parts on digitec.ch, amazon.de, and fs.com.

You can also get the fiber patch cable and the transceiver directly from init7, for 77 CHF (10G) or for 222 CHF (25G) (because SFP28 transceivers are most expensive). With a referral code, you get 111 CHF off the hardware.

If you (like me) buy the parts yourself, you need to do your research to make sure you buy the right cable and transceiver. The wall outlets (OTO) in Switzerland are generally LC APC. SFP transceivers are almost always UPC (aka. PC), but can be either SC or LC. APC is green, UPC is blue. fs.com’s blog is a great resource to learn about the various types of cables and connectors. For the transceiver, follow the specs provided by init7 here and here. People seem to buy fiber equipment either on fs.com or on flexoptics.net.

Assembling it

Here are a few photos from assembling the server.

The inside: You can see the two SSDs on the left, the big silver heat sink that hides the CPU behind it, and the two pieces of RAM on the right. The box on the very right (facing the front of the NUC) is the power supply. Lying at the front, you can see the box with the fan that goes on top of the CPU/SSD/RAM. The two grey bars will go on top of the SSDs, meaning that you don’t need dedicated heatsinks for the SSDs (also, they wouldn’t fit).

Installing the Mellanox card in the blue PCIe slot:

The view of the back: The top RJ45 port is 10 Gbit/s, the bottom one is 2.5 Gbit/s. On the right, the two SFP28 ports of the Mellanox card. Matchstick for scale.

Installing it

This is what the NUC looks like with everything installed:

And the back: The yellow fiber cable goes into the wall and out to init7. The black Direct Attach Cable (DAC) below goes into my NAS. The blue copper cables go into switches.

Because the NUC Extreme is designed for gamers, it has LEDs (of course it does):

Luckily, there is a physical button on the bottom of the NUC allowing you to turn the LEDs on and off. No need to install any drivers.

As you can see e.g. on its Digitec page, the front of the NUC usually features a skull. Weird taste, probably reaching only a subset of gamers, and an even smaller subset of the general population, but okay. Luckily, the skull is just a small semi-transparent mask. You can unscrew the front of the panel and take the skull mask out, leaving you with the full, square LED front (as seen in my picture). (I learnt this from this review of the Intel NUC 11 Extreme.) Optionally, you can cut your own thin sheet to mask the light to another shape. I haven’t tried this yet since I leave the LEDs off anyway.

Speedtest

So does it work? Does it make the internet go swoosh?

First, an iPad Pro, with a 2.5 Gbit/s USB-C to RJ45 adapter:

(Note that unlike Android, iPadOS doesn’t have a dedicated status icon in the top right for Ethernet (like for WiFi).)

Next, the system monitor on my laptop when connected with the 2.5 Gbit/s adapter (290 MiB/s = 2430 Mbit/s):

The first half shows an iperf3 test to speedtest.init7.net. The second half shows a speedtest.net test in Firefox. Clearly my CPU (a 2 core, 4 threads i5 7th gen) is the bottleneck. Speedtest.net, Firefox, the Snap sandbox, the network stack, or something else somewhere, makes the CPU go more spinny than iperf3 in a terminal does. Either I can buy a new latptop, or someone can fix the software.

Apropos iperf3 in a terminal. Here is iperf3 running on the shiny new router:

A mere 15–16 Gbit/s. Far from the 25 Gbit/s we were hoping for. But at least more than 10 Gbit/s, so not a complete waste, I guess?

Doing an iperf3 between the new router and the old NAS, both with their SFP28 cards and the SFP28 DAC cable, I also only get ~15 Gbit/s. So the bottleneck is not init7, but somewhere on my setup. I just need to find the time to track it down. Using PCI passthrough for the NIC (since OPNsense is running in a VM) already improved the performance by ~1 Gbit/s compared to using Proxmox’ virtual bridge.

Conclusion

In this post I described my journey to building a home router. Compute-wise, the new server is more powerful than originally envisioned. That’s fine, I will use it to run VMs, e.g. a remote desktop to extend the lifetime of my old laptop. Speed-wise, the router is practically capable of 15 Gbit/s, and theoretically capable of 25 Gbit/s (once I managed to tune it).

Do I need 15 Gbit/s? No. Working, reliable 2.5 Gbit/s would be nice though.

In practice, the download performance for many things is still at most 1 Gbit/s, and 1.5 Gbit/s if you are lucky. I briefly tested this by downloading various installation images (Debian, Ubuntu, LineageOS, GrapheneOS).

It is a chicken-and-egg problem. If no one has fast internet at home, no one will provide fast servers, and no one will optimise the software stacks and transport protocols to “just work”. By having the hardware for 25 Gbit/s, we have eliminated the first bottleneck. Next, we need to eliminate the software and configuration bottlenecks.

Hopefully, as more and more people have faster internet, more developers will optimise for it, and those speeds will become usable out-of-the-box. I am looking forward to the day when I can download the 10 GB folder of photos my friend shared with me from our holiday trip at 2.5 Gbit/s in 30 seconds, and not in 2 minutes at the currently realistic 0.7 Gbit/s.

And maybe even in 8 seconds at 10 Gbit/s, if I decide to spent way too much money on a Thunderbolt-to-SFP+ adapter for my laptop.

One solution is to meet up in person and verify their public key directly. This is what Signal and WhatsApp do with their Safety Numbers.

But this doesn’t scale. How would you meet up with Google to verify their website’s public key? This is where certificates come in.

A Certificate Authority (CA) verifies the identity (“thore.io”) and that this identity controls a keypair (pk, sk). The CA then signs a certificate that basically states “I, the CA, certify that the secret key belonging to the public key pk is controlled by the entity thore.io”. Now anybody who trusts that the CA is honest and that it correctly verifies identifies before issuing certificates can also trust that this public-key-to-identity binding is correct.

But what is the flow of creating certificates? What are the data structures?

Certificates are often seen as this big complex beast. And correctly so! The many extensions and options mean that it gets complicated fast. But the core data structures are not that complicated.

This post tries to break down these data structures. Using the data structures to guide us, we step through the process of creating a certificate.

Step 1: Certificate Request

It starts with what is colloquially known as “Certificate Signing Request”. It usually uses the following PKCS#10 data structures (defined in RFC 2986).

PKCS#10 looks like this:

CertificationRequestInfo ::= SEQUENCE {
    version       INTEGER { v1(0) } (v1,...),
    subject       Name,
    subjectPKInfo SubjectPublicKeyInfo{{ PKInfoAlgorithms }},
    attributes    [0] Attributes{{ CRIAttributes }}
}

CertificationRequest ::= SEQUENCE {
    certificationRequestInfo CertificationRequestInfo,
    signatureAlgorithm AlgorithmIdentifier{{ SignatureAlgorithms }},
    signature          BIT STRING
}

If subject S has a keypair and wants to obtain a certificate, it first creates a CertificationRequestInfo and fills out the subject name (“thore.io”) and the details of their public key. Then S signs the CertificationRequestInfo and together with the signature puts it into a CertificationRequest.

Step 2: X.509 Certificate

S then sends the CertificationRequest to a CA of their choice.

Now the CA needs to verify that the subject S really controls the public key S wants to bind to its identity. It can e.g. do this using ACME (defined in RFC 8555).

When the CA is satisfied it issues the certificate. Certificates generally use the X.509 format (defined in RFC 5280).

X.509 certificates look like this:

TBSCertificate  ::=  SEQUENCE  {
    version         [0]  EXPLICIT Version DEFAULT v1,
    serialNumber         CertificateSerialNumber,
    signature            AlgorithmIdentifier,
    issuer               Name,
    validity             Validity,
    subject              Name,
    subjectPublicKeyInfo SubjectPublicKeyInfo,
    issuerUniqueID  [1]  IMPLICIT UniqueIdentifier OPTIONAL,
    subjectUniqueID [2]  IMPLICIT UniqueIdentifier OPTIONAL,
    extensions      [3]  EXPLICIT Extensions OPTIONAL
}

Certificate  ::=  SEQUENCE  {
    tbsCertificate       TBSCertificate,
    signatureAlgorithm   AlgorithmIdentifier,
    signatureValue       BIT STRING
}

The CA first fills out the TBSCertificate (“to-be-signed” certificate). It copies over the values from the CertificationRequest, assigns a new serial number, chooses a validity period, and adds any extensions that it wants.¹

The CA then signs the TBSCertificate and puts everything together into the Certificate. We now have an X.509 certificate!

Step 3: Signed Certificate Timestamp

Back in the day, we would now be done. The CA returns the X.509 certificate to the requester subject S who can use it to prove their identity (e.g. in a TLS handshake).

However, after a series of CAs misbehaving and wrongly issuing certificates Google introduced Certificate Transparency (CT). With CT the CAs need to insert every certificate that they issue into a public append-only log. For efficiency, this log is a Merkle Tree with the leaves being the log entries.

CT is defined in RFC 9162. The relevant data structures are:

struct {
    VersionedTransType versioned_type;
    select (versioned_type) {
        case x509_entry_v2: TimestampedCertificateEntryDataV2;
        case precert_entry_v2: TimestampedCertificateEntryDataV2;
        case x509_sct_v2: SignedCertificateTimestampDataV2;
        case precert_sct_v2: SignedCertificateTimestampDataV2;
        case signed_tree_head_v2: SignedTreeHeadDataV2;
        case consistency_proof_v2: ConsistencyProofDataV2;
        case inclusion_proof_v2: InclusionProofDataV2;
    } data;
} TransItem;

opaque TBSCertificate<1..2^24-1>;

struct {
    uint64 timestamp;
    opaque issuer_key_hash<32..2^8-1>;
    TBSCertificate tbs_certificate;
    Extension sct_extensions<0..2^16-1>;
} TimestampedCertificateEntryDataV2;

struct {
    LogID log_id;
    uint64 timestamp;
    Extension sct_extensions<0..2^16-1>;
    opaque signature<1..2^16-1>;
} SignedCertificateTimestampDataV2;

First the CA sends the X.509 certificate to a CT log operator. (Alternatively the CA can also send a “precertificate”. This signals the CA’s binding intent to later issue a certificate.)

The log operator extracts the TBSCertificate from the X.509 certificate (after verifying the signature). The log operator then builds the TimestampedCertificateEntryDataV2 and puts it into a TransItem of type x509_entry_v2. This TransItem will eventually be inserted into the log (as one of the Merkle Tree leaves).

The log operator the creates a “Signed Certificate Timestamp (SCT)”: It signs the TransItem and puts the signature into the SignedCertificateTimestampDataV2. The timestamp and the sct_extensions are copied over from the TimestampedCertificateEntryDataV2. With this signature a log operator promises to include the corresponding TimestampedCertificateEntryDataV2 in the log.

Note that the SCT does not contain the certificate. Thus the SCT is only useful in combination with a certificate. Also, the SCT is not included in the log, only the certificate.

The log operator then returns the SCT to the CA.

Step 4: X.509 Certificate again

Chrome and Safari require CT and will only accept TLS connections with certificates that are included in two or more CT logs. Firefox does not enforce CT (at the time of writing in July 2023).

The browsers enforce this by verifying the >= 2 SCTs together with the certificate. There are three ways for the webserver to send the SCT to the browser: include the SCT directly in the certificate, or serve the SCT over OCSP, or as a TLS extension (see Section 6 of RFC 9162).

If the SCT is directly embedded into the certificate, the flow is slightly different: the CA does not submit a X.509 certificate to the log but instead it submits a precertificate. When the CA gets the SCT from the log operator, includes the SCT in the extensions field of the X.509 certificate. Only then the CA signs and issues the X.509 certificate.

Finally, the CA returns the X.509 certificate and the SCT to the subject.

Conclusion

This post gave a brief overview of the process of creating certificates and the data structures that the certificate data flows through.

We started with a Certificate Request, then issued an X.509 certificate, and then also obtained an SCT.

If you want to dive deeper read the linked RFCs.

If you want to inspect PEM/DER-encoded ASN.1 objects (e.g. a CSR) you can use openssl asn1parse. Alternatively der2ascii (Github) provides a nice hierarchical view.

TLDR: Don’t log sensitive information. Not on Android, and not anywhere else. Still, in 2022 I found two Android apps that do exactly that.

This post starts with an introduction to logging on Android. I describe common pitfalls that lead to sensitive information being leaked to Android logs. I describe why this is bad and what impact it can have. Finally, I show two real-world apps that were/are leaking sensitive information in this way.

Logs on Android

Logs are useful because they allow developers to trace application behaviour. It allows developers to understand what happened to an application, what state it was in, what events occurred. This is useful when troubleshooting.

Apps on Android usually write logs by calling one of the methods of the android.util.Log class:

Log.v("MY_TAG", "Something happened");

These print statements are collected into a log stream. It’s like printing to stdout, but separate.

There are different log levels: verbose, debug, info, warning, error.

Normally, apps can only read their own logs, but not other apps’ logs. However, certain (system) apps can read all logs if they have the READ_LOGS permission (see below).

If debugging is enabled in the phone’s developer settings, you can view all logs on a computer using the ADB command adb logcat or in the logcat tab in Android Studio. Logcat can filter logs by tag and by log level.

Common Pitfalls

There are two common pitfalls that I have seen Android developers fall into:

Pitfall 1: The log level DEBUG is only another tag to filter the stream when viewing logs.

Logs written with Log.d are included in builds with build type “release”. Even when BuildConfig.DEBUG == False these log statements are included and will be printed!

Developers, you need to manually wrap these Log.d statements in an if (BuildConfig.DEBUG) { ... }. Or better, you should strip them completely from the release apk/dexfile (which is non-trivial).

Google could make this more clear, or make it easier for developers to strip DEBUG level statements.

Pitfall 2: Including network logging in release builds.

Logging network requests is useful during development, since you don’t need to setup a proxy and break open TLS. And with OkHttp’s HttpLoggingInterceptor it is as easy as adding three lines of code:

HttpLoggingInterceptor logging = new HttpLoggingInterceptor(); // add this
OkHttpClient client = new OkHttpClient.Builder()
  .addInterceptor(logging) // add this
  .build();

// add this to the build.gradle
implementation("com.squareup.okhttp3:logging-interceptor:1.0.0")

// and - bam! - all network requests are logged and viewable in logcat

Developers, you should only add the HttpLoggingInterceptor temporarily while debugging a specific issue. There is no real need to even commit the HttpLoggingInterceptor to git. And if you do, at least wrap it in if (BuildConfig.DEBUG) { ... } and add the dependency only to debug builds (with flavourDebugImplementation(...), see the docs).

Impact of Logging Sensitive Information

Logging sensitive data can result in data exposure to third parties.

On Android specifically, logging crosses the trust boundary between the app sandbox and the logging system.

For example, an app that uses an access token for network requests can store the token in its “shared preferences”. Despite the name, shared preferences are sandboxed and cannot be accessed by other apps. But when the access token is written to logcat it leaks outside the sandbox. Other apps can read these logs (with the appropriate permission, see below) and can now access the token!

The corresponding CWE is CWE-532 (Insertion of Sensitive Information into Log File).

Exploitability

Reading logs requires the android.permission.READ_LOGS permission.

User apps usually do not have this permission. It can be granted via adb (some apps use this), but for the average user this is an unrealistic hurdle.

System apps that come pre-installed sometimes have this permission.

And starting in Android 13, in addition to requiring the READ_LOGS permission, Android also prompts the user:

But how widespread is this?

In my own test 44 system apps had the READ_LOGS permission on OxygenOS 11 (OnePlus 7, as of June 2022), and 21 apps on LineageOS 20 (without GApps, OnePlus 5, as of May 2023).

AppCensus also noted a similar problem of sensitive data exposure via logs back in April 2021. They found that 59 apps can READ_LOGS on a Xiami Redmi Note 9, and 89 apps on a Samsung Galaxy A11.

The following apps/packages can READ_LOGS on OxygenOS 11 (produced with this script):

android
cn.oneplus.nvbackup
com.android.dynsystem
com.android.inputdevices
com.android.keychain
com.android.localtransport
com.android.location.fused
com.android.providers.settings
com.android.server.telecom
com.android.settings
com.android.wallpaperbackup
com.dsi.ant.server
com.fingerprints.fingerprintsensortest
com.google.android.feedback
com.google.android.gms
com.google.android.gsf
com.oem.autotest
com.oem.logkitsdservice
com.oem.nfc
com.oem.oemlogkit
com.oem.rftoolkit
com.oneplus
com.oneplus.backuprestore.remoteservice
com.oneplus.brickmode
com.oneplus.camera.service
com.oneplus.config
com.oneplus.coreservice
com.oneplus.factorymode
com.oneplus.filemanager
com.oneplus.gamespace
com.oneplus.minidumpoptimization
com.oneplus.opbackup
com.oneplus.opbugreportlite
com.oneplus.orm
com.oneplus.screenrecord
com.oneplus.screenshot
com.oneplus.sdcardservice
com.oneplus.security
com.oneplus.setupwizard
com.oneplus.sound.tuner
com.qti.diagservices
com.tencent.soter.soterserver
net.oneplus.commonlogtool
net.oneplus.odm

And on LineageOS 20 (without GApps):

android
com.android.dynsystem
com.android.inputdevices
com.android.keychain
com.android.localtransport
com.android.location.fused
com.android.providers.settings
com.android.server.telecom
com.android.settings
com.android.shell
com.android.wallpaperbackup
com.dsi.ant.server
com.stevesoltys.seedvault
com.tencent.soter.soterserver
lineageos.platform
org.lineageos.lineageparts
org.lineageos.lineagesettings
org.lineageos.pocketmode
org.lineageos.settings.device
org.lineageos.settings.doze
org.lineageos.setupwizard

Notice that this includes Google Play Services (com.google.android.{gms, gsf}) and various proprietary OnePlus apps. It is unclear whether these apps do access the logs and/or sent them to vendors. But they do have the permission do to so.

In addition, both OnePlus and Google Pixel devices have features that allow users to explicitly submit logs to the vendor.

• OnePlus: Settings > System > Experience Improvement Programmes > System Stability program
• Googe Pixel: Settings > Tips & Support > Send feedback

Overall, the exploitability is low. Still, sensitive information could end up on the vendors’ servers.

Fixing it

Fixing this is easy. Just remove all logging statements (network and other) that expose sensitive information to logcat (at least in production builds).

Real-World Examples

Below I show the two real-world apps that I found logging sensitive information, prompting me to write this post.

How did I find these? In both cases, the apps were buggy. Some unrelated feature was not working for me. And when I investigate bugs the first thing I do is open logcat. This is where I went “oh dear”.

Both apps have one thing in common (despite the logging issue): they don’t have a security.txt or a Vulnerability Disclosure Policy (let alone a Bug Bounty).¹ In both cases I had to jump through hoops to find a contact. For app 1 I luckily had a shared contact at the company that I could reach out to. For app 2 I had to go through normal support.

So in 2022, it was still challenging to report vulnerabilities, even for two apps that both handle financial transactions and that both have 100K+ downloads on Google Play.

In the logcat excerpts below I replaced sensitive information with 🚫🚫🚫.

App 1: Unnamed

The first app is a financial app. It was logging passwords, but also network traffic:

D Action  : Logon(vertragsNr=🚫🚫🚫, password=🚫🚫🚫, appInfo=AppInfo(appIdentifier=🚫🚫🚫, versionNumber=42.0, buildNumber=42, deviceName=OnePlus, ONEPLUS A5000, osVersion=Android 12 (32) | Code name: REL, unlocked=true, pushServiceType=GOOGLE), isBiometricLogon=false, biometryType=null)
D Network : GET /mobile/home/context -> HomeContext(sections=[HomeSection(type=FINANZEN, position=0, title=null), HomeSection(type=SHORTCUTS, position=1, title=null), HomeSection(type=ZU_ERLEDIGEN, position=2, title=You have), HomeSection(type=MARKETING_CONTENT, position=3, title=Discover)])
<!-- more network traffic omitted -->

This log leaks the username (“vertragsNr”) and password. Also notice how the network responses are logged as data classes. They reflect the sections on the app’s home screen. This is bad, because the app was effectively streaming the screen state to logcat.

The log level “D” (debug) demonstrates Pitfall 1.

Disclosure Timeline

• Wed 2022-06-22 21:30: I reach out to my shared contact at the company.
• Thu 2022-06-23 08:18: They reply and say they will find the right contact internally.
• Thu 2022-06-23 10:22: The security team reaches out to me.
• Thu 2022-06-23 11:35: I send them my writeup.
• Thu 2022-06-23 15:22: The security team acknowledges that they reproduced and patched it and will make a new release soon.
• Thu 2022-06-23 22:45: I test their fix, and verify that it works.

Within a single day the issue was acknowledged and fixed! They clearly had the internal processes in place to handle security issues, even if they did not have a public security contact yet (now they do¹).

(Why am I not naming them? Their handling was exemplary, it is fixed, there is no public benefit knowing the name. If you keep your apps up-to-date, there is nothing else you as a user need to do to protect yourself. The only public benefit is in knowing that I found more than one app, because this demonstrates that this is a recurring issue that we as a security and developer community need to be aware of.)

App 2: Circuit Laundry

The second app is the app for a laundry service in the UK, called Circuit Laundry. This app allows you to log in, pay with your credit card within the app to top up your account balance, and use that account balance to start washing machines and tumble driers.

Just like App 1, Circuit also streams all network traffic to logcat. This includes credentials (passwords, bearer tokens), personally identifiable information (first name, last name, email address), account balance, and account id.

Logs written during login:

I okhttp.OkHttpClient: --> POST https://phoneadmin.flashcashonline.com/api/user/authenticate/?password=🚫🚫🚫&version=2&email=🚫🚫🚫&platform=Android
I okhttp.OkHttpClient: Content-Length: 0
I okhttp.OkHttpClient: --> END POST (0-byte body)
I okhttp.OkHttpClient: <-- 200 https://phoneadmin.flashcashonline.com/api/user/authenticate/?password=🚫🚫🚫&version=2&email=🚫🚫🚫&platform=Android (1201ms)
I okhttp.OkHttpClient: cache-control: no-cache
I okhttp.OkHttpClient: pragma: no-cache
I okhttp.OkHttpClient: content-type: application/json; charset=utf-8
I okhttp.OkHttpClient: expires: -1
I okhttp.OkHttpClient: vary: Accept-Encoding
I okhttp.OkHttpClient: set-cookie: ARRAffinity=6eaf2cedfb705ed6ce633e6c1ba37f34686cbd9dce05635559dff6fd9e92ea1a;Path=/;HttpOnly;Secure;Domain=phoneadmin2.azurewebsites.net
I okhttp.OkHttpClient: set-cookie: ARRAffinitySameSite=6eaf2cedfb705ed6ce633e6c1ba37f34686cbd9dce05635559dff6fd9e92ea1a;Path=/;HttpOnly;SameSite=None;Secure;Domain=phoneadmin2.azurewebsites.net
I okhttp.OkHttpClient: x-aspnet-version: 4.0.30319
I okhttp.OkHttpClient: x-powered-by: ASP.NET
I okhttp.OkHttpClient: x-cache: CONFIG_NOCACHE
I okhttp.OkHttpClient: x-azure-ref: 0xYNjYwAAAAAEtq35BO3kQa1pmW2pi1HQTFRTRURHRTEzMTAAYWFiZjcyNTctMmE5NC00MDY5LTlkMGMtMWM1NTQzYjNlZWIz
I okhttp.OkHttpClient: date: Thu, 03 Nov 2022 09:03:01 GMT
I okhttp.OkHttpClient: {"Data":{"AppUserId":🚫🚫🚫,"UserIdentification":"🚫🚫🚫","HasPromotions":false,"Token":{"Value":"🚫🚫🚫","Expires":null},"PrimaryLocation":"","AccountBalance":6.80,"InternalId":🚫🚫🚫,"ExternalKey":"🚫🚫🚫","AccountName":"🚫🚫🚫","AccountOperatorID":5,"AccountMinimumPurchaseAmount":5.00,"AccountLowBalanceIndicator":5.00,"AccountCurrencyTypeID":3,"AccountCurrencyUniCode":"20AC","IsRoomViewAvailable":true,"AccountWelcomeTitle":"Welcome to Circuit","AccountWelcomeText":"Welcome to Circuit","FirstName":"Thore","LastName":"Goebel","OptInNotification":false,"OptInMarketing":false,"EmailAddress":"🚫🚫🚫","AppVersion":"2","AppPlatform":"Android","MessageForUser":""},"Success":true,"Message":"Authenticated"}
I okhttp.OkHttpClient: <-- END HTTP (1727-byte body)

Logs written during app usage later:

I okhttp.OkHttpClient: --> GET https://phoneadmin.flashcashonline.com/api/user/
I okhttp.OkHttpClient: authorization: bearer 🚫🚫🚫
I okhttp.OkHttpClient: --> END GET
I okhttp.OkHttpClient: <-- 200 https://phoneadmin.flashcashonline.com/api/user/ (217ms)
I okhttp.OkHttpClient: cache-control: no-cache
I okhttp.OkHttpClient: pragma: no-cache
I okhttp.OkHttpClient: content-type: application/json; charset=utf-8
I okhttp.OkHttpClient: expires: -1
I okhttp.OkHttpClient: vary: Accept-Encoding
I okhttp.OkHttpClient: set-cookie: ARRAffinity=6eaf2cedfb705ed6ce633e6c1ba37f34686cbd9dce05635559dff6fd9e92ea1a;Path=/;HttpOnly;Secure;Domain=phoneadmin2.azurewebsites.net
I okhttp.OkHttpClient: set-cookie: ARRAffinitySameSite=6eaf2cedfb705ed6ce633e6c1ba37f34686cbd9dce05635559dff6fd9e92ea1a;Path=/;HttpOnly;SameSite=None;Secure;Domain=phoneadmin2.azurewebsites.net
I okhttp.OkHttpClient: x-aspnet-version: 4.0.30319
I okhttp.OkHttpClient: x-powered-by: ASP.NET
I okhttp.OkHttpClient: x-cache: CONFIG_NOCACHE
I okhttp.OkHttpClient: x-azure-ref: 0yYNjYwAAAACoPdEYF50OSrWF+8q6w4IZTFRTRURHRTEzMTAAYWFiZjcyNTctMmE5NC00MDY5LTlkMGMtMWM1NTQzYjNlZWIz
I okhttp.OkHttpClient: date: Thu, 03 Nov 2022 09:03:04 GMT
I okhttp.OkHttpClient: {"Data":{"AppUserId":🚫🚫🚫,"UserIdentification":"🚫🚫🚫","HasPromotions":false,"Token":null,"PrimaryLocation":null,"AccountBalance":6.80,"InternalId":🚫🚫🚫,"ExternalKey":"🚫🚫🚫","AccountName":"JLA LAB","AccountOperatorID":5,"AccountMinimumPurchaseAmount":5.00,"AccountLowBalanceIndicator":5.00,"AccountCurrencyTypeID":3,"AccountCurrencyUniCode":"20AC","IsRoomViewAvailable":false,"AccountWelcomeTitle":null,"AccountWelcomeText":null,"FirstName":"Thore","LastName":"Goebel","OptInNotification":false,"OptInMarketing":false,"EmailAddress":"🚫🚫🚫","AppVersion":"2","AppPlatform":"Android","MessageForUser":null},"Success":true,"Message":"Authorized"}
I okhttp.OkHttpClient: <-- END HTTP (669-byte body)

Disclosure Timeline

• Mon 2022-11-07: Initial outreach to {security, info, contact, postmaster}@circuit.co.uk. {security, contact}@ bounced.
• Mon 2022-11-14: Outreach to info@circuitcardtopup.com (found on Google Play) and dataprotection@jla.com (found in the Privacy Policy). Interestingly, the data protection contact does not reply.
• Tue 2022-11-15: Circuit support (info@circuitcardtopup.com) replies. I send them the report.
• Sun 2023-02-04: I ask for an update and notify them that I plan to publish in 90 days (i.e. on 2023-05-05). I decided to count the 90 days from today since I did not mention my intent to publish in November.
• Wed 2023-02-08: Circuit support replies: “I have passed this on to our app developers again to look into it for you and have escalated it. I will update you once they come back to me.”
• Mon 2023-04-17: I ask for an update.
• Tue 2023-04-18: Circuit support replies: “There have been no updates provided to us as of yet but I will chase up with the app development team and let you know of any updates when we get them.”
• 2023-05-10: I publish this blog post.

The issue remains unfixed, ~6 months after I first reported it, and despite the fix being super simple. The affected version is 4.1.0 which was released on 31 March 2022.²

As a user there is very little you can do. You could reboot your phone after using the Circuit app (since this wipes logcat). You could also avoid explicitly submitting logs to vendors (Google Pixel’s “Send feedback”). Or you could take your laundry somewhere else.

Conclusion

First, if you are a developer, don’t write sensitive information to logs. On Android, by writing logs you are sending information outside of the app’s sandbox. As I have shown, tens of preinstalled apps can read logs. It is an easy issue to fix and avoid. But it also easily slips through PR reviews, and there are two common pitfalls for developers.

Second, if you are a business, have a security contact and a security.txt. Even if you have never received a report before. Some day you will. And even if your main business is not selling software, you still need to have a security contact.

P.S.: The Expires field in the security.txt is required.

You’re in luck! Here are the tables:

Table 1: Taken from NIST’s “Recommendation for Key Management” (SP 800-57 Part 1, Section 5.6.1.1). You can find similar tables on keylength.com.

Table 2: Assembled by me.

My motivation for collecting this information and writing this post is two-fold:

1. To answer the “why”: Where do these numbers come from? You can find the NIST table quoted a lot around the internet. But I always missed a concise summary of why the parameter sizes and the security strengths relate in this way.
2. To cover quantum: All tables I found only contained the security strengths in the classical case.

Point 2 is already solved by Table 2 above. For the remainder of this post I will address point 1 and give an explanation of the “why”. That is, I will explain why Table 2 has these values.

This post is not meant to explain all areas in depth. Rather, it is intended as a quick, summarising overview that pulls all the concepts together in one place. It assumes prior knowledge, refreshing only the points that are important in this context.

THIS POST DOES NOT CONSTITUTE CRYPTOGRAPHIC ADVICE. DO NOT TRUST A RANDOM PERSON ON THE INTERNET. YOU SHOULD SEEK PROPER CRYPTOGRAPHIC COUNSEL INSTEAD.

Preliminaries

Let’s start with some background. We will need this when we discuss for each type of algorithm how its parameter sizes relate to its classical and quantum strength.

Representing Numbers

We represent integers \( \leq 2^n = N \) as bit strings of length \( n \). We use capital \( N \) for integers, and lower case \( n \) for bit string lengths.

Note that \( \frac{2^n}{2} = 2^{n-1} = \mathcal{O}(2^n) \). Also note that \( \sqrt{2^n} = 2^{n/2} \).

Parameter Size

The “parameter size” is what we can tune about an algorithm to achieve different security strengths.

In most cases the parameter size is the key size. But not always! For example, for unkeyed hash functions (such as SHA-256) the parameter size is the size of the output.

Security Strength: Bits of Security

We measure the “strength” of an algorithm in “bits of security”, or short “bits”. An algorithm has “\( n \) bits of security” if it takes \( \mathcal{O}(2^n) \) operations to break it.

Usually, an “operation” is equal to one algorithm invocation and one comparison. For example, one invocation of SHA-256 and one comparison of the output against a target value.

Ideally, an algorithm can only be “broken” by brute-forcing, i.e. by trying all possible values. For keys of length \( n \) there are \( 2^n \) many possible keys, so an average brute-force search succeeds after \( \frac{2^n}{2} \) attempts. But as noted above, this is still \( \mathcal{O}(2^n) \). So instead of “127 bits” we say “128 bits”. Thus you can read all the bit-of-security values in the tables above with a big-\( \mathcal{O} \) in mind.

Classical: Discrete Logarithm

The Discrete Logarithm Problem is that given two numbers \( a, b \), find a number \( x \) such that \( b^x = a \). That is, find \( x = \log_b a \). We work in a cyclic group \( G \) of order \( N \) (i.e. there are \( N \) group elements). For cryptography \( N \) is usually prime.

There exist classical algorithms such as Pollard’s rho that solve the DLP in time \( \mathcal{O}(\sqrt{N}) = \mathcal{O}(2^{n/2}) \).

Hence classically, for parameter size \( n \) bits we only get \( n/2 \) bits of security.

Quantum: Integer Factorisation and Discrete Logarithms – Shor’s Algorithm

Shor’s period finding algorithm can be adapted to solve both the Discrete Logarithm Problem and Integer Factorisation Problem. It is “efficient”, meaning that it runs in polynomial time.

Hence for quantum, for parameter size \( n \) bits we get \( \approx 0 \) bits of security.

Quantum: Brute-Force Search – Grover’s Algorithm

Grover’s algorithm speeds up unstructured search from \( \mathcal{O}(N) \) classically to \( \mathcal{O}(\sqrt{N}) \).

Hence for quantum, for parameter size \( n \) bits we only get \( n/2 \) bits of security.

Hashing: Pre-images, Collisions, and the Birthday Attack

For pre-image resistance, the problem is that given an output \( y \) to find an input \( x \) such that \( x = h(y) \). Both classically and quantum the best known approach is brute-force search.

For collision resistance, the problem is to find two arbitrary inputs \( x_1, x_2 \) such that \( h(x_1) = h(x_2) \). Classically, this can be solved with a birthday attack in time \( \mathcal{O}(\sqrt{N}) = \mathcal{O}(2^{n/2}) \). For quantum, there is a paper claiming to be able to find collisions in time \( \mathcal{O}(\sqrt[3]{N}) = \mathcal{O}(2^{n/3}) \). However, Daniel Bernstein vigorously disagrees, concluding that the best practical collision attack even with quantum is still in time \( \mathcal{O}(2^{n/2}) \).

Tying It All Together

With the above in mind, we obtain the following relations that describe how the security strength \( S \) varies with the parameter size \( n \).

• Symmetric crypto (AES, HMAC):
  • Random symmetric key, breakable by brute force search.
  • \( S = n \) classically.
  • \( S = n/2 \) quantum (Grover’s).
• Asymmetric crypto (RSA, DH, ECC):
  • Discrete Logarithm and Integer Factorisation.
  • \( S = n/2 \) classically (Pollard’s rho and others).
  • \( S \approx 0 \) quantum (Shor’s).
• Hashing:
  • Pre-image resistance:
    • \( S = n \) classically.
    • \( S = n/2 \) quantum (Grover’s).
  • Collision-resistance:
    • \( S = n/2 \) classically (Birthday attack).
    • \( S = n/2 \) OR \( S = n/3 \) quantum (Bernstein OR Brassard, Høyer, and Tapp).

Note that just like the value “128” in the security strength is an approximation, the \( = \) is also an approximation.

Also note that these relations and the values in Table 2 are upper bounds. They only represent the security strength under generic attacks. There can be algorithm-specific attacks. For example, SHA-1 has an output size of 160 bit, which would imply 80 bit security against generic collision attacks. However, the SHAttered attack estimates that it only needs \( \approx 2^{64} \) operations.

What Security Strength Should You Target?

For a long time, 128 bit was commonly recommended. However, the NSA’s latest guidance in its Commercial National Security Algorithm Suite is at least 192 bit (except for RSA-3072, which is 128 bit).

Conclusion

We started this post with concrete values: a table of common cryptographic algorithms, their parameter size, and their security strength in both the classical and the quantum setting.

We ended this post with the formal relations that relate the parameter size to the security strength.

Inbetween, we refreshed our knowledge to understand the why. We explained where these relations come from.

I hope this helps as a quick reference to look up the values, the relations, and the derivation of the relations.

Motivation

Before the pandemic, my laptop (a ThinkPad) was on and off the charger. I was walking around university, sitting in lectures, going to the library. I would charge my laptop when the battery was getting low, and then disconnect the charger again.

During the pandemic, my laptop was always sitting on my desk, connected to a monitor, a docking station — and a charger. It spent several months at full charge.

As of early 2020, my laptop was 3 years old and the battery capacity still at 85%. Over the course of the next 12 months, its capacity dropped to 65%. With a design capacity of 57 Wh, this is a drop from 48 Wh to 37 Wh. At a power consumption of 5 W, this is 2 hours less runtime! And it is a drop by 20% over 1 year, versus a drop of 15% over 3 years.

Factors Impacting Battery Life

Why did this happen? What can I do to prolong the life of my laptop battery?

According to Lenovo, battery life is a function of:

• age
• number of charge cycles
• amount of time at full charge
• high temperature

In other words: anything that causes physical stress to the battery/electrons/chemicals decreases battery life. The battery “wears out” and is no longer able to hold the same amount of charge.

The “amount of time at full charge” killed my laptop battery. Sitting at 100% for a large portion of 2020 was what accelerated its death so significantly.

The “high temperature” killed my phone battery. This is a different story. But keep this in mind next summer when it is over 30°C!

Setting an upper charging limit

To protect my battery, I needed to prevent it from being at 100% for long periods of time (read: days non-stop).

To do this, I wanted to configure the battery to only charge up to a maximum value. One option are command-line tools, such as TLP (more details below). Another option are graphical tools, such as KDE Plasma’s system settings:

Personally, I have set the “value to start charging at” and the “value to stop charging at” to 75% and 80% respectively.¹ They seem like a reasonable tradeoff, while still having a nearly-full battery for when I pack up and leave my desk. 75% and 80% are also the default values that TLP sets. And Lenovo agrees: “For systems which are always connected to an AC power source, Lenovo recommends setting the upper charge limit to 80% or less.”

Now when my laptop is at the docking station for 2 days, it sits their at 80% instead of at 100%.

The added benefit is that when it sits there at 80% and I run compute-heavy tasks, the 20 or so watt come directly from the charger. It does not impact the battery, i.e. it is saving me some charging cycles (see below).

Reading Battery Statistics

The Linux kernel reads battery statistics from the hardware and exposes them. To read the current power consumption in milli-Watt and the amount of full charging cycles:

cat /sys/class/power_supply/BAT0/power_now
cat /sys/class/power_supply/BAT0/cycle_count

For the former I made myself a bash alias: alias watt="cat /sys/class/power_supply/BAT0/power_now". I look at watt regularly to get an intuition of how my laptop is doing.

I found the following rule-of-thumb numbers for my system (4-core i5-7200U @ 2.5 GHz):

I also found that screen brightness is not a significant factor (on my laptop!). At least, it is barely noticeable in the watt readings. Instead of forcing brightness down to 10% to save power, I now have it at a comfortable 30% most of the time.

For the cycle_count, note that this is the number of full cycles. I have had my new battery for 8 months (= 240 days) now. Despite using it daily, I am only at 60 cycles. One contributing factor is that my laptop now spends weekends and home-office days on my desk at the charger.

If you like graphs, KDE Plasma also ships with an Energy Monitor that shows you the historical power consumption:

Using TLP to read statistics and set thresholds

TLP is a command-line tool that can read battery statistics and configure various settings, such as charging thresholds. In this section I will give a short primer on how I use TLP.

First, install it.

Now you can view the battery statistics:

$ sudo tlp-stat --battery
--- TLP 1.5.0 --------------------------------------------

+++ Battery Care
Plugin: thinkpad
Supported features: charge thresholds, recalibration
Driver usage:
* natacpi (thinkpad_acpi) = active (charge thresholds)
* tpacpi-bat (acpi_call)  = active (recalibration)
Parameter value ranges:
* START_CHARGE_THRESH_BAT0/1:  0(off)..96(default)..99
* STOP_CHARGE_THRESH_BAT0/1:   1..100(default)

+++ ThinkPad Battery Status: BAT0 (Main / Internal)
/sys/class/power_supply/BAT0/manufacturer                   = LGC
/sys/class/power_supply/BAT0/model_name                     = 01AV494
/sys/class/power_supply/BAT0/cycle_count                    =     60
/sys/class/power_supply/BAT0/energy_full_design             =  57000 [mWh]
/sys/class/power_supply/BAT0/energy_full                    =  59180 [mWh]
/sys/class/power_supply/BAT0/energy_now                     =  30780 [mWh]
/sys/class/power_supply/BAT0/power_now                      =   4883 [mW]
/sys/class/power_supply/BAT0/status                         = Discharging

/sys/class/power_supply/BAT0/charge_control_start_threshold =     75 [%]
/sys/class/power_supply/BAT0/charge_control_end_threshold   =     80 [%]
tpacpi-bat.BAT0.forceDischarge                              =      0

Charge                                                      =   52.0 [%]
Capacity                                                    =  103.8 [%]

As mentioned above, you can also obtain the power_supply values by cat-ing the files directly (this is my watt shortcut). The tlp-stat command just nicely aggregates them.

Next, configure the charging thesholds. To do this, create a file /etc/tlp.d/01-charge-thresholds.conf with the following content: ²

START_CHARGE_THRESH_BAT0=75
STOP_CHARGE_THRESH_BAT0=80

Make sure this file has the correct owner and correct permissions:

$ sudo chown root:root /etc/tlp.d/01-charge-thresholds.conf
$ sudo chmod 644 /etc/tlp.d/01-charge-thresholds.conf

Then apply the TLP config by running sudo tlp start. You should now see these changes when you run sudo tlp-stat -b.

What if you have a trip coming up, and you want to temporarily charge to 100%? Instead of messing with the config, just connect your charger and run:

$ sudo tlp fullcharge
Setting temporary charge thresholds for BAT0:
  stop  = 100
  start =  96
Charging starts now, keep AC connected.

sudo tlp-stat -b shows you the updated thresholds. Simply run sudo tlp start to revert back to the settings in your config file.

Note: The command tlp start takes the value from the config file, and writes them to the embedded controller (EC) via ACPI. tlp fullcharge does the same, but with the default values 96 and 100. The EC is a piece of hardware that controls the charging process. This way, charging will stop at 80% even if the laptop is turned off. But this also means that you have to manually run start or fullcharge for your changes to take effect!

For more information, read the docs, the FAQ or the manpage (man tlp).

Replacing the Battery

Now I knew how to slow down battery aging, and how to monitor the battery. But what if the battery is already dead?

Disclaimer: repeat this at your own risk. In my case, my laptop was 4 years old, so the warranty was void anyway.

When my old battery reached 65% capacity, I decided to replace it. Only having 6 hours of runtime was getting too impractical.

I ordered a replacement battery on AliExpress for 35 €, including shipping. For comparison: official replacement batteries cost 100-150 €.

Replacing the battery requires only a few screw drivers. I recommend searching YouTube for any replacement video (even if it is not your exact model). It is a good way to get a rough view of what to expect, and to build confidence before you open up your own laptop.

Here are the steps for my ThinkPads (others should be similar):

1. Disconnect the charger.
2. Discharge the battery to 20-30%. Less electrical charge reduces the risk if something goes badly wrong.
3. Touch some grounded metal to ensure you are grounded as well. Prevent electrostatic discharge (ESD).
4. Shut down the laptop and reboot into the BIOS.
5. In the BIOS, go to Config > Power > Disable Built-In Battery. Then click Enter. (This ensures the power is cut and the battery is fully disconnected from the other electronics inside the laptop. You won’t be able to turn the laptop on again without a charger.)
6. Close the lid and turn the laptop upside down.
7. Remove the screws that are holding the back plate and remove the back plate.
8. Disconnect the internal battery cable.
9. Remove the screws that fix the battery in-place.
10. Take the old battery out, and put the new one in.
11. Screw the new battery in place. Then re-connect the internal battery cable.
12. Close the back-plate and screw it tight.
13. Connect a charger and boot your laptop.
14. Done! Proceed to calibrating the battery.

Calibrating the new battery

After I installed the new battery, it was time to calibrate it. Calibration means pushing the battery to both the low and the high extreme. This helps the battery controller figure out what level of charge constitutes “full charge” and “empty charge”.

This changes over the lifetime of the battery, because as the battery ages it is able to hold less charge. The controller remembered a lower charge value as “full” from the old battery. Calibrating it means teaching it that “full” is now at a higher charge.

This paper card that iFixit ships with phone replacement batteries sums up the steps:

Determining the Quality of the Replacement Battery

35 € for a laptop battery is rather cheap. Sure, I did some diligence. I compared batteries on AliExpress. I read through comments and reviews. I looked at clues indicating seller reputation and quality. But it is still a guessing game. With seller and buyer so far away, there is little accountability.

How can I verify that the cheap battery is not of low quality? What if the battery reports a higher capacity than it actually has? Can I trust the hardware readings? After all, the “Last full charge = 59.18 Wh” (see Energy Monitor screenshot above) seems strange given that “Design capacity = 57 Wh”.

It turns out that batteries are smart. ³ They have a built-in chip that reports the voltage, current, etc to the laptop. This means that the laptop relies on the values measured and reported by the battery.

This also means that a scam battery can report wrong values. For example, it can report a larger capacity than it actually has. I know that this happens with smartphones: the Android system settings on phones that are cheap “offers” on AliExpress claims that the phone has 16 GB RAM but in reality it only has 0.5 GB RAM! (German news article)

Without taking the battery apart and doing more analysis, I just have to trust the battery.

One data point that I can control, though, is the overall runtime. With a nominal capacity of 57 Wh and a power consumption of 5.5 W, the new battery should last about 10 hours. So I measured how long the new battery would last during normal laptop usage. And indeed: the new battery got me back to running an entire day at university without having to charge. This has held up so far over the past 8 months. Because of this, I now also trust the watt readings – they vary with my laptop usage patterns in the way I expect them to.

I assume the 59.18 > 57 is either due to the uncertainty in the physical measurement/calculation. Or the factory capacity really is too large, to account for manufacturing uncertainties, that is, to prevent returns and angry customers.

Conclusion

In this post I explained what factors impact the lifespan of a modern laptop battery (age, number of charge cycles, amount of time at full charge, high temperature). Based on this, I also explained what you can do to prolong the lifespan (don’t keep it at 100% for long times). I also described how you can view battery statistics (use TLP). Finally, I showed how I replaced my laptop battery.

Note that I only focused on ThinkPads running Linux. Please let me know if you find similar ways to set the charging thresholds on MacOS. There might be, but I haven’t researched it. Apple has Optimised Charging, but this seems to only work when charging overnight. It is not a solution if you keep your MacBook connected to your monitor via USB-C-with-power-delivery during an entire workday — it will charge to 100%.

This is not new. All three companies have long supported FIDO to some extent. Both Google Chrome and Microsoft Edge have had WebAuthn and CTAP2 support for a few years now. iOS also has WebAuthn support since 14.5.

The two hot new things are:

1. smartphones as roaming authenticators via Bluetooth, and
2. multi-device credentials (“passkeys”), via cloud sync. ¹

The announcement has brought some renewed attention to FIDO. Reason enough to write down my thoughts on the current state of the FIDO ecosystem!

First, I will give a short introduction/refresher about FIDO and its concepts. Then, I will go into what I think is great about FIDO, as well as some of the problems that I see.

Background

Back in the early days, there was the password. Unfortunately, people took that too literally and chose words like “password” or “qwerty” (is that a word?) as their password.

In response to that (and recalling that good passwords should be long) we collectively decided to call them passphrase instead (phrase, as in “sentence”). Surely sentences are easier to remember! But that didn’t work either – people kept chosing “letmein”.

So now, “the industry has introduced the term passkeys” (source). “The industry” meaning Apple. ^{2 3}

This is where FIDO comes in. The basic idea behind FIDO is that instead of user-chosen passwords we can use public key cryptography to authenticate users. Hence the word “passkey”.

Here is how FIDO replaces the registration and login processes (roughly speaking):

1. Upon registration, you create a public-private key pair. The server provides you with a challenge, you sign, and – bam – you are now registered. This is trust-on-first-use.
2. Upon authentication (aka login), the server again sends you a challenge. You sign it, the server verifies it, and you are logged in!

Basically, this is all that FIDO is. Public key crypto and a challenge-response protocol.

Unfortunately, you (the human) already have a hard time keeping 256-bit numbers in your head, let alone computing ECDSA signatures with those numbers. Luckily, all this cryptography is done for you by an authenticator. There are two main types of authenticators: First, platform authenticators, which are built into your device. Think Touch ID, Face ID, or Windows Hello. Second, roaming authenticators, that can “roam” around, between multiple devices. Think physical keys (YubiKeys, …). They communicate with your computer via USB, NFC or BLE (using the CTAP protocol). ⁴

In addition, there is WebAuthn. WebAuthn is the API that websites or desktop/mobile apps interact with (see the Mozilla docs). The browser or OS then handle the communication with the authenticator (either internally for platform authenticators, or via CTAP for roaming authenticators).

So overall, when people speak about FIDO or FIDO2, they mean WebAuthn and CTAP. Here is my amateur drawing of an overview of FIDO:

For a deeper dive, see this glossary and this blog post as starting points for further reading. Adam Langley also has multiple great posts on security keys. Apple’s WWDC22 video nicely shows how the UX flows can look like.

By the way, did you know that the new SwissPass can act as a roaming authenticator over NFC? Every public transport user in Switzerland now gets a security key included in their travelcard! Since it is FIDO-compliant, you can use it to log in to any website that supports WebAuthn (on your phone, or on a laptop with an NFC reader).

What’s new?

One big UX problem for FIDO so far was that keys were bound to a single device. That meant you would need to manually enroll two or more security keys (in case you loose one), or manually enroll both your laptop and your phone. What is more, you would need to manually do this for every single service where you used FIDO. Just imagine switching phones, having to manually enrol your new phone in 10+ services again.

FIDO now proposed to solve this with multi-device keys. Effectively it lifts the requirement that a key needs to be bound to a secure hardware element, and allows it to be synced. Notably, the standard does not impose any requirements (security or other) on the nature of the sync service, and instead leaves it up to the vendor (e.g. iCloud).

Yes, this lowers the high bar that you get from hardware-bound secrets. But for the average user reusing their passwords, it still a major step up. In high-security contexts, you can continue to use device-bounds keys.

A second UX problem was the need to buy a security key. Without such a roaming authenticator that can plug into a USB port, how would you log into a new device?

This second problem is solved by specifying how smartphones can act as roaming authenticators. If you have a registered FIDO key on your smartphone, and you want to log into your email account on a workstation in a public library, you can scan a QR count on the workstation. The QR code contains information for local pairing over Bluetooth, thus proving proximity. Then your phone and the workstation will run the normal CTAP protocol, with your phone acting as a roaming authenticator. ⁵

These changes are exciting because they promise to clear the way and remove the two main blockers of mainstream adoption.

The good

The good: things that are great about FIDO and that we should do more of.

No shared secrets

One thing I love about FIDO is that it eliminates shared secrets. Instead, the secret (aka. private) key stays solely on the user’s device.

The long history of password leaks has shown that it is difficult to protect the treasure chests that password databases are. It is much easier to break into a single central server, than millions of end-user devices.

By using public key crypto we also get phishing protection: the secret never leaves the device, so there is nothing to be phished. ⁶

No human-generated secrets

With FIDO we move away from mostly human-generated secrets and towards (pseudo-)randomly generated secrets. Sure, we have mitigations like salting or special hash functions (scrypt, Argon2). But in the end, they are still band-aids working around a low-entropy source.

Industry push

Any technology can only make a difference if it has broad user adoption and widespread software support. Unfortunately, widespread adoption doesn’t just happen. Companies and people need to invest money and time. And sometimes, you need peer pressure to make a step.

We have seen this before with biometrics on laptops: Thinkpads have had fingerprint sensors for generations. Yet only after Apple pushed Touch ID on MacBooks we started seeing improved software support (imho because people know it from Apple and also demand it from other products). Even on Linux fingerprint sensors are starting to “just work”!

With FIDO we can hope for the same effect. Google, Apple, Microsoft and Mozilla laid the foundations over the past few years (and took on the initial risk of investment!).

We are still far away from people being so used to passkeys that they start demanding it from all products. But with industry leaders committing and leading the way, we can get there.

The bad

The bad: things that I think are problematic about FIDO, and that should be talked about more.

This is not to say we should throw FIDO out of the window and start anew! In fact you will see that some problems are very familiar and keep coming up in different areas of IT.

TLS client certificates revisited

First of all, why are we inventing a new protocol?

FIDO and its use of public key cryptography inherently reminds me of TLS client certificates. In both cases, the user/client holds a private key and uses it to authenticate itself to the server.

The only difference is where the server’s trust in the private key comes from (or rather, its trust in the binding from the key to a human). In FIDO, it is trust-on-first use, i.e. the server remembers the (public key, identity)-binding. In TLS client authentication on the other hand, the server derives its trust from the certificate chain that is presented alongside the public key.

Other than that, the two are pretty similar:

• Both use public key crypto.
• Both can have their secret key material either stored simply on disk or in secure hardware (YubiKeys, smartcards). ⁷
• Both have some form of freshness. This guarantees to the server that the client was recently alive (FIDO: challenge response, TLS: signing the session transcript).

Which begs the question: why are we not using TLS client certificates?

Or, in other words: if nobody outside enterprise uses TLS client certs, if no consumer site offers client certs, if client cert usability is so bad – why should FIDO succeed where client certs did not?

And, more concretely: what does FIDO do better than TLS?

I already mentioned that FIDO has hardware attestation, and TLS client certs don’t. ⁷ Also, you might argue that it makes sense to separate transport and authentication into separate protocols.

On the other hand, I would argue that having both together is beneficial, since it allows you to bind the security of your channel to an identity. That is, you can run FIDO over a TLS connection that is being MITMed (assuming the TLS client trusted the server cert, or a trusted CA has been compromised) – the server won’t notice that the connection to the WebAuthn client is not private.

Also, in terms of performance it makes sense to bind transport and authentication together. In almost all cases you need a secure channel anyway to communicate further after authentication. Why spend extra round trips with FIDO if you can integrate it into TLS? At Google-scale, the extra requests will make a difference.

I would love to hear your thoughts on this question (FIDO vs TLS client certs)!

Proprietary solutions

Open standards don’t lead to open solutions.

For Android, Google implements FIDO support in their proprietary Play Services (rather than in AOSP). That means all the CustomROMs and Android-based OSes without ties to Google won’t have FIDO support for a long while.

Meanwhile, Microsoft’s announcement on World Password Day (while everybody else was announcing their push for FIDO) used the terms “Azure”, “Windows”, “Microsoft” noticably more often than it did say “FIDO” or “Webauthn”. So far, my impression is that they are boasting with FIDO and open standards, all while pushing their proprietary Authenticator app.

Additionally, banks love to build their own 2FA apps rather than use TOTP for their online banking logins. Will they switch to FIDO?

Furthermore, how will the new multi-device credentials work? Will users be locked into the iCloud ecosystem? Or will Apple offer a way for 1Password or Keepass to plug into its FIDO APIs and function as a passkey sync provider? This is especially critical since FIDO does not specify how multi-device keys are synced. The sync services will be the Achilles’ heel for the security of users’ passkeys.

Higher complexity for developers

Implementing FIDO is more complex than username+password. For the time being, this will continue to pose a barrier for entry.

You need to parse binary messages, handle keys, verify signatures, generate challenges, adhere to a specification, and so on. Of course, all of this will be (and should be!) outsourced into libraries – after all, you also don’t implement your own TLS code.

But even with the bit-juggling abstracted away, developers still need to learn the concepts and the terminology. It is not simply a matter of new HTTPSConnection(url). The WebAuthn API seems innocent: it is just create() and get(). But you need to understand how to prepare your inputs and how to interpret the outputs.

FIDO has many options and knobs that you can configure. What’s the difference between attestation and assertion? What are PublicKeyCredentials, and which ones should I choose? What level of user verification do I want? Do I really need to verify that counter? Should I allow multi-device credentials, or only keys that reside in TPMs?

I can imagine a world where web frameworks and CMS systems offer FIDO support out of the box as part of the user management. But this will take time. Until then, developers will need to go through this learning curve. It is not impossible – but it is an barrier that I think is often neglected in the hype around FIDO.

Widespread support

With all the hype around FIDO: how good and how widespread will FIDO support actually be in reality?

On the authenticator side:

• Firefox still does not support CTAP2 on MacOS and Linux, only on Windows using Windows Hello (but they are working on it). Which means even though it has a WebAuthn API exposed to JavaScript, you cannot use features like discoverable credentials. Discoverable credentials are important because they allow usernameless signin (as opposed to “only” passwordless). They were introduced in CTAP2. In practice, this means that you still cannot use your YubiKey for passwordless sign-in to Microsoft’s ecosystem using Firefox.

  I.e. “FIDO support” in Firefox in July 2022 means “WebAuthn wrapped around CTAP1/U2F”.

• Support on the Linux desktop (for desktop apps, similar to Windows Hello) will (most likely) take a while.

• AOSP won’t support FIDO either any time soon (due to it being implemented in Play Services, see above).

On the website/client side:

• I have over 200 entries in my password manager. Only a fraction of those offers TOTP (20%). And an even smaller fraction offers WebAuthn (5%). Given FIDO’s complexity: will FIDO surpass TOTP? Will it become the password-killer?

  A real world example: ETH Zürich is planning to roll out 2FA in autumn 2022 for its single sign-on. But it will use OTP (presumably TOTP), not WebAuthn.

• Almost all consumer websites offer WebAuthn merely for 2FA. That is, they are not replacing passwords. One exception is Nextcloud, which offers it for passwordless signin. Not a single website I use offers usernameless signins via WebAuthn.

• On the upside, FIDO support is already significantly more widespread on consumer websites than TLS cert support is.

Research & academia

From an academic perspective, FIDO is a relatively new protocol, and research into it is just getting started.

There are some relatively recent papers formally modelling and formally verifying it (formal as in formal methods). One using ProVerif, another looking at the crypto in the computational model. Others are already presenting real-world attacks.

This reminds me of SSL/TLS and WEP/WPA: industry builds it, academia breaks it, industry patches it, repeat.

TLS 1.3 broke this cycle: it involved academia early on in the standardisation process. There was a lot of research into both the handshake and the record layer that was incorporated into the drafts. Also the research attacked the spec from multiple angles: proving cryptographic security computationally (as opposed to information theoretically), tool-based approaches (Tamarin, ProVerif), and formally verifying implementations.

This is not a central criticism of FIDO! FIDO is just starting out, you cannot expect dozens of researchers already jumping onto it. But if FIDO does prove itself to be there to stay and gains significant adoption, FIDO3 or FIDO4 should involve academia more early on in the process, just like TLS 1.3 did.

The name

WebAuthn has not even reached broad adoption, yet the name is already outdated: WebAuthn started out as a web standard, but with native/mobile/desktop support, it is now breaking out of the browser. Native iOS and Mac apps can use the WebAuthn protocol over native, non-browser APIs. Thus the term WebAuthn has become too narrow.

Even within FIDO (or should I say FIDO2? Or FIDO/WebAuthn?) there are legacy names that keep getting changed (e.g. resident keys aka. resident credentials aka. discoverable credentials).

In addition, it is often unclear what someone means when they talk about having “FIDO support”. Do they mean only WebAuthn? Do they mean CTAP2 as well (cf. Firefox)? What about U2F? And CTAP1? Wait, U2F and CTAP1 are the same thing.

The naming is not (yet) as messy as with USB (hello, USB 3.0 and USB 3.1 Gen 1 and USB 3.2 Gen 1x1). But it is definitely not very human friendly either – neither for consumers, nor for developers.

Will we see a broad migration from passwords to passkeys?

Mainstream adoption of passkeys depends on two questions: First: will passkeys be broadly available? And second: will users actually be willing to use them?

In terms of availability, we need both service-side and client-side support. For the service providers (social media, online shops, email providers, …) it is relatively straight-forward: just pull in a WebAuthn library, consider all my discussions above, and don’t forget to get the recovery process right. On the client-side big vendors such as Microsoft, Google and Apple are finally making a push. But even they have more work to do: the reality of syncable credentials is still very much open. It is unclear if and how third-party services (such as today’s password managers) can hook into the client and sync passkeys.

And when everything is ready, all my favourite services, and even my obscure Linux distro and my smart TV support FIDO – will users adopt it? The UX of passkeys is remarkably similar to the UX of using a password manager:

• One click to generate a new password/passkey.
• One click to auto-fill an existing password/use the passkey.
• Various features to list and manage all your passwords/keys.

Yet password managers have not taken off. If we as security professionals could never provide enough incentives to convince the broad public to use password managers – how will we be able to convince them to adopt passkey managers? How will we convince someone with an Apple iPad, a Windows laptop and an Android phone that passkeys are easier and faster than passwords?

Conclusion

Are passwords bad? Yes. Is FIDO the solution? (Some of) the industry definitely thinks so.

In reality however we are still a long way away. Even Google admits that “passwords will continue to be part of our lives” for a while. There are a lot of challenges yet to be solved.

This all sounds very pessimistic, but it shouldn’t. I am excited about replacing passwords. I am excited to see cryptography being applied to solve a real everyday problem. And I do want to be able to use my YubiKey on more than just 10 out of 200 accounts.

Apple has a history of pushing things forward, simply by adopting something and everybody else following suit. And with Microsoft and Google on board as well, there is a good chance that they can bridge the chicken-and-egg problem: website don’t offer FIDO because there is poor client support, and clients don’t offer FIDO because it is not a feature that is widely requested among websites.

For us computer scientists and engineers there are a lot of interesting challenges ahead. Naming, discussing and understanding the good things as well as the bad things, the opportunities as well as the weaknesses, is key to tackling these challenges.

Please let me know if you found this useful, have any comments, found an error, or outright disagree with me – I would love to hear your opinion!

When working on the lab I was googling around and by chance came across this OpenSSL man page. Specifically, its “NOTES” section at the bottom. You can read it on your own, but the TLDR is: under certain operating systems and TCP settings, and given a not-too-large amount of application data, 0-RTT may inadvertently end up being 1-RTT. In other words, you go to all the lengths to build a protocol that has low latency but wind up back on square one. (You can find the original issue here.)

For me this is a reminder of two things:

1. Measuring performance matters (just like writing tests). Without measuring it, you wouldn’t have noticed that not all of your packets are doing 0-RTT.

2. You can have a nice higher level protocol, but the lower part of the stack can cause unexpected effects. Abstraction works only so far. Thus even if you generally work higher up, you need to have a solid understanding of what is happening underneath.

A beautiful interface.

What is “beautiful”? How do we grasp that intuition and put it into something that we as designers and developers can implement in our apps?

In this post, I want to explore a number of things that I found make me stumble when I use an app. Things that I now watch out for when building an app. Things that I go to my company’s designers to complain discuss about. Things that I open issues and PRs with open source projects on Github for.

Basically, I describe a number of characteristics that a Beautiful UI has.

Disclaimer:

Firstly, I am not a designer. I have no clue about the concepts and structured approaches they teach in design school. I am writing this from the view of a user with a very keen eye, very high standards and a background in Android development.

Secondly, the last thing I want to do is to point fingers. The (anti-)examples I give are to illustrate. We simply learn more from our errors than from the things we already do right.

Lastly, I am biased towards Android, since this is what I know most about. However, the general rules should equally apply to other platforms.

Rule 1: Stick to the guidelines

Design systems exist for a reason: to provide a consistent set of rules how things are to be done, and to save us time since a lot of design problems are already solved for us.

Material Design is the go-to choice for many Android apps. It contains a huge list of building blocks that we can use to build an app, allowing us to create a consistent and beautiful experience.

Choose a design system, and then stick to its guidelines. Be consistent. Not doing so gives the user the impression that we don’t really care about our brand’s appearance after all.

For example, Material defines how an app bar should look like. Jitsi however does not adhere to that. First, the app bar height is too small – notice how the space to the left of the back arrow is greater than the space below it. Second, the title is centered, whereas in Material for Android it should be left-aligned.

Another issue here is colouring: if we define a primary brand colour, we should use it everywhere. The ugly default green on the switches does not go well with the blue.

Jitsi also provides us with an example of a modal navigation drawer, colloquially known as the “hamburger menu” (since it hides behind a button that vaguely looks like a hamburger). To be fair, the Material specs are not particularly helpful, as they imply that (on a phone) the drawer should always take the full width minus the size of the app bar. Not even Google does this with their own apps, and there are various different articles discussing what might be the right width. Anyway, the width that Jitsi uses is far smaller than what anybody else uses. This break with practical Material Design is distracting.

Icons need to be consistent, too: ProtonVPN uses an old Holo icon amidst their Material-themed app (but hopefully not for much longer!).

Finally, leaving random, unthemed and seemingly unfinished screens in our app is not just not beautiful. It’s ugly. This is an actual settings page from the Publibike app: in the default Android theme, without any app bar or back button.

Stick to the guidelines. Everywhere.

To end on a good note, I would like to point out two apps that have done an awesome job in encorporating the latest Material Design: this files app and Threema. Case studies of well-executed and more custom design systems can be found with Slack, ASVZ and SwissCovid.

Rule 2: Be responsive

By “responsive” I mean fast, not skipping animation frames, lagging, or jumping around. This rule concerns the dynamic aspect of our interface.

Lack of responsiveness quickly results in the user experience being consciously or subconsciously worsened.

Most often this happens to apps that are not native to Android, but written in some cross-platform framework like React Native or Cordova. This is not to say that we cannot build responsive apps with them, but we need to go to greather lengths to achieve that. One cross-platform framework for which this is not true is Flutter, at least in my experience so far. I am yet to see a non-responsive Flutter app.

Rule 3: Have touch states

Touch states are the difference between interacting with a panel of glass and interacting with something natural and beautiful.

It doesn’t matter whether it’s the classic default-hover-focus-press state list or the animated Material ripple effect. I believe that touch states contribute a great length to an app feeling “native”, “fluent” or “responsive”. In web apps the effect is even stronger: distinct hover states make the app feel much more “native”, even though it only runs in the browser. Thus touch states are closely related to responsiveness, but deserve to be considered on their own.

Unfortunately, there are numerous examples where touch states are missing. I’m sure you’ll find some in an app near you. If the user can click it and it does something, it needs a touch state.

When they are present, however, touch states should be a single colour. Not two, like in this example:

Rule 4: Align and pad

Consistent alignment gives an app a structured appearance. Malaligned items on the other hand make the app look messy and cluttered.

This effect is increased by too little padding. Remember Jitsi’s settings screen from above? Everything is crammed together, making texts harder to read and sections harder to differentiate at first glance.

One alignment example from ProtonMail:

And a very subtle centering example from Threema:

Subconsciously, something in the green header feels odd. It is only when we turn on layout bounds in the developer tools, that we see that the white content is top-aligned instead of being aligned to the horizontal center line:

Knowing this, it is easier to see (in the original image) that the green space above Hang up is smaller than the green space below. It’s like the arrow in the FedEx logo: once you know it, you can’t stop noticing it.

Summary

In this post, I described a number of points that I found to be common pitfalls when giving an app its final polish. I gave a few examples that I learned from and that continue to help me in not making the same mistakes.

These points complement Clean Code: just like Clean Code should be elegant, Beautiful UI should be delightful. Just like Clean Code should be pleasing to read for the developer, Beautiful UI should be pleasing to look at for the user.

Beautiful UI can be summarised in four rules:

1. Stick to the guidelines
2. Be responsive
3. Have touch states
4. Align and pad

Listing those is easy. Implementing them is hard.

Luckily, I found this Reddit post. I will repeat the steps here, both as a mental aid and in case that the original posts disappears. Also, I have slightly adjusted the steps.

0. In another tool (e.g., Gwenview), crop the screenshots so that the top and bottom system bars that are in the overlapping regions are gone.
1. Open GIMP
2. File > Open the top most screenshot
3. File > Open as Layers... all other screenshots
4. ∀ layers: set opacity to ~50%. (Select layer in the layer list on the left, then click opacity bar above.)
5. Image > Canvas Size...: liberally increase the canvas height. Make sure that “Resize layers” is set to “None”.
6. Click M to select the Move tool
7. ∀ layers: select the layer and use the arrow keys to move them up/down. Use Shift+⇅ for rough moving. Zoom in on the canvas to get the alignment pixel-perfect. Increase the canvas height again if necessary.
8. ∀ layers: set the opacity back to 100%
9. Image > Fit Canvas to Layers
10. File > Export As...

Consequently, there has been a lot of press about tracing apps. However, I feel like most news sites - even the tech media - have failed to underline some crucial points on how these apps (could) work. Sure, some have noted, maybe even explained them. But many have not, and those that have did not sufficiently stress their importance.

What do I mean by important? Important, in the sense that understanding them will enable you to develop a much better mental model of what tracing apps are. Important, because understanding them enables you to better participate in the current debate on tracing apps.

My goal is to explain the concepts, while striking a balance between too little detail and overwhelming the reader. If you want little detail, stay with the mainstream media. If you want to really dive into the topic, read the various white papers. (Especially if you are studying Computer Science. They are a great case study of security protocols.)

A note before I start:

• I won’t go into the pros and cons of tracing apps and the concerns they raise. I will simply try to explain a couple technical points.
• Whenever I speak of “tracing apps”, I mean tracing apps based on the DP^3T system. It is the one currently being implemented in Switzerland (among other countries). I cannot speak for other approaches.
• Contact tracing or proximity tracing? The Swiss Federal Office for Public Health FOPH as well as Google and Apple (mostly) use the term proximity tracing, so for the sake of consistency I will, too.
• I am going to make simplifications in order to strike a middle ground.

1. What data is uploaded to the server?

TLDR:

• always: random seed
• opt-in: contact event data only for contacts with infected patients
• never: location data, contact event data for unifected contacts

Some terminology: a “contact” is not an entry in your address book, but rather the event of being in close proximity with another person for some time.

Understanding what data is uploaded and why requires a brief explanation of what happens during your use of the app.

In the design described in section 2 of the DP^3T white paper, your app creates a long and random (and thus unguessable) sequence, the so-called seed. That seed is initally secret and only stored on your phone. From that seed a list of so-called ephemeral IDs are derived. Note that the other way around is unfeasible: Given an ephemeral ID you cannot compute the seed. Also note that without knowledge of the seed, you cannot learn whether or not two given IDs come from the same seed, i.e. come from the same user.

While using the app, it sends out these ephemeral IDs, going through the list one-by-one. One ID is broadcasted for some time, then the next, and so on. Regularly changing the IDs is what makes it harder to track you, since for anyone but you, the IDs look random. Nevertheless, your app stores all the other apps’ ephemeral IDs that it receives.

If and only if you are tested positive (and you enter the code the test center gives you), then the app uploads your seed to a central server, usually run by the local health authority. (Then, your app generates a new seed to start fresh.)

One reason for uploading the seed is to save data: the list of ephemeral IDs is huge since you change them regularly. And you do need to change IDs in order to prevent tracking – recall that there is no obvious connection between any two IDs.

The server collects a day’s worth of seeds from newly infected patients. Once a day, your app downloads the list of seeds of infected patients.

Then locally on your phone, your app goes through this list of “infected seeds”. For each seed it computes the corresponding ephemeral IDs, just like the infected user did to send them out. Your app then checks whether it previously “saw” any one of them. Under certain conditions (e.g. if it finds a minimum number of contact events with infected persons) it issues a warning. Thus, only now your app can make sense of the ephemeral IDs that it “met”.

DP^3T also describes a slight variant of the above (white paper section 3). This approach is currently not implemented by Google/Apple/Swiss PT, but it is still interesting enough to be mentioned as an outlook.

In this variant, your app would in the last step not receive a list of “infected seeds” and would not compute all the ephemeral IDs of the infected patients. Here is what happens instead:

The server still receives the seeds from the infected users like before. From them, it builds a so-called Cuckoo filter and sends that to all apps. Your app can check whether an ephemeral ID of another phone that it encountered was generated from a seed that the server put into the Cuckoo filter. Thus it can still issue a warning if needed. However it cannot in practise extract the “infected seeds” from the Cuckoo filter, thanks to the underlying maths.

Furthermore, DP^3T describes that the app can upload data about your contact with infected patients (e.g. when, how long, how often). This is to support the work of epidemiologists. It is purely optional and currently not implemented in the Swiss app.

2. Where do I have privacy?

Privacy in this is many-fold:

Privacy against other users/apps

Privacy against other users is preserved.

Other apps do not learn your identity, since the only link between your random seed and your identity is the code from the test centre. But that is not send to other users, it stays on the server.

Privacy is even stronger in DP^3T’s second variant: here, other users do not even learn the “infected seeds”. They only learn a Cuckoo filter, and from that the only thing that they can learn is whether or not they have a match.

Privacy against the central server

Privacy against the server depends on your situation:

1. You never test positive (or if you do, never enter the test centre code). Then no data is sent to the server.
2. You test positive and enter the code in the app. Then the server learns:

• that you are infected. Okay, the health authority saw your positive test anyway.
• your seed. Useless on its own though. Only useful to someone who met with you and can check for matching ephemeral IDs.

Crucially however, since the ephemeral IDs of your contacts do not leave your phone, the server is unable to build a social graph, a network of who is meeting whom. The only data your app sends to the server is “I am infected and this is my random sequence” but not “this is who I met”.

3. How are Google and Apple involved?

The Swiss proximity tracing app is going to use the software interface (API) that Google and Apple provide via system updates. What are the implications of this?

The idea behind using the operating systems’s API is to split responsibilities: The OS handles the exchange of ephemeral IDs and the proximity measuring. Since it has much deeper access to Bluetooth, the hope is that the accuracy of the system will be better. The app can then locally query the operating system for the encounters, and handle all the logic for storing and comparing the IDs, communicating with the server, and displaying a pretty UI.

Another advantage arising from the collaboration of the two tech companies is increased compatibility, since they agreed on a common interface. Have you ever tried sending a file via Bluetooth from an Android phone to an iPhone? See what I mean.

Also, both Android and iOS give you control over which app has access to the proximity data, similar to the way you can control permissions.

Note though, that the latest FAQ (as of writing) seems to be very careful not to explicitly say that no data will be shared with Google or Apple. It only talks about not sharing the identities of users with anyone, including Google and Apple.

Final words

I hope this helps a little bit in understanding conceptually what is going on with these tracing apps.

Yet, in the end, it all comes down to trust. Trust that the implementation really does what the specification says. Trust that the software contains no critical bugs. Trust that the maths and formalisms of the specification are correct. How this trust is earned and lost is a different matter.

Should you find any error, feel free to contact me.

Sources and further reading:

• DP^3T documents
• Apple ressources
• Google ressources

There are three significant Student Cluster Competitions: ASC in Asia, ISC in Frankfurt (Germany) and SC in the US. The last two are paired with supercomputing conferences. The oldest and “original” Supercomputing Conference (SC) takes place since 1988 and is the largest with over 11,000 visitors.

During each conference a Student Cluster Competition takes place: teams of 6 students build a small supercomputer with the help of their advisors and industry partners. They are given a set of benchmarks and real-world applications that they need to run within a power limit of 3 kW. Teams are then judged on the performance of their system as well as interviews with a jury where they need to demonstrate their knowledge of HPC.

I was lucky to be part of RACKlette, the first team from Switzerland to take part in one of these competitions. We first competed at ISC19 in June and now again in November at SC19. Despite being a newcomer we were quite successfull: in Frankfurt we placed third overall and won the LINPACK award and in Denver we placed 5th in a highly competitive field.

In this post, I will outline the main learnings our team had in the hope that it will be helpful to future teams. Among others, I will go over what you should remember to bring with you, some tips on how to tune both your software and hardware and finally what to expect from the competition in general.

What to bring

It’s easy, right? Your local supercomputing centre organises the shipment of the cluster, you bring your laptop, and you are all set.

Unfortunately, that’s far from it. Here is a - most likely incomplete - list of things you should bring to the competition:

0. Cabling:

   • 6 Ethernet cables: each of you needs to connect to the cluster via wire.
   • Ethernet-to-USB-C adapters (thank you Tim Apple)
   • Multiple socket outlet
   • CH-to-US adapter: different sockets!

1. Monitoring:

   • External monitor: Provided at SC but not at ISC, ask CSCS to ship one with the cluster
   • HDMI cable
   • Raspberry Pi: Setup InfluxDB and Grafana to read out IPMI readings
   • Second laptop: Always good if yours fails, plus you can use it do display the Grafana dashboard (the Pi is quite busy running the DB). Alternatively bring your iPad to be more productive with Sidecar (please can somebody build an equivalent for Linux?)

2. Varia:

   • Coffein: Very important at SC where the competition lasts 48 hours non-stop. The organisers also provide coke, but quickly there is only diet coke left, so buy some yourself at the local supermarket.
   • Replacement SSD: One team lost their boot drive. It’s unlikely, but if it happens you want to be prepared.
   • Paper clip: In case your team mate locked the rack but is now asleep and you need to reboot after a power outage.
   • National flag: a Swiss flag always attracts extra love from your human fans, and more people stop by a nicely decorated booth. Looks great if blowing in the wind of the cooling fans.

Tuning applications

Tuning things is what this competition is all about. There are a thousand different means to that end, and thanks to the complexity of modern systems much of it comes down to lots of time invested and trial-and-error. But if you suddenly get a 2x speedup it can feel very worthwhile! (Remember, running inputs naively can take 6ish hours each!)

Other than getting speedup, the other thing to keep in mind is the power limit. You cannot throw 16 Nvidia V100s at it, each consuming 250 W peak power, if the committee’s alarm goes off at 3 kW!

Hardware

First GPUs. When we competed we - like most teams - had Nvidia cards. You can use the CLI tool nvidia-smi to instruct your GPUs to run at a maximum frequency or maximum power. Try both, depending on the task one may be faster or one may result in a more stable power consumption.

Second CPUs. Use cpupower frequency-info -p to read out and cpupower frequency-set to set frequencies. We wrote a script for that to quickly set frequencies across all nodes using pdsh.

And frequency matters! At SC19, for SST Ember the input had ten iterations with each repetitively doing the same kind of work. When running at 2.6 GHz one iterations took 38 min on our hardware. At 2.8 GHz it only took 33 min. For ten iterations that saves you 50 minutes!

You can also set frequencies manually during a run. (Or automatically, if your so inclined to script it.) One prime example is LINPACK which has a big spike at the beginning and than uses less power if kept capped at the same frequency. In addition, here is an example of us manually adjusting the CPU frequency for SST Ember during SC19 (notice how each iteration has this bump at the beginning, and how close we are driving it to 3 kW):

Also be aware of turbo boost. To avoid sudden unexpected (and unnecessary) spikes over the power limit, you can limit the CPU frequency to something which the CPU can continuously maintain.

And third, cooling. Fans use power, too. During specially important things like LINPACK it is possible (but not necessarily recommended) to slow down the fans via your motherboard’s IPMI. Luckily our Gigabyte motherboards had a webinterface to do that, but there is also a CLI available.

DISCLAIMER: You MUST test this before the competition. If you set the fans too slow and your systems becomes too hot during the run, it becomes unstable, components go into lets-rescue-myself mode and you need to reboot!

Temperature matters too. At SC19 we ran HPCG in the morning just fine, though reproducibly very close below the power limit. When we re-run it in the afternoon, all three attempts spiked over the 3 kW limit (with the exact same configuration, i.e. same inputs, frequencies, fan speeds). We currently acknowledge this to a 1-2°C increase in temperature in the conference hall (because the IPMI was reading a 1-2°C higher CPU temperature, and our cooling fans were at fixed speeds).

Software

Try different compilers: gcc, icc, you name it. -O3 is always good, plus -march=native to tell the compiler it can use vectorisation. Do not overdo it with flags, at some point you will get worse results! Also be aware of flags that icc only has for compatibility reasons: e.g use -xSKYLAKE-AVX512 instead of -march=native.

Also try different MPIs: OpenMPI, IntelMPI, MVAPICH, MPICH. According to their docs MPICH has checkpointing. We only found that out during SC19, but that sounds like something worth looking into. Note that all of them may have slightly different syntax.

Input files

Understand your input files! The organisers - especially at SC - give you inputs that are not optimal and are meant to test your understanding of the application. For example for SST the input we got used --partitioner=linear but there are something like five others to partition the workload.

Or if there are matrices involved think about how much you assign to each MPI rank.

Ideally you had a chance to play with these parameters beforehand and know what to do. However it’s not unheard of that people (that is, I) have found out about other tuning opportunities while the competition was already going on. So don’t be afraid of a blind shot if you think it can reasonably give you more performance!

Varia

Always check the power reading on the provided official PDU! For one, it generally updates in real-time (i.e. every second) while you or the organisers’ Grafana may only update every five to thirty seconds. Our team lost two hours of compute time during 3am and 5am because we were only watching our Grafana, which unfortunately decided to update with outdated data so we didn’t notice our job had finished. So watch the PDU!

The Cluster Competition Experience: On the conference floor

Very exciting, you are finally there at the Supercomputing Conference! You spend a day searching your cluster because it was shipped to a completely different booth somewhere else in the hall, but now you are all set up and ready to go!

Here are some tips:

• Be ready to mingle with the other teams. Everyone is really nice and very outgoing, and chatting about how you optimise applications or what results you got for that input file is completely normal and okay. It great to see more people doing HPC!
• Expect to be interviewed. Not only by the judges, but also by the conference organisers, your sponsors and Dan Olds. Dan publishes articles about HPC (e.g. on HPCwire) and is an incredibly kind, chatty though eloquent and entertaining character. You will notice when you see him! He interviews all teams every year. Because he has been at the cluster competitions since the beginning he knows them inside out.
• At SC (and maybe at ISC in the future?) there is a power outage, planned by the committee. It is meant to test your ability to recover from such an event. This year at SC19 they took everyone by surprise by pulling the power not only once but twice!
• If you can, walk around the conference hall and check out the exhibition. There is cool stuff on display, and sometimes you even get merchandise. Some booths host raffles, so there’s your chance to win a NUC!

Final words

The student cluster competition is an incredible experience. Meeting like-minded students, experiencing the conference itself, learning a thousand things about HPC, computer systems, networking, compiling and application optimisation in the progress makes it a very unique challenge but also opportunity.

Although secretly, it’s only an international sweets exchange really: we brought Swiss chocolate and received sweets from Estonia, Poland and Shanghai in return.

This post also appeared on the RACKlette team blog

So you might ask, what’s the difference between a supercomputer and one of Google’s datacentres?

First is their configuration: even though both have lots of cabinets stuffed with computer hardware, the hardware in a normal datacentre is set up in a way such that it functions as many “small” computers (but those are still bigger than your PC at home!). The hardware in a supercomputer on the other hand is all tied together to form one massive computer where you can run your program on.

And the second difference is their use case: common services like websites, databases or application backends usually run in datacentres. Supercomputers on the other hand are mainly used for scientific simulations. Those might be academic (simulating expanding galaxies or protein reactions) but can also be industrial (simulating airflow over a new airplane design iteration rather than having to physically build and test it in a windtunnel). So both require a huge amount of compute power, but the first does so because it needs to be able to handle large amounts of clients and the second because it solves inherently difficult problems with brute force.

And even if you didn’t fly lately, HPC directly affects you every single day: Every few hours, a supercomputer in the Swiss Supercomputing Centre (CSCS) in Lugano crunches the latest weather data and runs the meteorologists’ simulations to come up with a new weather forecast(*). Meteo Swiss is a leader in this area and thanks to being one of the first to run these simulations on GPUs it is able to achieve predictions for a much finer grid than neighbouring countries.

That’s a short overview of what HPC is and does. If you want to find out more have a look at the field’s two big news websites HPCwire and insideHPC.

(*) Please stop giving bad reviews in the App Store if the weather prediction is bad, it’s usually not the app developer’s fault but the meteorologist’s!