Under certain circumstances, it can be necessary to build a customized Linux kernel, for example, with a different configuration, or with patches applied.

This is a relatively typical task, so there is plenty of information around, however, I’ve found lack of clarity about the concepts involved, outdated and incomplete information, etc.

For this reason, I’ve decided to write a small guide about this task, which can be used only as copy/paste reference, but also fully read, in order to get a better understanding of the concepts involved..

Content:

• Requirements
• Installing the required packages
• Choosing the source code
• Downloading the source code
• Patching the kernel
• Kernel configuration concepts
• Tools to set up and modify the kernel configuration
• Necessary changes
• Applying the desired customizations
• Building the kernel
• Conclusion

It’s a well-known fact that when moving from metal to the cloud, costs will typically increase (hopefully, trading it for reduced maintenance and/or increased system resilience).

There are some important things that are very hard to assess, before moving to the cloud service itself.

We moved, long ago, to AWS, and we had certain surprises; in this article, I’ll describe them, so that companies that plan to move to the cloud can make more informed decisions.

This article is updated to Jul/2022, and I will update it if/when I’ll found other notable things.

Content:

• Disks (EBS), also for database services, have an I/O budget
  • The bottom line
• Storage services can’t be stopped
  • The bottom line
• Last generation database services may not be necessarily reserved if they’re Intel/AMD
  • The bottom line

Github Actions are the new de facto standard for projects CI (and many other tasks).

Through a certain action, actions support data caching. I was very surprised though, when I’ve noticed that caching, as frequently described, has a very severe limitation - it’s not shared across PRs; this limits its performance severely.

In this small article, I’ll describe the problem, the solution, and two preset workflows, in Ruby and Rust.

Content:

• The Problem
• The solution
• Implementations
  • Ruby
  • Rust
• Conclusion

I’ve recently released my mini-book “Learn Bevy’s ECS by ripping off someone else’s project”; it’s a mini-book that uses the game Rusty Roguelike from the book Hands-on Rust as a base, in order to explain Bevy’s ECS.

The idea is for a beginner to learn ECS concepts from the base book, then apply them using Bevy; the structure of the game is ideal for a gentle introduction to ECS architecture.

Read it here!

There are a few strategies to improve Bevy’s compilation time; one of them is to enable dynamic linking (features = ["dynamic"]).

While this works fine when manually invoking Cargo, attempting to launch a debug session from Visual Studio Code will raise this error:

/path/to/project/target/debug/project: error while loading shared libraries: libbevy_dylib-ae04813e8bd66866.so: cannot open shared object file: No such file or directory

This is a relatively common topic on the net, but the solutions presented are not very clear.

What one exactly needs to do is to add this entry to the launch configuration (in launch.json):

"env": {
  "LD_LIBRARY_PATH": "${workspaceFolder}/target/debug/deps:${env:HOME}/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib",
}

This assumes that the dev uses Rustup and the nightly toolchain; if one uses the stable toolchain, replace nightly with stable.

Happy debugging 🙂

I’m reading the book Machine Code Games Routines For The Commodore 64; since there is no errata, I’m publishing my findings.

The pages referred are the printed ones.

Content:

• Page 012: JSR/RTS operation
• Page 047: Spiral fill: LDR and comment
• Page 071: Small memory fill: Address off by one
• Page 080: Fundamental Bomb Update: Start location
• Page 081: Hail Of Barbs BASIC: Data read cycle and entry point
• Page 085: 256 Bytes Continous Scroll: Wrong addressing mode
• Page 090: Joystick handling: Misplaced comment
• Page 094: Attribute Flasher: Off-by-1 error
• Page 095: Alternate Sprite System: Encoded sprite missing entry
• Page 111: Invalid tune entry
• Pages 114/115: Mixed up Window Projection concepts
• Page 116: Projecting a Landscape: Many bugs

I’ve recently completed the fantastic book “The Art of 64-Bit Assembly, Volume 1” (Randall Hyde, No Starch Press).

Since the errata of this book is hard to find, I’m posting here the link(s), in the hope that it gets more visible to the search engines.

The current errata discussions are in the book board of Randall Hyde’s forum.

The dicussion “Errata Table 2-9” contains several errata; other errata are reported in self-standing threads. I’ve contributed two 🙂

There isn’t anything particularly confusing, except one erratum (the subtraction/carry one).

Happy assembling 😁

Today is Christmas Eve, a notoriously famous day for experimenting with shell scripts! 😁

While updating a script of mine, I came across the problem of padding a filename.

Padding a filename is simple task, however, finding the simplest possible way of doing it in a shell script, is actually not so simple.

In this (quick) article I’ll explain the problem, and my solution.

Content:

• Design disclaimer
• The problem
• Issues with a typical find pipeline
• The st00pid simple solution
  • Regex-boosting the st00pid simple solution
• The epic solution
• Conclusion

The book The Art of 64-Bit Assembly has been recently released.

The book is an update in the series “The Art of Assembly Language”; while very popular, it has a problem: it’s based on the MASM assembler, which is tied to the Windows platform.

MASM has a multiplatform fork called UASM, however, it hasn’t been possible to build the project on Linux for a long time now.

I’ve therefore created a fork with all the fixes required to build it on Linux; this will hopefully be useful to people who, like me, are studying the book using a Linux system.

Check it out here.

I’ve recently completed the book Beginning x64 Assembly Programming Errata.

The book has several errors, at least a couple of which are significant; since there is no official errata, I’m publishing my findings.

Content:

• Introduction
• Page 68: Addressing forms
• Page 156: Using objdump
• Page 160: Working with I/O
• Page 206: Moving Strings
• Page 217: Using cpuid
• Page 328: Matrix Print: printm4x4
• Page 384: Using More Than Four Arguments
• Conclusion

Compiler Explorer (often known as “Godbolt”) is a very convenient tool for exploring the disassembly of Rust programs.

It has a significant limitation: it can’t disassemble projects depending on external crates.

After some hairpulling, I’ve found how to achieve Godbolt functionality in Visual Studio Code, without such limitation.

In this (quick) article, I’ll show how to do it.

Content:

• Setup
• Preparation of the assembly file
• Compiler Exploration :)
• Inlining notes
• Conclusion

I’m very annoyed by how slow installing Ruby gems is, even with a large number of available hardware threads (and even with many Bundler jobs configured).

I had a quick look a the Rubygems/Bundler source code today, and I’ve found a quick win.

Content:

• Context
• The tweak
• Conclusion

Today (Jul/15/2021), on Hacker News, it’s been published the CVE-2021-22555: Turning \x00\x00 into 10000$ (entertaining) article.

In the related GitHub repository, the source of the proof of concept is provided, however, for programmers not familiar with C, it’s not obvious how to compile it.

A list of kernel versions is provided, however, there are cases where identifying the kernel exploitability is not immediate.

In order to avoid hairpulling to fellow sysadmins, this article shows the procedure to compile the PoC on modern Ubuntu/Debian machines, with comments.

Content:

• Procedure

Those who work on many open source projects will surely find irritating to handle the different naming of the development/default branch.

In this short article, I’ll show how conveniently handle it, and as usual, I’ll sprinkle it with a bit of Linux tooling.

Content:

• Conventions and general information
• Finding the default branch
  • Offline version
  • Online version
• Integrating with git aliases
• Handling repositories with a development branch that is non-default
• Conclusion

The 0.5 series of the ZFS installer has been released.

The new version integrates parts of the updated official procedure for installing ZFS on Ubuntu-based distributions.

The main changes are:

• ability to specify the root pool datasets, for users who want a granular subdivision of the filesystem;
• KUbuntu support;
• removal (at least temporary) of Debian support, as it diverges more than expected from the base procedure.

Users can follow the instructions on the home page, which is simply:

GET https://git.io/JelI5 | sudo bash

A common concept to deal with when working with floating point numbers, is approximate equality. While the implementation itself is simple (a specialized function will do), Rust gives a more elegant option.

In this article, I’ll explain how to implement approximate equality via traits.

Content:

• The basic approach to the problem
• The doubt
• The solution
• Implementing the solution on a trait
• Conclusion

RISC-V is a hot topic lately. There’s lots of talk, and experimentation/research.

Experimenting on the platform is actually easy, through open source tools. In this article I’ll explain how to easily setup the RISC-V development and emulation tools, and how to build and run, as example, the parallel gzip compressor (pigz).

I’ll also show how to configure the QEMU disk image, and a few useful related functionalities.

Content:

• Introduction to the tooling options
• Setup
  • Preliminary steps
  • Building the toolchain
  • Building zlib and pigz
  • Downloading and preparing the Fedora RISC-V image
  • Installing QEMU
• Execution!
• Conclusion
• Footnotes

Associating file types (extensions, in Windows) to applications is a fundamental functionality of desktop environments.

Some people may still have a troubled experiences while doing so; some others may want to script the operating system configuration. For both, I’m writing a small article about how to make such association via commandline, using standard Unix programs.

Content:

• Introduction
• Procedure
  • Finding the desktop file
  • Finding the MIME type
  • Registering the association
• Association by extension
• Conclusion

This is the beginning of my “Rust Lulz” articles; the subject is simply things that, for one reason or another, I’ve found amusing while learning/applying Rust.

In this article, I’ll explain how to compute the cofactors of a matrix via bit manipulation (assuming an existing minor() function).

Content:

• The boring approach to the problem
• Walkthrough
  • Introduce a factor
  • Sign in number binary representations
  • Bit manipulation logic
  • IEEE-754 format
    • Rust bit operations on floats
  • Solution outline
• Implementation
• Conclusion

Email subjects are prone to “prefix piling” (e.g. Re: AW: Fwd: Subject); this easily gets very annoying, requiring the user to manually edit the subject.

Some email client tools allow automated subject editing, which, with the right rules, solves the problem.

In this article I’ll explain which regular expressions to use in order to solve the problem, along with the regular expression concepts required.

Content:

• Requirements
• The problem
  • Repetition metacharacters
  • Groups
  • Beginning of string metacharacter
  • Lookarounds
  • Negative lookarounds
  • The solution
  • The null-character
• Conclusion

Chef users, more or less frequently, run a resource with alternate credentials (user/group). This is easily done by configuring the user/group property of the resource, however, this is only part of the picture.

Setting only those two attribute will, in some cases, cause the resource to run with unexpected environment values, leading to subtly broken system configurations.

In this article, I’ll explain why and how to properly run a resource with alternate credentials.

Content:

• The problem
• Reviewing the cause, and the issue details
• Solution
• Conclusion

The “CPU Steal time” issue is a well known phenomenon in the cloud area. We’ve been recently hard hit by this problem, so we’ve decided to start monitoring it.

In this article I’ll explain how to easily configure an AWS CloudWatch metric in order to report it.

Updated on 02/Nov/2020: Fixed reload (restart) logic.

Content:

• Background and disclaimer
• Procedure: preliminary information
  • Overview
  • Requisites
  • nmon and its output
• Procedure
  • Packages/libraries
  • Variables
  • Systemd unit
  • Log rotation
  • Processing script
  • Processing script scheduling
• Conclusion

Yesterday (17/Jul/2020) I’ve added a separate feed for MySQL topics. It can be accessed at the address https://saveriomiroddi.github.io/feed/mysql.xml (also represented by the dolphin icon in the navigation bar of the blog).

A common MySQL strategy to perform updates with accumulating functions is to employ user-defined variables, using the UPDATE [...] SET mycol = (@myvar := EXPRESSION(@myvar, mycol)) pattern.

This pattern though doesn’t play well with the optimizer (leading to non-deterministic behavior), so it has been deprecated. This left a sort of void, since the (relatively) sophisticated logic is now harder to reproduce, at least with the same simplicity.

In this article, I’ll have a look at two ways to apply such logic: using, canonically, window functions, and, a bit more creatively, using recursive CTEs.

• Requirements and background
• The problem
• Setup
• The old-school approach
• Modern approach #1: Window functions
  • High-level logic
  • LAG() window function
  • Technical aspects
  • Named windows
  • PARTITION BY clause
  • Ordering
  • Considerations
• Modern approach #2: Recursive CTE
  • Working version
  • Performance considerations
  • Alternative for suboptimal plans
• Conclusion

(Those looking for a ready Ruby PPA, please have a look at the announcement article )

Recently, the ubiquitous Brightbox Ruby PPA has been discontinued.

This caused a problem, because there aren’t other stable Ruby PPAs, thus requiring engineers to manually package their own Ruby.

In this article, I’ll explain how to setup a Ruby PPA and the related scripts, so that engineers can package and automatically deploy their own Ruby, and, very importantly, benefit from automated updates, virtually without any manual operation.

I’ve dealt with the PPA subject in the past; this article is an “updated, extended, and more automated version”™.

Content:

• Disclaimer
• Requirements
  • System preparation
• Procedure
  • High-level overview
  • Procedure style
  • Preparation: variables
  • Preparing the package base metadata
  • Preparing the debian/rules
  • Builder configuration
  • Per-distro build steps
  • Building the source package
  • Testing the build
  • Uploading the package
• Automating the operation
• Conclusion

As a companion of the PPA learning article, we (Ticketsolve) are publishing a PPA with the stable, vanilla Ruby versions, which we’ll keep up to date for the foreseeable future.

Content:

• Disclaimer
• Installation
• How about Fullstaq Ruby?

For a variety of reasons, writing (Bash) shell scripts requires lots of care. When writing non-trivial shell scripts, it’s crucial to take precautions; shell options are part of this practice.

Typically, three shell options are suggested: errexit, nounset and pipefail, and occasionally, noglob.

However, there are other ones that developers will find important in this context: errtrace and inherit_errexit; in this article, I’ll explain them.

Content:

• A recap of the common shell options for solid programming
• errtrace (-E)
• inherit_errexit
• Conclusion

Another “missing and missed” functionality in MySQL is a data type for arrays.

While MySQL is not there yet, it’s now possible to cover a significant use case: storing denormalized columns (or arrays in general), and accessing them via index.

In this article I’ll give some context about denormalized data and indexes, including the workaround for such functionality on MySQL 5.7, and describe how this is (rather) cleanly accomplished on MySQL 8.0.

• Terminology
• Storing and indexing arrays in MySQL 5.7: an approach, and problems
• The MySQL 8.0 implementation: data type and index
• Performance expectations
  • Why multiple arrays can’t be indexed
• How do I declare an ARRAY UNSIGNED column?
• Conclusion
• Footnotes

Another interesting feature released with MySQL 8.0 is full support for functional indexes.

Although this is not a strictly new concept in the MySQL world (indexed generated columns provided the same functionality), I find it worth reviewing, through some applications, notes and considerations.

All in all, I’m not 100% bought into functional indexes (as opposed to indexed generated columns); I’ll elaborate on this over the course of the article.

As a natural fit, generated columns are included in the article; additionally, some constructs build on my previous article, in relation to the subject of CTEs.

Updated on 12/Mar/2020: Found another bug.

Contents:

• Terminology
• Generated columns, and their application on JSON data
• Functional indexes
• JSON functional index gotchas
  • Expression exactness
  • Inconsistent behavior between generated columns with index, and functional indexes
  • Encoding inconsistency based on the index usage
• An example of functional index with dates
  • Gotcha: JOINs don’t use functional key parts
• Bugs
  • Bug on CREATE TABLE ... SELECT
  • Bug on LOAD DATA INFILE
• Conclusion

A long-time missing (and missed) functionality in MySQL, is sequences/ranges.

As of MySQL 8.0, this functionality is still not supported in a general sense, however, it’s now possible to generate a sequence to be used within a single query.

In this article, I’ll give a brief introduction to CTEs, and explain how to build different sequence generators; additionally, I’ll introduce the new (cool) MySQL 8.0 query hint SET_VAR, and a pinch of virtual columns and functional indexes (“functional key parts”, another MySQL 8.0 feature).

Contents:

• A brief introduction to Common Table Expressions (CTEs)
• Recursive CTEs, and generating a linear sequence of integers
  • Per-statement variables setting
• Generating a sequence of random integers
• Generating a characters interval
• Generating a dates interval
• Conclusion
• Footnotes

Sometimes, either in scripts or direct commands, there is a series of repetitive, similar, commands, which could be executed in parallel.

Bash offers means for very basic parallelization (& and wait), however, they’re not very practical for generic solutions.

In this article, I’ll explain how to use GNU Parallel, which makes parallelization trivial, and, as usual, introduce some other useful shell concepts.

Contents:

• Using Bash built-in commands
  • Conclusion
• Enter the stage: GNU Parallel
  • Important to known: CPU count, from a Linux perspective
  • Base example: untarring multiple archives in parallel
  • Using positional parameters
  • Running multiple commands from a file
• Escaping strings in Bash
• Sending the content of a file to a process
• $XDG_RUNTIME_DIR
• Conclusion
• Post-conclusion

I’ve been a long time user of SBCs; as part of my equipment, I’ve always had the need for a small home server, which they fit increasingly well.

My previous SBC has been an ODROID XU4, which I’ve covered in a previous article.

I’ve been reasonably happy of it, however, my requirements overgrew its limitations, therefore, as soon as the second batch of ODROID H2 was available, I jumped at the chance to switch.

Like in the XU4 article, I will make some general considerations about small home servers and how to H2 fits the (my) requirements, and how ARM and x86 board nowadays relate.

Contents:

• Part 1: History and considerations
  • What do I do with a home server?
  • The bumps
  • ARM boards come with an expiry date
    • A specific example of the difference between a modified kernel and a vanilla one
  • Requirements
  • A brief look at the evolution of SBCs
• Part 2: The ODROID H2
  • The bad: the enclosure
  • The OK
  • The good: all the rest
  • The alternatives
• Part 3: A comparison of H2, N2 and Pi 4
  • The ODROID N2
  • Benchmarks
  • Pricing
• Part 4: The future
  • ARM
  • x86
• Conclusion

In this post I’ll expand on the subject of my MySQL pre-FOSDEM talk: what dbadmins need to know and do, when upgrading from MySQL 5.7 to 8.0.

I’ve already published two posts on two specific issues; in this article, I’ll give the complete picture.

As usual, I’ll use this post to introduce tooling concepts that may be useful in generic system administration.

The presentation code is hosted on a GitHub repository (including the the source files and the output slides in PDF format), and on Slideshare.

Contents:

• Summary of issues, and scope
• Requirements
• The new default character set/collation: utf8mb4/utf8mb4_0900_ai_ci
  • Summary
  • Tooling: MySQL RLIKE
  • How the charset parameters work
  • String, and comparison, properties
  • Collation coercion, and issues general <> 0900_ai
    • Comparisons utf8_general_ci column <> literals
    • Comparisons utf8_general_ci column <> columns
    • Summary of the migration path
  • The new collation doesn’t pad anymore
  • Triggers
    • Sort-of-related suggestion
  • Behavior with indexes
  • Consequences of the increase in (potential) size of char columns
• Information schema statistics caching
• GROUP BY not sorted anymore by default (+tooling)
• Schema migration tools support
• Obsolete Mac Homebrew default collation
  • Modify the formula, and recompile the binaries
  • Ignore the client encoding on handshake
• Good practice for (major/minor) upgrades: comparing the system variables
• Conclusion

When managing Ubuntu Server installations, for example image templates in the cloud, one of the main issues one comes across is apt locking, which causes the the annoying error Could not get lock /var/lib/apt/lists/lock, typically “at the worst times”.

This seems to be a matter of discussion on the Stack Overflow network, but there are no working and stable solutions.

In order to get an idea of the confusion about the subject, check out the number and scope of solutions in this Stack Overflow question.

In this post I’ll talk about a few approaches, and the stable solution I’ve implemented.

Readers interested in just solving the problem can skip to the Eureka™ section.

Contents:

• The general setup
• Assumptions
• The problem
• Potential solutions
  • Retrying in case of errors
  • Inspecting the running processes
  • Disabling the services
  • Waiting on cloud-init to complete
  • Using flock
  • Using fnctl
  • The “screw it” solution: replace /usr/lib/apt/apt.systemd.daily
• Eureka™
• Conclusion

Every month, I purge the files trashed more than one month before.

Since it’s been scientifically proven that manual operations cause PTSD in system administrators, I’ve made a script.

In this small article, I’ll explain some concepts involved, most notably, working with dates and numbers in Bash, and some other scripting-related concepts.

Contents:

• The base structure
• Writing safer Bash scripts
  • errexit
  • nounset, with pattern
  • pipefail
    • pipefail versus grep -q
• First step: Handling dates, cycling the data
• Splitting a string via Perl
• Splitting a string via cut
• Dates, arithmetic, and putting all together
• Conclusion
• Footnotes

While researching a MySQL subject, I needed to process the MySQL server status variables (output), in order to extract and process some.

A computed value in particular (the max checkpoint age) required more elaborate processing.

This post explains a few Perl text processing functionalities (optionally aided by Awk), through many approaches to solving the problem.

Contents:

• Input data, problem, and clarifications
• The approaches
  • Slurp mode, with single regex
  • Slurp mode, with global match
  • Conditional line matching
  • Two-passes processing, with single-line intermediate output
  • Two-passes processing, with readline in the second pass
• Conclusion: should cryptic (Perl) scripts be avoided in a team context?

Yesterday, I’ve officially released my ZFS Ubuntu installer.

This project will hopefully help the adoption of ZFS, at least by appealing curious/interested sysadmins.

Canonical is about to officially release an Ubuntu with built-in ZFS support, however, it has extremely limited functionality.

Create a VM, launch the program, and have fun playing with ZFS!

Although there is a complexity threshold for the convenience of shell scripting, there is still plenty that (in my opinion) can be done before hitting such ceiling.

In this brief article, I’ll explain how to use indirect references to write a function that prints the names and content of the variables passed; this is useful when writing debugging code for a script.

Contents:

• Specification
• Quick introduction to Bash array data types
  • Arrays
  • Associative arrays
• Other notions
• Indirect references, and writing the function
• Debugging goodies
• Conclusion

Debugging a live/stuck Ruby process is a well-known subject.

The way it’s generally exposed is simply a series of instructions and their outcome; given the expectation, this is fine of course, however, when I saw that a manual copy/paste operation was required, I decided to… step in (pun intended 😂).

This article adds only a few concepts, operatively speaking, but it clarifies all the concept involved, and employs neat approaches to accomplish the task. I will also employ several goodies available to Linux systems.

Contents:

• Premises
• A brief overview of debugging a process, and ptrace
• Using pgrep
• A brief overview of file descriptors
• Putting together Ruby and GDB (with fancy grep!)
• Basic GDB usage
• The procedure
  • The procedure, neater!
• Safety of messing with file descriptors
• Other GDB/Ruby tools
  • Evaluating Ruby statements
  • An alternative option to redirecting stdout/stderr
  • Other interesting sources
• Conclusion

In my latest sysadmin experimentations, I’ve configured the unattended upgrades on my new server (an Odroid H2 😍).

Among the other things, I had to add a PPA to the list of allowed origins to automatically upgrade. This required some interesting text processing, so I decided to dig into Perl’s text processing functionalities.

This post describes how to enable unattended upgrades for a PPA, while explaining several useful Perl features, in particular, how to process key/value tuples; a few regular expression functionalities are also used.

Contents:

• Update
• A high level overview of how to enable unattended upgrades for a PPA
• Premises, and structuring the procedure
  • Capturing groups
  • Processing the input text as a single string
    • The “multiple lines” match operator modifier (m)
  • The “global” match operator modifier (g)
  • The join() function
  • Adding a line to an input, after a match
• The final commands
• Conclusion

A functionality that I require, relatively frequently, when scripting, is to print the last N fields of a stream, with awk.

A typical example, is to copy part of the commits of a git cherry/log command:

+ 81061edd2023c399539f1ff5cfdc267fd41c5c43 Ruby GUI development article: add `Some references` section
+ 5c60b7ef357683137b5f772f8590ab7d12c8e218 Ruby GUI development article: add `Footnotes` section
+ b6fe3469bc2ae59a7eba629c16ab11c67b7fbcbf Ruby GUI development article: add note about browser-based toolkits
[...]

This post explains how to do it, with some extra goodies.

Contents:

• Reformulating the approach
• The pieces
  • Automatically printing the newline (in Perl 5.x)
  • Autosplit
  • Slicing an array
  • Send the stdout of a process to the global clipboard
• Putting everything together
• Conclusion
• Footnotes

2019 is a very exciting year for people with at least a minor interest in storage. First, in May, the ZFS support for encryption and trimming has been added, with the release 0.8; then, in August, Canonical has officially announced the plan to add ZFS support to the installer¹ in the next Ubuntu release.

As of now, achieving a full-ZFS system (with a ZFS root (/)) is possible, although non trivial. In this walkthrough, I’ve made the procedure as simple as possible, additionally setting up encryption and mirroring (RAID-1).

I’ll also give an introduction about the concepts involved, so that one has better awareness of the purpose of all the steps (but feel free to jump directly to the procedure).

Note that this guide is stable, but I’ll keep adding a few more sections and content in general.

Contents:

• Update
• An introduction to the concepts involved
  • The boot process: BIOS and EFI
  • GRUB
  • The /boot directory/mount
  • /boot vs. /boot/efi
  • Ubuntu’s ZFS support
  • Chroot
  • Udev
  • ZFS pools, filesystems and volumes
• Philosophy and limitations
• Requirements and specifications
• Procedure
  • EFI check
  • Preparations
  • Install the ZFS module in the live (loaded) kernel
  • Find the disks
  • Prepare the disk(s)
  • Install Ubuntu
  • Chroot and prepare the jail
  • Install ZFS 0.8 packages
  • Install and configure the bootloader
  • Cloning the EFI partition
  • Configure the boot pool import, and remaining settings
  • End
• Tweaks
  • Databases
• Conclusion
• Footnotes

Today we’ve had an emergency in production, which caused us to process log files.

An operation that comes very helpful in these situations is to quickly compute aggregates for values found in logs (or text files, in general).

This is possible, and actually easy, with the usual *nix tools; in this post I’ll explain three approaches to this problem.

Contents:

• Premises
• The generic and flexible, but a bit clunky solution: Perl + SQLite
• The simple and elegant solution: GAWK or Perl
• Working around OS X
• Bonus
• Conclusion

Fairly recently, we’ve upgraded to MySQL 8; it’s been a relatively smooth transition, however, some minor differences needed to be handled. One of them is the behavior of trailing spaces.

Trailing spaces are a (not in a good way) surprising, but also widely covered argument. This article gives a short overview, and relates it to how this affects people upgrading to MySQL 8.0.

Contents:

• Premises/Requirements
• Behavior in different contexts
  • Comparison (=) predicate (1)
    • Inspecting the collations
  • Comparison (=) predicate (2)
  • LIKE predicate
  • Unique indexes
  • DISTINCT predicate
  • GROUP BY clause
• Conclusion

Perhaps surprisingly, the ISO standard documents are not publicly available; they must to be puchased at a - significant - price (see a commentary on Quora).

Below there are links to documents that can be found on the web (documents are not provided in this post, only links). If some are offline, the Wayback Machine may have a copy.

During my last vacation, I wanted to watch some very old cartoons, during the daily break, on the digital media player that the apartment provided.

The player was very old, and couldn’t handle the video file (a modern MP4). Therefore, this was a good occasion to exercise the FFmpeg conversion and processing capabilities.

This post will give a glimpse of the tools that FFmpeg provides, showing how easy it is to perform the video/audio conversion, with extra processing.

I will also explain general concepts of media handling, like containers and encoding formats.

Contents:

• What is FFmpeg (and Libav)
• A brief introduction to video file related concepts and formats
• First attempt: a basic audio/video conversion
• Interlude: encoding bitrate modes
• Scaling (down) the video
• Backpedaling to CBR
• Mixing (down) the audio channels
• Splitting the file
• Conclusion

When it comes to configuring MySQL, a fundamental step is to find out which configuration files the MySQL server reads.

The operation itself is simple, however, if we want to script the operation, using text processing in a sharp way, it’s not immediate what the best solution is.

In this post I’ll explore the process of looking for a satisfying solution, going through grep, perl, and awk.

Contents:

• Assumptions
• Input data (finding the configuration files read by MySQL)
• First step: grep+tail
• Second step: expanding the tilde
• Final step: awk’s super powers
• Extra step: using the output
• Conclusion

Recently, I needed to test some operations to be performed during the installation of Ubuntu on a system.

This requires multiple sessions and reboots, so retaining the changes on the installation media would considerably streamline the task.

VirtualBox doesn’t support booting a VM from a USB flash drive though, so a workaround is required.

This small article explains how to do this.

Contents:

• Options and procedure(s)
  • Using the VMDK format
  • Using the key dump as a disk
• Conclusion and considerations

Due to some constraints, at Ticketsolve we sometimes need to work with an ancient file format: the Paradox Database.

This file format was in use between the 80s and 90s. In order to perform some operations on Paradox databases, nowadays, there are libraries based on the file format reverse engineering work by individual open source programmers, or ad hoc commercial programs.

Additionally, one can use Paradox 7, the reference commercial software originally published in 1992, now abandoned.
This software works good enough in Wine, however, the installer generally raises an error on installation, complaining that there isn’t enough disk space.

In this post we’ll use Radare2, one of the most powerful open source reverse engineering frameworks, in order to statically analyze and patch the installation binary, so that the pesky error is not triggered anymore.

There are no requirements for the reader; knowing x86/32 assembler and the PE executable format will improve the understanding, but it’s not a requirement by any means.

Contents:

• Disclaimer
• Tools
• Problem
• Finding a starting point
• Examining the relevant function
• Deciding and approach, testing the theory, and patching the binary
• Digging further
• Taking the simplest approach, and creating the final patch
• Distributing the patch
• Conclusion

I wanted to tweak a script of mine; it included the conventional find </path> -maxdepth 1 -type f. I wasn’t fully convinced it was the best choice, so I checked out what’s the ls equivalent.

This post is about the research I’ve done; as usual, it’s an exercise in extensive usage of the available tools.

Note that I’ve been notified by a reader of the ls parameter -A, which simplifies the logic. I’ve kept both sections, for two reasons:

• the concepts not needed with the new approach are interesting to know regardless;
• in the new section I don’t explain in detail the concepts already explained in the old one.

Contents:

• Introduction to the problem, and the find tool
• Exploring ls
• A better approach, via ls -A
• Conclusion

We’re in the process of upgrading our MySQL databases from v5.7 to v8.0; since one of the differences in v8.0 is that the default encoding changed from utf8 to utf8mb4, and we had the conversion in plan anyway, we anticipated it and performed it as preliminary step for the upgrade.

This post describes in depth the overall experience, including tooling and pitfalls, and related subjects.

Contents:

• Introduction
• Migration plan: overview and considerations
  • !! COLLATION WARNING !!
  • Free step: connection configuration
    • How do charset settings affect database operations?
  • Step 2: Preparing the the ALTER statements
    • Issue: Column/index size limits
    • Issue: Triggers/Functions
    • Issue: Joins between columns with heterogeneous charsets
  • Step 3: Altering the schema and tables
  • Warnings
    • Other schemas
    • Always run ANALYZE TABLE
    • Don’t rush the DROP TABLE
• Notes about Mathias Bynens’ post on the same subject
• Conclusion
• Footnotes

We recently had to drop a column in production, from a relatively large (order of 10⁷ records) table.

On modern MySQL setups, dropping a column doesn’t lock the table (it does, actually, but for a relatively short time), however, we wanted to improve a very typical Rails migration scenario in a few ways:

1. offloading the column dropping time from the deploy;
2. ensuring that in the time between the column is dropped and the app servers restarted, the app doesn’t raise errors due to the expectation that the column is present;
3. not overloading the database with I/O.

I’ll give the Gh-ost tool a brief introduction, and show how to fulfill the above requirements in a simple way, by using this tool and an ActiveRecord flag.

This workflow can be applied to almost any table alteration scenario.

Contents:

• Gh-ost
• Setup and workflow
  • Existing configuration
  • Configure ActiveRecord for ignoring the column, and performing the deploy
  • Using gh-ost to drop the column
  • Remove the ignored_columns and redeploy
• Conclusion

The bindings for the standard Ruby GUI toolkit, Tk, need some trickery in order to be installed on Ubuntu. This article shows how to do it.

Updated on 29/Apr/2021: Updated procedure to use Rubygems’ provided options for build flags.

Contents:

• Some history
• Prerequisites
• Workaround and installation

I’ve recently moved from Bash to Zsh, and I needed to port my tab completion scripts. Zsh has a sophisticated built-in tab completion, however, the documentation is not very beginner-friendly; moreover, Bash scripts can be used with no or little change in Zsh. Therefore, I’ve opted for using them directly.

This article will explain how to write tab-completion scripts in any language, with an example in Ruby, and how to use them in both Bash and Zsh.

As typical of this blog, the script is also used as an exercise in shell scripting, therefore, it contains additional (arguably) useful/interesting commands/concepts.

Contents:

• General notes
• Ruby Library
• Writing a tab-completion script
• Specification of the target program
• Writing the tab completion script
• Setting up autocompletion
• Debugging
• Conclusion

Seldom, the subject of how to wipe the disk(s) of a headliness linux server comes up; there are a few resources online about it. This blog summarizes all the information around into a clean, stable and generic script that can be used in order to perform this task.

As typical of this blog, the script is also used as an exercise in shell scripting and system administration, therefore, it contains (arguably) useful/interesting commands/concepts.

Contents:

• Preamble/specifications
• Procedure

Important! I’ve published an updated and extended guide on the same subject; check that out before reading this one!

Although the concepts involved in preparing a Debian (.deb) source package and publishing it on an Ubuntu PPA are simple, due to the many moving parts involved, it’s not easy to find a single source of information.

This article provides all the information required to perform the process, using a trivial program as an example.

Contents:

• About the approach and standards
• Conventions
• The procedure
  • Setup
    • Preparing the system
    • Setting up and publishing the PGP key(s)
    • Setting up a Launchpad account
    • Creating the PPA
  • Preparing the source package
    • Introduction to Debian packaging
    • Preparing the source code
    • The makefile
    • Debian packaging metadata creation and core concepts
    • Updating changelog
    • Updating control
    • Updating copyright
  • Building the source package
  • Uploading the package
  • Deleting a package
• Using the PPA
• Conclusion

With the large diffusion of SBCs [Single Board Computers], and subsequent maturation of their ecosystem, it’s now relatively easy to setup a home server.

I’ve had three SBCs until now; a Raspberry Pi 2 model B, a 3 model B, and recently, an Odroid XU4.

In this post, I’m going to share some considerations about their usage as home servers.

Updated on 3/Jul/2019: added new generation boards section.

Contents:

• Considerations about new generation (A7x and Goldmont Plus) boards
• General characteristics of an SBC/home server
• Brief informations about ARM processors
• Raspberry Pi 3 Model B
  • Specifications
  • Support and documentation
    • BEWARE: Stay far from the Ubuntu Pi Flavour Maker
  • Usage impressions
• Odroid XU4
  • Specifications
  • Support and documentation
  • Usage impressions
  • Power draw
  • The infamous fan noise
    • Introduction
    • Base setup
    • Solutions and references
  • Performance tweaking
• Alternatives
• Benchmark
• Conclusions
• Footnotes

In system administration, it’s typical to perform long-running commands on remote hosts.

With GNU Screen and one supporting script, it’s possible to efficiently perform such operations, by running them asynchronously and receiving an email at the end, all in a single command.

Contents:

• Some GNU Screen concepts
• Purpose and overview of the script
• Script requirements
• Script source
• Script explanation
• Alternatives
• Conclusion
• References

Ruby GUI development is a seldom mentioned subject, but it has value. Probably after some Rails development (cough…), developing a desktop tool may be an interesting diversion (or even a requirement).

During the development of my PM-Spotlight desktop application, I evaluated most of the Desktop Ruby GUI toolkits, and prototyped the application with three of them (Shoes 3, FXRuby, and Tk).

This article presents a summary of what I’ve experienced (or gathered) while I was “Developing GUI applications with Ruby”!

Updated on 03/Oct/2019: Added update section about the article archival.

Contents:

• Update
• TL;DR: The summary
• Extension/mistakes
• Frameworks
  • Shoes 3/4
  • FXRuby
  • Ruby-GNOME2
  • qtbindings
  • Tk
  • wxRuby
• Conclusion
• Some references
• Footnotes

It’s very convenient to run service processes (for development purposes!) without admin permissions, rather starting it as system service.

This guide will show how to easily setup PostgreSQL (both via package and binary tarball) to be run by an unprivileged user, with the data in any directory owned by him/her.

Contents:

• Setup notes/requirements
• Installation via apt package (Ubuntu)
• Installation via binary tarball (universal)
• Persisting the PGSQL data location, and adding the binaries to the $PATH
• Conclusion

GitLab offers a functionality for cherry picking a merge request (PR).

This functionality doesn’t exist in GitHub, and in Git, either; it is useful in some cases.

In this article I’ll explain some git fundamentals, and in the last section, how to cherry pick a PR/merge request.

Contents:

• Disclaimer
• What is a git merge
• Rebasing onto a new base
• Finding the merge base
• Cherry-picking a PR/merge request
• Merging the PR
  • Maintaining the PR exactly as it is
  • Changing the PR branch history
• Conclusion

In a project of mine, I’m implementing a feature that runs a background job in order to perform a search; in particular, it needs to support stopping at any time.

There is a variety of strategies to do this in Ruby. In this article I will expose what is the exact outcome of the common strategies, examining how this affects the underlying operating system.

The analysis is targeted to POSIX operating systems, although, at least part of it applies to Windows machines as well.

Contents:

• Brief introduction to Ruby concurrency
• Technical context and preliminary notes
• Problem statement
• Effect of the common strategies on a Linux operating systems
  • Threads, and Thread#kill
  • Kernel#fork
  • Dealing with zombie processes: Process.detach
  • Kernel#spawn
• Introduction to process groups and their usage
  • Non-working usage of process groups
  • Working implementation, using setsid
• Conclusion

This is the Part 3 (of 3) of the shell scripting adventures.

Updated on 11/Oct/2019: Use mapfile for the check list widget returned entries.

The following subjects are described in this part:

• Introduction to Whiptail
• The mysterious redirections (3>&1 1>&2 2>&3)
• Widgets, with snippets
  • Message box
  • Yes/no box
  • Gauge
  • Radio list
  • Check list
  • Password box
• Other widgets
  • Input box
  • Text box
  • Menus

Since Whiptail is simple to use, the objective of this post is rather to show some useful code snippets/patterns.

The examples are taken from my ZFS installer project and RPi VPN Router project installation scripts.

Previous chapters:

• Introduction
• Part 1
• Part 2

This is the Part 2 (of 3) of the shell scripting adventures.

The following subjects are described in this part:

• Awk/sed/perl considerations
• Perl text processing
  • Regex modifiers
  • Replace multiple lines
    • m modifier
    • s modifier
  • Isolating ambiguous group references
  • Printing a matching group (of matching lines)
• Awk text processing
• Progress bars processing with awk (and stdbuf)
  • Processing single-line progress outputs
  • Working around the buffering
  • Streams processing
  • Putting things together

The examples are taken from my RPi VPN Router project installation script.

Previous/following chapters:

• Introduction
• Part 1
• Part 3

This is the Part 1 (of 3) of the shell scripting adventures.

The following subjects are described in this part:

• Associative arrays (hash maps)
• ANSI-C quoting
• Escape strings
• Expand strings into separate options
• Regular expressions matching
• Find a filename’s basename
• Replace the extension of a filename
• Cycle a multi-line variable
• Heredoc

The examples are taken from my RPi VPN Router project installation script.

Previous/Following chapters:

• Introduction
• Part 2
• Part 3

I’ve always thought of shell scripting as a second class scripting form, being awkward and limited. After a deep dive in my RPi VPN router project, I still think it’s awkward and limited, but I do appreciate it as first class choice under specific conditions.

Shell scripting generally can’t be disassociated from system/infrastructure administration, so developing the project has been actually, as a whole, a very interesting and pleasant undertaking.

This post lays the structure of the future posts I’ll publish about the experience.

Following chapters:

• Part 1
• Part 2
• Part 3

Mysql 5.7 added native support for JSON data type. This opens up several interesting possibilities, but it’s not natively supported in Rails 4 (only in v5).

I’ve released a gem, JSON on Rails, for supporting this functionality on Rails 4.

This article describes how the gem works, for those interested in the inner workings, or who want to implement the functionality by themselves.

MySQL 5.7.19 fixes a quite dangerous functionality that causes corruption of a slave when changing the delay (CHANGE MASTER TO MASTER_DELAY=<seconds>) while the slave threads are stopped; since one wouldn’t expect this condition to cause harm, users of such setup should upgrade, if possible.

Users trying to perform the update on Ubuntu 14.04 LTS will face the mysql service not starting.

In my VGA Passthrough guide, I explain how to configure an Ubuntu machine for VFIO.

With the release of QEMU 2.10, some bugs in the OVMF firmware surfaced, which make Windows guests unstable; such bugs have been fixed in the master branch of the EDK II Project repository, so, in order to use QEMU 2.10, it’s required to build the firmware from scratch.

In the vein of the previous article, we’ll examine here the impact of using the ENUM data type.

Such data type is very interesting, although it requires a careful examination, since it has very important design implications.

Recently, we decided to clean our data model, in particular, the OLTP section of our schema. We have a reasonably well structured model, but it falls short when it comes to the data type definitions.

In this article we explore the storage requirements of a (InnoDB) table of ours, and the impact of choosing more strictly defined columns.