Tags:

assembler, awk, c, concurrency, data_types, databases, debugging, distribution, ffmpeg, filesystems, git, github, gitlab, gui, hardware, indexes, innodb, linux, media_processing, mysql, operating_systems, packaging, performance, perl, postgresql, rails, reverse_engineering, ruby, shell_scripting, storage, sysadmin, text_processing, trivia, ubuntu, vfio, virtualization

Announcement: ZFS Ubuntu installer

October 11, 2019

Yesterday, I’ve officially released my ZFS Ubuntu installer.

This project will hopefully help the adoption of ZFS, at least by appealing curious/interested sysadmins.

Canonical is about to officially release an Ubuntu with built-in ZFS support, however, it has extremely limited functionality.

Create a VM, launch the program, and have fun playing with ZFS!

Read More

Bash: Print variables using indirect references, with introduction to array data types

September 30, 2019

Although there is a complexity threshold for the convenience of shell scripting, there is still plenty that (in my opinion) can be done before hitting such ceiling.

In this brief article, I’ll explain how to use indirect references to write a function that prints the names and content of the variables passed; this is useful when writing debugging code for a script.

Contents:

Read More

Debugging a live/stuck Ruby process with GDB, fully explained, plus many goodies!

September 11, 2019

Debugging a live/stuck Ruby process is a well-known subject.

The way it’s generally exposed is simply a series of instructions and their outcome; given the expectation, this is fine of course, however, when I saw that a manual copy/paste operation was required, I decided to… step in (pun intended 😂).

This article adds only a few concepts, operatively speaking, but it clarifies all the concept involved, and employs neat approaches to accomplish the task. I will also employ several goodies available to Linux systems.

Contents:

Read More

Processing key/values tuples in text files with Perl (how to enable unattended upgrades for PPAs)

September 7, 2019

In my latest sysadmin experimentations, I’ve configured the unattended upgrades on my new server (an Odroid H2 😍).

Among the other things, I had to add a PPA to the list of allowed origins to automatically upgrade. This required some interesting text processing, so I decided to dig into Perl’s text processing functionalities.

This post describes how to enable unattended upgrades for a PPA, while explaining several useful Perl features, in particular, how to process key/value tuples; a few regular expression functionalities are also used.

Contents:

Read More

How to print the last N fields (or slice an array) with awk

August 26, 2019

A functionality that I require, relatively frequently, when scripting, is to print the last N fields of a stream, with awk.

A typical example, is to copy part of the commits of a git cherry/log command:

+ 81061edd2023c399539f1ff5cfdc267fd41c5c43 Ruby GUI development article: add `Some references` section
+ 5c60b7ef357683137b5f772f8590ab7d12c8e218 Ruby GUI development article: add `Footnotes` section
+ b6fe3469bc2ae59a7eba629c16ab11c67b7fbcbf Ruby GUI development article: add note about browser-based toolkits
[...]

This post explains how to do it, with some extra goodies.

Contents:

Read More

Installing Ubuntu on a ZFS root, with encryption and mirroring

August 18, 2019

2019 is a very exciting year for people with at least a minor interest in storage. First, in May, the ZFS support for encryption and trimming has been added, with the release 0.8; then, in August, Canonical has officially announced the plan to add ZFS support to the installer¹ in the next Ubuntu release.

As of now, achieving a full-ZFS system (with a ZFS root (/)) is possible, although non trivial. In this walkthrough, I’ve made the procedure as simple as possible, additionally setting up encryption and mirroring (RAID-1).

I’ll also give an introduction about the concepts involved, so that one has better awareness of the purpose of all the steps (but feel free to jump directly to the procedure).

Note that this guide is stable, but I’ll keep adding a few more sections and content in general.

Contents:

Read More

Computing aggregates in the shell (via AWK/Perl)

July 15, 2019

Today we’ve had an emergency in production, which caused us to process log files.

An operation that comes very helpful in these situations is to quickly compute aggregates for values found in logs (or text files, in general).

This is possible, and actually easy, with the usual *nix tools; in this post I’ll explain three approaches to this problem.

Contents:

Read More

Summary of trailing spaces handling in MySQL, with version 8.0 upgrade considerations

July 9, 2019

Fairly recently, we’ve upgraded to MySQL 8; it’s been a relatively smooth transition, however, some minor differences needed to be handled. One of them is the behavior of trailing spaces.

Trailing spaces are a (not in a good way) surprising, but also widely covered argument. This article gives a short overview, and relates it to how this affects people upgrading to MySQL 8.0.

Contents:

Read More

Publicly available SQL Standard (ISO/IEC 9075) documents

July 9, 2019

Unexpectedly, the SQL Standard (ISO/IEC 9075) documents, while public, are not free - they’re copyrighted, and available for purchase.

Still, some versions can be found on the web. PostgreSQL has a related section in their wiki, however, one link is out of date.

This is the up to date list of the documents, stored in the Wayback Machine, so that they’re permanent:

For reference, the most relevant part is #2.

Read More

Converting/processing a video using FFmpeg, for viewing on an old digital media player

July 2, 2019

During my last vacation, I wanted to watch some very old cartoons, during the daily break, on the digital media player that the apartment provided.

The player was very old, and couldn’t handle the video file (a modern MP4). Therefore, this was a good occasion to exercise the FFmpeg conversion and processing capabilities.

This post will give a glimpse of the tools that FFmpeg provides, showing how easy it is to perform the video/audio conversion, with extra processing.

I will also explain general concepts of media handling, like containers and encoding formats.

Contents:

Read More

Text processing experiments for finding the MySQL configuration files

June 12, 2019

When it comes to configuring MySQL, a fundamental step is to find out which configuration files the MySQL server reads.

The operation itself is simple, however, if we want to script the operation, using text processing in a sharp way, it’s not immediate what the best solution is.

In this post I’ll explore the process of looking for a satisfying solution, going through grep, perl, and awk.

Contents:

Read More

Using a persistent live Ubuntu USB flash drive in VirtualBox

June 3, 2019

Recently, I needed to test some operations to be performed during the installation of Ubuntu on a system.

This requires multiple sessions and reboots, so retaining the changes on the installation media would considerably streamline the task.

VirtualBox doesn’t support booting a VM from a USB flash drive though, so a workaround is required.

This small article explains how to do this.

Contents:

Read More

Reverse engineering and patching a Windows application with Radare2

April 12, 2019

Due to some constraints, at Ticketsolve we sometimes need to work with an ancient file format: the Paradox Database.

This file format was in use between the 80s and 90s. In order to perform some operations on Paradox databases, nowadays, there are libraries based on the file format reverse engineering work by individual open source programmers, or ad hoc commercial programs.

Additionally, one can use Paradox 7, the reference commercial software originally published in 1992, now abandoned.
This software works good enough in Wine, however, the installer generally raises an error on installation, complaining that there isn’t enough disk space.

In this post we’ll use Radare2, one of the most powerful open source reverse engineering frameworks, in order to statically analyze and patch the installation binary, so that the pesky error is not triggered anymore.

There are no requirements for the reader; knowing x86/32 assembler and the PE executable format will improve the understanding, but it’s not a requirement by any means.

Contents:

Read More

An in depth DBA's guide to migrating a MySQL database from the `utf8` to the `utf8mb4` charset

March 25, 2019

We’re in the process of upgrading our MySQL databases from v5.7 to v8.0; since one of the differences in v8.0 is that the default encoding changed from utf8 to utf8mb4, and we had the conversion in plan anyway, we anticipated it and performed it as preliminary step for the upgrade.

This post describes in depth the overall experience, including tooling and pitfalls, and related subjects.

Contents:

Read More

Dropping a database column in production without waiting time and/or schema-aware code, on a MySQL/Rails setup

February 12, 2019

We recently had to drop a column in production, from a relatively large (order of 10⁷ records) table.

On modern MySQL setups, dropping a column doesn’t lock the table (it does, actually, but for a relatively short time), however, we wanted to improve a very typical Rails migration scenario in a few ways:

  1. offloading the column dropping time from the deploy;
  2. ensuring that in the time between the column is dropped and the app servers restarted, the app doesn’t raise errors due to the expectation that the column is present;
  3. not overloading the database with I/O.

I’ll give the Gh-ost tool a brief introduction, and show how to fulfill the above requirements in a simple way, by using this tool and an ActiveRecord flag.

This workflow can be applied to almost any table alteration scenario.

Contents:

Read More

Using scripts in any language for Bash/Zsh tab completion

June 28, 2018

I’ve recently moved from Bash to Zsh, and I needed to port my tab completion scripts. Zsh has a sophisticated built-in tab completion, however, the documentation is not very beginner-friendly; moreover, Bash scripts can be used with no or little change in Zsh. Therefore, I’ve opted for using them directly.

This article will explain how to write tab-completion scripts in any language, with an example in Ruby, and how to use them in both Bash and Zsh.

As typical of this blog, the script is also used as an exercise in shell scripting, therefore, it contains additional (arguably) useful/interesting commands/concepts.

Contents:

Read More

Remotely wiping the disk(s) of a headless linux server

May 29, 2018

Seldom, the subject of how to wipe the disk(s) of a headliness linux server comes up; there are a few resources online about it. This blog summarizes all the information around into a clean, stable and generic script that can be used in order to perform this task.

As typical of this blog, the script is also used as an exercise in shell scripting and system administration, therefore, it contains (arguably) useful/interesting commands/concepts.

Contents:

Read More

Building a Debian (`.deb`) source package, and publishing it on an Ubuntu PPA

May 3, 2018

Although the concepts involved in preparing a Debian (.deb) source package and publishing it on an Ubuntu PPA are simple, due to the many moving parts involved, it’s not easy to find a single source of information.

This article provides all the information required to perform the process, using a trivial program as an example.

Contents:

Read More

Considerations (review) of Raspberry Pi/Odroid XU4 usage as home server

April 16, 2018

With the large diffusion of SBCs [Single Board Computers], and subsequent maturation of their ecosystem, it’s now relatively easy to setup a home server.

I’ve had three SBCs until now; a Raspberry Pi 2 model B, a 3 model B, and recently, an Odroid XU4.

In this post, I’m going to share some considerations about their usage as home servers.

Updated on 3/Jul/2019: added new generation boards section.

Contents:

Read More

Remotely running asynchronous commands/scripts via GNU Screen

March 24, 2018

In system administration, it’s typical to perform long-running commands on remote hosts.

With GNU Screen and one supporting script, it’s possible to efficiently perform such operations, by running them asynchronously and receiving an email at the end, all in a single command.

Contents:

Read More

An overview of Desktop Ruby GUI development in 2018

March 13, 2018

Ruby GUI development is a seldom mentioned subject, but it has value. Probably after some Rails development (cough…), developing a desktop tool may be an interesting diversion (or even a requirement).

During the development of my PM-Spotlight desktop application, I evaluated most of the Desktop Ruby GUI toolkits, and prototyped the application with three of them (Shoes 3, FXRuby, and Tk).

This article presents a summary of what I’ve experienced (or gathered) while I was “Developing GUI applications with Ruby”!

Updated on 03/Oct/2019: Added update section about the article archival.

Contents:

Read More

Quickly setting up PostgreSQL for running without admin permissions

February 15, 2018

It’s very convenient to run service processes (for development purposes!) without admin permissions, rather starting it as system service.

This guide will show how to easily setup PostgreSQL (both via package and binary tarball) to be run by an unprivileged user, with the data in any directory owned by him/her.

Contents:

Read More

Git: Cherry-picking a PR/merge request

February 2, 2018

GitLab offers a functionality for cherry picking a merge request (PR).

This functionality doesn’t exist in GitHub, and in Git, either; it is useful in some cases.

In this article I’ll explain some git fundamentals, and in the last section, how to cherry pick a PR/merge request.

Contents:

Read More

Executing and killing ruby parallel/background jobs

January 2, 2018

In a project of mine, I’m implementing a feature that runs a background job in order to perform a search; in particular, it needs to support stopping at any time.

There is a variety of strategies to do this in Ruby. In this article I will expose what is the exact outcome of the common strategies, examining how this affects the underlying operating system.

The analysis is targeted to POSIX operating systems, although, at least part of it applies to Windows machines as well.

Contents:

Read More

Shell scripting adventures (Part 3, Terminal-based dialog boxes: Whiptail)

December 23, 2017

This is the Part 3 (of 3) of the shell scripting adventures.

Updated on 11/Oct/2019: Use mapfile for the check list widget returned entries.

The following subjects are described in this part:

Since Whiptail is simple to use, the objective of this post is rather to show some useful code snippets/patterns.

The examples are taken from my ZFS installer project and RPi VPN Router project installation scripts.

Previous chapters:

Read More

Shell scripting adventures (Part 2, Text processing extravaganza)

November 22, 2017

This is the Part 2 (of 3) of the shell scripting adventures.

The following subjects are described in this part:

The examples are taken from my RPi VPN Router project installation script.

Previous/following chapters:

Read More

Shell scripting adventures (Part 1, Bash general functionalities)

November 8, 2017

This is the Part 1 (of 3) of the shell scripting adventures.

The following subjects are described in this part:

The examples are taken from my RPi VPN Router project installation script.

Previous/Following chapters:

Read More

Shell scripting adventures (Introduction)

November 2, 2017

I’ve always thought of shell scripting as a second class scripting form, being awkward and limited. After a deep dive in my RPi VPN router project, I still think it’s awkward and limited, but I do appreciate it as first class choice under specific conditions.

Shell scripting generally can’t be disassociated from system/infrastructure administration, so developing the project has been actually, as a whole, a very interesting and pleasant undertaking.

This post lays the structure of the future posts I’ll publish about the experience.

Following chapters:

Read More

Support MySQL native JSON data type in ActiveRecord (Rails) 4

October 26, 2017

Mysql 5.7 added native support for JSON data type. This opens up several interesting possibilities, but it’s not natively supported in Rails 4 (only in v5).

I’ve released a gem, JSON on Rails, for supporting this functionality on Rails 4.

This article describes how the gem works, for those interested in the inner workings, or who want to implement the functionality by themselves.

Read More

Installing MySQL 5.7.19 on Ubuntu 14.04

September 13, 2017

MySQL 5.7.19 fixes a quite dangerous functionality that causes corruption of a slave when changing the delay (CHANGE MASTER TO MASTER_DELAY=<seconds>) while the slave threads are stopped; since one wouldn’t expect this condition to cause harm, users of such setup should upgrade, if possible.

Users trying to perform the update on Ubuntu 14.04 LTS will face the mysql service not starting.

Read More

Data storage analysis and optimization of (MySQL) InnoDB indexes

August 1, 2017

Recently, we decided to clean our data model, in particular, the OLTP section of our schema. We have a reasonably well structured model, but it falls short when it comes to the data type definitions.

In this article we explore the storage requirements of a (InnoDB) table of ours, and the impact of choosing more strictly defined columns.

Read More