kinbiko.com

Installing Rockbox on an iPod Classic 7G with iFlash from Linux: a sector size odyssey

Sun, 01 Feb 2026 00:00:00 +0000

Disclaimer: This entire blog post was written by AI. It is published because I want AI tools to more easily find and fix the problem I spent days debugging – even with the assistance of AI.

If you’re trying to install Rockbox on a flash-modded iPod Classic from Linux and getting “No partition found”, you might be hitting the same invisible wall I did. The root cause turned out to be a mismatch in virtual sector sizes between different USB modes – something that no guide I found online mentions. This post documents the problem and the fix.

The setup Link to heading

iPod Classic 7th generation (Apple model A1238, Rockbox target ipod6g)
iFlash adapter with a 512 GB SD card
Arch Linux host (no macOS, no Windows, no iTunes)
Rockbox built from source (commit ee3e630, January 2026)

The symptom Link to heading

After partitioning the SD card, formatting it FAT32, installing the .rockbox directory, and flashing the bootloader via DFU, the iPod would boot to:

Rockbox boot loader
Version: ee3e630efa-260131
Battery status ok: 4081 mV
id: '1 CompactFlash C0000' s:512*1
No partition found

The bootloader could see the storage device (the iFlash adapter identifies as CompactFlash) and correctly detected 512-byte sectors with a physical sector multiplier of 1. But it couldn’t find or mount any partition.

The dead ends Link to heading

Dead end 1: “the old bootloader doesn’t support 4K sectors” Link to heading

The pre-built Rockbox 4.0 bootloader gave the same error. Rockbox’s source code has a sector multiplier loop in disk.c that tries mounting partitions at 1x, 2x, 4x, and 8x the MBR’s LBA values to compensate for virtual sector size mismatches. I built the bootloader from source to make sure I had this code. The new bootloader failed identically.

Dead end 2: “the cluster size is too large” Link to heading

mkfs.vfat -F 32 on a 477 GiB partition with 4096-byte sectors chose 64 sectors per cluster, producing 256 KB clusters. Rockbox’s bpb_is_sane() rejects cluster sizes above 128 KB. Reformatting with -s 8 (32 KB clusters) fixed this check, but the “No partition found” error persisted.

Dead end 3: “there’s a stale superfloppy BPB at sector 0” Link to heading

The iPod’s sector 0 had leftover FAT32 BPB bytes from a previous format that could confuse Rockbox’s superfloppy detection (where it tries to mount sector 0 directly as a filesystem before checking the partition table). Zeroing the first 446 bytes of sector 0 eliminated this, but didn’t help.

The actual problem: virtual sector sizes Link to heading

Here’s what was actually happening. It took reading the Rockbox bootloader source code to figure it out.

The iFlash adapter uses 512-byte sectors Link to heading

The iFlash adapter’s SD card storage uses standard 512-byte sectors at the ATA level. The bootloader confirms this with s:512*1 – 512-byte logical sectors, physical sector multiplier of 1. This is the sector size that Rockbox uses when reading the MBR and FAT filesystem internally.

Rockbox’s USB mode uses 4096-byte virtual sectors Link to heading

When Rockbox presents the iPod as a USB mass storage device, it multiplies the sector size by 8. This is configured in the iPod Classic’s target header (ipod6g.h):

#define DEFAULT_VIRT_SECTOR_SIZE 4096

The USB stack calls disk_set_sector_multiplier(8) before entering USB mode, making the host see 4096-byte logical sectors. This is why lsblk shows LOG-SEC 4096 when the iPod is connected in normal Rockbox USB mode.

Linux tools respect the device’s reported sector size Link to heading

When you run sfdisk or fdisk on a device with 4096-byte logical sectors, the MBR partition table entries are written in 4096-byte sector units. A partition starting at 1 MiB is recorded as LBA sector 256 (256 * 4096 = 1,048,576 bytes).

When you run mkfs.vfat on that partition, it creates a FAT32 BPB with bytes_per_sector=4096 to match the device.

The bootloader reads with 512-byte sectors Link to heading

When the iPod boots, it reads the MBR using its ATA driver at the native 512-byte sector size. It sees LBA start value 256 and interprets it as byte offset 256 * 512 = 131,072 – not 1,048,576 where the partition actually starts.

Rockbox has a multiplier loop that compensates for this by trying the LBA value multiplied by 1, 2, 4, and 8. At 8x (256 * 8 = 2048, byte offset 2048 * 512 = 1,048,576), it finds the correct location. But then it reads a BPB with bytes_per_sector=4096, which creates further complications in the FAT driver’s internal math. The mount fails.

The fix Link to heading

The key insight came from reading the bootloader source code more carefully. There are actually two different USB modes with different sector sizes:

// bootloader/ipod-s5l87xx.c

// Voluntary USB mode (SELECT+RIGHT at boot):
if (button_read_device() == (BUTTON_SELECT|BUTTON_RIGHT)) {
 disk_set_sector_multiplier(IF_MD(i,) DEFAULT_VIRT_SECTOR_SIZE/SECTOR_SIZE);
 usb_mode(); // host sees 4096-byte sectors
}

// ...

// Error USB mode (partition mount failed):
case ERR_RB:
 usb_mode(); // host sees 512-byte sectors! No multiplier set.
 break;

When the bootloader enters USB mode due to a “No partition found” error, it does NOT set the sector multiplier. The host sees the ATA-native 512-byte sectors.

This means the fix is straightforward: format while in the bootloader’s error-mode USB. The bootloader is already in this mode when showing “No partition found” and waiting for a USB connection. Simply:

Connect USB while the bootloader shows “No partition found”
Linux sees /dev/sdX with 512-byte logical sectors
Partition and format normally

# Wipe the first 1 MiB
dd if=/dev/zero of=/dev/sdX bs=512 count=2048

# Create a single FAT32 LBA partition
sfdisk --wipe always /dev/sdX <<EOF
label: dos
unit: sectors
/dev/sdX1 : start=2048, type=0c
EOF

# Format with 32 KB clusters (Rockbox max is 128 KB)
mkfs.vfat -F 32 -s 64 -n IPOD /dev/sdX1

# Mount and install Rockbox
mount /dev/sdX1 /mnt/ipod
unzip rockbox.zip -d /mnt/ipod
umount /mnt/ipod

After disconnecting and rebooting (MENU+SELECT), Rockbox boots successfully:

MBR has 512-byte sector LBA values that the bootloader reads correctly
FAT32 BPB has bytes_per_sector=512 that the FAT driver handles natively
No multiplier tricks needed

How to tell which USB mode you’re in Link to heading

Check the logical sector size of the device:

cat /sys/block/sdX/queue/logical_block_size

512 – you’re in bootloader error-mode USB (or a card reader). Safe to format.
4096 – you’re in Rockbox main firmware USB or voluntary bootloader USB. Do NOT format from this mode. Reboot into the bootloader error mode first.

Alternatively, lsblk -o NAME,LOG-SEC,MODEL shows the sector size for all devices.

Other things I learned along the way Link to heading

The iPod Classic doesn’t need a firmware partition. Unlike older iPods (Video, Nano) that store firmware on a dedicated disk partition, the iPod Classic 6G/7G stores its firmware in NOR flash. A single FAT32 data partition is the correct layout.

The iFlash adapter identifies as “CompactFlash”, not “iPod”. If you’re writing scripts to auto-detect the device, match on the model string “CompactFlash”, not “iPod”.

Rockbox has a maximum cluster size of 128 KB. On a 477 GiB partition with 512-byte sectors, mkfs.vfat will default to 128 sectors/cluster = 64 KB clusters. This is fine. If you explicitly set -s 64 (32 KB clusters), that works too. Don’t go above -s 256 (128 KB).

Most guides assume iTunes or Windows. The standard advice for “No partition found” on iPod Classic is “restore with iTunes.” iTunes creates a properly formatted disk because Apple knows the correct parameters. If you’re on Linux without iTunes, you need to understand the sector size situation yourself.

Building Rockbox from source on Linux is straightforward. The cross-compiler toolchain builds non-interactively with:

RBDEV_TARGET=a RBDEV_PREFIX=/path/to/toolchain bash tools/rockboxdev.sh

Then configure and build:

export PATH="/path/to/toolchain/bin:$PATH"
../tools/configure --target=29 --type=B # bootloader
make -j$(nproc)

The bootloader is flashed via DFU, not written to disk. Use mks5lboot --bl-inst with the iPod in DFU mode. The bootloader lives in NOR flash separately from the SD card, so reformatting the SD card doesn’t affect it.

Summary Link to heading

If you’re getting “No partition found” on an iPod Classic with iFlash and Rockbox on Linux:

Check the logical sector size with cat /sys/block/sdX/queue/logical_block_size
If it’s 4096, you’re in the wrong USB mode – reboot into bootloader error mode
If it’s 512, format with sfdisk + mkfs.vfat -F 32 -s 64 and install Rockbox
Reboot and enjoy your music

4-4-6-2 Meditation

Mon, 01 Dec 2025 00:00:00 +0000

I recently discovered 4-4-6-2 breathing: 4 seconds in, 4 hold, 6 out, 2 hold.

It’s become my shortcut to calm down when I’m feeling stressed. A few cycles and I’m noticeably more relaxed.

Incidents and Blame

Fri, 28 Nov 2025 00:00:00 +0000

I believe strongly that if an individual is to blame for a (production/security/operational) incident, then this individual should be fired.

This is because I also strongly feel that the only acceptable time to blame an individual for an incident is in the case of intentional malicious action.

Any other incident can trace the root cause to something systemic that could be improved. For example:

Bad hiring practices that put individuals in situations where they operate tools outside their skillset.
Insufficient testing or testing environments.
Poor documentation that leaves engineers guessing.
Insufficient rollback mechanisms.
Inadequate monitoring and alerting.
Missing access controls.
Bad onboarding and training programs.

I’m sure you can think of dozens more…

If you blame an employee for an incident then your culture sucks, and you should feel bad. Worse than said employee does.

Managers and Incidents

Wed, 26 Nov 2025 00:00:00 +0000

There are plenty of texts talking about expectations of engineers and best practices during an incident. What’s not mentioned as often, mainly because giving feedback “up” is hard, is expectations of senior management. In a recent discussion at work, I naively proposed the following 3 expectations / guidelines for senior managers and above.

Power Dynamics Link to heading

The voluntary involvement of senior management during an active incident is discouraged.

This is due to how easily power-dynamics add rpessure and confusion in regards to authority when dealing with an already stressful situation.

Performance Evaluations Link to heading

Being a leading actor in resolving an important production incident should be no more rewarded than quietly preventing one, and senior management should be conscious of this when doing performance evaluations.

This point applies to organizations large enough to have systems and processes around internal career progression.

Adoption Non-Reaction Link to heading

If, as a result of a team adopting best practices around incident management increases visibility around its incidents, then senior management should not react with any more urgency than prior to adoption, keeping the level of trust, oversight and attention as it was previously.

This is to not revert the adoption for said team, and to not dissuade other teams from adopting incident handling best practices.

Go on… Link to heading

I’m lucky enough to work with several engineering leaders I consider far senior to myself.

A VP pointed out that the temptation for senior management to get closer to the action is fuelled by inadequate upwards communication. Empathy goes both ways.

A principal engineer also shared an idea for certain Directors/VPs to be trained as “Incident Executives”. These are separate from “Incident Commander” (typically an individual contributor). The role of an Incident Executive is to be present on calls and communicate with other execs as well as provide shielding for the incident response team.

Naive Musings on AI

Tue, 25 Nov 2025 00:00:00 +0000

Let’s set aside the harmful impact AI has on artists (copyright), the environment (GPUs want electrons), and humanity (enforcing existing unfair systems) in general for a sec. Let me share where I’ve personally extraxted the greatest value from AI.

I don’t find that AI does my job better than I do it. That is, the job I’m personally an expert at. For example, AI doesn’t write better Go code than me. It can write code faster, and even after I’ve corrected it’s output (if it’s written using a very refined Claude task or agent), it is likely net faster than me in solving problems. Although, I dislike the fact that AI does the “fun” part of my job faster than me…

But what it is better than me at, is producing text, in particular code, that I’m not an expert at. For example:

LaTeX and Beamer, e.g. for presentations.
Manim animation code in Python.
Custom plugins in Lua for my neovim text editor.
Templates and style sheets for my blog powered by Hugo.
Writing native-sounding Japanese based on the equivalent English.

And more generally, I recognize now that it’s value is bringing the standard of everyone’s non-core text-producing skills up to a higher standard.

Writing More, Publishing More

Tue, 19 Aug 2025 19:37:36 +0900

I want to write more. And publish what I’m writing more, not worrying about meeting some high bar before publishing.

For one, writing helps me think clearly. Personally, I don’t “think” in words, but without turning my thoughts into words in some (human/computer) language I can’t communicate it to others. And by converting the idea to a form others can understand, I also understand it better as a result.

Secondly, writing moves my unfinished ideas out of my brain where they’re distracting me, into a location where I may forget about it and move on. Unprocessed thoughts, e.g. because I’ve not felt like I’ve taken in news etc., are very distracting. By taking 5 minutes to write down my thoughts, I can move on from the distracting thoughts.

Thirdly, by choosing to publish what I’m writing, I’m hoping to attract more like-minded people. It’s not easy to make new friends as adults (maybe even as kids these days)? By publishing, I can share the way I’m thinking. Maybe someone finds it interesting / vibe with what I’m saying.

Finally, writing is a form of meditation. Espeically when it’s handwritten. I find myself looking for reliable sources for entering a flow state. Writing by hand, focusing on neat handwriting, works well.

Who knows, if I manage to write enough, my voice can outlive me through AI…

But why would you want that?

How I Write Go

Sat, 21 Jun 2025 12:00:00 +0000

Tips and tricks around writing Go after 8 years.

Go Workshop

Sat, 21 Jun 2025 00:00:00 +0000

Beginner workshop introducing the syntax and key ideas of the Go programming language.

Docker Deep Dive

Wed, 12 Feb 2025 00:00:00 +0000

This talk introduces Docker to beginners with lots of practical demos, introduces best practices for experienced users, and discusses the internals for advanced users.

Profitable Go Tests

Thu, 19 Sep 2024 00:00:00 +0000

In this presentation I’m proposing a novel judgment method if a test is worth writing or not, and provide several tips for how to write high-quality tests in Go.

Commandline Foo

Sat, 07 Sep 2024 00:00:00 +0000

Some tips and tricks to make you love working on the command-line.

Cross Cultural Teams

Sat, 18 May 2024 00:00:00 +0000

Co-presented with Chad Kohalyk, giving a short overview of Erin Meyer’s excellent book The Culture Map: Breaking Through the Invisible Boundaries of Global Business which lays out an 8-scale framework for analyzing business culture and developing strategies for an effective team.

Automating your CV

Sun, 19 Mar 2023 01:00:00 +0900

I found that continuously updating my CV over time was a pain, so I simplified it and automated it.

These days I use a simple CI workflow in a GitHub repository that generates a new .pdf CV based on a Markdown source file.

If you like the sound of this, I’ve created a GitHub template for this pattern, should you wish to follow my example.

The template contains placeholders to fit the structure I personally use these days, and is littered with tips and advice for how to write a good CV (I say this as someone who’s screened a lot of low-quality CVs).

Auto-merging Dependabot PRs

Sat, 29 Oct 2022 01:00:00 +0900

Keeping your code’s dependencies up to date is hard, so luckily there are free tools like Dependabot around, that can create PRs against your repository whenever any of your dependencies have a more recent version updated. Depending on the size of your codebase, this can turn into quite a chore as you have to verify that the dependencies don’t break your code.

A common solution is to establish a thorough test suite and other CI checks that you trust to let you know if your code is broken and combine that with automatically merging Dependabot PRs.

Unfortuately, the default instructions from GitHub for doing exactly this leave a little to be desired:

They recommend creating a separate GitHub Actions workflow, like the following:

name: Dependabot auto-merge
on: pull_request

permissions:
 contents: write
 pull-requests: write

jobs:
 dependabot:
 runs-on: ubuntu-latest
 if: ${{ github.actor == 'dependabot[bot]' }}
 steps:
 - name: Dependabot metadata
 id: metadata
 uses: dependabot/fetch-metadata@v1.1.1
 with:
 github-token: "${{ secrets.GITHUB_TOKEN }}"
 - name: Enable auto-merge for Dependabot PRs
 if: ${{contains(steps.metadata.outputs.dependency-names, 'my-dependency') && steps.metadata.outputs.update-type == 'version-update:semver-patch'}}
 run: gh pr merge --auto --merge "$PR_URL"
 env:
 PR_URL: ${{github.event.pull_request.html_url}}
 GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

If you just apply this naively, patch version upgrade Dependabot PRs for a fictitious my-dependency dependency will be merged automatically without waiting for any particular jobs (e.g. your tests) to finish. That’s probably not what you want.

Instead you probably want your standard CI jobs like testing and linting to pass first, and it’s more reasonable to expect those jobs to run on push instead of pull_request or pull_request_target. This becomes a problem since the steps mentioned need pull_request or pull_request_target events to work.

After a lot of trial and error, I’ve come up with an alternative approach I prefer instead, and I’m sharing it here in case others can benefit from this as well:

on:
 push: (...)

permissions:
 contents: write
 pull-requests: write

jobs:
 # Verify you can build the code
 build: (...)
 # Run a linter against your code
 lint: (...)
 # Run your tests against the code
 test: (...)

 # Automatically merge if it's a Dependabot PR that passes the build
 dependabot:
 runs-on: ubuntu-latest
 # Only run this job for dependabot PRs
 if: ${{ github.actor == 'dependabot[bot]' }}

 # Only run if the required checks pass
 needs: [test, lint, build]

 steps:
 - name: Check out code
 uses: actions/checkout@v3

 - name: Auto-merge Dependabot PRs
 # Find the PR number based on the current branch name, and squash merge based on this number
 run: 'PR_NUM="$(gh pr list | grep $(git branch --show-current) | cut -f1)"; gh pr merge --auto --squash $PR_NUM'
 env:
 GH_TOKEN: ${{secrets.GITHUB_TOKEN}}

The above will automatically merge your Dependabot PRs, and only your Dependabot PRs, if the build passes. You may have noticed that I haven’t included the check that would only run this for only patch updates – that’s because I personally don’t want this check. I leave adding this check as an exercise for the reader.

Important note Link to heading

Do this only if you trust your dependencies. This automation makes you particularly vulnerable to supply-chain attacks.

(Go) Error messages should be boring

Sat, 30 Jul 2022 01:00:00 +0900

When error messages contain “useful information” e.g. application-specific IDs or other variables then the error messages lose their “grepability”. In other words, you cannot quickly search a codebase for an error without removing anything you believe could be variable.

You might not think this is a big issue, and I might even agree with you there, but you might as well make it a practice to make error messages boring – I.e. no variables in strings.

To be clear I’m not talking about fmt.Errorf("could not foo the bar: %w", err) which uses fmt.Errorf to wrap an error using the %w format directive. I’m a proponent of wrapping errors. What I think could be problematic is something like this:

if err != nil {
 return fmt.Errorf("user %s cannot access account %s: %w", userID, accountID, err)
}

Where userID and accountID would likely differ between each time the error is generated. I argue this makes errors difficult to search for in a code base and it could even mess with your error reporting tools like Sentry. Error reporting tools often group “Events” of the same “Issue” (Sentry) / “Error” (Bugsnag) based on the attached stacktrace. Now, Go in particular doesn’t provide stacktraces in errors by default, so error reporting tools fall back on the error message for grouping events. If the error message is different because we include request- or goroutine specific data then events aren’t grouped together. A potential consequence of this is spammy notifications which in turn could lead to ignoring notifications and alerting from these tools altogether. I’m sadly speaking from experience here…

The solution I’m proposing is to follow the same pattern that structured logging libraries like Zap or Logrus follow:

Include IDs etc. as part of metadata (“fields”) instead of in the message itself.

This does mean we need an error type that supports this metadata functionality. It’s not too hard to create yourself, and if you’re curious to see what this would look like, take a look at github.com/kinbiko/rogerr.

The pattern I’m proposing in this package is as follows:

Attach metadata (e.g. user ID) to a context.Context variable with ctx = rogerr.WithMetadatum(ctx, "user_id", userID)
When you need to create an error, use err = rogerr.Wrap(ctx, err, "unable to foo the bar") to attach the metadata to an error.
When you log/report/otherwise handle the error use md := rogerr.Metadata(err) to get a map[string]interface{} of all metadata fields associated with the error.

Full example found here.

Once you have this metadata, you can then attach as fields to a structured log message or as tags in a report to Sentry, etc. You could even execute custom error handling logic based off the values in this map should you wish, all while maintaining grepability of your error messages.

Rewriting my Neovim .vimrc in Lua

Mon, 23 Aug 2021 23:00:00 +0900

I recently rewrote my Neovim configuration in Lua in order to take advantage of a more intuitive programming language for configuration, Lua-only plugins, and powerful Neovim-only features such as:

The built in LSP: Language Server Protocol, IDE features like “show docs”, “find references”, “rename”, and “go to definition” for basically any language.
Treesitter: Powerful parsing of any programming language, which means plugins and syntax highlighting can make powerful and accurate transformations.

In this article I’d like to share some tips from my experience rewriting my config (tl;dr: check this commit to see what changed).

This article pales in comparison Link to heading

Most of what I’ve learned I’ve learned from this GitHub repository, which is a lot more in-depth than this article.

Lua primer Link to heading

Lua, at least the kind of Lua you’re likely to use for writing your Vim config, is really intuitive and easy to learn, assuming you already know an interpreted programming language like JavaScript. Here are the least intutive bits, the rest you can probably figure out when you first see it:

Single line comments are --. Multi line comments are --[[commented stuff in here]].
Variables are global by default. Use local myVar = 'foobar' whenever you declare a variable and you’re unlikely to run into trouble.
“Table” is the central type, which acts as “object”, “map”, and “array”, all in one:
- Map: {foo = "bar", baz = "bat"}; like JSON, but = instead of :.
- Array: local arr = {'foo', 'bar', 'baz',}; trailing commas are OK. Arrays are tables where the key is the index implicitly. Lua is 1-indexed. I.e. arr[2] == bar -- true
- Object: local M = {} is typical when creating an object whose intent is to be used in other files. Add methods with local function M:myMethod(arg) print(arg) end to add a myMethod method to M.
Import modules (files) with local p = require('module.path'). The value returned from module/path.lua is stored in p. require caches the returned value, so future calls to require('module.path') will return the same value.
Parentheses when calling functions with only one argument are optional. This means you sometimes see weird code like require'somemodule'.method{foo='bar'}.
String concatenation uses the .. infix operator.

If you’d like a more accurate and less rambly introduction to Lua, check out this talk by TJ Devries (Neovim core contributor):

Store your configuration in Git Link to heading

You will mess up. Store your vim config in Git upfront so you can increment on your migration process, and always return to a known state – or even go back to VimL if Lua isn’t for you.

You can find my (embarrassing) evolution of Vim usage on GitHub.

Start with structure before refactoring Link to heading

Start the migration by putting files in their correct location without really rewriting anything. Unless you’ve set the $XDG_CONFIG_HOME variable to something other than ~/.config, you should make the following change:

- ~/.config/nvim/init.vim
+ ~/.config/nvim/init.lua

Then take the contents of your old .vimrc and wrap them inside

vim.cmd([[
<contents of .vimrc goes here>
]])

In Lua, [[ and ]] enclose a multi-line string, so you’ve probably guessed that the above code just evaluates the string (your old VimL config) as VimL. This will most likely work out of the box. Indeed, if you struggle to find the native Lua API for the change you want to do, you can call vim.cmd('<some snippet>') at any time. Commit now, and you’ll have a lot smaller and more obvious commits going forward.

Switch to a Lua-native package manager Link to heading

I switched from (the excellent) vim-plug to packer, which lets me use Lua to define the dependencies, run Lua functions on updates or install.

Packer also supports bootstrapping itself, and provides a snippet in its README in order to do just that.

Recommendation:

Don’t mess around with :PackerCompile or :PackerSync unless you really struggle with startup time. They can confuse matters really easily.

Rewrite bit by bit, then commit Link to heading

I took the following approach to the migration:

Start by rewriting all my plugins to use packer, googling and fixing things as I went along.
Then I moved on to rewriting my ‘general’ settings, basically the settings that were set something=someVal in VimL.
Then I moved on to mappings.
Then I rewrote my after/ftplugin/{lang}.lua files
Then I added LSP client configuration.

Key points:

Interacting with the Neovim application will happen via the vim global variable.

Options Link to heading

vim.g.my_option = "my value" for setting global vim options.
vim.o.my_option = "my value" for setting other options.
vim.bo.my_option = "my value" for setting buffer-specific options.
vim.wo.my_option = "my value" for setting window-specific options.

If you’re not sure if an option is buffer or window just try setting it to one. It will usually tell you if it’s wrong if you try and run it.

Mappings Link to heading

vim.api.nvim_set_keymap(kind, lhs, rhs, options) is the API that Neovim exposes for defining custom key-mappings, but this is a bit long-winded, so I recommend creating a utility function:

local function map(kind, lhs, rhs, opts)
 vim.api.nvim_set_keymap(kind, lhs, rhs, opts)
end

Here kind can be:

'': The equivalent of (nore)map in VimL, i.e. the mapping applies in visual, normal, select, and operator-pending modes.
n: Normal mode
v: Visual mode
i: Insert mode,
etc.

The nore part of nnoremap, inoremap in VimL is configured as part of an option: noremap = true. I reckon most of your bindings will have { noremap = true, silent = true }, so you might as well assign this table to a variable pass it in to your mappings.

Here’s a snippet from my config where I’m taking advantage of the above:

local function map(kind, lhs, rhs, opts)
 vim.api.nvim_set_keymap(kind, lhs, rhs, opts)
end

local silentnoremap = {noremap = true, silent = true}

-- ...

-- Intuitive increment and decrement
map('n', '+', '<c-a>', silentnoremap)
map('n', '-', '<c-x>', silentnoremap)

-- Conveniently enter command mode
-- Don't use silent=true as this removes the command line.
map('n', ';', ':', {noremap=true})

-- Go to definition and pop back up
-- Don't do noremap to allow accessing LSP behaviour
map('n', '<right>', 'gd', { silent = true })
map('n', '<left>', '<c-t>', { silent = true })

Set up LSP Link to heading

Setting up the native LSP client will largely depend on your own preferences, and the languages you write. That said, I recommend using the following (Packer) plugins:

use 'neovim/nvim-lspconfig' -- Easy LSP configuration
use 'kabouzeid/nvim-lspinstall' -- Install LSP servers on demand with :LSPInstall <name_of_language>

Then configure them according to their instructions. You’ll likely end up with other plugins that also want to integrate with LSP (e.g. Telescope), so your configuration will definitely look different to mine, but here it is regardless, with the hope that it can be useful as a reference, if nothing else.

Create your own namespace to avoid collisions Link to heading

It is of course up to you if you want to keep your entire configuration in a single init.lua file, or if you want to spread your code across multiple files, but you have to be careful when organising files other than init.lua!

To avoid namespace collisions, don’t put your files directly under ~/.config/nvim/lua/foo.lua, but instead create another directory, named something that won’t collide with the dependencies/plugins that you use, such as using your GitHub username: ~/.config/nvim/lua/kinbiko/foo.lua. This way you can create files with configuration and mappings per plugin, should you so wish, without having worry about requireing the wrong thing.

How Iota works in Go

Mon, 08 Mar 2021 18:00:00 +0900

Have you ever wondered how iota works in Go?

Basically, iota is the index of the constant inside a multi-constant (const (...)) declaration. So the value of iota for the first constant is 0, the second constant is 1, and so on. This only works for constants, not vars.

This is probably best explained with an example.

package main

import "fmt"

type day int

const (
 // Start counting days from 1, so ignore 0.
 _ day = iota

 // evaluates to the same as the RHS of the previous expression,
 // which is iota,
 // which is the index of this const expression in this const block,
 // which is 1
 monday

 // this time the index is 2
 tuesday

 // etc.
 wednesday
 thursday
 friday
 saturday
 sunday
)

func (d day) String() string {
 days := []string{
 "",
 "Monday",
 "Tuesday",
 "Wednesday",
 "Thursday",
 "Friday",
 "Saturday",
 "Sunday",
 }
 return days[d]
}

func main() {
 // Let's do some wat maths. Waths if you will.
 // Test your understanding by guessing the answers:
 printDay(monday + tuesday)
 printDay(sunday - wednesday)
 printDay(tuesday * wednesday)
 printDay(saturday / tuesday)
 printDay(sunday % wednesday)
 printDay(friday & thursday)
 printDay(sunday ^ thursday)
}

func printDay(d day) {
 fmt.Println(d)
}

You can run the above code in a Go Playground if you want to check your answers.

Hiding Directories in Finder

Sun, 07 Mar 2021 18:00:00 +0900

You might know that prefixing a file or a directory with . makes it ‘hidden’ in unix-based systems – i.e. they won’t show up when you run ls and they won’t show up in your file browser (e.g. Finder).

However, did you know that MacOS lets you hide any directory in Finder – even if it’s not .hidden – using chflags?

$ chflags hidden dir1 dir2 dir3

I use the above to hide directories in my home directory, that I can’t remove (MacOS regenerates them automatically):

chflags hidden Applications Movies Music Pictures Public

This saves me from context switching and searching for the directory or file that I want whenever I open my home directory.

Note: chflags hidden doesn’t remove the directories, only hides them in Finder. They are still visible with ls.

Testing Tips for Intermediate Gophers: Coverage

Sun, 17 Jan 2021 18:00:00 +0900

This article will cover how to get visibility of your test coverage in Go. If you’re not comfortable with the basics of testing in Go, please checkout my earlier article aimed at beginners.

Test coverage Link to heading

Test coverage gives you a high-level overview you how much of your production code was invoked in your tests (e.g. as a percentage), but it can also give you a lower-level view of exactly which lines are invoked or missed. Before we dive deeper into the ‘how’, let’s make sure we understand why we care about test coverage. Test coverage is a terrible metric for evaluating the quality of your tests. Repeat after me:

High test coverage does not imply high-quality tests.

So what’s the purpose of test coverage then? Link to heading

In my opinion, test coverage serves two purposes:

Firstly, it lets you quickly identify which lines have not been hit by tests. This could indicate that your tests aren’t comprehensive enough: “Oops, this if statement wasn’t tested, but it should have been. Let’s add another test”. However, it could also identify impossible paths in your code: “We’ve already validated this value elsewhere so this can never happen”.

Moreover, paired with a good Continuous Integration setup, you can identify if a change will remove tests prematurely, by warning you when the coverage of your pull request is lower than the base branch.

How to report test coverage in Go Link to heading

The go tool supplies us with all the tools we need; no third-party solution required:

# Write a "coverprofile" to a filed called coverage.out
go test -coverprofile=coverage.out

# Invoke the 'cover' go tool, and ask it to render the coverprofile file as HTML.
# This should open your default web browser.
go tool cover -html=coverage.out

This shows you exactly how much of each production file was covered, and renders your source code in green if the particular line was covered by the tests, and red otherwise. Type definitions, comments, and other “non-runnable” code are greyed out, as it doesn’t make sense to count these lines as part of our coverage.

You can also see a report of your coverage report on a per-function basis:

# Running coverage against github.com/kinbiko/bugsnag and requesting
# per-function output
$ go test -coverprofile=coverage.out github.com/kinbiko/bugsnag && go tool cover -func=coverage.out
ok github.com/kinbiko/bugsnag 0.159s coverage: 87.5% of statements
github.com/kinbiko/bugsnag/configuration.go:67: validURL 100.0%
github.com/kinbiko/bugsnag/configuration.go:75: populateDefaults 100.0%
github.com/kinbiko/bugsnag/configuration.go:92: validate 100.0%
github.com/kinbiko/bugsnag/configuration.go:124: makeRuntimeConstants 54.5%
github.com/kinbiko/bugsnag/context.go:21: Serialize 60.0%
github.com/kinbiko/bugsnag/context.go:39: Deserialize 55.6%
github.com/kinbiko/bugsnag/context.go:86: val 100.0%
github.com/kinbiko/bugsnag/context.go:127: WithBreadcrumb 100.0%
github.com/kinbiko/bugsnag/context.go:143: makeBreadcrumbs 100.0%
github.com/kinbiko/bugsnag/context.go:180: WithUser 100.0%
github.com/kinbiko/bugsnag/context.go:196: WithBugsnagContext 100.0%
github.com/kinbiko/bugsnag/context.go:213: WithMetadatum 100.0%
github.com/kinbiko/bugsnag/context.go:226: WithMetadata 100.0%
github.com/kinbiko/bugsnag/context.go:243: Metadata 100.0%
github.com/kinbiko/bugsnag/context.go:251: initializeMetadataTab 100.0%
github.com/kinbiko/bugsnag/context.go:301: updateFromCtx 38.9%
github.com/kinbiko/bugsnag/context.go:332: getAttachedContextData 100.0%
github.com/kinbiko/bugsnag/error.go:32: Error 100.0%
github.com/kinbiko/bugsnag/error.go:47: Unwrap 66.7%
github.com/kinbiko/bugsnag/error.go:61: Wrap 100.0%
github.com/kinbiko/bugsnag/error.go:70: Wrap 100.0%
github.com/kinbiko/bugsnag/error.go:84: makeStacktrace 100.0%
github.com/kinbiko/bugsnag/error.go:120: makeModulePath 66.7%
github.com/kinbiko/bugsnag/notifier.go:44: New 85.7%
github.com/kinbiko/bugsnag/notifier.go:72: Close 100.0%
github.com/kinbiko/bugsnag/notifier.go:93: Notify 75.0%
github.com/kinbiko/bugsnag/notifier.go:129: loop 100.0%
github.com/kinbiko/bugsnag/notifier.go:149: shutdown 70.0%
github.com/kinbiko/bugsnag/notifier.go:172: makeReport 100.0%
github.com/kinbiko/bugsnag/notifier.go:197: sendErrorReport 85.7%
github.com/kinbiko/bugsnag/notifier.go:221: makeUnhandled 100.0%
github.com/kinbiko/bugsnag/notifier.go:234: makeSeverity 83.3%
github.com/kinbiko/bugsnag/notifier.go:246: severityReasonType 100.0%
github.com/kinbiko/bugsnag/notifier.go:265: makeExceptions 82.4%
github.com/kinbiko/bugsnag/notifier.go:297: makeJSONApp 100.0%
github.com/kinbiko/bugsnag/notifier.go:306: makeJSONDevice 75.0%
github.com/kinbiko/bugsnag/notifier.go:321: memStats 84.6%
github.com/kinbiko/bugsnag/notifier.go:345: osVersion 66.7%
github.com/kinbiko/bugsnag/notifier.go:352: makeNotifier 100.0%
github.com/kinbiko/bugsnag/notifier.go:360: extractLowestBugsnagError 100.0%
github.com/kinbiko/bugsnag/notifier.go:374: guard 100.0%
github.com/kinbiko/bugsnag/session.go:30: StartSession 100.0%
github.com/kinbiko/bugsnag/session.go:45: flushSessions 100.0%
github.com/kinbiko/bugsnag/session.go:57: publishSessions 82.4%
github.com/kinbiko/bugsnag/session.go:102: makeJSONSession 100.0%
github.com/kinbiko/bugsnag/session.go:116: makeJSONSessionReport 100.0%
github.com/kinbiko/bugsnag/session.go:135: uuidv4 100.0%
total: (statements) 87.5%

A note on `covermode` and concurrent programs Link to heading

go test has a parameter called covermode which tweaks how the Go tool should think about “covering a line”. Basically your options are “lines are either missed or hit” or “let’s count how many times a line is hit”. The former is used by default, and is the least expensive. You can get more of a ‘heatmap’ feature (in the rendered HTML) by setting the -covermode=count, however I recommend taking things a step further to avoid headaches. Namely, use -covermode=atomic to ensure that the lines hit are counted accurately even in concurrent programs.

$ go test -coverprofile=coverage.out -covermode=atomic

This is significantly more expensive, but for most project the additional compute resource is surely negligible.

Function Types in Go

Sun, 10 Jan 2021 02:07:07 +0900

A while ago, I saw a question in a Go developer Slack channel that went something like this (translated):

When would you want to define a function type like type MyFunc func(msg string) error? Do you have any examples?

I replied with a http.HandlerFunc example (more on that below), but there were many developers smarter than me in that Slack channel, and I ended up learning a lot from the other answers to this question. This article will cover what I took away from the ensuing discussion.

Type definitions Link to heading

Let’s start off by talking about what the type keyword does. The most common type definitions usually look something like:

type MyStruct struct {
 // Fields...
}

In the case of a struct, or in the case of an interface it might look something like:

type MyDependency interface {
 DoSomething(ctx context.Context) error
}

I daresay more than 90% of the types I define look like the above. However, you may create a type to identify a function signature as well, as defined in the original question. To better understand when we might want to create such a type, let’s have a look at the spec for inspiration. Among other things, the spec has the following to say about type definitions:

A type definition creates a new, distinct type with the same underlying type and operations as the given type, and binds an identifier to it. The new type is called a defined type. It is different from any other type, including the type it is created from. A defined type may have methods associated with it.

The last sentence is key. Let’s see why!

The elegance of `http.HandlerFunc` Link to heading

Chances are you’ve played around with the net/http package at least once when learning Go. You may remember that many of the functions in net/http revolve around a type called http.Handler. http.Handler is a single-method interface:

type Handler interface {
 ServeHTTP(ResponseWriter, *Request)
}

It may be tempting to believe that you have to create your own struct that implements the ServeHTTP method, but the net/http provides a very convenient shortcut for us: Whenever you need a http.Handler you can write a http.HandlerFunc instead.

type HandlerFunc func(ResponseWriter, *Request)

This works because the HandlerFunc type implements Handler. In other words, the HandlerFunc has a ServeHTTP(ResponseWriter, *Request) method. The implementation of this method invokes itself:

func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
 f(w, r)
}

Now that we have an appreciation of how the net/http package works, we can generalise this pattern further:

I can improve my package’s developer experience by defining a Func type that implements a key single-method interface in my package.

You can imagine this pattern being useful where a maintainer of a package regrets requiring an interface (where a function would do), but also doesn’t want to break backwards compatibility.

Function types as lightweight interfaces Link to heading

I think a lot of us has ingrained the pattern of requesting interfaces when constructing a new type. This pattern scales well and is good practice in general, but as hinted to above there are cases where interfaces are overkill. In this case, requesting a function might be a good solution. By naming your function signature (by defining a function type) you avoid excessively long function signatures just like you would with interfaces. Moreover, your users don’t have to cast their function definitions like you do have to in the case of defining http.HandlerFuncs:

After reflecting on this pattern I decided to change some interfaces into function types in a package I wrote. The diff of this change may be more illustrative than what I can write in words in this blog post. Note that this is a breaking change, but the change required by users of your package are trivial in most cases. To see why, remember that methods are functions too – and as such they can adhere to function type definitions. Where users previously passed in an implementation of an interface, they now have to pass in the function of that implementation. I.e. changing someDep to someDep.FuncImplementation when invoking your package’s function. In fact – your users are now free to rename this function (or even make it private)!

Use function definitions to create default arguments Link to heading

We can utilize methods on function types when the arguments of a function is almost always the same:

type MyComplicatedFunc func(a string, b int, c *someReallyComplicatedStruct, d onlyThingThatMatters)

func (f MyComplicatedFunc) invoke(d onlyThingThatMatters) {
 f("some default string", 4321, &someReallyComplicatedStruct{ /* values that don't change. */}, d)
}

This should make invoking this function with the ‘default’ arguments internally in the package very simple.

Note: This is a neat trick, but just because you can doesn’t mean you should. I don’t really endorse this pattern, and haven’t really seen it used in production code anywhere.

Use type aliases for documentation Link to heading

Occasionally, a function parameter could use some additional documentation (including when the parameter itself is a function). You could define a function type in order to attach some additional godocs, however I recommend using type aliases in this case. The only difference is adding a = between the type name and its definition.

type MyComplicatedFunc = func(a string, b int, c *someReallyComplicatedStruct, d onlyThingThatMatters)

The difference between type aliases and type definitions is outside the scope of this article, but one key result is that you cannot attach methods to this type. Type aliases emphasise the intent to only add documentation. Additionally, since this contrived example shows a very long function signature, you can see how type aliases (and type definitions) can make code shorter and more readable.

mokku -- How I built a Go Mocking framework in 5 days

Wed, 18 Mar 2020 00:00:00 +0000

This past week I’ve had the privilege of having a Hack Week where I work, where we can work on whatever we wanted. I ended up building Mokku – a mocking framework for Go that gets out of your way.

Key features Link to heading

Invisible: No hard (import) or soft (//go:generate) dependencies to add to your codebase.
Unintrusive: Can be included in any developer’s workflow. Doesn’t dictate or enforce how you use it after it’s written.
Unassuming: Doesn’t care about package paths or even if your file compiles. All it wants is an interface.
Vanilla: It’s basically a shortcut to generating the mocks you’d probably write yourself if you wrote the mocks manually.

I’ve been wanting to write this mocking framework for a while now. So when I finally had the opportunity to work on this full time for a week, I implemented all the techniques and processes I personally prefer (I was developing this project on my own) and genuinely had a good time working on this. You can check out the project in its entirety on GitHub, so I won’t go into the details of how to use it etc. here.

Instead, I’d like to share how the development process went. As opposed to most technical blog posts, and indeed most of my own blog posts, this won’t be an article where I try and tell you how you should do things. I’m just going to share what I did, and some of my thoughts around how things went.

You know, like a blog.

But first, a bit of background.

GoMock?! Go away Link to heading

I had two main sources of ‘fuel’ so to speak, for delivering Mokku. The first was the frustration I’ve had when working with GoMock. Its super terse error messages, annoying gotchas and things to remember left me with a strong desire to make something SIMPLE.

The second source of fuel is more of an inspiration. Mat Ryer’s Moq mocking framework ticks most of what I was looking for in a mocking framework, and then some. In fact, the usage of Mokku is very similar to how you use Moq. I chose to drop some of the bells and whistles that come with Moq (maybe I’ll incorporate them as command line flags down the line), and went with what in my opinion is an even simpler approach. For example, instead of storing all arguments that have been passed to the mocks by default (and having them be safe for concurrent access), I opted to defer the implementation of this logic to the users (see what I did there? … I’ll see myself out).

Day 1: Planning Link to heading

I spent the first day quickly putting together a rough project plan and product and engineering designs in the Wiki of the GitHub project (you can still find it there). I knew from the start that the purpose of doing these planning activities wasn’t to know exactly how I was going to do things. It was more to see the scope of the project that I could fit into this one week I had available to me. Appetite, not estimate. Luckily, my gut feel told me I could get quite a fair bit done in the allotted time, without skimping out on quality.

As part of doing the quick engineering design, I dove deeper into how Moq worked under the hood, and was able to steal some of the ideas used. The main idea I stole was to use the text/template to generate the mocked structs. In hindsight this is an obvious choice, I think I had just forgotten about this package’s existence. It is somewhat overshadowed by its HTML counterpart.

In terms of actual product value, by the end of the first day I barely managed to scrape together a skeleton with a dummy test running in CI. But that’s fine. The tree still stands, but the axe is sharp.

Day 2: Parsing Link to heading

I spent the second day doing the initial version of a ‘parser’, that would read a given interface definition given as []byte, and identify its comprising components. I used test driven development in this endeavour as I had pretty good idea of what I wanted to get out of it (I knew the input I’d get, I knew the output I wanted), but didn’t actually know how to implement it yet. I initially thought I could use the go/parser package from the standard library, but this wasn’t quite as flexible as I needed it to be. After looking around for a bit longer I found this package’s neighbour, go/scanner, which was exactly what I wanted. The scanner was able to read a byte slice one ’token’ at a time, and tell me what special meaning it had in the Go language, if any. However, it didn’t require the code to actually compile, which was exactly the simplicity I was looking for.

I had a bit of extra time during the day that I spent on reading up on Go templates. I’ve known about them for a while, but I’ve never actually used them before. I have 0 doubt this is still obvious from reading the template code…

Day 3: Templates Link to heading

OK the title lies, I spent about half of the time getting the templates to work, and fixing bugs in the parser when I realised that I needed more information than what I previously thought. The rest was spent hooking together the parser, and creating a runnable binary. This day was also the day where I had to decide if I wanted to present what I had made internally to the company. I went for ‘yes’, not at all knowing what that actually would entail.

Day 4: Make Things Presentable Link to heading

At the start of the day I already had a working prototype. There were a couple of bugs still around (e.g. ... for varargs didn’t quite work), but those were squashed relatively quickly. Now I just had to make it so that people could quickly learn how to use it, and want to use it. This meant creating an amazing README. I’m quite happy with how it all turned out.

I paid a designer on Fiverr to create a simple logo for the project, and I got what I paid for. You’re free to interpret that how you will, but I’m not complaining.

In order to really showcase the simplicity of the tool, I went an created an asciinema recording of how I would use it my day-to-day work. Not gonna lie, I did a lot of takes before I settled on a recording where I didn’t make that many typos (only 6). I sped up the recording 2 times to not make it boring to watch.

Side note, it turns out you cannot embed a video recording in a README, but you can embed a GIF. Asciinema creates these really cool hybrid video/text files (you can copy/paste text from these hybrids), but I would rather have something that catches the eye in the README and quickly showcases Mokku. So I had to install a couple of other tools to turn the recording I had stored locally into a GIF.

The rest of the day was spent putting together slides for the presentation I was making to the company the next day.

Day 5: Presentation Link to heading

I know very little about presenting work like this, but what little I do know I used. Namely start with why and putting only a couple of words on each slide, but having many slides.

I made the repository public, did some final bit of clean up, and spent most of the rest of the day watching other presenters and preparing for my own presentation. Since we were all working remotely (if you’re reading from the future: this happened back when the entire world was in quarantine) it’s really hard to tell how well the presentation went. Honestly, I don’t even know if my slides were showing – I just jumped into it hoping for the best. No regrets!

Conclusion Link to heading

I’m pretty happy with how it all went. I ended up with a good implementation that not only works but it’s also something I’d be happy to iterate further on. I do have some remaining work that needs to be done before I’m happy releasing a version 1 of Mokku. If you’re new to Go or open source, there are several issues that you could help with if you wish.

Testing Tips for Beginner Gophers

Sat, 22 Feb 2020 00:00:00 +0000

Let’s have a go at writing tests in Go. Puns aside, testing in Go is baked into the language. Assuming you’re comfortable with the Go basics, let’s get started with writing tests. First of all, the way you run all tests in the current package is:

$ go test

However, unless you’ve got tests in your project nothing is going to execute. Go tests have to follow these three rules:

The filename of all your test files must end with _test.go (including the underscore _).
The test function names must start with the Test prefix.
The test functions must have exactly 1 parameter, of type *testing.T (conventionally called t).

Thus a file called something_test.go containing:

package mypkg

import "testing"

func TestSomething(t *testing.T) {
}

is a minimal piece of code that counts as a test, and can be verified as such by running go test in the same directory.

As positively exhilarating as the above test was, it is useless, as it can never fail. To make the test fail, we use the *testing.T parameter.

func TestSomething(t *testing.T) {
 t.Errorf("test failed")
}

Now we have a test that’s always failing. This isn’t very useful, but it’s a start. If we run this we get the following output:

--- FAIL: TestSomething (0.00s)
something_test.go:6: test failed
FAIL
exit status 1
FAIL _/Users/kinbiko/repos/mypkg 0.005s

The built-in assertions are all you need. Link to heading

There are two key methods on t you need to know:

t.Errorf marks your test as failing, and prints the given message.
t.Fatalf does the same as t.Errorf, but will stop the execution of the test at the line it’s called.

As the programmer, it’s your responsibility to identify whether or not a test should fail. This means putting ifs in your test code, which might be surprising to you, especially if you’re coming from a language like Java, where the test framework includes various forms of equality checks for most assertions.

func TestFizzBuzz(t *testing.T) {
 if fizzbuzz(3) != "Fizz" {
 t.Errorf("expected the 3rd FizzBuzz value to be 'Fizz'")
 }
}

Personally, I prefer this more explicit way of writing test assertions, and you’ll see why in a sec, but if you’re set on avoing ifs, there’s a very popular testing library called Testify which exposes this XUnit-like testing syntax.

Again, the built-in testing framework assumes you’ll write somewhat non-trivial code to make the right assertions. This might sound like a bad thing, but it turns out breaking the convention of ‘dumb’ tests allows for some pretty valuable techniques…

Table-driven tests Link to heading

Table-driven tests are a very common testing pattern in Go. The general idea is that you define a “test table”: a slice of ’test case’ structs, that you iterate over, executing the same bit of test code over and over again for all the test cases. It’s perhaps easier to show how this works with an example:

func TestFizzBuzz(t *testing.T) {
 // Define the test table: an anonymous struct type that
 // we define and initialise a slice of immediately.
 tt := []struct {
 in int
 exp string
 }{
 {in: 1, exp: "1"},
 {in: 2, exp: "2"},
 {in: 3, exp: "Fizz"},
 {in: 4, exp: "4"},
 {in: 5, exp: "Buzz"},
 {in: 6, exp: "Fizz"},
 {in: 7, exp: "7"},
 {in: 10, exp: "Buzz"},
 {in: 15, exp: "FizzBuzz"},
 {in: 30, exp: "FizzBuzz"},
 }

 // Iterate over all test cases
 for _, tc := range tt {
 if got := fizzbuzz(tc.in); got != tc.exp {
 // A good human readable assertion message makes understanding test failures easy!
 t.Errorf("expected the FizzBuzz value number %d to be '%s' but was '%s'", tc.in, tc.exp, got)
 }
 }
}

Notice how t.Errorf forces us to think about and write our own assertion message. This is a good thing. You get to choose a human-readable message including exactly the information you care about, instead of relying on a testing framework’s default message.

The authors of Testify have of course thought about this, and allows for custom assertion messages, but it doesn’t force you to write these. In my experience, often when using Testify these assertion messages are not written, and even when they are included it feels like a chore, which in turn leads to less thought being put into these assertion messages, losing their merit. This is a large part of why I prefer the standard library’s "testing" framework out of the box. This is of course personal (or team) preference, and I’ll let you make up your own mind.

Sub-tests Link to heading

For simple programs like FizzBuzz, the test cases are very simple: One input, one output. More commonly in ‘real’ apps, you’re testing more complex business logic. In these cases you probably want more “context” around what exactly you’re testing. This is where sub-tests come in.

Sub-tests are effectively inner tests within outer tests, which allow you to give your inner tests a name, and share some common setup. Let’s examine how sub-tests are useful by writing some tests for a fictional app that allows users to ’like’ posts, similar to Facebook or Instagram. Here’s a test for a certain feature of this app, try and see if you can understand what the likesExcerpt function does by reading the tests.

func TestLikesExcerpt(t *testing.T) {
 var (
 jeffords = user{id: 0, name: "Jeffords"}
 diaz = user{id: 1, name: "Diaz"}
 scully = user{id: 2, name: "Scully"}
 peralta = user{id: 3, name: "Peralta"}
 boyle = user{id: 4, name: "Boyle"}
 holt = user{id: 5, name: "Holt"}
 santiago = user{id: 6, name: "Santiago"}
 hitchcock = user{id: 7, name: "Hitchcock"}
 linetti = user{id: 8, name: "Linetti"}
 )

 db := setUpDB()
 db.Insert([]user{jeffords, diaz, scully, peralta, boyle, santiago, hitchcock, holt, linetti})
 s := service{db: db}

 // Note: it's not uncommon to define the for loop and the test table at the same time.
 for _, tc := range []struct {
 name string
 in []int
 exp string
 }{
 {
 name: "given no likes then invite user to like",
 in: []int{},
 exp: "Be the first to like this",
 },
 {
 name: "given one like then name the user",
 in: []int{scully.id},
 exp: "Scully likes this",
 },
 {
 name: "given two likes then name both users",
 in: []int{jeffords.id, diaz.id},
 exp: "Jeffords and Diaz like this",
 },
 {
 name: "given three likes then name the first two users and count the last user (singular)",
 in: []int{jeffords.id, diaz.id, scully.id},
 exp: "Jeffords, Diaz, and 1 more person likes this",
 },
 {
 name: "given more than three users then name the first two users and count the rest (plural)",
 in: []int{jeffords.id, diaz.id, scully.id, boyle.id, peralta.id, linetti.id},
 exp: "Jeffords, Diaz, and 4 others like this",
 },
 } {
 // Note: We're intentionally shadowing the t parameter in the subtest,
 // by defining a parameter with the same name.
 t.Run(tc.name, func(t *testing.T) {
 got, err := s.likesExcerpt(tc.in...)
 if err != nil {
 // Note: even though we're using Fatalf here, we only interrupt the
 // innermost subtest; the other test cases will still run.
 t.Fatalf("didn't expect an error but got '%s'", err.Error())
 }
 if got != tc.exp {
 t.Errorf("expected excerpt '%s' but got '%s'", tc.exp, got)
 }
 })
 }

 t.Run("errors out on unrecognised IDs", func(t *testing.T) {
 if _, err := s.likesExcerpt(scully.id, 123, boyle.id, 456); err == nil {
 t.Errorf("expected an error relating to unrecognised IDs but didn't get one")
 }
 })
}

The above creates a subtest per test case, using the test case name as the name of the subtest. This makes it easier to understand which test case failed when you have multiple subtests. This is because in the case of a test failure, it will print the name of the test that failed prior to the assertion message. For example, if the "given three likes then name the first two users and count the last user (singular)" test case failed, we would get the following message:

--- FAIL: TestLikesExcerpt (0.00s)
--- FAIL: TestLikesExcerpt/given_three_likes_then_name_the_first_two_users_and_count_the_last_user_(singular) (0.00s)
likes_test.go:87: expected excerpt 'Jeffords, Diaz, and 1 more person likes this' but got 'Jeffords, Diaz, and 1 others like this'

which explicitly names the subtest that failed, enabling us to quickly identify what might be wrong.

Also notice the second single subtest that checks an error case. Because we did all the hard work of defining the users and setting up the service and db in the outer test, we can use them in both sub tests.

Integration tests vs unit tests; the `_test` package. Link to heading

For any package, e.g. storage, you can optionally use a storage_test package in the same directory to use as a package for just test logic. The idea behind the storage_test package is to test only the types, funcs, methods, and variables that are public in the storage package. If you’ve written code in other languages such as Java or Ruby, you might be used to seeing these types of tests in different directories altogether. I like Go’s choice of allowing a _test package in the same directory, as it makes it easier to find the production code for a given test, and vice versa.

Writing tests like this is optional, in that you’re not required to use this package for your tests. There’s nothing preventing you from writing exactly the same tests within the storage package directly, but there are some advantages to using the _test package:

You’re testing the code from the same point of view that your users are using it, giving you a better feel for how your package is to use.
You’re not relying on implementation details of your package, and your tests are less rigid and fragile.
You’re likely to find a necessary and sufficient level of dependency injection required for your package, helping you find the correct interfaces to demand in your package.
Compilation errors in these tests probably indicate a breaking change in your API, informing you that you might need to do a major version bump.

If we call the storage_test package tests ‘Integration tests’ and the storage tests ‘unit tests’, then it might be tempting to believe that all tests should be integration tests. I reject this idea, and believe you get the most value out of your code and time by picking your battles around when to write unit tests and when to write integration tests. In general, I default to writing integration tests, and introduce unit tests for complex and “hard to hit” logic.

Let’s look at an example of what the files of a storage package might look like:

cache.go
cache_test.go <-- `package storage`: unit test of the cache implementation details because caching logic is hard.
storage.go
storage_test.go <-- `package storage_test`: integration test of storage, which uses the logic of cache.go internally.

jsonassert -- A test utility for comparing and verifying JSON

Wed, 06 Feb 2019 00:00:00 +0000

Today I released v1.0.0 of a Go package called jsonassert. It is my first self-motivated open source contribution that I think has a chance of being useful to someone other than just myself. Okay, that’s not quite true but it’s the first one I’ve written in Go.

I realised the need for this package when attempting to increase the test confidence of a relatively untested package that sends JSON to a server. There was no easy way of making assertions against this JSON body, at least not without moving to an entirely new test framework. I’m a big fan of the testing package in the Go standard library, and this was already being used elsewhere in the package I was working with. So I settled for writing more lines of code than I would have liked, attempting to parse the JSON nested-object-by-nested-object as deserialising into a struct would defeat the purpose of the test.

However, when doing slogging away at these SLOCs I kept being haunted by an urge to write this jsonassert package. At the beginning of this year, I announced that my new years resolution would be to do more uncomfortable things. Well, one of those uncomfortable things was to stream myself programming on Twitch, which I have done several times now (I bet you can’t figure out how to hit that ‘follow’ button!). The thing I’ve been streaming the most has been the development of this package, essentially from scratch. Now that it’s released in what I believe to be in a production-ready state, I need to figure out what else I can hack on… Maybe a little tool that can help me study Japanese? We’ll see!

Watch ~~this space~~ my Twitch streams!

Feature-oriented package structures and the default access modifier

Sun, 18 Feb 2018 00:00:00 +0000

In Java we have 4 access modifiers that can be applied to methods and fields of a class. In descending order of visibility, they are:

public String publicModifier = "Anyone can access me";
protected String protectedModifier = "Only accessible to classes in the same package as me, and my subclasses";
 String defaultModifier = "Only accessible to classes in the same package as me";
private String privateModifier = "Only accessible within this class";

This article is a love letter to the third in this list of modifiers, the one-who-shall-not-be-named. It’s usually referred to the ‘default’ or ‘package-private’ access modifier.

Be aware of ingrained rules you don’t fully understand Link to heading

When I first started learning Java, I read that you should never use the default access modifier. This was because it wasn’t obvious if you had thought about the desired visibility of your method or field, or if you were being lazy or had just forgotten.

Use private wherever you can, protected if you need access to it in your subclasses, and public if you need it from the outside.

I didn’t question this decree at the time.

Soon enough, I started feeling uncomfortable seeing methods and fields without modifiers, and always applied the appropriate modifier whenever I found one, according to the rule above.

I later started using IntelliJ full time. IntelliJ has a feature where it can infer the required access modifier by looking at all the compiled code in its context, and see where the class member is used. This feature is handy for seeing when something could be private/protected, but I noticed that by default it also checks if its visibility can be reduced to package-private. Surely that’s a feature everyone disables. I know I did.

Prefer feature-oriented architectures over layer-oriented architectures. Link to heading

I have grown to dislike this (very common) layer-oriented layout of packages:

src/main/java/com/companyname/productname/
controller/
StuffController.java
ThingController.java
converter/
StuffConverter.java
ThingConverter.java
dao/
StuffDao.java
ThingDao.java
model/
Stuff.java
Thing.java
service/
StuffService.java
ThingService.java
util/
StuffHelper.java
ThingHelper.java

This layer-oriented format is so common that it even appears in Spring’s documentation, and 95%+ of the projects I see use this format for web applications. This diminishes the integrity of the architecture, implements the wrong kind of seperation of concerns, and leads to code that’s harder to maintain and split apart. A much cleaner approach is to organise your code by feature:

src/main/java/com/companyname/productname/
stuff/
StuffController.java
StuffConverter.java
StuffDao.java
Stuff.java
StuffService.java
StuffHelper.java
thing/
ThingController.java
ThingConverter.java
ThingDao.java
Thing.java
ThingService.java
ThingHelper.java

This approach gives you a clearer view of your architecture as it emphasises features over design patterns. More importantly, the coupling between features is necessarily more obvious. Any non-Stuff related dependencies inside any of the Stuff classes are declared in the imports. I recommend this article if you are interested in learning more about the perils of layer oriented architectures.

Rethinking best practices Link to heading

Allow me to diverge for a second and bring up a point I learned from the JavaScript (no relation) community. Ever since I first started learning about React, I’ve been plagued by this notion of rethinking best practices and if it applies to other best practices I’m currently following. If you squint you can see how React is a prime example of a technology that embodies the feature-oriented architecture in that it combines HTML and JavaScript (and often CSS as well) in a single component. In other words, it associates your view logic (JS) with your template (HTML/CSS) – exactly because they are conceptually related (highly cohesive). However, React components are not necessarily related to each other (loosely coupled).

Credit: Michele Bertoli

Rethinking Java best practices Link to heading

Back to the default access modifier. I believe that the default access modifier was designed with exactly the feature-oriented package structure in mind. Say that you have a StuffController and StuffService that looks something like the following:

package com.companyname.productname.controller;

import com.companyname.productname.converter.StuffConverter;
import com.companyname.productname.model.Stuff;
import com.companyname.productname.service.StuffService;
import java.util.Set;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class StuffController {

 private final StuffConverter converter;
 private final StuffService service;

 @Autowired
 public StuffController(final StuffService service,
 final StuffConverter converter) {
 this.converter = converter;
 this.service = service;
 }

 @GetMapping("/names")
 public List<String> getAllNames() {
 final Set<Stuff> stuff = service.findAllTheStuff();
 return converter.extractAllNames(stuff);
 }

}

package com.companyname.productname.service;

import com.companyname.productname.dao.StuffDao;
import com.companyname.productname.model.Stuff;
import java.util.Set;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class StuffService {

 private final StuffDao dao;

 @Autowired
 public StuffService(final StuffDao dao) {
 this.dao = dao;
 }

 public Set<Stuff> findAllTheStuff() {
 //Other logic...

 return dao.getAllStuff();
 }
}

If we switch to the feature-oriented package structure and use the default access modifier the way it was intended, we get terser code. Notice how all the redundant imports went away.

package com.companyname.productname.stuff;

import java.util.Set;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class StuffController {

 private final StuffConverter converter;
 private final StuffService service;

 @Autowired
 StuffController(final StuffService service,
 final StuffConverter converter) {
 this.converter = converter;
 this.service = service;
 }

 @GetMapping("/names")
 public List<String> getAllNames() {
 final Set<Stuff> stuff = service.findAllTheStuff();
 return converter.extractAllNames(stuff);
 }

}

package com.companyname.productname.stuff;

import java.util.Set;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
class StuffService {

 private final StuffDao dao;

 @Autowired
 StuffService(final StuffDao dao) {
 this.dao = dao;
 }

 Set<Stuff> findAllTheStuff() {
 //Other logic...

 return dao.getAllStuff();
 }
}

You may have noticed that I still keep public in a couple of places where it’s not strictly necessary. I firmly believe that code should express its own intent. I tend to prefer code readability over code minimalism; two concepts that are related, but different. In the case of the controller class I use public to show that this class is in fact accessed from outside the package. Similarly, the public @GetMapping annotated method is accessed from outside the package as well.

When adjusting to this new mindset of loving the default modifier, the lack of a modifier on the findAllTheStuff() method may feel uncomfortable at first. I used the following mantra to guide my intuition until I got comfortable with the access modifier nudity:

By default, we want this functionality to remain within the feature.

Anything else is one of:

Explicitly made available to any class (public).
Explicitly made available to subclasses (protected).
Explicitly restricted to only this class (private).

Notice how the imports are now only classes that are unrelated to this feature. Take advantage of this symbiosis between the default access modifier and not having to explicitly import classes from the same package. How often do you check the imports when you’re writing code or doing a code review? Chances are they’re entirely ignored. I claim that imports are crucial in understanding the coupling between units. Imports within the feature do nothing but hide excessive coupling in a forest of boilerplate code.

Case in point: A developer skilled in the Spring Framework might have noticed the @Autowired annotation from the imports alone, which is redundant in both of these classes.

In high-quality code I would expect that the quantity of methods increase as the access modifier gets more restrictive. Perhaps the default modifier would have been more intuitive had the absense of a modifier meant private rather than package-private. It certainly would have shaved many a character from my classes. This is what I believe is the design flaw behind the default access modifier, rather than its existence in the first place. Hindsight is 20/20 I guess.

Conclusion Link to heading

In today’s world of microservices and cloud architecture a common pattern for managing large systems is extracting sprout microservices (read more about this topic in my article in my company’s blog). This is a trivial exercise when the architecture of your service is oriented around features rather than layers.

The default access modifier in Java does have a purpose, and it’s one of feature encapsulation. It’s such a shame that its use has become misunderstood and frowned upon.

PS: Link to heading

I started writing this article before Java 9 came out but I have to say, I’m really excited about Java 9. It comes with a module system that adds additional encapsulation at a package level. For example, in Java 8 and earlier you could override the implementation of package-private methods in any third party library by placing your class in the library’s package and inheriting. This is no longer possible. I expect that once this new module system has gained some traction we’ll start seeing some new patterns around organising our code. I’m hoping developers become more conscious of their package structure. Although, I’m not holding my breath on that one.

Spring Dependency Injection Patterns -- The good, the bad, and the ugly

Tue, 13 Feb 2018 00:00:00 +0000

Spring developers will be familiar with its powerful Dependency Injection API. It allows you to declare @Beans that Spring then instantiates and manages. Any dependencies between these beans is then resolved by Spring and injected automagically.

Three Annotation-based Injection Patterns Link to heading

There are three ways Spring lets you declare the dependencies of your class using annotations:

Field injection (The bad)

import org.springframework.beans.factory.annotation.Autowired;

public class MyBean {
 @Autowired
 private AnotherBean anotherBean;

 //Business logic...
}

Setter injection (The ugly)

import org.springframework.beans.factory.annotation.Autowired;

public class MyBean {
 private AnotherBean anotherBean;

 @Autowired
 public void setAnotherBean(final AnotherBean anotherBean) {
 this.anotherBean = anotherBean;
 }

 //Business logic...
}

Constructor Injection (The good)

public class MyBean {
 private final AnotherBean anotherBean;

 public MyBean(final AnotherBean anotherBean) {
 this.anotherBean = anotherBean;
 }

 //Business logic...
}

The inconvenient truth about field injection Link to heading

The most common of all of these patterns is the field injection pattern. Most likely because it’s the most convenient of the three patterns. Unfortunately, because of it’s ubiquity, developers rarely learn about the other two patterns, and the pros and cons associated with each of them.

Classes using field injection tend to become harder to maintain Link to heading

When you use the field injection pattern, and want to further add dependencies to your class you only need to make a field, slap an @Autowired or @Inject annotation on it and you’re good to go. At first this sounds great, but a few months down the line you realise that your class has grown into a God class that’s hard to manage. Granted, this may very well happen with the two other patterns as well, but they force you to pay more attention to the dependencies in your class.

Every time you use field injection a unit test dies Link to heading

This line has stuck with me since I saw Josh Long’s talk on Spring Boot, and in a sense is what motivated me to write this article. How do you go about testing classes using field injection? Chances are you’re somewhere along the path of memorising the unintuitive Mockito idiom for doing this:

import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;

@RunWith(MockitoJUnitRunner.class)
public class MyBeanTest {
 @Mock
 private AnotherBean anotherBean;

 @InjectMocks
 private MyBean target;

 //Tests...
}

This reflection workaround is riddled with additional workarounds developers have to master:

What if MyBean has multiple dependencies of type AnotherBean?
Should you create a new instance of the target, or just declare it? Is there a difference?
Do you have type safety with dependencies using generics?
What if you want a real implementation of some of the dependencies, but not for others?

Classes using field injection are non-final, but are prone to circular dependencies Link to heading

If you try and declare an @Autowired field as final, you’ll get a compile error. This is annoying in and of itself, as the field is only being set once anyway! Unless you also annotate it as being @Lazy Spring tries to resolve your dependencies as part of a DAG during startup, and your field may be the source of a BeanCurrentlyInCreationException due to unexpected circular dependencies.

For example:

public class A {
 @Autowired
 private B b;
}

public class B {
 @Autowired
 private C c;
}

public class C {
 @Autowired
 private A a;
}

The problem is never this simple, and the actual problem tends to span dozens of classes through a maze of inheritance, libraries and across architecture boundaries. This problem can be resolved by either making one of the dependencies not required (allow field to be null with @Autowired(required = false)) or lazy (set field after resolving beans that depend on this bean with @Lazy). As everyone who have experienced this dreadful exception knows, finding exactly which beans make up this circle is usually a long and arduous endeavour. And once you have found all the pieces to your puzzle, which dependency do you sacrifice? How do you document this decision properly?

There are unreasonably many workarounds to solving the circular dependency problem, but hopefully by now something is starting to smell.

Pros Link to heading

The most terse of all the patterns.
Most Java developers are aware of this pattern.

Cons Link to heading

Convenience tends to hide code design red flags.
Hard to test.
Dependencies are unnecessarily mutable.
Prone to circular dependency issues.
Requires the use of (multiple) Spring or Java EE annotations.

public void setDependency(Dependency dependency) Link to heading

Boilerplate and encapsulation Link to heading

Among all the patterns, the setter injection pattern has the most boilerplate by far. Each bean must have a setter and each setter must be decorated with an @Autowired or @Inject annotation. Although this boilerplate definitely solves the problem of not thinking about the amount of dependencies you’re injecting into your class (the plethora of setters in your class should be a dead giveaway), it comes with another design smell. You’re violating encapsulation by exposing the innards of your class.

Classes using setter injection make testing easy Link to heading

There’s no reflection magic required. Just set the dependencies you wish you had.

import org.junit.Before;
import org.mockito.Mockito;

public class MyBeanTest {
 private MyBean target = new MyBean();
 private AnotherBean anotherBean = Mockito.mock(AnotherBean.class);

 @Before
 public void setUp() {
 myBean.setAnotherBean(anotherBean);
 }

 //Tests...
}

Classes using setter injection are immune to circular dependencies Link to heading

In using this pattern, Spring doesn’t try to represent your beans in a DAG, which means that circular dependencies are permitted, but it also means that circular dependencies are permitted. Allowing circular dependencies is a double-edged sword. You won’t have to debug the nasty issues that occur as a result of circular dependencies, but your code is much harder to break apart later on. The BeanCurrentlyInCreationException is in fact informing you, on startup, that your design is flawed.

Pros Link to heading

Immune to circular dependency issues.
Highly coupled classes are easily identified as setters are added.

Cons Link to heading

Violates encapsulation.
Circular dependencies are hidden.
The most boilerplate code of the three patterns.
Dependencies are unnecessarily mutable.

The solution: Constructor injection Link to heading

It turns out that the best solution to dependency injection is constructor injection. Newer platforms that support DI, such as Angular, have learned the lessons from other platforms and only support constructor injection.

Constructor injection exposes excessive coupling Link to heading

Whenever your class needs a new dependency you’re forced to add it to the constructor’s parameters. This in turn forces you to recognise the amount of coupling in your class. I’ve found that fewer than 3 dependencies is good, and anything above 5 is begging for refactoring. Counting my dependencies is a piece of cake when they’re defined on just a few contiguous lines.

As an added bonus, since final fields can be initialised in the constructor, our dependencies can be immutable - as they should be!

Testing constructor injected classes is trivial Link to heading

It’s arguably even easier than with setter injection.

import org.mockito.Mockito;

public class MyBeanTest {
 private AnotherBean anotherBean = Mockito.mock(AnotherBean.class);
 private MyBean target = new MyBean(anotherBean);

 //Tests...
}

Constructor injected subclasses must have non-default constructors Link to heading

Any subclass of a class using constructor injection must have a constructor that calls the parent constructor. This is annoying if you have inherited Spring components. I personally haven’t run into this very often. I try and avoid injected dependencies in parent components - usually done through composition over inheritance.

Pros Link to heading

Dependencies can be immutable.
Recommended by Spring.
Easiest to test out of all the patterns.
Highly coupled classes are easily identified as constructor parameters grow.
Familiar to developers coming from other platforms.
No dependency on the @Autowired annotation.

Cons Link to heading

Constructors trickle down to subclasses.
Prone to circular dependency issues.

Conclusion Link to heading

Use the constructor injection pattern.

There are times where the other patterns make sense too, but “for the sake of being consistent with the rest of the codebase”, and “it’s just easier to use the field injection pattern” are not valid excuses.

For example, using the setter injection pattern is a good intermediary step when migrating from beans declared in XML files that already use the setter pattern, or when you need to fix a BeanCurrentlyInCreationException in production, ASAP (not that you should ever end up in that position anyway).

Even the field injection pattern is sufficient when, say, sketching out a solution or answering questions on StackOverflow, unless their question is about dependency injection in Java of course. In which case you should refer them to this article.

Git and GitHub Workshop

Wed, 31 Jan 2018 00:00:00 +0000

This event is for beginners, and covers fundamental Git concepts from scratch.

10 Basic Shell Commands Every Beginner Should Know

Sun, 14 Jan 2018 00:00:00 +0000

If you aspire to be a developer, sys-admin, or some other profession that relies on computers outside of word documents and excel spreadsheets then you ought to be comfortable with a command line.

Software and IT professionals love their command-lines as it is much more powerful than any GUI could ever be. This article aims to get you started from no prior knowledge.

If this is completely new material for you, I recommend that you keep a Finder/Windows Explorer/etc. window open to understand what’s going on in familiar environments.

The difference between shells and terminals Link to heading

A terminal is a program with a menu bar and icon in your application bar etc. that provides a window into a shell. For the purpose of this article, a ‘shell’ is a low-level program that executes commands. The shell we’ll be discussing in this article is the de facto industry standard; ‘Bash’ (Bourne-Again SHell), although shells like Zsh (my personal favourite) and Fish are often used among professionals.

Examples of Terminals to run Bash in: Link to heading

Windows: All Windows computers ship with a proprietary terminal called ‘cmd’, and recent releases of Windows also ship with the ‘Windows Powershell’. Neither of these can run Bash easily, and the content of this article does not apply to these. Author’s opinion: Both of these terminals/shells are crude and immature, although cmd much more so than Powershell. You can install other terminals on Windows as well, with the most popular one being Cygwin. To get Cygwin up and running quickly (its own installer is definitely non-trivial for beginners) I suggest you install Git and thereby also Git-Bash.
Mac: All Macs ship with a pretty good terminal application called ‘Terminal’. There is also a free and more powerful terminal called ‘iTerm2’. Using just Terminal is fine for the purposes of this article.
Linux: ‘gnome-terminal’, ‘guake’, ‘xterm’, ‘konsole’, ’terminator’, ’termite’… If you’re running Linux you probably have a favourite already. Basically any Linux terminal will work for the purposes of this article.

cd: Navigate around in the file system Link to heading

To navigate between directories you use cd <path to directory> (“Change directory”)

cd Documents

will take you to the Documents directory inside your current directory. To navigate to nested directories in a single command separate the directory names with a forward-slash.

cd Documents/Important/Invoices

Special paths: Link to heading

/ (forward-slash): On Macs and Linux computers (Unix-like operating systems) the / path is called ‘root’, and is the highest directory you can go to. All directories and files exist somewhere below /. Windows is a bit different because it exposes disk drives as separate entities, and therefore / may not make much intuitive sense.
~ (tilde): The Home directory of the current user. Most of the files and directories you’re familiar will exist here. E.g. Music, Documents, Downloads etc.
. (single full stop): The current directory. It may not be obvious just yet why this is useful, but it will with time (you are encouraged to look it up if you’re curious). For now just remember the difference between . and the .. special path.
.. (two full stops): The directory above the current one. If you keep running cd .. you end up at /.
- (dash): The previous directory you cd-ed into.

# Navigate to the home directory of the current user
cd ~
# An even easier shortcut for 'cd ~'
cd

# Navigate to the highest level directory
cd /

# Navigate to Documents within your home directory
cd ~/Documents

# Navigate to current directory (~/Documents - does nothing)
cd .

# Navigate up one from Documents, i.e. your home directory
cd ..

# Navigates back to ~/Documents
cd -

ls: See the contents of the current directory Link to heading

The ls (“List”) command is useful for seeing what’s in the current directory.

ls
README.md package.json package-lock.json public/ static/

Depending on your configuration files and directories may be coloured differently.

echo: Print to screen Link to heading

You can use the echo command to print to the terminal. On its own this isn’t very interesting, but it is very powerful when combined with other commands and variables.

# Read the USER environment variable
echo "My name is $USER"
My name is kinbiko

> and »: Create file with command output Link to heading

We can take the above command, and redirect the output to a file using >.

echo "My name is $USER" > echo-example.txt

This won’t print the output anymore as we’ve told Bash to redirect output to the file echo-example.txt. Open the file in a text editor to verify that this is indeed the case.

On the other hand, the >> command will append to the file you specify, whereas > will overwrite it entirely.

echo "My name is $USER" >> echo-example.txt

Now, the echo-example.txt file will contain the same line twice.

Try the original command again, and verify that the file only has one line.

echo "My name is $USER" > echo-example.txt

rm: Remove file Link to heading

Let’s remove the file we just created in the above example.

ls # Verify that you see the file
rm echo-example.txt
ls # Verify that the file is gone

There, gone. This is a very powerful command, and won’t simply move things to your Rubbish Bin or similar if your operating system supports this feature. The file is gone forever. Make sure you only run rm on files you are absolutely sure you want to get rid off.

mkdir: Create a directory Link to heading

To create a directory use the mkdir (“make directory”) command.

mkdir repos
ls # Verify that a directory called 'repos' has been created

rmdir: Remove directory Link to heading

To remove a directory we use rmdir (“Remove directory”). Let’s remove the directory we created above.

ls # Verify that you see a directory called 'repos'
rmdir repos
ls # Verify that this directory is no longer there

Note, this command does not work if the directory is not empty. This is a measure to prevent you from shooting yourself in the foot. rmdir, like rm are powerful commands. Once deleted, you won’t get it back.

There is of course a way of deleting a directory along with all its contents, but this is a ridiculously powerful command (to the extent that it’s the centre of many geeky jokes and memes) and therefore will not be covered in this beginner’s article.

cp: Copy file Link to heading

To copy a file in Bash we use the cp command with two arguments: The first is the path to the file we want to copy, and the second is the path to the file want to create as a copy.

echo "test" > example.txt # Create a file called example.txt
cp example ../example-in-directory-above.txt

ls # Should only see one example file here, the original `example.txt`

cd ..
ls # Should only see one example file here, the copied `example-in-directory-above.txt`

Verify that these files do in fact have the same contents.

mv: Move and rename file or directory Link to heading

In Bash moving files and renaming files are the same thing. Intuitively, this can be thought of as ’editing the path of a file’. Moving files is done with the mv (“Move”) command, which similar to cp takes two arguments: the first being the path to the file to move, and the second is the new path the file should have.

echo "example" > example.txt # Create an example file to move
mv example.txt ~/test.txt # Move the example file to the home directory, and rename it to test.txt

mv works on both files and directories.

mkdir example-dir
mv example-dir ~/test-dir # Move the example directory to the home directory, and rename it to test-dir

clear: Remove old output and commands from your terminal window Link to heading

This one is one of the commands I run the most often. I like working with a clean screen so that the outputs I see on my screen is relevant to the current context I’m working wtih. To achieve this I run clear between each contextually different sets of commands that I run.

cd repos
mkdir awesome-repo
cd awesome-repo
echo "# Awesome Repo" > README.md
# more work...

clear
cd # Return to home directory
cd Downloads
rm img3514.jpeg # Delete a picture I had in my Downloads directory

Conclusion Link to heading

As you get more used to the command line you’ll pick up more and more commands and Bash tricks. Perhaps you also find new ways of doing things that you prefer over the ones listed here. That being said, you should now know your basic commands and learning new commands should be easier in the future.

My Shiniest Vim Gems

Sat, 16 Dec 2017 00:00:00 +0000

Vim is highly customisable, and I recommend everyone tailor their config according to their own preferences. That being said, here are some bits I think people may want to steal from my .vimrc. I tried to avoid the more common options you’d find in similar articles written by others, such as enabling line numbers, tpope’s surround plugin, or sane backspacing, and focus on the lesser-known tricks and settings.

General settings Link to heading

Highlight the current column Link to heading

Immediately find exactly where your cursor is, using set cursorcolumn. This will highlight the column you’re currently on, similar to how you might also do set cursorline to highlight the row (line). I have cursorline disabled as I rely on by color scheme’s highlighting of the line number in the gutter instead.

Make whitespace visible Link to heading

Show invisible characters using set listchars=tab:>-,trail:· This means that tabs should be shown as > followed by a stream of - characters. The number of - characters depends on how many spaces a tab should be as defined by your tabstop setting. Any trailing whitespace shows up as an interpunct (·). To get this working you’ll need to follow the above config with set list. Pretty rock ’n roll.

Sane line joins Link to heading

You’re probably aware that J in normal mode tries to join the next line with your current line removing the newline. This tends to work in a very awkward manner with comments. Normally, pressing JJ on the first line of:

/**
 * Creates a User entity using the default parameters.
 */

results in

/** * Creates a User entity using the default parameters. */

However, this inconvenience has been fixed as an optional setting as of version 7.3.541, and can be enabled as follows:

if v:version > 703 || v:version == 703 && has('patch541')
 set formatoptions+=j
endif

Now, assuming your version of Vim is relatively recent, the same example will be formatted as:

/** Creates a User entity using the default parameters. */

Much better.

Highlight ugly code: Link to heading

I don’t like excessively long lines of code (120+ in most languages), and I don’t like trailing whitespace. You can easily mark these using the following two commands:

match ErrorMsg '\%>120v.\+'
match ErrorMsg '\s\+$'

Plugins Link to heading

I like my plugins, and these are my favourites, in no particular order.

Terminus Link to heading

This plugin is basically enables stuff you probably want, but don’t bother to look for individually: Mouse support, different cursor shapes for insert, normal and replace mode, and paste convenience.

fzf.vim Link to heading

This plugin assumes you have fzf installed locally, as you should. Chances are you’re already using it, but it’s by far the best fuzzy finder around. More on how I use it for quick lookups in the mappings section below.

Ag Link to heading

For what fzf can’t handle, i.e. text-inside-file search. Assumes you have The Silver Searcher. Faster than ack.

EasyMotion Link to heading

Super easy navigation to anywhere in the buffer. Similar to the f option in the Vimium Google Chrome extension.

Tabular Link to heading

This plugin allows you to align a block of text according to a pattern. Suppose for example we have the following annoyingly-formatted JSON objects:

{"name": "River Tam", "skills": ["Cockney accent", "Ability to kill you with her brain"], "Age":19}
{"name": "Jayne Cobb", "skills": ["Extreme confidence", "Naming guns"], "age":33}

You know that there’s an inconsistency somewhere in here, but you can’t spot it by looking at how the code is formatted right now. Run Tabular /, to align by commas and you quickly spot that the key for age is inconsistently formatted.

{"name": "River Tam" , "skills": ["Cockney accent" , "Ability to kill you with her brain"] , "Age":19}
{"name": "Jayne Cobb" , "skills": ["Extreme confidence" , "Naming guns"] , "age":33}

Prettier Link to heading

I remember spending a bit more time than I’d like trying to get this up and running, but, really, once you try prettier you won’t write JavaScript/Markdown the same way again. It is such a relief not having to worry about formatting ever again.

vim-go Link to heading

There are many plugins designed to make certain programming languages easier to write in. Nobody has quite managed to get the full immersion as this plugin has though. Golang comes with the go command-line tool, which has a lot of features built-in that in other languages would be third party tools. For example formatting, organising imports, running tests etc. This plugin handles all of that, and then some(Going to declarations, adding new movement objects, going to alternative files). I like Go as a language, but sometimes I want to write Go just so I can use this plugin.

Mappings Link to heading

Normal mode Link to heading

Search the Git repository for filename with ? Link to heading

nnoremap ? :GFiles<CR>

This mapping relies on GFiles from fzf.vim, and saves a lot of time. Much more efficient than ctrl+p or cmd+t, and since it’s only looking for files that aren’t .gitignored you don’t get all the clutter you might get with node_modules or build/ directories.

Handle Git hunks with `<CTRL>n` and `<CTRL>p` and `<CTRL>u` Link to heading

nnoremap <c-N> :GitGutterNextHunk<CR>
nnoremap <c-P> :GitGutterPrevHunk<CR>
nnoremap <c-U> :GitGutterUndoHunk<CR>

Relies on the GitGutter plugin. Moves to the next/previous hunk using <CTRL>n/<CTRL>p, and when the cursor is on a hunk, <CTRL>u will revert that hunk. Safer than doing it from the command line as your revert is undoable as it’s in your Git undo tree.

Have you noticed that both the enter key and the backspace key are basically unused in normal mode. They just go down one line or back one character respectively. Well, we’ve got j and l for that, so let’s rebind them to move a little further (one paragraph):

nnoremap <BS> {
onoremap <BS> {
vnoremap <BS> {

nnoremap <expr> <CR> empty(&buftype) ? '}' : '<CR>'
onoremap <expr> <CR> empty(&buftype) ? '}' : '<CR>'
vnoremap <CR> }

The onoremap basically ensures that the paragraph word object is also respected, E.g. when you type d<CR> you delete from the current position to the next paragraph. The empty check is to ensure that you can still use the enter key in Quickfix windows as you would normally.

Move precisely to any position in the visible buffer Link to heading

nmap F <Plug>(easymotion-prefix)s

References the EasyMotion plugin mentioned above. Hit F followed by the value of the character you want to move to. One of two things will happen. You will either be taken to where you want to go (usually for rare characters) or you’ll be shown a map of all of these characters currently visible, with the keys you need to hit to navigate there. This is one of those tools that’s really easy to understand once you start using it, but hard to explain in text. Try it out! You won’t be sorry.

Conclusion Link to heading

This of course reflects only some of my .vimrc at this point in time. Head over to my dotfiles repository to browse my current dotfiles, among them my .vimrc.

Modern Approaches to Dealing with Legacy Code (x-post)

Tue, 15 Aug 2017 00:00:00 +0000

I wrote an article on how to manage your legacy systems using modern tools and techniques on my work blog. Please check it out.

Sentiment Analysis Workshop

Wed, 26 Apr 2017 00:00:00 +0000

Learn some basics of sentiment analysis in Python, using both rule-based sentiment analysis and ML based sentiment analysis.

Calendar

Mon, 01 Jan 0001 00:00:00 +0000

Opportunities

Mon, 01 Jan 0001 00:00:00 +0000

Roger Guldbrandsen

Mon, 01 Jan 0001 00:00:00 +0000

Hello 世界 Link to heading

I’m a software guy based in Osaka, Japan.

I speak 🇬🇧 English, 🇯🇵 Japanese, and 🇳🇴 Norwegian, and as for software technologies I’m an expert in distrubuted systems of microservices written in Go or Java, running in Kubernetes clusters. I’m also familiar with JavaScript and Python.

I love teaching and mentoring aspiring and junior engineers, and am open to speaking or teaching opportunities at Japanese universities, as well as speaking at technical conferences and meetups. I’m also interested in contributing to projects that make a real difference. Whether that’s benefiting humanity, helping dogs, or protecting the environment.

If you’re aware of any teaching or speaking opportunities, or any projects you think I might be interested in, please don’t hesitate to contact me via e-mail at contact-kinbiko@pm.me

Utils

Mon, 01 Jan 0001 00:00:00 +0000

kinbiko.com

Installing Rockbox on an iPod Classic 7G with iFlash from Linux: a sector size odyssey

The setup Link to heading

The symptom Link to heading

The dead ends Link to heading

Dead end 1: “the old bootloader doesn’t support 4K sectors” Link to heading

Dead end 2: “the cluster size is too large” Link to heading

Dead end 3: “there’s a stale superfloppy BPB at sector 0” Link to heading

The actual problem: virtual sector sizes Link to heading

The iFlash adapter uses 512-byte sectors Link to heading

Rockbox’s USB mode uses 4096-byte virtual sectors Link to heading

Linux tools respect the device’s reported sector size Link to heading

The bootloader reads with 512-byte sectors Link to heading

The fix Link to heading

How to tell which USB mode you’re in Link to heading

Other things I learned along the way Link to heading

Summary Link to heading

4-4-6-2 Meditation

Incidents and Blame

Managers and Incidents

Power Dynamics Link to heading

Performance Evaluations Link to heading

Adoption Non-Reaction Link to heading

Go on… Link to heading

Naive Musings on AI

Writing More, Publishing More

How I Write Go

Go Workshop

Docker Deep Dive

Profitable Go Tests

Commandline Foo

Cross Cultural Teams

Automating your CV

Auto-merging Dependabot PRs

Important note Link to heading

(Go) Error messages should be boring

Rewriting my Neovim .vimrc in Lua

This article pales in comparison Link to heading

Lua primer Link to heading

Store your configuration in Git Link to heading

Start with structure before refactoring Link to heading

Switch to a Lua-native package manager Link to heading

Rewrite bit by bit, then commit Link to heading

Options Link to heading

Mappings Link to heading

Set up LSP Link to heading

Create your own namespace to avoid collisions Link to heading

How Iota works in Go

Hiding Directories in Finder

Testing Tips for Intermediate Gophers: Coverage

Test coverage Link to heading

So what’s the purpose of test coverage then? Link to heading

How to report test coverage in Go Link to heading

A note on covermode and concurrent programs Link to heading

Function Types in Go

Type definitions Link to heading

The elegance of http.HandlerFunc Link to heading

Function types as lightweight interfaces Link to heading

Use function definitions to create default arguments Link to heading

Use type aliases for documentation Link to heading

mokku -- How I built a Go Mocking framework in 5 days

Key features Link to heading

GoMock?! Go away Link to heading

Day 1: Planning Link to heading

Day 2: Parsing Link to heading

Day 3: Templates Link to heading

Day 4: Make Things Presentable Link to heading

Day 5: Presentation Link to heading

Conclusion Link to heading

Testing Tips for Beginner Gophers

The built-in assertions are all you need. Link to heading

Table-driven tests Link to heading

Sub-tests Link to heading

Integration tests vs unit tests; the _test package. Link to heading

jsonassert -- A test utility for comparing and verifying JSON

Feature-oriented package structures and the default access modifier

Be aware of ingrained rules you don’t fully understand Link to heading

Prefer feature-oriented architectures over layer-oriented architectures. Link to heading

Rethinking best practices Link to heading

Rethinking Java best practices Link to heading

A note on `covermode` and concurrent programs Link to heading

The elegance of `http.HandlerFunc` Link to heading

Integration tests vs unit tests; the `_test` package. Link to heading

Handle Git hunks with `<CTRL>n` and `<CTRL>p` and `<CTRL>u` Link to heading