Using isync

2019-09-18T00:00:00Z

Isync is a command-line utility used for synchronising (downloading) remote and local mailboxes. Isync can be used as part of a tool-chain to be able to retrieve, store and send mail all from the command line.

Benefits

The benefits of using isync combined with neomutt(a command-line mail client) over just neomutt is that isync stores a local cache, so when offline, you can still view your emails. There is also the benefit of increased speed and a reduction in bandwidth usage; as isync only downloads new emails to the local cache. Whereas neomutt, when hooked into IMAP, doesn't keep a local cache. Therefore it needs to re-download all emails each time; meaning you can't view your email without an internet connection.

Installation

For Arch Linux isync is in the main repositories and can be installed easily via pacman.

sudo pacman -S isync

Setting up isync

Isync's configuration file resides in ${HOME}/.mbsyncrc. Within this configuration file, you need to define how to download each mailbox and each 'channel' for every email account.

We will start by defining the remote IMAP servers configuration. As an example, I will pretend to be setting up this configuration for a Gmail account.

Note to use isync with Gmail you need to 'Enable Less Secure Apps', you can follow the steps here https://www.dev2qa.com/how-do-i-enable-less-secure-apps-on-gmail/.

IMAP setup

# Global Channel configuration.
Create Both
Expunge Both
Remove Both
Sync All
SyncState *

################################
####### example@gmail.com ######
################################
# IMAP setup
IMAPAccount example-gmail
Host imap.gmail.com
User example@gmail.com
PassCmd "pass Communications/Gmail"
SSLType IMAPS

...

The variables under # Global Channel configuration. will be explained later when talking about channel configuration.

################################
####### example@gmail.com ######
################################

Is not required, but I use it to denote the different sections of the configuration file for the various emails; helping me to find what I want quickly.

IMAPAccount example-gmail Denotes the IMAP's account name, and tells isync the following options are related to IMAP. The name does not affect anything; so you can name it anything you want.

Host imap.gmail.com Denotes the IMAP server's URL. The IMAP URL is email provider-specific, check your email providers website to find the provider's specific URL.

User example@gmail.com Is the username to login in as to the IMAP server. This is typically your email, but it can vary depending on the provider.

PassCmd "pass Communications/Gmail" PassCmd tells isync the shell command it can execute to fetch the IMAP servers password for the user. I use the utility 'pass' to manage all my passwords, it is a useful utility, and you should check it out.

However, you can do Pass examplePassword if you would prefer to enter the password in plain text. But be warned now anyone who can get hold of your configuration file has YOUR PASSWORD!

Note you don't need to specify either PassCmd or Pass if the options are missing when executing isync's command mbsync then you are just prompted for the password.

Local & Remote storage setup

...

# Remote storage setup
IMAPStore example-gmail-remote
Account example-gmail

# Local storage setup
MaildirStore example-gmail-local
Path ~/.neomutt/mail/example-gmail/
Inbox ~/.neomutt/mail/example-gmail/inbox

...

Now that we have set up the IMAP server configuration, we will move on to define the properties of remote and local locations we will be synchronising.

IMAPStore example-gmail-remote Denotes the IMAP's store name, and tells isync the following configurations are related to the IMAP store. Again the name does not affect anything; so you can name it anything you want.

Account example-gmail Specifies which IMAP configuration to use for the IMAPStore, here we are referencing the IMAPAccount configuration we specified earlier, using the name we gave it.

MaildirStore example-gmail-local Gives your local Maildir Store a name, also informing isync a Maildir Store is about to be configured. As previously stated; the alias can be anything you want.

Path ~/.neomutt/mail/example-gmail/ tells isync the location on the file system to store the local mail cache.

Inbox ~/.neomutt/mail/example-gmail/inbox defines the location to contain the local cache's inbox folder. Typically it is inside the Path directory defined above.

NOTE - Path needs to end in a / whereas Inbox does not.

Setting up the channels to synchronise

...

# Inbox
Channel example-gmail-inbox
Master :example-gmail-remote:
Slave  :example-gmail-local:inbox

# Sent
Channel example-gmail-sent
Master :example-gmail-remote:"[Gmail]/Sent Mail"
slave  :example-gmail-local:sent

# Drafts
Channel example-gmail-drafts
Master :example-gmail-remote:"[Gmail]/Drafts"
slave  :example-gmail-local:drafts

# Trash
Channel example-gmail-trash
Master :example-gmail-remote:"[Gmail]/Bin"
slave  :example-gmail-local:trash

# Spam
Channel example-gmail-spam
Master :example-gmail-remote:"[Gmail]/Spam"
slave  :example-gmail-local:spam

...

Now we have defined the local/remote storage locations and entered the IMAP credentials; we now need to define which 'channels' (folders) we want to synchronise with our local cache. We don't have to download every remote folder we can be selective, change the folders name and various other custom configurations to meet your needs!

Channel example-gmail-inbox Denotes the channels name and tells isync the following parameters are related to this channel. Again the name does not affect anything; so you can name it anything you want.

Master :example-gmail-remote: Defines the Master folder for the channel as the remote IMAP, from which it will take directions from. We reference the remote IMAPStore by the name we gave it prior. The option follows the format Master :<remote IMAP>:<folder> because we are defining the inbox which is the default we do not need to specify the folder, but for other folders, they will need to be detailed.

Slave :example-gmail-local:inbox Declares the slave repository as the local cache via the naming we gave it. Like Master the format has the format Slave :<local cache name>:<folder>. However, note that even though it is still the inbox channel being defined, you need to give the folder to save it to. The remote host will be synchronised to $PATH/<folder>, i.e. the path we defined earlier for MaildirStore.

Channel example-gmail-sent
Master :example-gmail-remote:"[Gmail]/Sent Mail"
slave  :example-gmail-local:sent

Above is another example of a channel; for this, we are defining the sent channel. Note the remote Master now contains a folder, this is the same naming and structure as on the remote host. The remote data in [Gmail]/Sent Mail is now synchronised to our local cache at $PATH/sent.

Global/Local Channel Configurations

# Global Channel configuration.
Create Both
Expunge Both
Remove Both
Sync All
SyncState *

...

At the beginning of the configuration file, there were the above options which I did not explain. They are configurations for channels, having them at the top of the file makes them global options which apply to every channel.

However, you can override global settings by defining them locally for a channel.

Channel example-gmail-sent
Master :example-gmail-remote:"[Gmail]/Sent Mail"
slave  :example-gmail-local:sent
Create None

...

For the channel above the global setting Create Both has been overridden by Create None, this local option is set for this channel only, and all the other channels use the global setting.

Create Both Tells isync to create missing mailboxes automatically. Create can take the options {None|Master|Slave|Both}.

Expunge Both Permanently deletes all messages marked for deletion. Expunge can take the options {None|Master|Slave|Both}.

Remove Both Notifies isync if it should propagate mailbox deletions. Remove can take the options {None|Master|Slave|Both}.

Sync All Informs isync to synchronise all operations, pull, push, delete etc.

SyncState * Tells isync to keep the channel's synchronisation state files inside the channel's local cache of the folder, in the file .mbsyncstate.

Grouping channels


...

Group example-gmail
Channel example-gmail-inbox
Channel example-gmail-sent
Channel example-gmail-drafts
Channel example-gmail-trash
Channel example-gmail-spam

We can group channels into a group. Meaning you can use the command mbsync <group> to synchronise all the channels in that group. So you can only update select email accounts or specific folders across all accounts if we are limited by download speed or size etc.

However you don't need to define any groups, you can use mbsync -a to synchronise all channels in the mbsync configuration file.

Multiple Email's

As I previously alluded to when mentioning the inclusion of :

################################
####### example@gmail.com ######
################################

You can configure multiple email accounts from which to synchronise channels. To achieve this, you define them sequentially one after another. I use the header to signify where one email's configuration stops and another begins. I use it to quickly find my place among a wall of text annotating several account's configuration and options.

...

################################
####### example@gmail.com ######
################################

...

################################
##### example@outlook.co.uk ####
################################

...

TIP: How to find out remote IMAP folder names

Sometimes you have no idea of the remote IMAP folders naming or the structure, meaning you can not configure the channels for the email address.

To overcome this problem, you can download the whole remote folder, without altering the structure or naming, then using the structure of the unaltered local cache you can work out how to configure the channels you desire.

...

# Local storage
MaildirStore example-gmail-local
Subfolders Verbatim
Path ~/.neomutt/mail/example-gmail/
Inbox ~/.neomutt/mail/example-gmail/inbox

Channel example-gmail-default
Master :example-gmail-remote:
Slave  :example-gmail-local:
# Include everything
Patterns "*"

Subfolders Verbatim specifies that the local cache should use the directory hierarchical structure $PATH/top/sub/subsub. Meaning it will copy the remote structure verbatim.

Patterns "*" is some simple regex telling isync to synchronise every remote folder.

Notice how on the end of the Slave for the channel that there is no directory specified, meaning it will default to the remote naming and structure.

Now you need to synchronise the email then inspect your local cache to know the remote server's structure and naming conventions.

TIP: Documentation

If you want to know more about a specific configuration and the options available you can read the man page man mbsync and scroll past the command line usage description to get to the configuration file documentation!

References

Linux Foundation Certified SysAdmin

2020-02-13T00:00:00Z

I recently took and passed my Linux Foundation Certified SysAdmin certification. The certification examination consists of SSHing into a remote machine and performing a set of tasks. The tasks cover all the basic operations you would be expected to perform while working as a system administrator; creating and configuring file systems, using SELinux/AppArmor, adding users/groups, setting up RAID, etc.

I believe practical examinations are an improvement over written examinations at demonstrating the holder's knowledge of Linux. However, I have a singular issue with the LFCS exam, that being the maximum time allowance. During the examination, you are asked to perform specific exercises, with no access to the internet, you can easily spend ten or fifteen minutes scrolling through the man pages. Yet you could find the specific argument/flag within seconds via a search engine.

Allowing the usage of the internet to search for command arguments/flags quickly, this would stop capable system administrators from being bogged down and failing just because they could not remember a specific command/argument. This would make the examination easier but they could counter-balance this by increasing the complexity of the questions, so someone just using Google not understanding what they are doing can not copy and paste their way into becoming certified.

Because of the specific nature of the questions, I would not advise anyone who has not already used Linux and had years of experience as a systems administrator to take the exam. Without already having any experience, you would need to memorise a lot of different commands and arguments to be able to keep up with the exam's pace to achieve a passing mark. Online I have seen numerous people with no prior experience saying they have re-take it four or five times to pass.

Git Hooks and Rust!

2020-04-22T00:00:00Z

I have started writing a lot of Rust recently, but that is a story for another time. Like with all my open source projects my Rust repositories have CI pipelines. These CI pipelines run for every pull request; enforcing basic code quality by checking the code is properly formatted etc.

As part of my development workflow, I usually have formatting and other basic code quality issues automatically fixed or flagged. However, sometimes issues slip through.

Having pipeline failures for issues that otherwise could have quickly been checked and fixed locally, is a problem because of the lengthy feedback loop. Therefore ideally, you want to stop wasting time by not committing code that does not pass basic code quality checks.

Enter Git Hooks, which is essentially event-driven execution upon scripts before or after events within Git's lifecycle. Git Hooks are part of core Git, so no additional installation is necessary. For a more in-depth introduction to Git Hooks, you can visit https://githooks.com/.

The only requirement for a Git Hook is that it is executable, but Shell/Bash are the most commonly used languages. To react to a specific event you place an executable inside the .git/hooks/ directory with the event's name, e.g. .git/hooks/pre-commit.

So we can utilise Git hooks to either perform code quality checks locally before creating the commits or pushing them remotely. To provide the best developer experience we want to fix or flag a commit before it is created, to avoid rebasing/commit editing headaches.

The pre-commit hook is fired after the commit message and content have been provided; but before the commit has been created. So using this event we can alter the contents of a commit, or if the hook exits with a non-zero status code stop it from being created.

Beneath I will show you multiple examples of using Git hooks being used to enforce formatting. The examples can easily be modified to automate other tasks such as code linting etc. Hopefully, afterwards, you will see the power in Git hooks and begin to use them to help you deliver better quality software.

Examples

Formatting

Formatting

Formatting - Single Project

Checking code formatted is very quick, which makes it one of many ideal code quality checks that can preemptively be reviewed locally. Using the pre-commit event when we find incorrectly formatted code we have two choices. Either we can stop the commit from being created, or alter the content so it is correctly formatted. I prefer to stop the commit being created, otherwise, you are never sure what has actually been committed.

.git/hooks/pre-commit

#!/usr/bin/env bash

# Exit immediately on first non-zero status from a command in the script.
set -o errexit

# Ensure all tools being used are available.
command -v 'cargo' >/dev/null 2>&1 || { echo >&2 "The 'cargo' command is not installed. Aborting."; exit 1; }
command -v 'rustfmt' >/dev/null 2>&1 || { echo >&2 "The 'rustfmt' command is not installed. Aborting."; exit 1; }

# Check the formatting of all Rust code.
cargo fmt --all -- --check

Using the above as a Git hook it will stop you from committing unformatted code, because cargo fmt --all -- --check will exit with a non-zero status code cancelling the commit.

E.g. with src/main.rs incorrectly formatted as follows,

src/main.rs

fn  main() {
println!("Hello, world!");
	}

The Git hook will print out that src/main.rs is incorrectly formatted, indicating which lines need to be corrected.

Diff in /home/rust/hello-world/src/main.rs at line 1:
-fn  main() {
-println!("Hello, world!");
-    }
+fn main() {
+    println!("Hello, world!");
+}

You can then format it correctly before attempting to commit it again.

Formatting - Monorepo

The Git hook example above falls short when monorepos are introduced which contain multiple packages. We can perform checks upon each package by programmatically getting each package’s manifest and then checking each package independently. Sorting the list of package’s manifest before working upon it will also improve usability by providing a consistent output.

.git/hooks/pre-commit

#!/usr/bin/env bash

# Exit immediately on first non-zero status from a command in the script.
set -o errexit

# Ensure all tools being used are available.
command -v 'cargo' >/dev/null 2>&1 || { echo >&2 "The 'cargo' command is not installed. Aborting."; exit 1; }
command -v 'rustfmt' >/dev/null 2>&1 || { echo >&2 "The 'rustfmt' command is not installed. Aborting."; exit 1; }

# Check the formatting of all Rust code for every project, if in monorepo.
for project in $(du -a | grep "Cargo[.]toml$" | awk '{print $2}' | sort); do
		# Check project is correctly formatted.
		cargo fmt --all --manifest-path "${project}" -- --check
done

The above example works for both monorepos and non-monorepos alike. However there is a subtle usability issue, the Git hook will fail and exit on the first incorrectly formatted project. This may mean multiple rounds of attempting to commit before learning of another project that needs formatted. It would be more useful to list every package that needs formatting the first time. To make it more usable we can remove exiting on the first error and store the status code, only returning it at the end.

.git/hooks/pre-commit

#!/usr/bin/env bash

# Ensure all tools being used are available.
command -v 'cargo' >/dev/null 2>&1 || { echo >&2 "The 'cargo' command is not installed. Aborting."; exit 1; }
command -v 'rustfmt' >/dev/null 2>&1 || { echo >&2 "The 'rustfmt' command is not installed. Aborting."; exit 1; }

status=0

# Check the formatting of all Rust code for every project, if in monorepo.
for project in $(du -a | grep "Cargo.toml$" | awk '{print $2}' | sort); do
		# Check project is correctly formatted.
		cargo fmt --all --manifest-path "${project}" -- --check
		project_is_formatted=$?

		if [ "${project_is_formatted}" -ne 0 ]; then
				status=${project_is_formatted}
		fi
done

exit $status

A lot of noise may be generated by all the packages within a monorepo when the formatting issues are reported in detail. To reduce the noise you can instead list just the files which need formatted, by changing to cargo fmt --all --manifest-path "${project}" --message-format short -- --check.

Formatting - Monorepo & Workspaces

The above examples all work fine for separate Rust packages, however, Rust has the idea of workspaces. The Rust documentation describes a workspace as

A workspace is a collection of one or more packages that share common dependency resolution (with a shared Cargo.lock), output directory, and various settings such as profiles.

see https://doc.rust-lang.org/cargo/reference/workspaces.html for additional details about workspaces.

.git/hooks/pre-commit

#!/usr/bin/env bash

# Ensure all tools being used are available.
command -v 'cargo' >/dev/null 2>&1 || { echo >&2 "The 'cargo' command is not installed. Aborting."; exit 1; }
command -v 'rustfmt' >/dev/null 2>&1 || { echo >&2 "The 'rustfmt' command is not installed. Aborting."; exit 1; }

status=0

# Check the formatting of all Rust code for every project, if in monorepo.
for project in $(du -a | grep "Cargo.toml$" | awk '{print $2}' | xargs -I {} sh -c 'cargo metadata --manifest-path {} | jq ".workspace_root"' | sort  | uniq); do
		# Check project is correctly formatted.
		cargo fmt --all --manifest-path "${project}" -- --check
		project_is_formatted=$?

		if [ "${project_is_formatted}" -ne 0 ]; then
				status=${project_is_formatted}
		fi
done

exit $status

AWS Lambda Language Benchmarking

2021-02-02T00:00:00Z

I have seen several articles from years prior detailing the performance for various languages on AWS Lambda. Frankly, I did not believe the results the articles were reporting.

With no source code, infrastructure as code, or any details on conducting the performance testing, I could not reproduce and verify the results myself. Therefore I set out to create an open-source repository containing all the source code, infrastructure, etc. So anyone can reproduce my findings, even in years to come as the performance characteristics may change.

To test the various languages, they will all be performing a straightforward task. They are to take a String as an input and perform 100,000 SHA256 hashes upon it and return the results, to simulate an application performing work. For larger and more complex applications, I would expect the performance margins to widen even further.

The implementations for the various languages are what you could expect the typical developer to produce. No arcane wizardry or obscure performance optimisations have been introduced to tip the results to favour a particular language.

I will be investigating the languages Node.js, Python3, Go and Rust, this will cover the majority of languages used in AWS Lamdba implementations.

I am open to pull requests for both additional languages and performance optimisations and will update the article accordingly.

https://gitlab.com/DeveloperC/aws_lambda_language_benchmarking

Local Performance

AWS Lambda & API Gateway Performance

Symbolic links and gitignore

2022-09-27T00:00:00Z

I've often seen developers, myself included, use .gitignore patterns[1] that do not work when used with symbolic links (symlinks)[2]. While not universal, it's common to leave a trailing / on the end of a directory name to indicate it is a directory.

However, if you use a trailing / in a gitignore pattern you will have issues with symbolic links. My gut feeling was that Git would treat a symbolic link like what it's linking to. But since symbolic links are a 'special kind of file that points to another file'[2], Git seems to treat them as files. As this has tripped me up a couple of times I want to confirm this and come up with some best practices for the future.

Symbolic links

So from the results below we can confirm that Git treats symbolic links as files, regardless of what they are linking to. However, another reason for using a trailing / is to notate the end of the pattern. So we should confirm that omitting it doesn't cause unintended files or directories to be ignored, before recommending to not include trailing /'s.

Type	`target`	`target/`
File	🚫	📦
Symbolic Link → File	🚫	📦
Directory	🚫	🚫
Symbolic Link → Directory	🚫	📦

gitignore-symlink-test.sh

#!/usr/bin/env sh

set -o errexit
set -o nounset

# Emoji output: 🚫 = ignored, 📦 = tracked
check_ignored() {
	git ls-files . --exclude-standard --others | grep -q -E "$1" && echo "📦" || echo "🚫"
}

run_test() {
	test_dir=$(mktemp -d)
	cd "${test_dir}"
	git init >/dev/null 2>&1

	case "$1" in
		real_file)
			touch "target"
			;;
		symlink_file)
			touch "source_file"
			ln -s "source_file" "target"
			;;
		real_dir)
			mkdir "target"
			touch "target/inner.txt"
			;;
		symlink_dir)
			mkdir "source_dir"
			touch "source_dir/inner.txt"
			ln -s "source_dir" "target"
			;;
	esac

	echo "target" > .gitignore
	result_no_slash=$(check_ignored "^target($|/)")

	echo "target/" > .gitignore
	result_with_slash=$(check_ignored "^target($|/)")

	echo "| $2 | ${result_no_slash} | ${result_with_slash} |"
}

echo "| Type | \`target\` | \`target/\` |"
echo "|---|---|---|"
run_test "real_file" "File"
run_test "symlink_file" "Symbolic Link → File"
run_test "real_dir" "Directory"
run_test "symlink_dir" "Symbolic Link → Directory"

Prefix

From the results we can confirm that Git does not prefix match. A pattern of target will not match target-extra, so omitting the trailing / is safe.

Type	`target`	`target/`
File	📦	📦
Symbolic Link → File	📦	📦
Directory	📦	📦
Symbolic Link → Directory	📦	📦

gitignore-prefix-test.sh

#!/usr/bin/env sh

set -o errexit
set -o nounset

# Emoji output: 🚫 = ignored, 📦 = tracked
check_ignored() {
	git ls-files . --exclude-standard --others | grep -q -E "$1" && echo "📦" || echo "🚫"
}

run_test() {
	test_dir=$(mktemp -d)
	cd "${test_dir}"
	git init >/dev/null 2>&1

	case "$1" in
		file)
			touch "target-extra"
			;;
		symlink_file)
			touch "source_file"
			ln -s "source_file" "target-extra"
			;;
		dir)
			mkdir "target-extra"
			touch "target-extra/inner.txt"
			;;
		symlink_dir)
			mkdir "source_dir"
			touch "source_dir/inner.txt"
			ln -s "source_dir" "target-extra"
			;;
	esac

	echo "target" > .gitignore
	result_no_slash=$(check_ignored "^target-extra($|/)")

	echo "target/" > .gitignore
	result_with_slash=$(check_ignored "^target-extra($|/)")

	echo "| $2 | ${result_no_slash} | ${result_with_slash} |"
}

echo "| Type | \`target\` | \`target/\` |"
echo "|---|---|---|"
run_test "file" "File"
run_test "symlink_file" "Symbolic Link → File"
run_test "dir" "Directory"
run_test "symlink_dir" "Symbolic Link → Directory"