Random Ramblings

Hardening the Unpacakgeable: A systemd-run Sandbox for Third-Party Binaries

2026-04-12T12:53:00+05:30

The Shift in Software Consumption

Historically, I have been a "distribution-first" user. Sticking to tools packaged within the Debian archives provides a layer of trust; maintainers validate licenses, audit code, and ensure the entire dependency chain is verified. However, the rapid pace of development in the Generative AI space—specifically with new tools like Gemini-CLI—has made this traditional approach difficult to sustain.

Many modern CLI tools are built within the npm or Python ecosystems. For a distribution packager, these are a nightmare; packaging a single tool often requires packaging a massive, shifting dependency chain. Consequently, I found myself forced to use third-party binaries, bypassing the safety of the Debian archive.

The Supply Chain Risk

Recent supply chain attacks affecting widely used packages like axios and LiteLLM have made it clear: running unvetted binaries on a personal system is a significant risk. These scripts often have full access to your $HOME directory, SSH keys, and the system D-Bus.

After discussing these concerns with a colleague, I was inspired by his approach—using a Flatpak-style sandbox for even basic applications like Google Chrome. I decided to build a generalized version of this using OpenCode and Qwen 3.6 Fast (which was available for free use at the time) to create a robust, transient sandbox utility.

The Solution: safe-run-binary

My script, safe-run-binary, leverages systemd-run to execute binaries within an isolated scope. It implements strict filesystem masking and resource control to ensure that even if a dependency is compromised, the "blast radius" is contained.

Key Technical Features

1. Virtualized Home Directory (tmpfs)

Instead of exposing my real home directory, the script mounts a tmpfs over $HOME. It then selectively creates and bind-mounts only the necessary subdirectories (like .cache or .config) into a virtual structure. This prevents the application from ever "seeing" sensitive files like ~/.ssh or ~/.gnupg.

2. D-Bus Isolation via xdg-dbus-proxy

For GUI applications, providing raw access to the D-Bus is a security hole. The script uses xdg-dbus-proxy to sit between the application and the system bus. By using the --filter and --talk=org.freedesktop.portal.* flags, the app can only communicate with necessary portals (like the file picker) rather than sniffing the entire bus.

3. Linux Namespace Restrictions

The sandbox utilizes several systemd execution properties to harden the process:

RestrictNamespaces=yes: For CLI tools, this prevents the app from creating its own nested namespaces.
PrivateTmp=yes: Ensures a private /tmp space that isn't shared with the host.
NoNewPrivileges=yes: Prevents the binary from gaining elevated permissions through SUID/SGID bits.

4. GPU and Audio Passthrough

The script intelligently detects and binds Wayland, PipeWire, and NVIDIA/DRI device nodes. This allows browsers like Firefox to run with full hardware acceleration and audio support while remaining locked out of the rest of the filesystem.

Usage

To run a CLI tool like Gemini-CLI with access only to a specific directory:

safe-run-binary -b ~/.gemini-config -- npx @google/gemini-cli

For a GUI application like Firefox:

safe-run-binary --gui -b ~/.mozilla -b ~/.cache/mozilla -b ~/Downloads -- firefox

Conclusion

While it is not always possible to escape the need for third-party software, it is possible to control the environment in which it operates. By leveraging native Linux primitives like systemd and namespaces, high-grade isolation is achievable.

PS: If you spot any issues or have suggestions for improving the script, feel free to raise a PR on the repo.

Using Gemini CLI to Configure the Hyprland Window Manager

2026-03-15T12:35:00+05:30

What led to this experiment? Well, for one, Well, for one, there was a thought shared by Andrej Karpathy regarding the shift towards "Agentic" workflows.

"The future of software is not just 'tools', but 'agents' that can navigate complex tasks on your behalf."

—Andrej Karpathy

Recently, I spoke with Ritesh, who mentioned his success using the Gemini CLI to debug an idle power drain issue on his laptop. I wanted to experiment with this myself, and I had the perfect use case: configuring the Hyprland Window Manager on my aging laptop.

The machine is nearly eight years old with 12GB of RAM (upgraded from the original 4GB). I found that GNOME and KDE were becoming overkill, often leading to system freezes when running multiple AI-powered IDEs like Antigravity and VS Code with Co-pilot. Coincidentally, I noticed my Jio number had a "Google One 2TB" and "Google AI Premium" plan available to claim. I claimed it, and now here I am, experimenting with the Gemini CLI.

Getting Started

First, you need to install geminicli. It is an open-source project, and currently, the easiest way to install it is via the Node Package Manager (npm):

npm install -g @google/gemini-cli

Next, we need to create a context for Gemini—a set of instructions for it to follow throughout the project. This is managed via a GEMINI.md file. I went to Google Gemini, explained my requirements, and asked it to generate one for me.

My requirements were:

A minimalist but fully functional session, comparable to my existing GNOME setup.
Basic functionalities including wallpaper, screen locks, and a status bar with system icons.
Swapping Control and Caps Lock (a must for Emacs users).
Mandatory permission prompts for privileged operations; otherwise, it can work freely within a specified directory.
Persistent memory/artifacts for the session.
Permission to inspect my current session to understand the existing hardware and software configuration.

The goal was to reduce bloat and reclaim memory for heavy applications like Antigravity and VS Code. Gemini provided the following GEMINI.md file:

# Role: Hyprland Configuration Specialist (Minimalist & High-Performance)

You are a Linux Systems Engineer specializing in migrating users from heavy
Desktop Environments to minimalist, tiling-based Wayland sessions on Debian.
Your goal is to maximize available RAM for heavy applications while maintaining
essential desktop features.

## 1. Environment & Persona
- **Target OS:** Debian (Linux)
- **Target WM:** Hyprland
- **Hardware:** ThinkPad E470 (i5-7th Gen, 12GB RAM)
- **User Profile:** Emacs user, prioritizes "anti-gravity" (zero bloat).
- **Tone:** Technical, concise, and security-conscious.

## 2. Core Functional Requirements
- **Status Bar:** `waybar` (with CPU, RAM, Network, and Battery icons).
- **Wallpaper:** `swww` or `hyprpaper`.
- **Screen Lock:** `hyprlock` + `hypridle`.
- **Input Mapping:** Swap Control and Caps Lock (`kb_options = ctrl:nocaps`).

## 3. Operational Constraints
- **Permission First:** Ask before using `sudo` or writing outside the work directory.
- **Inspection:** Use `hyprctl`, `lsmod`, or `gsettings` for compatibility checks.
- **Artifact Management:** Update `MEMORY.md` after every major step.

Gemini also recommended creating a MEMORY.md file to track progress. Interestingly, Gemini remembered that I had previously shared dmidecode output, so it already knew my exact laptop specs. (Though it did include a note about me being a "daily rice eater"—I assume it meant Linux 'ricing,' though I actually use Debian Unstable, not Stable!).

The AI suggested starting with this prompt:

Read MEMORY.md and GEMINI.md. Based on my hardware, give me a shell script to inspect my current GNOME environment so we can start replicating the session basics.

How Did It Go?

I initialized a git repository for these files and instructed the Gemini CLI to update GEMINI.md and commit changes after every major step so I could track the progress.

The workflow looked like this:

Inspection: It created a script to extract my GNOME settings.
Configuration: Once I provided the output, it began configuring Hyprland.
Utilities: It generated an installation script for all required Wayland utilities.
Validation: All changes were staged in a hypr-config-draft folder. I had Gemini verify them using hyprland --verify-config before moving them to ~/.config/hypr.

Most things worked immediately, but I hit a snag with the wallpaper. Even after generating the config, hyprpaper failed to display anything. The AI got stuck in a loop trying to debug it. I eventually spawned a second Gemini CLI instance to review the code and logs.

The debug log showed: 'DEBUG ]: Monitor eDP-1 has no target: no wp will be created'. It turns out the configuration format was outdated. By feeding the Hyprpaper Wiki into the AI, it finally corrected the config, and the wallpaper appeared.

After that, it successfully fixed an ssh-agent issue and configured a clipboard manager with custom keybindings.

Learnings

I have used window managers for a long time because my hardware was rarely top-of-the-line. However, I had moved back to KDE/GNOME with the arrival of Wayland because most of my preferred WMs were X11-based.

Manually configuring a window manager is a painful, time-consuming process involving endless wiki-trawling and trial-and-error. What usually takes weeks took only a few hours with the Gemini CLI.

AI isn't perfect—I still had to step in and guide it when it hit a wall—but the efficiency gain is undeniable. If you're interested in the configuration or the history of the session, you can find the repository here.

I still have a few pending items in MEMORY.md, but I'll tackle those next time!

Learning Notes: Debsecan MCP Server

2026-02-21T17:13:00+05:30

Since Generative AI is currently the most popular topic, I wanted to get my hands dirty and learn something new. I was learning about the Model Context Protocol at the time and wanted to apply it to build something simple.

Idea

On Debian systems, we use debsecan to find vulnerabilities. However, the tool currently provides a simple list of vulnerabilities and packages with no indication of the system's security posture—meaning no criticality information is exposed and no executive summary is provided regarding what needs to be fixed. Of course, one can simply run the following to install existing fixes and be done with it:

apt-get install $(debsecan --suite sid --format packages --only-fixed)

But this is not how things work in corporate environments; you need to provide a report showing the system's previous state and the actions taken to bring it to a safe state. It is all about metrics and reports.

My goal was to use debsecan to generate a list of vulnerabilities, find more detailed information on them, and prioritize them as critical, high, medium, or low. By providing this information to an AI, I could ask it to generate an executive summary report detailing what needs to be addressed immediately and the overall security posture of the system.

Initial Take

My initial thought was to use an existing LLM, either self-hosted or a cloud-based LLM like Gemini (which provides an API with generous limits via AI Studio). I designed functions to output the list of vulnerabilities on the system and provide detailed information on each. The idea was to use these as "tools" for the LLM.

Learnings

I learned about open-source LLMs using Ollama, which allows you to download and use models on your laptop.
I used Llama 3.1, Llama 3.2, and Granite 4 on my laptop without a GPU. I managed to run my experiments, even though they were time-consuming and occasionally caused my laptop to crash.
I learned about Pydantic and how to use it to parse custom JSON schemas with minimal effort.
I learned about osv.dev, an open-source initiative by Google that aggregates vulnerability information from various sources and provides data in a well-documented JSON schema format.
I learned about the EPSS (Exploit Prediction Scoring System) and how it is used alongside static CVSS scoring to detect truly critical vulnerabilities. The EPSS score provides an idea of the probability of a vulnerability being exploited in the wild based on actual real-world attacks.

These experiments led to a collection of notebooks. One key takeaway was that when defining tools, I cannot simply output massive amounts of text because it consumes tokens and increases costs for paid models (though it is fine for local models using your own hardware and energy). Self-hosted models require significant prompting to produce proper output, which helped me understand the real-world application of prompt engineering.

Change of Plans

Despite extensive experimentation, I felt I was nowhere close to a full implementation. While using a Gemini learning tool to study MCP, it suddenly occurred to me: why not write the entire thing as an MCP server? This would save me from implementing the agent side and allow me to hook it into any IDE-based LLM.

Design

This MCP server is primarily a mix of a "tool" (which executes on the server machine to identify installed packages and their vulnerabilities) and a "resource" (which exposes read-only information for a specific CVE ID).

The MCP exposes two tools:

List Vulnerabilities: This tool identifies vulnerabilities in the packages installed on the system, categorizes them using CVE and EPSS scores, and provides a dictionary of critical, high, medium, and low vulnerabilities.
Research Vulnerabilities: Based on the user prompt, the LLM can identify relevant vulnerabilities and pass them to this function to retrieve details such as whether a fix is available, the fixed version, and criticality.

Vibe Coding

"Vibe coding" is the latest trend, with many claiming that software engineering jobs are a thing of the past. Without going into too much detail, I decided to give it a try. While this is not my first "vibe coded" project (I have done this previously at work using corporate tools), it is my first attempt to vibe code a hobby/learning project.

I chose Antigravity because it seemed to be the only editor providing a sufficient amount of free tokens. For every vibe coding project, I spend time thinking about the barebones skeleton: the modules, function return values, and data structures. This allows me to maintain control over the LLM-generated code so it doesn't become overly complicated or incomprehensible.

As a first step, I wrote down my initial design in a requirements document. In that document, I explicitly called for using debsecan as the model for various components. Additionally, I asked the AI to reference my specific code for the EPSS logic. The reasons were:

debsecan already solves the core problem; I am simply rebuilding it. debsecan uses a single file generated by the Debian Security team containing all necessary information, which prevents us from needing multiple external sources.
This provides the flexibility to categorize vulnerabilities within the listing tool itself since all required information is readily available, unlike my original notebook-based design.

I initially used Gemini 3 Flash as the model because I was concerned about exceeding my free limits.

Hiccups

Although it initially seemed successful, I soon noticed discrepancies between the local debsecan outputs and the outputs generated by the tools. I asked the AI to fix this, but after two attempts, it still could not match the outputs. I realized it was writing its own version-comparison logic and failing significantly.

Finally, I instructed it to depend entirely on the python-apt module for version comparison; since it is not on PyPI, I asked it to pull directly from the Git source. This solved some issues, but the problem persisted. By then, my weekly quota was exhausted, and I stopped debugging.

A week later, I resumed debugging with the Claude 3.5 Sonnet model. Within 20-25 minutes, it found the fix, which involved four lines of changes in the parsing logic. However, I ran out of limits again before I could proceed further.

In the requirements, I specified that the list vulnerabilities tool should only provide a dictionary of CVE IDs divided by severity. However, the AI instead provided full text for all vulnerability details, resulting in excessive data—including negligible vulnerabilities—being sent to the LLM. Consequently, it never called the research vulnerabilities tool. Since I had run out of limits, I manually fixed this in a follow-up commit.

How to Use

I have published the current work in the debsecan-mcp repository. I have used the same license as the original debsecan. I am not entirely sure how to interpret licenses for vibe-coded projects, but here we are.

To use this, you need to install the tool in a virtual environment and configure your IDE to use the MCP. Here is how I set it up for Visual Studio Code:

Follow the guide from the VS Code documentation regarding adding an MCP server.
My global mcp.json looks like this:

{
  "servers": {
      "debsecan-mcp": {
          "command": "uv",
              "args": [
                  "--directory",
                  "/home/vasudev/Documents/personal/FOSS/debsecan-mcp/debsecan-mcp",
                  "run",
                  "debsecan-mcp"
              ]
          }
  },
  "inputs": []
}

I am running it directly from my local codebase using a virtualenv created with uv. You may need to tweak the path based on your installation.
To use the MCP server in the Copilot chat window, reference it using #debsecan-mcp. The LLM will then use the server for the query.
Use a prompt like: "Give an executive summary of the system security status and immediate actions to be taken."
You can observe the LLM using list_vulnerabilities followed by research_cves. Because the first tool only provides CVE IDs based on severity, the LLM is smart enough to research only high and critical vulnerabilities, thereby saving tokens.

What's Next?

This MCP is not yet perfect and has the following issues:

The list_vulnerabilities dictionary contains duplicate CVE IDs because the code used a list instead of a set. While the LLM is smart enough to deduplicate these, it still costs extra tokens.
Because I initially modeled this on debsecan, it uses a raw method for parsing /var/lib/dpkg/status instead of python-apt. I am considering switching to python-apt to reduce maintenance overhead.
Interestingly, the AI did not add a single unit test, which is disappointing. I will add these once my limits are restored.
I need to create a cleaner README with usage instructions.
I need to determine if the MCP can be used via HTTP as well as stdio.

Conclusion

Vibe coding is interesting, but things can get out of hand if not managed properly. Even with a good process, code must be reviewed and tested; you cannot blindly trust an AI to handle everything. Even if it adds tests, you must validate them, or you are doomed!

Moving blog from self hosting to Github Pages

2025-11-16T10:32:00+05:30

I haven't been blogging as much as I used to. For a while, I've been hosting my blog on a single-core DigitalOcean droplet, which cost me around $7 per month. It also hosted my mail server. Most of the time, the droplet was idle, and I wasn't actively using my personal email much. Since it was self-hosted, I didn't have confidence that it would always be up, so I relied on Gmail as my personal email for everything—from banking to social media.

Now, I feel this cost is just a waste of money, even though it's not huge. So, I decided to move my blog back to GitHub Pages, now published using GitHub Workflows. I've stopped my mail server for the time being, and it won't be reachable for a while. For any personal queries or comments, please reach me on my Personal Gmail.

I'm not sure how active I'll be with blogging again, but I'll try my best to return to my old habit of writing at least a post every week or two. Not because people will read it, but because it gives me the option to explore new things, experiment, and take notes along the way.

Signing the systemd-boot on Upgrade Using Dpkg Triggers

2024-09-29T13:08:00+05:30

In my previous post on enabling SecureBoot, I mentioned that one pending improvement was signing the systemd-boot EFI binary with my keys on every upgrade. In this post, we'll explore the implementation of this process using dpkg triggers.

For an excellent introduction to dpkg triggers, refer to this archived blog post. The source code mentioned in that post can be downloaded from alioth archive.

From /usr/share/doc/dpkg/spec/triggers.txt, triggers are described as follows:

A dpkg trigger is a facility that allows events caused by one package but of interest to another package to be recorded and aggregated, and processed later by the interested package. This feature simplifies various registration and system-update tasks and reduces duplication of processing.

To implement this, we create a custom package with a single script that signs the systemd-boot EFI binary using our key. The script is as simple as:

#!/bin/bash

set -e

echo "Signing the new systemd-bootx64.efi"
sbsign --key /etc/secureboot/db.key --cert /etc/secureboot/db.crt \
       /usr/lib/systemd/boot/efi/systemd-bootx64.efi

echo "Invoking bootctl install to copy stuff"
bootctl install

Invoking bootctl install is optional if we have enabled systemd-boot-update.service, which will update the signed bootloader on the next boot.

We need to have a triggers file under the debian/ folder of the package, which declares its interest in modifications to the path /usr/lib/systemd/boot/efi/systemd-bootx64.efi. The trigger file looks like this:

# trigger 1 interest on systemd-bootx64.efi
interest-noawait /usr/lib/systemd/boot/efi/systemd-bootx64.efi

You can read about various directives and their meanings that can be used in the triggers file in the deb-triggers man page.

Once we build and install the package, this request is added to /var/lib/dpkg/triggers/File. See the screenshot below after installation of our package:

To test the functionality, I performed a re-installation of the systemd-boot-efi package, which provides the EFI binary for systemd-boot, using the following command:

sudo apt install --reinstall systemd-boot-efi

During installation, you can see the debug message being printed in the screenshot below:

To test the systemd-boot-update.service, I commented out the bootctl install line from the above script, performed a reinstallation, and restarted the systemd-boot-update.service. Checking the log, I saw the following:

Sep 29 13:42:51 chamunda systemd[1]: Stopping systemd-boot-update.service - Automatic Boot Loader Update...
Sep 29 13:42:51 chamunda systemd[1]: Starting systemd-boot-update.service - Automatic Boot Loader Update...
Sep 29 13:42:51 chamunda bootctl[1801516]: Skipping "/efi/EFI/systemd/systemd-bootx64.efi", same boot loader version in place already.
Sep 29 13:42:51 chamunda bootctl[1801516]: Skipping "/efi/EFI/BOOT/BOOTX64.EFI", same boot loader version in place already.
Sep 29 13:42:51 chamunda bootctl[1801516]: Skipping "/efi/EFI/BOOT/BOOTX64.EFI", same boot loader version in place already.
Sep 29 13:42:51 chamunda systemd[1]: Finished systemd-boot-update.service - Automatic Boot Loader Update.
Sep 29 13:43:37 chamunda systemd[1]: systemd-boot-update.service: Deactivated successfully.
Sep 29 13:43:37 chamunda systemd[1]: Stopped systemd-boot-update.service - Automatic Boot Loader Update.
Sep 29 13:43:37 chamunda systemd[1]: Stopping systemd-boot-update.service - Automatic Boot Loader Update...

Indeed, the service attempted to copy the bootloader but did not do so because there was no actual update to the binary; it was just a reinstallation trigger.

The complete code for this package can be found here.

With this post the entire series on using UKI to Secureboot with Debian comes to an end. Happy hacking!.

Disabling Lockdown Mode with Secure Boot on Distro Kernel

2024-09-26T22:13:00+05:30

In my previous post, I mentioned that Lockdown mode is activated when Secure Boot is enabled. One way to override this was to use a self-compiled upstream kernel. However, sometimes we might want to use the distribution kernel itself. This post explains how to disable lockdown mode while keeping Secure Boot enabled with a distribution kernel.

Understanding Secure Boot Detection

To begin, we need to understand how the kernel detects if Secure Boot is enabled. This is done by the efi_get_secureboot function, as shown in the image below:

Disabling Kernel Lockdown

The kernel code uses the value of MokSBStateRT to identify the Secure Boot state, assuming that Secure Boot can only be enabled via shim. This assumption holds true when using the Microsoft certificate for signature validation (as Microsoft currently only signs shim). However, if we're using our own keys, we don't need shim and can sign the bootloader ourselves. In this case, the Secure Boot state of the system doesn't need to be tied to the MokSBStateRT variable.

To disable kernel lockdown, we need to set the UEFI runtime variable MokSBStateRT. This essentially tricks the kernel into thinking Secure Boot is disabled when it's actually enabled. This is achieved using a UEFI initializing driver.

The code for this was written by an anonymous colleague who also assisted me with various configuration guidance for setting up UKI and Secure Boot on my system. The code is available here.

Implementation

Detailed instructions for compiling and deploying the code are provided in the repository, so I won't repeat them here.

Results

I've tested this method with the default distribution kernel on my Debian unstable system, and it successfully disables lockdown while maintaining Secure Boot integrity. See the screenshot below for confirmation:

Note to Self: Enabling Secure Boot with UKI on Debian

2024-09-24T11:30:00+05:30

Note

This post is a continuation of my previous article on enabling the Unified Kernel Image (UKI) on Debian.

Enabling Unified Kernel Image on Debian

In this guide, we'll implement Secure Boot by taking full control of the device, removing preinstalled keys, and installing our own. For a comprehensive overview of the benefits and process, refer to this excellent post from rodsbooks.

Key Components

To implement Secure Boot, we need three essential keys:

Platform Key (PK): The top-level key in Secure Boot, typically provided by the motherboard manufacturer. We'll replace the vendor-supplied PK with our own for complete control.
Key Exchange Key (KEK): Used to sign updates for the Signatures Database and Forbidden Signatures Database.
Database Key (DB): Used to sign or verify binaries (bootloaders, boot managers, shells, drivers, etc.).

There's also a Forbidden Signature Key (dbx), which is the opposite of the DB key. We won't be generating this key in this guide.

Preparing for Key Enrollment

Before enrolling our keys, we need to put the device in Secure Boot Setup Mode. Verify the status using the bootctl status command. You should see output similar to the following image:

Generating Keys

Follow these instructions from the Arch Wiki to generate the keys manually. You'll need the efitools and openssl packages. I recommend using rsa:2048 as the key size for better compatibility with older firmware.

After generating the keys, copy all .auth files to the /efi/loader/keys/<hostname>/ folder. For example:

❯ sudo ls /efi/loader/keys/chamunda
db.auth  KEK.auth  PK.auth

Signing the Bootloader

Sign the systemd-boot bootloader with your new keys:

sbsign --key <path-to db.key> --cert <path-to db.crt> \
   /usr/lib/systemd/boot/efi/systemd-bootx64.efi

Install the signed bootloader using bootctl install. The output should resemble this:

Note

If you encounter warnings about mount options, update your fstab with the `umask=0077` option for the EFI partition.

Verify the signature using sbsign --verify:

Configuring UKI for Secure Boot

Update the /etc/kernel/uki.conf file with your key paths:

SecureBootPrivateKey=/path/to/db.key
SecureBootCertificate=/path/to/db.crt

Signing the UKI Image

On Debian, use dpkg-reconfigure to sign the UKI image for each kernel:

sudo dpkg-reconfigure linux-image-$(uname -r)
# Repeat for other kernel versions if necessary

You should see output similar to this:

sudo dpkg-reconfigure linux-image-$(uname -r)
/etc/kernel/postinst.d/dracut:
dracut: Generating /boot/initrd.img-6.10.9-amd64
Updating kernel version 6.10.9-amd64 in systemd-boot...
Signing unsigned original image
Using config file: /etc/kernel/uki.conf
+ sbverify --list /boot/vmlinuz-6.10.9-amd64
+ sbsign --key /home/vasudeva.sk/Documents/personal/secureboot/db.key --cert /home/vasudeva.sk/Documents/personal/secureboot/db.crt /tmp/ukicc7vcxhy --output /tmp/kernel-install.staging.QLeGLn/uki.efi
Wrote signed /tmp/kernel-install.staging.QLeGLn/uki.efi
/etc/kernel/postinst.d/zz-systemd-boot:
Installing kernel version 6.10.9-amd64 in systemd-boot...
Signing unsigned original image
Using config file: /etc/kernel/uki.conf
+ sbverify --list /boot/vmlinuz-6.10.9-amd64
+ sbsign --key /home/vasudeva.sk/Documents/personal/secureboot/db.key --cert /home/vasudeva.sk/Documents/personal/secureboot/db.crt /tmp/ukit7r1hzep --output /tmp/kernel-install.staging.dWVt5s/uki.efi
Wrote signed /tmp/kernel-install.staging.dWVt5s/uki.efi

Enrolling Keys in Firmware

Use systemd-boot to enroll your keys:

systemctl reboot --boot-loader-menu=0

Select the enroll option with your hostname in the systemd-boot menu.

After key enrollment, the system will reboot into the newly signed kernel. Verify with bootctl:

Dealing with Lockdown Mode

Secure Boot enables lockdown mode on distro-shipped kernels, which restricts certain features like kprobes/BPF and DKMS drivers. To avoid this, consider compiling the upstream kernel directly, which doesn't enable lockdown mode by default.

As Linus Torvalds has stated, "there is no reason to tie Secure Boot to lockdown LSM." You can read more about Torvalds' opinion on UEFI tied with lockdown.

Next Steps

One thing that remains is automating the signing of systemd-boot on upgrade, which is currently a manual process. I'm exploring dpkg triggers for achieving this, and if I succeed, I will write a new post with details.

Acknowledgments

Special thanks to my anonymous colleague who provided invaluable assistance throughout this process.

Note to Self: Enabling Unified Kernel Image on Debian

2024-09-19T11:40:00+05:30

Note

These steps may not work on your system if you are using the default Debian installation. This guide assumes that your system is using systemd-boot as the bootloader, which is explained in the post linked below.

Install Debian from grml-liveboot

A unified kernel image (UKI) is a single executable that can be booted directly from UEFI firmware or automatically sourced by bootloaders with little or no configuration. It combines a UEFI boot stub program like systemd-stub(7), a Linux kernel image, an initrd, and additional resources into a single UEFI PE file.

systemd-boot already provides a hook for kernel installation via /etc/kernel/postinst.d/zz-systemd-boot. We just need a couple of additional configurations to generate the UKI image.

Installation and Configuration

Install the systemd-ukify package:
```
sudo apt-get install systemd-ukify
```
Create the following configuration in /etc/kernel/install.conf:
```
layout=uki
initrd_generator=dracut
uki_generator=ukify
```
This configuration specifies how to generate the UKI image for the installed kernel and which generator to use.
Define the kernel command line for the UKI image. Create /etc/kernel/uki.conf with the following content:
```
[UKI]
Cmdline=@/etc/kernel/cmdline
```

Generating the UKI Image

To apply these changes, regenerate the UKI image for the currently running kernel:

sudo dpkg-reconfigure linux-image-$(uname -r)

Verification

Use the bootctl list command to verify the presence of a "Type #2" entry for the current kernel. The output should look similar to this:

bootctl list
      type: Boot Loader Specification Type #2 (.efi)
     title: Debian GNU/Linux trixie/sid (2d0080583f1a4127ac0b073b1a9d3e61-6.10.9-amd64.efi) (default) (selected)
        id: 2d0080583f1a4127ac0b073b1a9d3e61-6.10.9-amd64.efi
    source: /boot/efi/EFI/Linux/2d0080583f1a4127ac0b073b1a9d3e61-6.10.9-amd64.efi
  sort-key: debian
     linux: /boot/efi/EFI/Linux/2d0080583f1a4127ac0b073b1a9d3e61-6.10.9-amd64.efi
   options: systemd.gpt_auto=no quiet root=LABEL=root_disk ro systemd.machine_id=2d0080583f1a4127ac0b073b1a9d3e61

      type: Boot Loader Specification Type #2 (.efi)
     title: Debian GNU/Linux trixie/sid (2d0080583f1a4127ac0b073b1a9d3e61-6.10.7-amd64.efi)
        id: 2d0080583f1a4127ac0b073b1a9d3e61-6.10.7-amd64.efi
    source: /boot/efi/EFI/Linux/2d0080583f1a4127ac0b073b1a9d3e61-6.10.7-amd64.efi
  sort-key: debian
     linux: /boot/efi/EFI/Linux/2d0080583f1a4127ac0b073b1a9d3e61-6.10.7-amd64.efi
   options: systemd.gpt_auto=no quiet root=LABEL=root_disk ro systemd.machine_id=2d0080583f1a4127ac0b073b1a9d3e61

      type: Automatic
     title: Reboot Into Firmware Interface
        id: auto-reboot-to-firmware-setup
    source: /sys/firmware/efi/efivars/LoaderEntries-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f

Cleanup and Reboot

Once the "Type #2" entries are generated, remove any "Type #1" entries using the bootctl unlink command. After this, reboot your system to boot from the UKI-based image.

Future Considerations

The primary use case for a UKI image is secure boot. Signing the UKI image can also be configured in the settings above, but this guide does not cover that process as it requires setting up secure boot on your system.

Cloning a laptop over NVME TCP

2024-03-10T17:15:00+05:30

Recently, I got a new laptop and had to set it up so I could start using it. But I wasn't really in the mood to go through the same old steps which I had explained in this post earlier. I was complaining about this to my colleague, and there came the suggestion of why not copy the entire disk to the new laptop. Though it sounded like an interesting idea to me, I had my doubts, so here is what I told him in return.

I don't have the tools to open my old laptop and connect the new disk over USB to my new laptop.
I use full disk encryption, and my old laptop has a 512GB disk, whereas the new laptop has a 1TB NVME, and I'm not so familiar with resizing LUKS.

He promptly suggested both could be done. For step 1, just expose the disk using NVME over TCP and connect it over the network and do a full disk copy, and the rest is pretty simple to achieve. In short, he suggested the following:

Export the disk using nvmet-tcp from the old laptop.
Do a disk copy to the new laptop.
Resize the partition to use the full 1TB.
Resize LUKS.
Finally, resize the BTRFS root disk.

Exporting Disk over NVME TCP

The easiest way suggested by my colleague to do this is using systemd-storagetm.service. This service can be invoked by simply booting into storage-target-mode.target by specifying rd.systemd.unit=storage-target-mode.target. But he suggested not to use this as I need to tweak the dracut initrd image to involve network services as well as configuring WiFi from this mode is a painful thing to do.

So alternatively, I simply booted both my laptops with GRML rescue CD. And the following step was done to export the NVME disk on my current laptop using the nvmet-tcp module of Linux:

modprobe nvmet-tcp
cd /sys/kernel/config/nvmet
mkdir ports/0
cd ports/0
echo "ipv4" > addr_adrfam
echo 0.0.0.0 > addr_traaddr
echo 4420 > addr_trsvcid
echo tcp > addr_trtype

cd /sys/kernel/config/nvmet/subsystems
mkdir testnqn
echo 1 >testnqn/allow_any_host
mkdir testnqn/namespaces/1

cd testnqn
# replace the device name with the disk you want to export
echo "/dev/nvme0n1" > namespaces/1/device_path
echo 1 > namespaces/1/enable

ln -s "../../subsystems/testnqn" /sys/kernel/config/nvmet/ports/0/subsystems/testnqn

These steps ensure that the device is now exported using NVME over TCP. The next step is to detect this on the new laptop and connect the device:

nvme discover -t tcp -a <ip> -s 4420
nvme connectl-all -t tcp -a <> -s 4420

Finally, nvme list shows the device which is connected to the new laptop, and we can proceed with the next step, which is to do the disk copy.

Copying the Disk

I simply used the dd command to copy the root disk to my new laptop. Since the new laptop didn't have an Ethernet port, I had to rely only on WiFi, and it took about 7 and a half hours to copy the entire 512GB to the new laptop. The speed at which I was copying was about 18-20MB/s. The other option would have been to create an initial partition and file system and do an rsync of the root disk or use BTRFS itself for file system transfer.

dd if=/dev/nvme2n1 of=/dev/nvme0n1 status=progress bs=40M

Resizing Partition and LUKS Container

The final part was very easy. When I launched parted, it detected that the partition table does not match the disk size and asked if it can fix it, and I said yes. Next, I had to install cloud-guest-utils to get growpart to fix the second partition, and the following command extended the partition to the full 1TB:

growpart /dev/nvem0n1 p2

Next, I used cryptsetup-resize to increase the LUKS container size.

cryptsetup luksOpen /dev/nvme0n1p2 ENC
cryptsetup resize ENC

Finally, I rebooted into the disk, and everything worked fine. After logging into the system, I resized the BTRFS file system. BTRFS requires the system to be mounted for resize, so I could not attempt it in live boot.

btfs fielsystem resize max /

Conclusion

The only benefit of this entire process is that I have a new laptop, but I still feel like I'm using my existing laptop. Typically, setting up a new laptop takes about a week or two to completely get adjusted, but in this case, that entire time is saved.

An added benefit is that I learned how to export disks using NVME over TCP, thanks to my colleague. This new knowledge adds to the value of the experience.

Using LUKS-Encrypted USB Stick with TPM2 Integration

2023-07-09T13:17:00+05:30

I use a LUKS-encrypted USB stick to store my GPG and SSH keys, which acts as a backup and portable key setup when working on different laptops. One inconvenience with LUKS-encrypted USB sticks is that you need to enter the password every time you want to mount the device, either through a Window Manager like KDE or using the cryptsetup luksOpen command. Fortunately, many laptops nowadays come equipped with TPM2 modules, which can be utilized to automatically decrypt the device and subsequently mount it. In this post, we'll explore the usage of systemd-cryptenroll for this purpose, along with udev rules and a set of scripts to automate the mounting of the encrypted USB.

First, ensure that your device has a TPM2 module. You can run the following command to check:

sudo journalctl -k --grep=tpm2

The output should resemble the following:

Jul 08 18:57:32 bhairava kernel: ACPI: SSDT 0x00000000BBEFC000 0003C6 (v02
LENOVO Tpm2Tabl 00001000 INTL 20160422) Jul 08 18:57:32 bhairava kernel:
ACPI: TPM2 0x00000000BBEFB000 000034 (v03 LENOVO TP-R0D 00000830
PTEC 00000002) Jul 08 18:57:32 bhairava kernel: ACPI: Reserving TPM2 table
memory at [mem 0xbbefb000-0xbbefb033]

You can also use the systemd-cryptenroll command to check for the availability of a TPM2 device on your laptop:

systemd-cryptenroll --tpm2-device=list

The output will be something like following:

blog git:(master) systemd-cryptenroll --tpm2-device=list
PATH        DEVICE      DRIVER
/dev/tpmrm0 MSFT0101:00 tpm_tis
➜  blog git:(master)

Next, ensure that you have connected your encrypted USB device. Note that systemd-cryptenroll only works with LUKS2 and not LUKS1. If your device is LUKS1-encrypted, you may encounter an error while enrolling the device, complaining about the LUKS2 superblock not found.

To determine if your device uses a LUKS1 header or LUKS2, use the cryptsetup luksDump <device> command. If it is LUKS1, the header will begin with:

LUKS header information for /dev/sdb1

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha256
Payload offset: 4096

Converting from LUKS1 to LUKS2 is a simple process, but for safety, ensure that you backup the header using the cryptsetup luksHeaderBackup command. Once backed up, use the following command to convert the header to LUKS2:

sudo cryptsetup convert --type luks2 /dev/sdb1

After conversion, the header will look like this:

Version:        2
Epoch:          4
Metadata area:  16384 [bytes]
Keyslots area:  2064384 [bytes]
UUID:           000b2670-be4a-41b4-98eb-9adbd12a7616
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

The next step is to enroll the new LUKS key for the encrypted device using systemd-cryptenroll. Run the following command:

sudo systemd-cryptenroll --tpm2-device=/dev/tpmrm0 --tpm2-pcrs="0+7" /dev/sdb1

This command will prompt you to provide the existing key to unseal the device. It will then add a new random key to the volume, allowing it to be unlocked in addition to the existing keys. Additionally, it will bind this new key to PCRs 0 and 7, representing the system firmware and Secure Boot state.

If there is only one TPM device on the system, you can use --tpm2-device=auto to automatically select the device. To confirm that the new key has been enrolled, you can dump the LUKS configuration and look for a systemd-tpm2 token entry, as well as an additional entry in the Keyslots section.

To test the setup, you can use the /usr/lib/systemd/systemd-cryptsetup command. Additionally, you can check if the device is unsealed by using lsblk:

sudo /usr/lib/systemd/systemd-cryptsetup attach GPG_USB "/dev/sdb1" - tpm2-device=auto

lsblk

The lsblk command should display the unsealed and mounted device, like this:

NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda           8:0    0 223.6G  0 disk
├─sda1        8:1    0   976M  0 part  /boot/efi
└─sda2        8:2    0 222.6G  0 part
  └─root    254:0    0 222.6G  0 crypt /
sdb           8:16   1   7.5G  0 disk
└─sdb1        8:17   1   7.5G  0 part
  └─GPG_USB 254:1    0   7.5G  0 crypt /media/vasudev/GPG_USB

Auto Mounting the device

Now that we have solved the initial problem of unsealing the USB device using TPM2 instead of manually entering the key, the next step is to automatically mount the device upon insertion and remove the mapping when the device is removed. This can be achieved using the following udev rules:

ACTION=="add", KERNEL=="sd*", ENV{DEVTYPE}=="partition", ENV{ID_BUS}=="usb", ENV{SYSTEMD_WANTS}="mount-gpg-usb@$env{DEVNAME}.service"
ACTION=="remove", KERNEL=="sd*", ENV{DEVTYPE}=="partition", ENV{ID_BUS}=="usb", RUN+="/usr/local/bin/umount_enc_usb.sh '%E{ID_FS_UUID}'"

When a device is added, a systemd service is triggered to mount the device at a specific location. Initially, I used a script with the RUN directive, but it resulted in an exit code of 32. This might be due to systemd-cryptsetup taking some time to return, causing udev to time out. To address this, I opted to use a systemd service instead.

For device removal, even though the physical device is no longer present, the mapping may still remain, causing issues upon reinsertion. To resolve this, I created a script to close the luks mapping upon device removal.

Below are the systemd service and script files:

mount_enc_usb.sh:

#!/bin/bash
set -x

if [[ "$#" -ne 1 ]]; then
    echo "$(basename $0) <device>"
    exit 1
fi

device_uuid="$(blkid --output udev $1 | grep ID_FS_UUID= | cut -d= -f2)"
if [[ "$device_uuid" == 000b2670-be4a-41b4-98eb-9adbd12a7616 ]]; then
    # Found our device, let's trigger systemd-cryptsetup
    /usr/lib/systemd/systemd-cryptsetup attach GPG_USB "$1" - tpm2-device=auto
    [[ -d /media/vasudev/GPG_USB ]] || (mkdir -p /media/vasudev/GPG_USB/ && chown vasudev:vasudev /media/vasudev/GPG_USB)
    mount /dev/mapper/GPG_USB /media/vasudev/GPG_USB
else
    echo "Not the interested device. Ignoring."
    exit 0
fi

umount_enc_usb.sh:

#!/bin/bash
if [[ "$#" -ne 1 ]]; then
  echo "$(basename $0) <fsuuid>"
  exit 1
fi

if [[ "$1" == "000b2670-be4a-41b4-98eb-9adbd12a7616" ]]; then
  # Our device is removed, let's close the luks mapping
  [[ -e /dev/mapper/GPG_USB ]] && cryptsetup luksClose /dev/mapper/GPG_USB
else
  echo "Not our device."
  exit 0
fi

mount-gpg-usb@.service:

[Unit]
Description=Mount the encrypted USB device service

[Service]
Type=simple
ExecStart=/usr/local/bin/mount_enc_usb.sh

With this setup, plugging in the USB device will automatically unseal and mount it, and upon removal, the luks mapping will be closed.

Note

This can be even done for LUKS2 encrypted root disk but will need some tweaking in initramfs.

References

Migrating my domain from copyninja.info to copyninja.in

2023-06-25T16:13:00+05:30

After holding the domain copyninja.info for almost 15 years, I finally let it expire and bought a new domain, copyninja.in. With this move, I also bid goodbye to my VPS, which I had been using for over 12 years on DigitalOcean. This particular VPS was initially set up with Debian Wheezy (7) and had been upgraded over the years to successive Debian versions and finally was running Debian Bullseye (11).

The main reason for the move was that the .info domain was becoming more expensive every year, and the VPS, which I had upgraded to the $10 USD range, cost around $12 USD per month with GST included. Since I wasn't really using the VPS anymore and had recently even broken my DNS and mail server settings, I decided it was the right time to reduce this additional cost.

Now I have a cheaper .in domain, and the VPS is on a minimal configuration at DigitalOcean, costing $5 USD per month (which becomes almost $7 USD with GST). Currently, I only run a blog and mail server on this VPS. I will assess if I really need to keep running the mail server for some more time. If not, I will move the blog to a hosting service like GitHub Pages and completely get rid of the VPS.

My email address has now changed, and the new mail can be obtained from this link <http://scr.im/newcopyninj>. I have updated my GPG key and added the new email as the new UID. I still need to revoke the old domain UID. The key has already been updated in the Debian Keyring.

Notes: Experimenting with ZRAM and Memory Over commit

2023-06-20T10:55:00+05:30

Introduction

The ZRAM module in the Linux kernel creates a memory-backed block device that stores its content in a compressed format. It offers users the choice of compression algorithms such as lz4, zstd, or lzo. These algorithms differ in compression ratio and speed, with zstd providing the best compression but being slower, while lz4 offers higher speed but lower compression.

Using ZRAM as Swap

One interesting use case for ZRAM is utilizing it as swap space in the system. There are two utilities available for configuring ZRAM as swap: zram-tools and systemd-zram-generator. However, Debian Bullseye lacks systemd-zram-generator, making zram-tools the only option for Bullseye users. While it's possible to use systemd-zram-generator by self-compiling or via cargo, I preferred using tools available in the distribution repository due to my restricted environment.

Installation

The installation process is straightforward. Simply execute the following command:

apt-get install zram-tools

Configuration

The configuration involves modifying a simple shell script file /etc/default/zramswap sourced by the /usr/bin/zramswap script. Here's an example of the configuration I used:

# Compression algorithm selection
# Speed: lz4 > zstd > lzo
# Compression: zstd > lzo > lz4
# This is not inclusive of all the algorithms available in the latest kernels
# See /sys/block/zram0/comp_algorithm (when the zram module is loaded) to check
# the currently set and available algorithms for your kernel [1]
# [1]  https://github.com/torvalds/linux/blob/master/Documentation/blockdev/zram.txt#L86
ALGO=zstd

# Specifies the amount of RAM that should be used for zram
# based on a percentage of the total available memory
# This takes precedence and overrides SIZE below
PERCENT=30

# Specifies a static amount of RAM that should be used for
# the ZRAM devices, measured in MiB
# SIZE=256000

# Specifies the priority for the swap devices, see swapon(2)
# for more details. A higher number indicates higher priority
# This should probably be higher than hdd/ssd swaps.
# PRIORITY=100

I chose zstd as the compression algorithm for its superior compression capabilities. Additionally, I reserved 30% of memory as the size of the zram device. After modifying the configuration, restart the zramswap.service to activate the swap:

systemctl restart zramswap.service

Using systemd-zram-generator

For Debian Bookworm users, an alternative option is systemd-zram-generator. Although zram-tools is still available in Debian Bookworm, systemd-zram-generator offers a more integrated solution within the systemd ecosystem. Below is an example of the translated configuration for systemd-zram-generator, located at /etc/systemd/zram-generator.conf:

# This config file enables a /dev/zram0 swap device with the following
# properties:
# * size: 50% of available RAM or 4GiB, whichever is less
# * compression-algorithm: kernel default
#
# This device's properties can be modified by adding options under the
# [zram0] section below. For example, to set a fixed size of 2GiB, set
# `zram-size = 2GiB`.

[zram0]
zram-size = ceil(ram * 30/100)
compression-algorithm = zstd
swap-priority = 100
fs-type = swap

After making the necessary changes, reload systemd and start the systemd-zram-setup@zram0.service:

systemctl daemon-reload
systemctl start systemd-zram-setup@zram0.service

The systemd-zram-generator creates the zram device by loading the kernel module and then creates a systemd.swap unit to mount the zram device as swap. In this case, the swap file is called zram0.swap.

Checking Compression and Details

To verify the effectiveness of the swap configuration, you can use the zramctl command, which is part of the util-linux package. Alternatively, the zramswap utility provided by zram-tools can be used to obtain the same output.

During my testing with synthetic memory load created using stress-ng vm class I found that I can reach upto 40% compression ratio.

Memory Overcommit

Another use case I was looking for is allowing the launching of applications that require more memory than what is available in the system. By default, the Linux kernel attempts to estimate the amount of free memory left on the system when user space requests more memory (vm.overcommit_memory=0). However, you can change this behavior by modifying the sysctl value for vm.overcommit_memory to 1.

To demonstrate this, I ran a test using stress-ng to request more memory than the system had available. As expected, the Linux kernel refused to allocate memory, and the stress-ng process could not proceed.

free -tg                                                                                                                                                                                         ──(Mon,Jun19)─┘
                total        used        free      shared  buff/cache   available
 Mem:              31          12          11           3          11          18
 Swap:             10           2           8
 Total:            41          14          19

sudo stress-ng --vm=1 --vm-bytes=50G -t 120                                                                                                                                                      ──(Mon,Jun19)─┘
 stress-ng: info:  [1496310] setting to a 120 second (2 mins, 0.00 secs) run per stressor
 stress-ng: info:  [1496310] dispatching hogs: 1 vm
 stress-ng: info:  [1496312] vm: gave up trying to mmap, no available memory, skipping stressor
 stress-ng: warn:  [1496310] vm: [1496311] aborted early, out of system resources
 stress-ng: info:  [1496310] vm:
 stress-ng: warn:  [1496310]         14 System Management Interrupts
 stress-ng: info:  [1496310] passed: 0
 stress-ng: info:  [1496310] failed: 0
 stress-ng: info:  [1496310] skipped: 1: vm (1)
 stress-ng: info:  [1496310] successful run completed in 10.04s

By setting vm.overcommit_memory=1, Linux will allocate memory in a more relaxed manner, assuming an infinite amount of memory is available.

Conclusion

ZRAM provides disks that allow for very fast I/O, and compression allows for a significant amount of memory savings. ZRAM is not restricted to just swap usage; it can be used as a normal block device with different file systems.

Using ZRAM as swap is beneficial because, unlike disk-based swap, it is faster, and compression ensures that we use a smaller amount of RAM itself as swap space.

Additionally, adjusting the memory overcommit settings can be beneficial for scenarios that require launching memory-intensive applications.

Note: When running stress tests or allocating excessive memory, be cautious about the actual memory capacity of your system to prevent out-of-memory (OOM) situations.

Feel free to explore the capabilities of ZRAM and optimize your system's memory management. Happy computing!

Reference

Installing Debian from GRML Live CD

2022-12-12T12:35:00+05:30

I had bought a Thinkpad E470 laptop back in 2018 which was lying unused for quite some time. Recently when I wanted to use it, I found that the keyboard is not working, especially some keys and after some time the laptop will hang in Lenovo boot screen. I came back to Bangalore almost after 2 years from my hometown (WFH due to Covid) and thought it was the right time to get my laptop back to normal working state. After getting the keyboard replaced I noticed that 1TB HDD is no longer fast enough for my taste!. I've to admit I never thought I would start disliking HDD so quickly thanks to modern SSD based work laptops. So as a second upgrade I got the HDD removed from my laptop and got a 240G SSD. Yeah I know its reduction from my original size but I intend to continue using my old HDD via USB SATA enclosure as an external HDD which can house the extra data which I need to save.

So now that I've a SSD I need to install Debian Unstable again on it and this is where I tried something new. My colleague (name redacted on request) suggested to me use GRML live CD and install Debian via debootstrap. And after giving a thought I decided to try this out. Some reason for going ahead with this are listed below

Debian Installer does not support a proper BTRFS based root file system. It just allows btrfs as root but no subvolume support. Also I'm not sure about the luks support with btrfs as root.
I also wanted to give a try to systemd-boot as my laptop is UEFI capable and I've slowly started disliking Grub.
I really hate installing task-kde-desktop (Yeah you read it right, I've switched to be a KDE user for quite some time) which will pull tons of unwanted stuff and bloat. Well it's not just task-kde-desktop but any other task-desktop package does similar and I don't want to have too much of unused stuff and services running.

Disk Preparation

As a first step I went to GRML website and downloaded current pre-release. Frankly, I'm using GRML for first time and I was not sure what to expect. When I booted it up I was bit taken a back to see its console based and I did not have a wired lan just a plain wireless dongle (Jiofi device) and was wondering what it will take to connect. But surprisingly curses based UI was pretty much straight forward to allow me to connect to Wifi AP. Another thing was the rescue CD had non-free firmware as the laptop was using ath10k device and needed non-free blobs to operate.

Once I got shell prompt in rescue CD first thing I did was to reconfigure console-setup to increase font size which was very very small on default boot. Once that is done I did the following to create a 1G (FAT32) partition for EFI.

parted -a optimal -s /dev/sda mklabel gpt
parted -a optimal -s /dev/sda mkpart primary vfat 0% 1G
parted -a optimal -s /dev/sda set 1 esp on
mkfs.vfat -n boot_disk -F 32 /dev/sda1

So here is what I did: created a 1G vfat type partition and set the esp flag on it. This will be mounted to /boot/efi for systemd-boot. Next I created a single partition on the rest of the available free disk which will be used as the root file system.

Next I encrypted the root parition using LUKS and then created the BTRFS file system on top of it.

cryptsetup luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 ENC
mkfs.btrfs -L root_disk /dev/mapper/ENC

Next is to create subvolumes in BTRFS. I followed suggestion by colleague and created a top-level @ as subvolume below which created @/home @/var/log @/opt . Also enabled compression with zstd and level of 1 to avoid battery drain. Finally marked the @ as default subvolume to avoid adding it to fstab entry.

mount -o compress=zstd:1 /dev/mapper/ENC /mnt
btrfs subvol create /mnt/@
cd /mnt/@
btrfs subvol create ./home
btrfs subvol create ./opt
mkdir -p var
btrfs subvol create ./var/log
btrfs suvol set-default /mnt/@

Bootstrapping Debian

Now that root disk is prepared next step was to bootstrap the root file system. I used debootstrap for this job. One thing I missed here from installer was ability to preseed. I tried looking around to figure out if we can preseed debootstrap but did not find much. If you know the procedure do point it to me.

cd /mnt/
debootstrap --include=dbus,locales,tzdata unstable @/ http://deb.debian.org/debian

Well this just gets a bare minimal installation of Debian I need to install rest of the things post this step manually by chroot into target folder @/.

I like the grml-chroot command for chroot purpose, it does most of the job of mounting all required directory like /dev/ /proc /sys etc. But before entering chroot I need to mount the ESP partition we created to /boot/efi so that I can finalize the installation of kernel and systemd-boot.

umount /mnt
mount -o compress=zstd:1 /dev/mapper/ENC /mnt
mkdir -p /mnt/boot/efi
mount /dev/sda1 /mnt/boot/efi
grml-chroot /mnt /bin/bash

I remounted the root subvolume @ directly to /mnt now, remember I made @ as default subvolume before. I also mounted ESP partition with FAT32 file system to /boot/efi. Finally I used grml-chroot to get into chroot of newly bootstrapped file system.

Now I will install the kernel and minimal KDE desktop installation and configure locales and time zone data for the new system. I wanted to use dracut instead of default initramfs-tools for initrd. I also need to install cryptsetup and btrfs-progs so I can decrypt and really boot into my new system.

apt-get update
apt-get install linux-image-amd64 dracut openssh-client \
                        kde-plasma-desktop plasma-workspace-wayland \
                        plasma-nm cryptsetup btrfs-progs sudo

Next is setting up crypttab and fstab entries for new system. Following entry is added to fstab

LABEL="root_disk" / btrfs defaults,compress=zstd:1 0 0

And the crypttab entry

ENCRYPTED_ROOT UUID=xxxx none discard,x-initrd.attach

I've not written actual UUID above this is just for the purpose of showing the content of /etc/crypttab. Once these entries are added we need to recreate initrd. I just reconfigured the installed kernel package for retriggerring the recreation of initrd using dracut. .. Reconfiguration was locales is done by editing /etc/locales.gen to uncomment en_US.UTF-8 and writing /etc/timezone with Asia/Kolkata. I used DEBIAN_FRONTEND=noninteractive to avoid another prompt asking for locale and timezone information.

export DEBIAN_FRONTEND=noninteractive
dpkg-reconfigure locales
dpkg-reconfigure tzdata

Added my user using adduser command and also set the root password as well. Added my user to sudo group so I can use sudo to elevate privileges.

Setting up systemd-boot

So now basic usable system is ready last part is enabling the systemd-boot configuration as I'm not gonna use grub. I did following to install systemd-boot. Frankly I'm not expert of this it was colleague's suggestion.

Before installing the systemd-boot I had to setup kernel command line. This can be done by writing command line to /etc/kernel/cmdline with following contents.

systemd.gpt_auto=no quiet root=LABEL=root_disk

I'm disabling systemd-gpt-generator to avoid race condition between crypttab entry and auto generated entry by systemd. I faced this mainly because of my stupidity of not adding entry root=LABEL=root_disk

apt-get install -y systemd-boot
bootctl install --make-entry-directory=yes --entry-token=machine-id
dpkg-reconfigure linux-image-6.0.0-5-amd64

Finally exit from the chroot and reboot into the freshly installed system.

systemd-boot already ships a hook file zz-systemd-boot under /etc/kernel so its pretty much usable without any manual intervention. Previously after kernel installation we had to manually update kernel image in efi partitions using bootctl

Conclussion

Though installing from live image is not new and debian-installer also does the same only difference is more control over installation and doing things which is installer is not letting you do (or should I say is not part of default installation?). If properly automated using scripts we can leverage this to do custom installation in large scale environments. I know there is FAI but I've not explored it and felt there is too much to setup for a simple installations with specific requirements.

So finally I've a system with Debian which differs from default Debian installation :-). I should thank my colleague for rekindling nerd inside me who had stopped experimenting quite a long time back.

Note to Self: Growing the Root File System on First Boot

2019-02-16T22:13:00+05:30

These 2 are the use cases I came across for expanding root file system.

RaspberryPi images which comes in smaller image size like 1GB which you write bigger size SD cards like 32GB but you want to use full 32GB of space when system comes up.
You have a VM image which is contains basic server operating system and you want to provision the same as a VM with much larger size root file system.

My current use case was second but I learnt the trick from 1, that is the RaspberryPi3 image spec by Debian project.

Idea behind the expanding root file system is first expanding the root file system to full available size and then run resize2fs on the expanded partition to grow file system. resize2fs is a tool specific for ext2/3/4 file system. But this needs to be done before the file system is mounted.

Here is my modified script from raspi3-image-spec repo. Only difference is I've changed the logic of extracting root partition device to my need, and of course added comments based on my understanding.

#!/bin/sh

# Just extracts root partition and removes partition number to get the device
# name eg. /dev/sda1 becomes /dev/sda
roottmp=$(lsblk -l -o NAME,MOUNTPOINT | grep '/$')
rootpart=/dev/${roottmp%% */}
rootdev=${rootpart%1}

# Use sfdisk to extend partition to all available free space on device.
flock $rootdev sfdisk -f $rootdev -N 2 <<EOF
,+
EOF

sleep 5

# Wait for all pending udev events to be handled
udevadm settle

sleep 5

# detect the changes to partition (we extended it).
flock $rootdev partprobe $rootdev

# remount the root partition in read write mode
mount -o remount,rw $rootpart

# Finally grow the file system on root partition
resize2fs $rootpart

exit 0fs

raspi3-image-spec uses sytemd service file to execute this script just before any file system is mounted. This is done by a making service execute before local-fs.pre target. From the man page for systemd.special

local-fs.target

systemd-fstab-generator(3) automatically adds dependencies of type Before= to all mount units that refer to local mount points for this target unit. In addition, it adds dependencies of type Wants= to this target unit for those mounts listed in /etc/fstab that have the auto mount option set.

Service also disables itself on executing to avoid re-runs on every boot. I've used the service file from raspi3-image-spec as is.

Testing with VM

raspi3-image-spec is well tested, but I wanted to make sure this works with my use case for VM. Since I didn't have any spare physical disks to experiment with I used kpartx with raw file images. Here is what I did

Created a stretch image using vmdb2 with grub installed. Image size is 1.5G
I created another raw disk using fallocate of 4G size.
I created a partition on 4G disk.
Loop mounted the disk and wrote 1.5G image on it using dd
Finally created a VM using virt-install with this loop mounted device as root disk.

Below is my vmdb configuration yml again derived from raspi3-image-spec one with some modifications to suit my needs.

# See https://wiki.debian.org/RaspberryPi3 for known issues and more details.
steps:
- mkimg: "{{ output }}"
  size: 1.5G

- mklabel: msdos
  device: "{{ output }}"

- mkpart: primary
  device: "{{ output }}"
  start: 0%
  end: 100%
  tag: /

- kpartx: "{{ output }}"

- mkfs: ext4
  partition: /
  label: RASPIROOT

- mount: /

- unpack-rootfs: /

- debootstrap: stretch
  mirror: http://localhost:3142/deb.debian.org/debian
  target: /
  variant: minbase
  components:
  - main
  - contrib
  - non-free
  unless: rootfs_unpacked

# TODO(https://bugs.debian.org/877855): remove this workaround once
# debootstrap is fixed
- chroot: /
  shell: |
    echo 'deb http://deb.debian.org/debian buster main contrib non-free' > /etc/apt/sources.list
    apt-get update
  unless: rootfs_unpacked

- apt: install
  packages:
  - ssh
  - parted
  - dosfstools
  - linux-image-amd64
  tag: /
  unless: rootfs_unpacked

- grub: bios
  tag: /

- cache-rootfs: /
  unless: rootfs_unpacked

- shell: |
     echo "experimental" > "${ROOT?}/etc/hostname"

     # '..VyaTFxP8kT6' is crypt.crypt('raspberry', '..')
     sed -i 's,root:[^:]*,root:..VyaTFxP8kT6,' "${ROOT?}/etc/shadow"

     sed -i 's,#PermitRootLogin prohibit-password,PermitRootLogin yes,g' "${ROOT?}/etc/ssh/sshd_config"

     install -m 644 -o root -g root fstab "${ROOT?}/etc/fstab"

     install -m 644 -o root -g root eth0 "${ROOT?}/etc/network/interfaces.d/eth0"

     install -m 755 -o root -g root rpi3-resizerootfs "${ROOT?}/usr/sbin/rpi3-resizerootfs"
     install -m 644 -o root -g root rpi3-resizerootfs.service "${ROOT?}/etc/systemd/system"
     mkdir -p "${ROOT?}/etc/systemd/system/systemd-remount-fs.service.requires/"
     ln -s /etc/systemd/system/rpi3-resizerootfs.service "${ROOT?}/etc/systemd/system/systemd-remount-fs.service.requires/rpi3-resizerootfs.service"

     install -m 644 -o root -g root rpi3-generate-ssh-host-keys.service "${ROOT?}/etc/systemd/system"
     mkdir -p "${ROOT?}/etc/systemd/system/multi-user.target.requires/"
     ln -s /etc/systemd/system/rpi3-generate-ssh-host-keys.service "${ROOT?}/etc/systemd/system/multi-user.target.requires/rpi3-generate-ssh-host-keys.service"
     rm -f ${ROOT?}/etc/ssh/ssh_host_*_key*

  root-fs: /

# Clean up archive cache (likely not useful) and lists (likely outdated) to
# reduce image size by several hundred megabytes.
- chroot: /
  shell: |
     apt-get clean
     rm -rf /var/lib/apt/lists

# TODO(https://github.com/larswirzenius/vmdb2/issues/24): remove once vmdb
# clears /etc/resolv.conf on its own.
- shell: |
     rm "${ROOT?}/etc/resolv.conf"
  root-fs: /

I could not run with vmdb2 installed from Debian archive, so I cloned raspi3-image-spec and used vmdb2 submodule from it. And here are rest of commands used for testing the script.

fallocate -l 4G rootdisk.img
# Create one partition with full disk
sfdisk -f rootdisk.img <<EOF
,+
EOF
kpartx -av rootdisk.img # mounts on /dev/loop0 for me
dd if=vmdb.img of=/dev/loop0
sudo virt-install --name experimental --memory 1024 --disk path=/dev/loop0 --controller type=scsi,model=virtio-scsi --boot hd --network bridge=lxcbr0

Once VM booted I could see the root file system is 4G of size instead of 1.5G it was after using dd to write image on to it. So success!.

SPAKE2 in Golang: Implementing Groups

2018-09-02T17:14:00+05:30

In my previous post I talked about SPAKE2 protocol and Ed25519 Curve group. This post is mostly implementation notes and decisions taken during implementing EC groups and number groups in Golang.

In the Beginning ...

Like always I had too many question on how to do? what to do? etc. etc. Well there is no definitive answer, you have to experiment to get the answer. So as a first step I had to decide what I need. Looking at Python implementation of SPAKE2 some basic group operations needed are as follows.

Element Addition for the group
Scalar Multiplication for the group
Scalar Multiplication with Base point/Generator of group

Along with this some other functions are needed, these are not exactly group related operations, but are needed in SPAKE2 calculation. We can call them helper operations and they are listed as follows.

Convert password into Scalar of the group (Scalar is nothing but big integer in the group)
Generate a Random scalar for the group.
Constants M, N and S.
Converting group element to bytes and creating element from bytes.
Element Size for the group.
Order of the subgroup

Now you might be wondering what S might be, as I only talked about M and N while explaining SPAKE2. This is for a special mode called Symmetric Mode. This special mode created by Brian Warner with help from other cryptographic folks for magic-wormhole use case. This mode removes the side/identity from the SPAKE2 protocol as we saw in previous post (A, B) and makes both side identical. This will reduce additional exchanges that had to be done to setup side/identities for both peers.

Next question was whether SPAKE2 implementation is going to be Ed25519 group specific or whether I plan to introduce other groups as well?. This was important decision I had to make, since Python version also has some integer groups I decided to make groups configurable in Golang as well with Ed25519 as default group.

Defining Group Interface

To make group as configurable in SPAKE2 package I had to define a generic type. Though Golang does not have generics, we can define interface type to get some flexibility. As a first try I defined Group interface as follows

type Group interface {
       ConstM() Group
       ConstN() Group
       ConstS() Group

       RandomScalar() (*big.Int, error)
       PasswordToScalar(pw []byte) *big.Int

       ElementToBytes(ele Group) []byte
       ElementFromBytes(b []byte) Group

       BasePointMult(s *big.Int) Group
       Add(other Group) Group
       ScalarMult(s *big.Int) Group

       ElementSize() int
}

Remember this was not final version which I reached but initial version. I kept this under group.go file. I also thought it would be better to keep group implementations outside SPAKE2. So I created SPAKE2 as salsa.debian.org/vasudev/gospake2 Ed25519 group operations under salsa.debian.org/vasudev/ed25519group and integergroup operations under salsa.debian.org/vasudev/integergroup.

I hit my first road block with this representation. I created a type Ed25519 in ed25519group package and implemented the Group interface above but instead of function accepting/returning Group I used Ed25519. This felt natural to me as Ed25519 was confimring to Group interface but that Go compiler thought otherwise. Go compiler refused to agree Ed25519 type implemented Group interface and told the defintion of functions do not match. For eg. it said for ConstM, expected return value of Group but found Ed25519. I did not yet figure out why this does not work, but as far as I can understand this happens because when Go compiler scans ConstM line it still does not know that type Ed25519 implements Group interface. I could have asked in forums but frankly I did not know how to phrase this question :). If you are a gopher and know the answer please do let me know :).

Without spending too much time to figure out why its not working, I changed the interface definition to look like below.

type Group interface {
       ConstM() interface{}
       ConstN() interface{}
       ConstS() interface{}

       RandomScalar() (*big.Int, error)
       PasswordToScalar(pw []byte) *big.Int

       ElementToBytes(ele interface{}) []byte
       ElementFromBytes(b []byte) interface{}

       BasePointMult(s *big.Int) interface{}
       Add(other interface{}) interface{}
       ScalarMult(s *big.Int) interface{}

       ElementSize() int
}

So now compiler is happy because interface{} means any type. Though I was not happy because I had to do lot of type assertions in the actual implementation of group.

After I did first version of ed25519group and successfuly used it in gospake2 0.1.0, I was feeling something was not correct and things needs to be improved. Then when I started to implement integergroup package things started becoming more clear to me. I finished writing integergroup with same interface definition as above.

After both groups are implemented and integrated into gospake2, I started to look at python code moe carefully. A pattern started emerging in my mind. Python code was structured to differentiate Group and its elements, and this seemed natural separation. Once you separat Elements your interface definition will become more simpler.

So after struggling a bit I wrote a new interface, now differentiating Elements and Group itself. The final code as of writing this post is below.

// Element represents the operation that needs to be satisfied by Group element.
type Element interface {
     Add(other Element) Element
     ScalarMult(s *big.Int) Element
     Negate() Element
}

// Group defines methods that needs to be implemented by the number / elliptic
// curve group which is used to implement SPAKE2 algorithm
type Group interface {
     // These functions are not really group operations but they are needed
     // to get the required group Element's needed for calculation of SPAKE2
     ConstM() Element
     ConstN() Element
     ConstS() Element

     // This operation is needed to get a random integer in the group
     RandomScalar() (*big.Int, error)

     // This operation is for converting user password to a group element
     PasswordToScalar(pw []byte) *big.Int

     // These operations are group operations
     BasePointMult(s *big.Int) Element
     Add(a, b Element) Element
     ScalarMult(a Element, s *big.Int) Element

     ElementToBytes(e Element) []byte
     ElementFromBytes([]byte) (Element, error)

     // This operation should return size of the group
     ElementSize() int

     // This operation returns order of subgroup
     Order() *big.Int
}

Element interface requires implementer to implement Add, ScalarMult and Negate function. Group interface also has Add and ScalarMult operation but Group functions require you to pass Element as input and returns Element as output. Though it may be redundant it gives a natural organization to code.

With new interface new group implementations don't have to do too much type assertions but there will still be some which can't be avoided (eg. Element to actual type).

Packages, Subpackages and....

Well there is no such thing called subpackage in Go, this is one of the learning I had while writing gospake2 and related group implementation. I first created the Group interface in file called group.go which was under salsa.debian.org/vasudev/gospake2 package. So to refer Group interface I just need to import gospake2 and refer it as gospake2.Group. In the beginning this seemed correct approach as I did not directly refer the Group interface in the first versions of ed25519group and integergroup. (The version where I used interface{} extensively). But when I refactored to have 2 interfaces above I got cyclic dependency error. ed25519group and integergroup both referred gospake2.Group and gospake2 referred these groups.

So to fix the error I moved the interface declaration from group.go to groups folder under gospake2 package, and made it package groups. Few points I learned while doing this is

Even if the package is inside your package you can't directly use it. i.e. there is no such thing as subpackage. gospake2 had to refer groups with its full namespace i.e. salsa.debian.org/vasudev/gospake2/groups
Folder structure inside package does not directly relate to each other, its just placing them in meaningful path like crypto/sha256 and crypto/sha512 they do not mean they are related its just that they fall under cryptography.
Standard library can refer to interface in parent package, for example crypto/ecdsa can refer to crypto.SignerOpts interface which is defined in crypto package just by importing "crypto" inside ecdsa package. This works because crypto is package name in GOPATH, but there is nothing special here. For us to refer something in so called parent package we need to use fullpath for example salsa.debian.org/vasudev/gospake2/groups because that is how user packages are namespaced under GOPATH.

So finally I got a proper layout for Group and Element interfaces, its now available under salsa.debian.org/vasudev/gospake2/groups package. If you intend to provide a new group implementation for gospake2 you need to implement these interfaces in your package.

Implementing Ed25519 Group

Implementing Ed25519 group was a bit of adventurous journey. First I searched for ready made implementation if any. Only thing I found was golang.org/x/crypto/ed25519/internal/edwards25519 which is port of DJB's original C code to Go by Adam Langley. Problem was this package was internal to ed25519 package and Go compiler would refuse to allow you import it outside the ed25519 package. So I decided to embed edwards25519 as a internal package with in gospake2. This was prior to second version of Group interface design.

Even with embedding I could not really use it properly. I could get the BasePointMult operation working but nothing else worked and naively I tried to use ScMulAdd for Scalar multiplication which was really a wrong thing to do. Later I understood that the module was specifically written for Ed25519 signature scheme. Though it might still be possible to use it now that I've understood the basics of curve, I will definitely give it a second try at later point in time.

After the failed attempt with edwards25519 Ramakrishnan suggested me to use big integer and implement those methods myself and finally that is what I did. I used math/big package to implement the operations required myself. So the experience of writing this module taught me a lot. Below are few of my learnings.

Annoyance with big.Int

While using big.Int I was annoyed by the specific syntax which invovled invoking the operation using a big.Int variable which will set the result to same variable and additionally return same value also. This design felt redundant to me and also I had to create so many intermediate variables to get operations like Add or Double implemented. Are you thinking why?. Then look at below formula for Add to add 2 points P1 = (X1,Y1,T1,Z1) and P2 = (X2, Y2, T2. Z2)

A = (Y1-X1)*(Y2-X2)
B = (Y1+X1)*(Y2+X2)
C = T1*k*T2
D = Z1*2*Z2
E = B-A
F = D-C
G = D+C
H = B+A
X3 = E*F
Y3 = G*H
T3 = E*H
Z3 = F*G

With big.Int I had to create every intermediate variable and then calculate their result, for eg. (Y1-X1) and (Y2 - X2) and then finally calculate A. At this point I started liking C++ more as it will allow me to override + operator ;-). But at later point I noticed some Go code where people dealt with the big.Int in following format

Y1MX1 := new(big.Int).Sub(Y1, X1)
Y2MX2 := new(big.Int).Sub(Y2, X2)
A := new(big.Int).Mul(Y1MX1, Y2MX2)

It reduced intermediate variables to some extent, additionally it avoids declaring required variables first and then use it. But still its bit of annoyance :).

Confusions with Pointers

When using local variables of type big.Int and returning a pointer to it, I started to have a doubt that if what I'm doing is correct. Being from C background where you are not supposed to be returning pointer to a stack variable, Go's ability to return address of local variable confused me. But it looks like Go compiler is smarter in this aspect. Basically Go compiler does escape analysis to figure out if variable leaves after function and if so it moves it to garbage collected heap. So basically as Go programmer I need not bother on where my variables are allocated. Thats a relief :).

Type Aliasing in Go

Type aliasing in languages like Rust or C++ is a handy way to create alternate name for previously created type. This just creates a new name for existing type and you can still use the original types methods or variables. But this is not the same case in Go. Go language spec. clearly says that new type does not derive anything from originl type. But compiler allows you to cast to-and-fro from original to new type and vice versa.

I was bit by this as I did not knew about it. I created a alias for ExtendedPoint type as Ed25519 to implement Group interface. But when I tried to access original functions from ExtendedPoint I noticed this behavior. So I had to write private conversion function just to cast types around.

Implementing Scalar Multiplication and stack exhaustion

Implementing scalar multiplication was one of the last adventure I tackled in the ed25519 group implementation. Scalar multiplication is multiplying a given elliptic curve point with a large integer (otherwise called as scalar) limited by subgroup order. Warner's python implementation was as follows

def scalarmult_element(pt, n): # extended->extended
  # This form only works properly when given points that are a member of
  # the main 1*L subgroup. It will give incorrect answers when called with
  # the points of order 1/2/4/8, including point Zero. (it will also work
  # properly when given points of order 2*L/4*L/8*L)
  assert n >= 0
  if n==0:
      return xform_affine_to_extended((0,1))
  _ = double_element(scalarmult_element(pt, n>>1))
  return _add_elements_nonunfied(_, pt) if n&1 else _

Though I don't exactly remember first version of my scalar multiplication function, it was mimicking the python code in Go with big.Int. The code worked well with small integers but when I gave big numbers generated using RandomScalar function of Group interface, code will panic as it will run out of stack.

Above python code is slightly optimized version, so I looked at Haskell implementation of SPAKE2 which looked like below

-- | Scalar multiplication parametrised by addition.
scalarMultiplyExtendedPoint :: (ExtendedPoint a -> ExtendedPoint a -> ExtendedPoint a) -> Integer -> ExtendedPoint a -> ExtendedPoint a
scalarMultiplyExtendedPoint _ 0 _    = extendedZero
scalarMultiplyExtendedPoint add n x
       | even n    = doubleExtendedPoint (scalarMultiplyExtendedPoint add (n `div` 2) x)
       | n == 1    = x
       | n <= 0    = panic $ "Unexpected negative multiplier: " <> show n
       | otherwise = add x (scalarMultiplyExtendedPoint add (n - 1) x)

So algorithm is like this

If scalar is 1 return same point
If scalar is 0 return identity element for group
If scalar is even then recursively call scalarmult by reducing the scalar to half and double the result.
Otherwise recursively scalarmultiply the point with scalar reduced by 1 and add the result to the point.

It might look slightly confusing explanation so I will just show the code below.

func (e *ExtendedPoint) ScalarMultSlow(s *big.Int) ExtendedPoint {
     if s.Cmp(big.NewInt(0)) == 0 {
             return Zero
     }

     if s.Cmp(big.NewInt(1)) == 0 {
             return *e
     }

     var result ExtendedPoint
     if IsEven(s) {
             // If scalar is even we recursively call scalarmult with n/2 and
             // then double the result.
             result = e.ScalarMultSlow(new(big.Int).Rsh(s, 1))
             result = result.Double()
     } else {
             // We decrement the scalar and recursively call scalarmult with
             // it then we add the result with point
             result = e.ScalarMultSlow(new(big.Int).Sub(s, big.NewInt(1)))
             result = AddUnified(&result, e)
     }

     return result
}

So instead of dividing by 2 I just right shift the scalar by 1 which is faster operation. (AddUnified is one of the algorithm for point addition which is more safer but slower alternative, hence the name ScalarMultSlow.)

So this implementation works with every input, except the negative one for which I modified ScalarMult definition in Group level to reduce input scalar to subgroup order L. Otherwise Group function just calls function from Element. Code below.

// ScalarMult multiples given point with scalar and returns the result
func (e Ed25519) ScalarMult(a group.Element, s *big.Int) group.Element {
     // First let's reduce s to curve order, this is important in case if we
     // pass negated value
     s.Mod(s, L)
     if s.Cmp(big.NewInt(0)) == 0 {
             return Zero
     }

     extendedPoint := a.(ExtendedPoint)
     result := extendedPoint.ScalarMult(s)
     return result
}

Probably I shoud move reducing the scalar to subgroup order into scalarmult inside Element's implementation.

Implementing Integer Group

Given the problems I faced and things I learnt, implementing Ed25519 group, implementing integer group was much straight forward. Only some design decisions had to be made.

How to Represent Group and Elements

Unlike Ed25519 where group elements are basically points on the curve, element in multiplicative integer groups are basically integers. So how do I represent various integer groups?. Various integer groups are differentiated by bit length of elements in it, group and subgroup order. Looking at python code I created a structure called GroupParameters which will contain necessary information for a given group.

type GroupParameters struct {
       p, q, g *big.Int
       elementSizeBytes, elementSizeBits, scalarSize int
}

I did not want to export these fields as they are not useful outside the package. Python code implemented 3 integer group of 1024,2048 and 3072 bit integers. All values for above variables were taken from NIST document.

In first iteration I only had a struct called IntegerGroup which had parameters as member and implemented Group interface from gospake2. But when I refactored Group interface to have Element interface refactoring the code for integergroup became bit challenging. I introduced IntegerElement struct to hold actual integer value, but since all operations needed access to order of group I had to modify it to also contain parameters defined above. So final definition of IntegerGroup and IntegerElement is as follows

type IntegerGroup struct {
       params *GroupParameters
}

type IntegerElement {
       params *GroupParameters
       e *big.Int
}

Operations in IntegerGroup were simply calling functions from IntegerElement. So its really redundant but to make sure I can distinguish between both group and its element I had to use it in this form.

Scalar Multiplication and Addition Operations

Since the integer groups are really multiplicative group, addition operation is really a multiplication modulo p. Scalar multiplication is just exponentiation modulo p. Since these operations are readily available in math/big I did not had to do anything much for integer group. These operations in IntegerElement are defined as follows.

// Add is actually multiplication mod `p` where `p` is order of the
// multiplicative group
func (i IntegerElement) Add(other group.Element) group.Element {
     a := i.e
     b := other.(IntegerElement).e

     if !i.params.equal(other.(IntegerElement).params) {
             panic("You can't add elements of 2 different groups")
     }

     result := new(big.Int).Mul(a, b)

     return group.Element(IntegerElement{params: i.params, e: result.Mod(result, i.params.p)})
}

// ScalarMult for multiplicative group is g^s mod p where `g` is group generator
// and p is order of the group
func (i IntegerElement) ScalarMult(s *big.Int) group.Element {
     reducedS := new(big.Int).Mod(s, i.params.q)
     return group.Element(IntegerElement{params: i.params, e: new(big.Int).Exp(i.e, reducedS, i.params.p)})
}

Conclussion

Well its been already a pretty long post, so without extending it more I would like to say that I had lot of learning experience in writing the Go code to implement these integer and ed25519 group. Main learnings were

There is no definite answer for any questions, may it be how to write a library or if I structured my library correctly. Of course there will be some best practice available but you have to start at some point and then improve it in iteration.
Go provides great tooling especially linters and formatters which makes you write a clean code. And also document all your exported functions as you write (else you will keep seeing warnings in your editor which is annoying).
Use your library yourself and you will see how you can improve it. If you are feeling uncomfortable with your own written API then that means others will too :).
Every language is designed for a specific purpose, if I'm feeling discomfort using some features of the language (I had problems with verbose error handling) then probably I'm less experienced with language and should see how others handle such things. There are many good projects which you can refer to and lern from.

In the next post which should be last in series I will write about design decisions I made writing gospake2 package. Code for both ed25519 and integer groups are now merged into gospake2 as that is the right place for them. You can find the code for them in my gospake2 repo.

SPAKE2 in Golang: ECDH, SPAKE2 and Curve Ed25519

2018-08-28T20:25:00+05:30

In my previous post I talked about finite field and how it helps in Elliptic Curve Cryptography. In this post we will see briefly how Diffie-Hellmann key exchange varies with use of Elliptic curve groups, then we will see SPAKE2 original variant followed by Elliptic curve version and finally we will have a look at curve Ed25519 which is used as default group in python-spake2 module.

Elliptic Curve Diffie-Hellman (ECDH)

In the previous post we defined the domain parameter of elliptic curve cryptography as $(p, a, b, G, n, h)$. Now we will see how we use this in Diffie-Hellman Key exchange.

Diffie-Hellman key exchange is a way to securely exchange cryptographic key over public channel. Original Diffie-Hellman protocol used multiplicative group of integers modulo p where p is the a large prime and g a generator for subgroup (as we saw in earlier post). The protocol can be explained as follows

Alice and Bob agrees on p and g belonging to group G
Alice selects a belonging to G and calculates $A = g^a \bmod{p}$
Bob selects b belonging to G and calculates $B = g^b \bmod{p}$
Alice sends Bob A
Bob sends Alice B
Now Alice calculates $s = B^a \bmod{p}$
Now Bob calculates $s = A^b \bmod{p}$

Mathematically s computed by both Alice and Bob are same and hence both share a shared secret now.

\begin{equation*} s = B^a \bmod{p} = g^{ba} \bmod{p} = A^b \bmod{p} = g^{ab} \bmod{p} \end{equation*}

Since group is Abelian $g^{ba} \bmod{p} = g^{ab} \bmod{p}$ and hence both side will come to same shared key.

Now in ECC,

private key $d$ is a random integer choosen from $\{1, \dots, n - 1\}$ where n is the order of subgroup
public key is the point $H = dG$ where G is the base point of subgroup.

With above now we can write Diffie-Hellman key exchange as

Alice selects private key $d_A$ and public key $H_A = d_AG$ and sends it to Bob
Bob selects private key $d_B$ and public key $H_B = d_BG$ and sends it to Alice
Alice calculates $S = d_AH_B$ and Bob calculates $S = d_BH_A$ which if you see carefully is one and same and Alice and Bob now share a secret key!.

\begin{equation*} S = d_A H_B = d_A (d_B G) = d_B (d_A G) = d_B H_A \end{equation*}

In both cases observer only sees the public key and will not be able to find discrete logarithm (hard problem), given the numbers are large prime.

Advantage of ECDH is its faster as its replacing the costly exponentiation operation with scalar multiplication without reducing hardness of the problem.

SPAKE2 Protocol

Now that we understood Diffie-Hellman exchange and saw how to apply Elliptic curve in Diffie-Hellman exchange, lets see what is SPAKE2 protocol. This paper by Abdalla and Pointcheval gives full explanation of SPAKE2 and proof of its security. I highly recommend reading the paper as I can only summarize my understadning here.

SPAKE2 is a variation of Diffie-Hellman problem we described above. Domain parameters for SPAKE2 are $(G, g, p, M, N, H)$

G is the group
g is the generator for group
p is the big prime which is order of group
$M, N \in G$ and are selected by Alice and Bob respectively
H is the hash function used to derive final shared key.

Along with this SPAKE2 both side will have common password $pw \in Z_p$. Protocol is defined as follows

Alice selects a random scalar $x \xleftarrow{R} Z_p$ and calculates $X \leftarrow g^x$.
Alice then computes $X^* \leftarrow X \cdot M^{pw}$
Bob selects a random scalar $y \xleftarrow{R} Z_p$ and calculates $Y \leftarrow g^y$.
Bob then computes $Y^* \leftarrow Y \cdot N^{pw}$.
$X^*, Y^*$ are called pake messages and are sent to other side. i.e. Alice sends $X^*$ to Bob and Bob sends $Y^*$ to Alice.
Alice computes $K_A \leftarrow (Y*/N^{pw})^x$ and Bob computes $K_B \leftarrow (X^*/M^{pw})^y$
Shared Key is calculated by Alice as $SK_A \leftarrow H(A,B,X^*,Y^*,pw,K_A)$ and Bob computes $SK_B \leftarrow H(A,B,X^*,Y^*, pw, K_B)$

$SK_A = SK_B$ because mathematically value $K_A = K_B$ (you can expand $X^*$ and $Y^*$ in step 6 aboveand see that they are really same).

In step 7 we calculate Hash of transcript, where A and B are identities of Alice and Bob. and rest is calculated during protocol execution.

One thing to note here is paper does not define what are identities or which hash function is used. This allows some creativity from implementer side to choose things. For python-spake2 interoperability Brian has written a detailed blog post describing decision he has taken for all these points which are not defined in original paper.

Curve Ed25519 Group

Now that we have seen the SPAKE2 protocol, we will next see the use of Elliptic Curve groups in it and see how it varies.

SPAKE2 uses Abelian Group with large number of "elements". We know that Elliptic curve groups are Abelian groups, so we can fit them in SPAKE2. Brian Warner has choosen elliptic curve group Ed25519 (some times also referred as X25519) as default group in python-spake2 implementation. This is the same group which is used in Ed25519 signature scheme. The difference between multiplicative integer group modulo p and elliptic curve group is that, element in integer group is just a number but in elliptic curve group its a point. (represented by 2 co-ordinates).

Curve Ed25519 which is actually called Edwards25519 is a twisted Edwards curve, defined in affine form as $ax^2 + y^2 = 1 + dx^2y^2$ where $d \in k\{0,1\}$.

$q = 2^{255} - 19$ is the order of curve groups
$l = 2^{252} + 27742317777372353535851937790883648493$ is the order of curve subgroup.
$a = -1$
$d = \frac{-121665}{121666}$
Base point $B$ is unique and has y-co-ordinate :math:4/5 and x co-ordinate is positive.

Curve itself is given as $E/\mathbb F_q$

\begin{equation*} -x^2 + y^2 = 1 - \frac{121665}{121666} x^2y^2 \end{equation*}

This curve is birationally equivalaent to the Montgomery curve known as Curve25519. If you are wondering what 25519 is?, well its in the order of group i.e. $2^{255} - 19$.

Till now we were working with elliptic curves with affine co-ordinates, i.e. each point is represented as $(x,y)$. But for fast operation twisted edwards curve introduces new type of co-ordinates called Extended Co-ordinates where x, y is represented as X,Y,T and Z, and affine co-ordinates are represented using the extended co-ordinates as follows

\begin{align*} x = X/Z \\ y = Y/Z \\ x*y = T/Z \\ \end{align*}

Initial base point is converted to extended co-ordinate using Z as 1. In all above case the operations are $mod q$. Additionally all division operations are actually multiplication with inverse of element.

We also noted above, Base point represented using only y co-ordinate. This is because x co-ordinate can be recovered from y, using twisted edwards curve equation we defined above. In most of libraries you will see that this compressed notation of representing a point as just y co-ordinate is used. (Its called CompressedEdwardsY in Rust's curve25519-dalek crate.)

In all above case the operations are $mod q$. Additionally all division operations are actually multiplication with inverse of element.Inverse of a element is calculated as point raised to power q-2 modulo q. I could not find the technical/mathematical reason behind this. If some one knows please let me know. So the inverse operation can be mathematically defined as follows.

\begin{equation*} x^{-1} = x^{q - 2} \bmod{q} \end{equation*}

The addition and doubling operations are as per algorithms defined in hyperelliptic.org post. We have seen scalar multiplication in second post of this series, which depends on addition and doubling operation.

SPAKE2 using Ed25519 group

Unlike normal elliptic curve here the domain parameters are slightly different. Ed25519 domain parameters are defined as $(q, d, B, l)$ where q gives the order of elliptic curve group and l is the order of subgroup. B is the base point of the group.

Now lets rewrite original SPAKE2 protocol using elliptic curve groups

Alice selects a random scalar $x \xleftarrow{R} E/\mathbb F_q$ and calculates $X \leftarrow B \cdot x$ and computes $X^* \leftarrow X + M \cdot pw$. Alice sends $X^*$ to Bob.
Bob selects random scalar $y \xleftarrow{R} E/\mathbb F_q$ and calculates $Y \leftarrow B \cdot y$ and computes $Y^* \leftarrow Y + N \cdot pw$. Bob sends $Y^*$ to Alice.
Alice now calculates $K_A \leftarrow (Y^* - N \cdot pw) \cdot x$
Bob now calculates $K_B \leftarrow (X^* - M \cdot pw) \cdot y$
Shared key is calculated by Alice $SK_A \leftarrow H(A, B, X^*, Y^*, pw, K_A)$ and by Bob $SK_B \leftarrow H(A,B, X^*, Y^*, pw, K_B)$

In 3 and 4 if you expand $X^*$ and $Y^*$ you will see that $K_A = K_B$. And given password used by both sides are same we will arrive at same shared key.

As you see above protocol for SPAKE2 remains same only things what changed from earlier is operations, exponentiation is changed to multiplication and division to substraction. Since we do not explicitly define substraction what we do is negate the password and do addition instead.

Conclusion

So we now have seen all the basics needed to start writing the actual Go code to implement SPAKE2 library. It was bit long I know but if you know the basics writing code is a cake walk!. (quoting from Ramakrishnan). So in the next post I will start writing implementation notes.

SPAKE2 In Golang: Finite fields of Elliptic Curve

2018-08-12T22:51:00+05:30

In my previous post I talked about elliptic curve basics and how the operations are done on elliptic curves, including the algebraic representation which is needed for computers. For usage in cryptography we need a elliptic curve group with some specified number of elements, that is what we called Finite Fields. We limit Elliptic Curve groups with some big prime number p. In this post I will try to briefly explain finite fields over elliptic curve.

Finite Fields

Finite field or also called Galois Field is a set with finite number of elements. An example we can give is integer modulo `p` where p is prime. Finite fields can be denoted as $\mathbb Z/p, GF(p)$ or $\mathbb F_p$.

Finite fields will have 2 operations addition and multiplications. These operations are closed, associative and commutative. There exists a unique identity element and inverse element for every element in the set.

Division operation in finite fields is defined as $x / y = x \cdot y^{-1}$, that is x multiplied by inverse of y. and substraction $x - y$ is defined in terms of addition as $x + (-y)$ which is x added by negation of y. Multiplicative inverse can be easily calculated using extended Euclidean algorithm which I've not understood yet myself as there were readily available library functions which does this for us. But I hear from Ramakrishnan that its very easy one.

Elliptic Curve in $\mathbb F_p$

Now we understood what is finite fields we now need to restrict our elliptic curves to the finite field. So our original definition of elliptic curve becomes slightly different, that is we will have modulo p to restrict the elements.

\begin{equation*} \begin{array}{rcl} \left\{(x, y) \in (\mathbb{F}_p)^2 \right. & \left. | \right. & \left. y^2 \equiv x^3 + ax + b \pmod{p}, \right. \\ & & \left. 4a^3 + 27b^2 \not\equiv 0 \pmod{p}\right\}\ \cup\ \left\{0\right\} \end{array} \end{equation*}

All our previous operations can now be written as follows

\begin{equation*} \begin{array}{rcl} x_R & = & (m^2 - x_P - x_Q) \bmod{p} \\ y_R & = & [y_P + m(x_R - x_P)] \bmod{p} \\ & = & [y_Q + m(x_R - x_Q)] \bmod{p} \end{array} \end{equation*}

Where slope, when $P \neq Q$

\begin{equation*} m = (y_P - y_Q)(x_P - x_Q)^{-1} \bmod{p} \end{equation*}

and when $P = Q$

\begin{equation*} m = (3 x_P^2 + a)(2 y_P)^{-1} \bmod{p} \end{equation*}

So now we need to know order of this finite field. Order of elliptic curve finite field can be defined as number of points in the finite field. Unlike integer modulo p where number of elements are 0 to p-1, in case of elliptic curve you need to count points from x to p-1. This counting will be $O(p)$. Given large p this will be hard problem. But there are faster algorithm to count order of group, which even I don't know much in detail :). But from my reference its called Schoof's algorithm.

Scalar Multiplication and Cyclic Group

When we consider scalar multiplication over elliptic curve finite fields, we discover a special property. Taking example from Andrea Corbellini's post, consider curve $y^2 \equiv x^3 + 2x + 3 ( mod 97)$ and point $P = (3,6)$. If we try calculating multiples of P

\begin{align*} 0P = 0 \\ 1P = (3,6) \\ 2P = (80,10) \\ 3P = (80,87) \\ 4P = (3, 91) \\ 5P = 0 \\ 6P = (3,6) \\ 7P = (80, 10) \\ 8P = (80, 87) \\ 9P = (3, 91) \\ ... \end{align*}

If you are wondering how to calculate above (I did at first). You need to use point addition formula from earlier post where P = Q with mod 97. So we observe that there are only 5 multiples of P and they are repeating cyclicly. we can write above points as

$5kP = 0P$
$(5k + 1)P = 1P$
$(5k + 2)P = 2P$
$(5k + 3)P = 3P$
$(5k + 4)P = 4P$

Or simply we can write these as $kP = (k mod 5)P$. We also note that all these 5 Points are closed under addition. This means adding two multiples of P, we obtain a multiple of P and the set of multiples of P form cyclic subgroup

\begin{equation*} nP + mP = \underbrace{P + \cdots + P}_{n\ \text{times}} + \underbrace{P + \cdots + P}_{m\ \text{times}} = (n + m)P \end{equation*}

Cyclic subgroups are foundation of Elliptic Curve Cryptography (ECC).

Subgroup Order

Subgroup order tells how many points are really there in the subgroup. We can redefine the order of group in subgroup context as order of P is the smallest positive integer such that nP = 0. In above case if you see we have smallest n as 5 since 5P = 0. So order of subgroup above is 5, it contains 5 element.

Order of subgroup is linked to order of elliptic curve by Lagrange's Theorem which says the order of subgroup is divisor of order of parent group. Lagrange is another name which I had read in my college, but the algorithms were different.

From this we have following steps to find out the order of subgroup with base point P

Calculate the elliptic curve's order N using Schoof's algorithm.
Find out all divisors of N.
For every divisor of n, compute nP.
The smallest n such that nP = 0 is the order of subgroup N.

Note that its important to choose smallest divisor, not a random one. In above examples 5P, 10P, 15P all satisfy condition but order of subgroup is 5.

Finding Base Point

Far all above which is used in ECC, i.e. Group, subgroup and order we need a base point P to work with. So base point calculation is not done at the beginning but in the end i.e. first choose a order which looks good then look for subgroup order and finally find the suitable base point.

We learnt above that subgroup order is divisor of group order which is derived from Lagrange's Theorem. This term $h = N/n$ is actually called co-factor of the subgroup. Now why is this term co-factor important?. Without going into details, this co-factor is used to find generator for the subgroup as $G = hP$.

Conclusion

So now are you wondering why I went on such length to describe all these?. Well one thing I wanted to make some notes for myself because you can't find all these information in single place, another these topics we talked in my previous post and this point forms the domain parameters of Elliptic Curve Cryptography.

Domain parameters in ECC are the parameters which are known publicly to every one. Following are 6 parameters

Prime p which is order of Finite field
Co-efficients of curve a and b
Base point $\mathbb G$ the generator which is the base point of curve that generates subgroup
Order of subgroup n
Co-factor h

So in short following is the domain parameters of ECC $(p, a, b, G, n, h)$

In my next post I will try to talk about the specific curve group which is used in SPAKE2 implementation called twisted Edwards curve and give a brief overview of SPAKE2 protocol.

SPAKE2 In Golang: Elliptic Curves Primer

2018-07-28T13:22:00+05:30

TLDR; this post is going to be too long and mostly theoretical. If you are wondering about where is Go in all these posts, please wait for few more posts before I can write about the actual implementation.

In my previous post I talked about starting of my new adventure to write a cryptographic library and how I tried to brute force to solve problem without knowing basics. In this post I'll talk about my learning of Elliptic Curves and their Groups. This post is just my notes and I'm not trying to explain all basics or mathematics behind the Elliptic curves.

Ramakrishnan gave me a good article to start with Elliptic curves. Its a series of posts by Andrea Corbellini, which starts with Elliptic Curve Cryptography: A Gentle Introduction. These posts gave me a great deal of understanding about elliptic curves and how elliptic curve cryptography works, and why its faster. If you want to learn about elliptic curve I highly recommend you to go through the series.

Elliptic Curves

Elliptic curves are set of points described by equation

\begin{equation*} y^2 = x^3 + ax + b \end{equation*}

where a and b are the coefficients which needs to satisfy $4a^3 + 27b^2 \neq 0$. This form of equation is called Weirstrass normal form of elliptic curves.

Along with normal point there is also point on curve which is considered ideal point and is point at infinity; it is denoted by symbol 0. So considering point at infinity we can define elliptic curve as

\begin{equation*} \{(x,y) \in \mathbb R ^2 | y^2 = x^3 + ax + b, 4a^3 + 27b^2 \neq 0 \} \cup \{0\} \end{equation*}

Groups

Before going to Elliptic curve groups we need to know what are groups. I now remembered my academics where I had studied about groups and number theory in general. That time never knew where exactly it was useful. And I should really say learning something without knowing its application is really difficult. Moving on to the groups,

A group in mathematics is basically a set for which there is a binary operation which is called as "addition" and idicated with + symbol. In order for set $\mathbb G$ the binary operation must be defined so that it has following properties.

closure: if a and b are members of of $\mathbb G$ then $a + b$ is also member of $\mathbb G$.
associativity: $(a + b) + c = a + (b + c);$
there exists an identity element 0 such that $a + 0 = 0 + a = a;$
Every element has a inverse, that is for every a there exists b such that $a + b = 0$

For a group to be Abelian there is a 5th rule.

commutativity: $a + b = b + a$

If a group satisfies we get some additional property such that unique identity element and unique inverse element which is either directly or indirectly very important.

Elliptic Curve Groups

Similar to number groups we defined above, its possible to have group over elliptic curves. Following are the property or rule for a Elliptic curve groups.

Elements of the groups are point on elliptic curve;
the identity element is the point at infinity; yes the one we described above while defining elliptic curves.
the inverse of a point $\mathrm R$ is the one symmetric on x-axis. i.e. if point is $(x,y)$ then inverse of the point is $(x, -y)$
adition is given by the rules: given three aligned, non-zero points, $\mathrm P,Q$ and $\mathrm R$ their sum is $P + Q + R = 0$. This means the line through the 3 point passes through the point at infinity.
For addition we want 3 points to be aligned and does not require any specific order which means $P + (Q + R) = Q + (P + R) = R + (P + Q) = ... = 0$ that is our + operator is both associative and commutative

By point 5 we conclude that Elliptic curve groups are Abelian.

Algebraic Addition

Since we know $P + Q + R = 0$ we can say $P + Q = -R$. This means we draw a line on elliptic curve which passes through point P and Q it intersects the curve at 3rd point R. Inverse of point R is -R and is the result of addition of 2 points P and Q. This is easy geometrically but for computers, we need to represent this with algebraic notation. This is where the slope of line comes to picture, another academic topic which I learnt without actually knowing its real world application.

So considering point P as $(x_P,y_P)$ and Q as $(x_Q, y_Q)$ we can calculate slope of line going from P to Q as follows

\begin{equation*} m = \frac{y_P - y_Q}{x_P - x_Q} \end{equation*}

And in a special case where $P = Q$ formula is slightly different

\begin{equation*} m = \frac{3x_P^2 + a}{2y_P} \end{equation*}

Intersection of this line with curve is point R $(x_R,y_R)$ and point $x_R$ and $y_R$ are calculated using following formula.

\begin{equation*} x_R = m^2 - x_P - x_Q \end{equation*}

\begin{equation*} y_R = \begin{cases} y_P + m(x_R - x_P), & \text{or} \\ y_Q + m(x_R - x_Q) \end{cases} \end{equation*}

I've not mentioned about special cases of elliptic curve additions which is explained in Andrea Corbellini blog. Please refer for more mathematical oriented explanation.

Scalar Multiplication

One more important operation we do with elliptic curve group is scalar multiplication. It allows you to calculate $nP$ where n is a natural number. Scalar multiplication can be implemented using addition as follows.

\begin{equation*} nP = \underbrace{P + P + \cdots + P}_{n\ \text{times}} \end{equation*}

This property is what makes elliptic curves attractive and faster than the normal integer groups where scalar multiplication is actually exponentiation operation.

If we see scalar multiplication formula above, we see that it requires n addition, which can be $O(n)$ operation or, if we consider n as k bit integer then algorithm will be $O(2^k)`which looks highly inefficient when `n$ is large prime number.

But there is another algorithm which is much faster for this purpose, its called double and add you can find wikipedia article on it. First I did not understand this, but after a while of struggling, and trying to implement it myself it became pretty clear. It works with first taking binary representation of n $n = n_0 + 2n_1 + 2^2_n2 + ...+2^kn_k$ where $[n_0..n_k] \in \{0,1\}$. Below is pseudo code for the recursive version of algorithm.

def scalarmult(P, n):
       if n == 1:
           return P # identity element
       if n & 1:
           # n is odd we add point
           return point_add(P, scalarmult(P, n - 1)
       else:
           return scalarmult(point_double(P), n >> 1) # we double point and multiply by n/2

Let's consider an example to understand what is this algorithm doing. Lets say we want to caculate 21P. First thing is we denote 21 as binary string 10101 now skipping all those 0's and consider only the 1's, so we will have $2^4 + 2^2 + 2^0$. Now if we multiply P to this representation. So this algorithm calculates 21P as $16P + 4P + P$.

If we consider doubling and adding as constant operations i.e. $O(1)$ then this algorithm will be $O(\log n)$ (in above 21 can be calculated in 3 operations which is $\log 21$) which is much better than earlier $O(n)$ algorithm.

Given n and P we can easily cacluate $Q = nP$ but given Q and P there is no easy way to find n especially when numbers are big. This is the "hard" problem or the discrete logarithm problem which makes cryptography work.

Conclusion

So I learned now about Elliptic curves, its group and operations like scalar multiplication and addition. But this is not directly useful, we need to learn about Finite Fields and Elliptic Curve over Finite Field and many more concepts before we can go to actual implementation!. I thought of writing finite field as part of this post, but then I've already written a lot and there is no point in expanding this post more. So finite fields, cyclic groups will be part of next post.

As a closing note, if you are wondering why there is no mention of anything about Go, you need to wait a bit more for my actual implementation notes in Golang. I would like to write down entire learning process rather than directly jumping to implementation. So please bare with me :-).

SPAKE2 In Golang: Journey to Cryptoland begins

2018-07-22T22:07:00+05:30

This or the series of SPAKE2 related posts are inspired from Jonathan Lange's series on SPAKE2 when he ported it to Haskell, which is also the reference I used for my implementation in Golang.

Brief Background

Before I can go to detail I should tell why/how I came to implementing SPAKE2 in Golang. Story starts a couple of month back when I started contributing to *magic-wormhole.rs*, a Rust port of original Python project of magic-wormhole. You can read this LWN article to understand more about what magic-wormhole is.

During contribution my friend Ramakrishnan Muthukrishnan said to me that I should try to port the magic-wormhole to Golang. I was not a expert Go programmer but had understanding of language basics and thought why not use it to improve my language understanding. And this is where it all started.

What is SPAKE2 why is it used?

At this point we need to know why SPAKE2 is needed and how magic-wormhole uses it. SPAKE2 is Simple Password Authenticated Key Exchange protocol. It allows 2 parties having a shared weak password to derive a strong shared key which can then be used by the parties to setup a encrypted+authenticated channel between them.

magic-wormhole uses SPAKE2 to negotiate a shared session key between communicating parties which is then used by magic-wormhole to derive different keys needed for different purpose.

So I hit my first road block after agreeing to implement the magic-wormhole in Golang, there is no SPAKE2 implementation readily available in Go!.

Enter Cryptography and My knowledge about it

Ram convinced me that its easy to implement SPAKE2 in Go and I agreed. But my knowledge on cryptography was limited. I knew Cryptography is

basically math with big numbers and relies on fact that factoring big prime number is hard for computers.
These big numbers are derived from Abelian Group and related concepts from number theory, which I have studied in academics but since I've never learnt a practical use case for them, its eating dust some where inside my memory.

I've taken some theoretical course on Cryptography but never thought of implementing one myself. With these weak foundation I set out for a new adventure.

Python SPAKE2 Implementation

Since magic-wormhole is implemented in python, SPAKE2 implementation in Python is considered as reference for other language implementation. SPAKE2 paper does not specify much about where or how the requirement public constants are defined, so implementer can take liberty at defining those. Python code uses 2 groups one is Twisted Edwards Curve group Ed25519 and others are multiplicative group over Integer of 1024, 2048 and 3072 bits.

In Python code Warner himself has defined Ed25519 and other Integer group and related operations. In Rust code though there is only Ed25519 group but its created using curve25519-dalek library. Haskell code also defines the group operations by itself instead of depending on any other libraries (possibly Cryptonite). So as a first step I started for searching any library equivalent to curve25519-dalek as I've no clue what is an Elliptic curve.(forget groups I don't know basic itself).

First Try, Bruteforce

I've this bad habit of tackling problem with brute force; some times it works but most time it just exhausts me, taking me nowhere. So with my normal habit I started looking for Ed25519 curve operations library. (Without actually knowing what operations are and how it works). I tried to read through curve25519-dalek but invain, nothing entered into my head. I found ed25519 package for Go written by Adam Langley but eventually it turned out to be actually a signature package. I found an internal package defined in ed25519 package called edwards25519, which seem to have some operations defined but I was unable to understand it nor figure out why its made internal. I later even took a dig at embedding edwards25519 as part of my implementation of ed25519 group but finally had to drop it for my own version that story will be part of another post in this series.

Conclusion

During all these I was constantly in talk with Ram and first thing he told me was to slow down a bit and go from scratch. And that's what was the learning point for me. In short I can say the following.

Nothing can be done in single day, before you code understand the basic concepts and then build it from there. As they say you can't build a stable house on weak foundation.

In the next post in this series I will write about my learning on Elliptic Curves and Elliptic curve groups, followed by experiment with number groups and finally learning and decision I had to make in writing gospake2.

Docker Private Registry and Self Signed Certificates

2018-04-14T19:47:00+05:30

I was recently experimenting with hosting a private registry on an internal LAN network for publishing docker private images. I found out that docker-pull works only with TLS secured registry. There is possible to run insecure registry by editing daemon.json file but its better to use self-signed certificates instead.

Once I followed the step and started registry I tried to docker-pull and it started complaining about certificate not having any valid names. But this same certificate worked fine with browsers too, of course you need to add exception but no other errors were encountered.

Documentation for Docker does not speaks any specific settings needs to be done prior to generating a self-signed certificate so I was bit confused at beginning.A bit of searching showed up following issue filed against docker and then later re-assigned against *Golang* for its method of handling x509 certificate. It appears that with valid Subject Alternative Name Go crypto library ignores the Common Name.

From thread on Security Stack Exchange I found the command to create a self-signed certificate to contain self-signed certificate. Command in excepted answer does not work until you add --extensions option to it as mentioned in one of the comments. Full command is as shown below.

openssl req -new -sha256 -key domain.key \
             -subj "/C=US/ST=CA/O=Acme, Inc./CN=example.com" \
             -reqexts SAN -extensions SAN \
             -config \
<(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:example.com,DNS:www.example.com")) -out domain.crt

You would need to replace values in -subj and under [SAN] extension. Benefit of this command is you need not modify the /etc/ssl/openssl.conf file.

If you do not have a domain name for the registry and using IP address instead consider replacing [SAN] section in above command to have IP: <ip-address> instead of DNS.

Happy hacking.!

Docker container as Development Environment

2018-04-14T13:15:00+05:30

When you have a distributed team working on a project, you need to make sure that every one uses similar development environment. This is critical if you are working on embedded systems project. There are multiple possibility for this scenario.

Use a common development server and provide every developer in your team an account in it.
Provide description to every one how to setup their development environment and trust them they will do so.
Use Docker to provide the developer with ready made environment and use build server (CI) for creating final deployment binaries.

1st approach was most common approach used, but it has some drawbacks. Since its a shared system you should make sure not every one is able to install or modify system and hence have a single administrator so that no one accidentally breaks the development environment. Other problems include forced to use older OS due to specific requirements of compilers etc.

2nd approach makes you put your trust on your developer to get the correct development environment. Of course you need to trust your team, but every one is a human being and humans can make mistake.

Enter Docker

Dockerfile is best way to document/setup development environment. A developer can read Dockerfile and understand what is being done to setup the environment and simply execute docker build command to generate his/her development environment, or better you build the image and publish it in either public registry and if that is not possible put it in a private registry, and ask developers to pull a image of development environment.

Don't be scared by private registry, setting one up is not a humongous task! Its just couple of docker commands and there is a pretty good documentation available to set one up.

While setting up a development environment you need to make sure last instruction in your Dockerfile is executing the shell of your choice. This is because when you start a container this last instruction is what actually is run by docker, and in our case we need to provide developer with a shell all build toolchain and libraries.

...
CMD ["/bin/bash"]

Now developer just needs to get/build the image, start the container and use it for their work!. To summarize below command is sufficient for developer to run a fresh environment.

$ docker build -t devenv . # If they are building it

# If they are pulling it from say private registry
$ docker pull private-registry-ip/path/to/devenv

$ docker run -itd --name development devenv
$ docker attach development

When container is started it will execute the shell and the next attach command will attach your shell's input/output to container. Now it can be used just like normal shell to build the application.

Another good thing which can be done is sharing the workspace with container so your container can just contain the toolchain and library that is needed and all version control, code editing and likewise can be done in the host machine. One thing that would be needed to make sure is your UID on host and UID of the user inside the container is same. This can be easily done by creating a separate user in the container with UID same as your UID on host system.

Advantages

Some advantages of using docker container include

They are easy to setup and saves a lot of time to the team as a whole compared to traditional approaches.
Easy to throwaway and start fresh: If a developer thinks he did something wrong with container he can prune it and create a fresh one based on the development environment image. So this gives a lot of freedom to developer to experiment.
Uniformity: You will be sure that all your team will be using same evnironment.

So you might ask what about other container technologies like systemd-nspawn or lxc etc. Of course they can also be used in the similar fashion, infact before experimenting with Docker I was a vivid user of systemd-nspawn container. You might have also seen my previous blog posts on systemd-nspawn too. Only reason I switched to Docker is its easy to setup unlike systemd-nspawn which needs so many tweaking and tuning of things besides it does not have a Dockerfile like approach which makes things more time consuming. So for me Docker won the war and I shifted to using Docker more.

This entire post was based on my experiment and experience with Docker. If you feel something can be done in a better way please feel free to write to me.

Biboumi - A XMPP - IRC Gateway

2018-03-11T10:49:00+05:30

IRC is a communication mode (technically a communication protocol) used by many Free Software projects for communication and collaboration. It is serving these projects well even 30 years after its inception. Though I'm pretty much okay with IRC I had a problem of not able to use IRC from the mobile phones. Main problem is the inconsistent network connection, where IRC needs always to be connected. This is where I came across Biboumi.

Biboumi by itself does not have anything to do with mobile phones, its just a gateway which will allow you to connect with IRC channel as if it is a XMPP MUC room from any XMPP client. Benefit of this is it allows to enjoy some of XMPP feature in your IRC channel (not all but those which can be mapped).

I run Biboumi with my ejabbered instance and there by now I can connect to some of the Debian IRC channel directly from my phone using Conversations XMPP client for Android.

Biboumi is packaged for Debian, though I'm co-maintainer of the package most hardwork is done by Jonas Smedegaard in keeping the package in shape. It is also available for stretch-backports (though slightly outdated as its not packaged by us for backports). Once you install the package, copy example configuration file from /usr/share/doc/biboumi/examples/example.conf to /etc/biboumi/biboumi.cfg and modify the values as needed. Below is my sample file with password redacted.

hostname=biboumi.localhost
password=xxx
db_name=/var/lib/biboumi/biboumi.sqlite
#log_file=/var/log/biboumi/biboumi.log
log_level=0
admin=vasudev@copyninja.info
port=8888
realname_customization=true
realname_from_jid=false

Explanation for all the key, values in the configuration file is available in the man page (man biboumi).

Biboumi is configured as external component of the XMPP server. In my case I'm using ejabberd to host my XMPP service. Below is the configuration needed for allowing biboumi to connect with ejabberd.

 listen:
-
  port: 8888
  ip: "127.0.0.1"
  module: ejabberd_service
  acess: all
  hosts:
    "biboumi.localhost":
       password: xxx

password field in biboumi configuration should match password value in your XMPP server configuration.

After doing above configuration reload ejabberd (or your XMPP server) and start biboumi. Biboumi package provides systemd service file so you might need to enable it first. That's it now you have an XMPP to IRC gateway ready.

You might notice that I'm using local host name for hostname key as well as ip field in ejabberd configuration. This is because TLS support was added to biboumi Debian package only after 7.2 release as botan 2.x was not available till that point in Debian. Hence using proper domain name and making biboumi listen to public will be not safe at least prior to Debian package version 7.2-2. Also making the biboumi service public means you will also need to handle spam bots trying to connect from your service to IRC, which might get your VPS banned from IRC.

Connection Semantics

Once biboumi is configured and running you can now use XMPP client of your choice (Gajim, Conversation etc.) to connect to IRC. To connect to OFTC from your XMPP client you need to following address in Group Chat section

%irc.oftc.net@biboumi.localhost

Replace part after @ to what you have configured in hostname field in biboumi configuration. To join a specific channel on a IRC server you need to join the group conversation with following format

#channel%irc.example.com@biboumi.localhost

If your nick name is registered and you would want to identify yourself to IRC server you can do that by joining in group conversation with NickServ using following address

nickserv%irc.example.org@biboumi.localhost

Once connected you can send NickServ command directly in this virtual channel. Like identify password nick. It is also possible to configure your XMPP clients like Gajim to send Ad-Hoc commands on connection to particular IRC server for identifying your self with IRC servers. But this part I did not get working in Gajim.

If you are running your own XMPP server then biboumi gives you best way to connect to IRC from your mobile phones. And with applications like Conversation running XMPP application won't be hard on your phone battery.

Resources

My personal Email setup - Notmuch, mbsync, postfix and dovecot

2017-12-23T16:43:00+05:30

I've been using personal email setup for quite long and have not documented it anywhere. Recently when I changed my laptop (a post is pending about it) I got lost trying to recreate my local mail setup. So this post is a self documentation so that I don't have to struggle again to get it right.

Server Side

I run my own mail server and I use postfix as SMTP server and Dovecot for the IMAP purpose. I'm not going into detail of setting those up as my setup was mostly done by using scripts created by Jonas for Redpill infrastructure. What redpill is?. (In jonas's own words)

<jonas> Redpill is a concept - a way to setup Debian hosts to collaborate across organisations <jonas> I develop the concept, and use it for the first ever Redpill network-of-networks redpill.dk, involving my own network (jones.dk), my main client's network (homebase.dk), a network in Germany including Skolelinux Germany (free-owl.de), and Vasudev's network (copyninja.info)

Along with that I have a dovecot sieve filtering to classify on high level mails into various folders depending on from where they originate. All the rules live in the ~/dovecot.sieve file under every account which has a mail address.

Again I'm not going into detail of how to set these things up, as its not goal of my this post.

On my Laptop

On my laptop I've following 4 parts setup

Mail syncing : Done using mbsync command
Classification: Done using notmuch
Reading: Done using notmuch-emacs
Mail sending: Done using postfix running as relay server and SMTP client.

Mail Syncing

Mail syncing is done using mbsync tool, I was previously user of offlineimap and recently switched to mbsync as I felt it more lighter and simpler to configure than offlineimap. mbsync command is provided by package isync.

Configuration file is ~/.mbsyncrc. Below is my sample content with some private things redacted.

IMAPAccount  copyninja
Host imap.copyninja.info
User vasudev
PassCmd      "gpg -q --for-your-eyes-only --no-tty --exit-on-status-write-error --batch --passphrase-file ~/path/to/passphrase.txt -d ~/path/to/mailpass.gpg"
SSLType IMAPS
SSLVersion TLSv1.2
CertificateFile /etc/ssl/certs/ca-certificates.crt


IMAPAccount gmail-kamathvasudev
Host imap.gmail.com
User kamathvasudev@gmail.com
PassCmd "gpg -q --for-your-eyes-only --no-tty --exit-on-status-write-error --batch --passphrase-file ~/path/to/passphrase.txt -d ~/path/to/mailpass.gpg"
SSLType IMAPS
SSLVersion TLSv1.2
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPStore copyninja-remote
Account copyninja

IMAPStore gmail-kamathvasudev-remote
Account gmail-kamathvasudev

MaildirStore copyninja-local
Path ~/Mail/vasudev-copyninja.info/
Inbox ~/Mail/vasudev-copyninja.info/INBOX

MaildirStore gmail-kamathvasudev-local
Path ~/Mail/Gmail-1/
Inbox ~/Mail/Gmail-1/INBOX

Channel copyninja
Master :copyninja-remote:
Slave :copyninja-local:
Patterns *
Create Both
SyncState *
Sync All

Channel gmail-kamathvasudev
Master :gmail-kamathvasudev-remote:
Slave :gmail-kamathvasudev-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
Patterns * ![Gmail]*
Create Both
SyncState *
Sync All

Explanation for some interesting part in above configuration. One is the PassCmd which allows you to provide shell command to obtain the password for the account. This avoids filling in the password in configuration file. I'm using symmetric encryption with gpg and storing password some where on my disk. Which is of course just safe guarded by Unix ACL.

I actually wanted to use my public key to encrypt the file but unlocking the file when script is run in background or via systemd looks difficult (or looked nearly impossible). If you have better suggestion I'm all ears :-).

Next instruction part is Patterns. This allows you to selectively sync mail from your mail server. This was really helpful for me to exclude all crappy [Gmail]/ folders.

Mail Classification

Once mail is locally on your device, we need a way to read the mails easily in a mail reader. My original setup was serving synced Maildir using local dovecot instance and read it in Gnus. This setup was bit of a over kill with all server software setups but inability of Gnus to not cope well with Maildir format this was best way to do it. This setup also has a disadvantage, that is searching a mail quickly when you have huge pile of mail to go through. This is where notmuch comes into picture.

notmuch allows me to easily index through Gigabytes of my mail archives and get what I need very easily. I've created a small script which combines executing of mbsync and notmuch execution. I tag mails based on the Maildirs which are actually created on server side using dovecot sieve. Below is my full shell script which is doing task of syncing classification and deleting of spams.

#!/bin/sh

MBSYNC=$(pgrep mbsync)
NOTMUCH=$(pgrep notmuch)

if [ -n "$MBSYNC" -o -n "$NOTMUCH" ]; then
   echo "Already running one instance of mail-sync. Exiting..."
         exit 0
fi

echo "Deleting messages tagged as *deleted*"
notmuch search --format=text0 --output=files tag:deleted |xargs -0 --no-run-if-empty rm -v

echo "Moving spam to Spam folder"
notmuch search --format=text0 --output=files tag:Spam and \
  to:vasudev@copyninja.info | \
    xargs -0 -I {} --no-run-if-empty mv -v {} ~/Mail/vasudev-copyninja.info/Spam/cur
notmuch search --format=text0 --output=files tag:Spam and
  to:vasudev-debian@copyninja.info | \
     xargs -0 -I {} --no-run-if-empty mv -v {} ~/Mail/vasudev-copyninja.info/Spam/cur


MDIR="vasudev-copyninja.info vasudev-debian Gmail-1"
mbsync -Va
notmuch new

for mdir in $MDIR; do
    echo "Processing $mdir"
    for fdir in $(ls -d /home/vasudev/Mail/$mdir/*); do
      if [ $(basename $fdir) != "INBOX" ]; then
          echo "Tagging for $(basename $fdir)"
          notmuch tag +$(basename $fdir) -inbox -- folder:$mdir/$(basename $fdir)
      fi
    done
done

So before running mbsync I search for all mails tagged as deleted and delete them from system. Next I look for mails tagged as Spam on both my accounts and move it to Spam folder. Yeah you got it right these are mails escaping the spam filter and landing in my inbox and personally marked as Spam.

After running mbsync I tag mails based on their folder (searching string folder:). This allows me easily get contents of lets say a mailing list without remembering the list address.

Reading Mails

Now that we have synced and classified mail its time to setup the reading part. I use notmuch-emacs interface to read the mails. I use Spacemacs flavor of emacs so I took some time to write down the a private layer which brings together all my keybindings and classification in one place and does not clutter my entire .spacemacs file. You can find the code for my private layer in notmuch-emacs-layer repository

Sending Mails

Well its not sufficient that if we can read mails, we need to be able to reply to mail. And this was the slightly tricky part where I recently got lost and had to write this post so that I don't forget it again. (And of course don't have to refer some outdated posts on web).

My setup to send mails is using postfix as SMTP client with my own SMTP server as relayhost for it. The problem of relaying is it's not for the hosts with dynamic IP. There are couple of ways to allow hosts with dynamic IP to use relay servers, one is put the IP address from where mail will originate into my_network or second use SASL authentication.

My preferred way is use of SASL authentication. For this I first had to create a separate account one for each machine which is going to relay the mails to my main server. Idea is to not use my primary account for SASL authentication. (Originally I was using primary account, but Jonas gave this idea of account per road runner).

adduser <hostname>_relay

Here replace <hostname> with name of your laptop/desktop or whatever you are using. Now we need to adjust postfix to act as relaying server. So add following lines to postfix configuration

# SASL authentication
smtp_sasl_auth_enable = yes
smtp_tls_security_level = encrypt
smtp_sasl_tls_security_options = noanonymous
relayhost = [smtp.copyninja.info]:submission
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

So here relayhost is the server name which your postfix instance will be using to relay mails forward into internet. :submission part tells postfix to forward mail on to port 587 (secure). smtp_sasl_tls_security_options is set to disallow anonymous connection. This is must so that relay server trusts your mobile host and agrees to forward the mail for you.

/etc/postfix/sasl_passwd is the file where you need to store password for account to be used for SASL authentication with server. Put following content into it.

[smtp.example.com]:submission    user:password

Replace smtp.example.com with your SMTP server name which you have put in relayhost configuration. Replace user with <hostname>_relay user you created and its password.

To secure the sasl_passwd file and create a hash of it for postfix use following command.

chown root:root /etc/postfix/sasl_passwd
chmod 0600 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd

The last command will create /etc/postfix/sasl_passwd.db file which is hash of your file /etc/postfix/sasl_passwd with same owner and permission. Now reload the postfix and check if mail makes out of your system using mail command.

Bonus Part

Well since I've a script created above bringing together mail syncing and classification. I went ahead and created a systemd timer to periodically sync mails in the background. In my case every 10 minutes. Below is mailsync.timer file.

[Unit]
Description=Check Mail Every 10 minutes
RefuseManualStart=no
RefuseManualStop=no

[Timer]
Persistent=false
OnBootSec=5min
OnUnitActiveSec=10min
Unit=mailsync.service

[Install]
WantedBy=default.target

Below is mailsync.service which is needed by mailsync.timer to execute our scripts.

[Unit]
Description=Check Mail
RefuseManualStart=no
RefuseManualStop=yes

[Service]
Type=oneshot
ExecStart=/usr/local/bin/mail-sync
StandardOutput=syslog
StandardError=syslog

Put these files under /etc/systemd/user and run below command to enable them.

systemctl enable --user mailsync.timer
systemctl enable --user mailsync.service
systemctl start --user mailsync.timer

So that's how I've sync and send mail from my system. I came to know about afew from Jonas Smedegaard who also proof read this post. So next step I will try to improve my notmuch configuration using afew and of course a post will follow after that :-).

Writing a UDP Broadcast Receiver as Python Iterator

2017-08-19T18:58:00+05:30

I had to write a small Python application to listen for some broadcast message and process the message. This broadcast messages are actually sort of discovery messages to find some peers in a network. Writing a simple UDP Server to listen on a particular port was easy; but while designing an application I was wondering how can I plugin this server into my main code. There are 2 possibility

Use threading module of python to send the server code in back ground and give it a callback to communicate the data to main thread.
Periodically read some messages from server code and then dispose of server.

I didn't like first approach because I need to pass a callback function and I some how will end up complicating code. Second approach sounded sane but I did want to make server more like iterator. I searched around to see if some one has attempted to write something similar, but did not find anything useful (may be my Googling skills aren't good enough). Anyway so I thought what is wrong in trying?. If it works then I'll be happy that I did something different :-).

The first thing for making iterator in Python is having function __iter__ and __next__ defined in your class. For Python 2 iterator protocol wanted next to be defined instead of __next__. So for portable code you can define a next function which in return calls __next__.

So here is my first shot at writing BroadCastReceiver class.

from socket import socket, AF_INET, SOCK_DGRAM


class BroadCastReceiver:

    def __init__(self, port, msg_len=8192):
        self.sock = socket(AF_INET, SOCK_DGRAM)
        self.sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
        self.sock.bind(('', port))
        self.msg_len = msg_len

    def __iter__(self):
             return self

    def __next__(self):
             try:
                 addr, data = self.sock.recvfrom(self.msg_len)
                 return addr, data
             except Exception as e:
                 print("Got exception trying to recv %s" % e)
                 raise StopIteration

This version of code can be used in a for loop to read from socket UDP broadcasts. One problem will be that if no packet is received which might be due to disconnected network the loop will just block forever. So I had to modify the code slightly to add timeout parameter. So changed portion of code is below.

...
    def __init__(self, port, msg_len=8192, timeout=15):
        self.sock = socket(AF_INET, SOCK_DGRAM)
        self.sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
        self.sock.settimeout(timeout)
        self.sock.msg_len = msg_len
        self.sock.bind(('', port))

 ...

So now if network is disconnected or no packet was received for timeout period we get a socket.timeout exception due to which StopIteration will be raised causing the for loop using server as iterator to exit. This avoids us just blocking our periodic code run forever when network is disconnected or no messages are received for long time. (may be due to connected to wrong network).

Now every thing looks fine but only part is if we create the server object each time our periodic code is called we will have binding issue as we did not properly close the socket once iterator has stopped. So I added socket closing code in __del__ function for the class. __del__ will be called when garbage collector try to recollect object when it goes out of scope.

...
    def __del__(self):
        self.sock.close()

So the server can be used in for loop or by passing the object of server to next built-in function. Here are 2 examples.

r = BroadCastReceiver(5000, timeout=10)
count = 0
for (address, data) in r:
    print('Got packet from %s: %s' % address, data)
    count += 1
    # do whatever you want with data
    if count > 10:
        break

Here we use an counter variable to track iteration and after some iteration we exit for loop. Another way is use for loop with range of iteration like below.

r = BroadCastReceiver(5000,  timeout=10)
for i in range(20):
    try:
        address, data = next(r)
        # do whatever you want with data
    except:
        break

Here an additional try block was needed inside the for loop to card call to next, this is to handle the timeout or other exception and exit the loop. In first case this is not needed as StopIteration is understood by for.

Both use cases I described above are mostly useful when it is not critical to handle each and every packet (mostly peer discovery) and packets will always be sent. So if we miss some peers in one iteration we will still catch them in next iteration. We just need to make sure we provide big enough counter to catch most peers in each iteration.

If its critical to receive each packet we can safely send this iterating logic to a separate thread which keeps receiving packets and process data as needed.

For now I tried this pattern mostly with UDP protocol but I'm sure with some modification this can be used with TCP as well. I'll be happy to get feed back from Pythonistas out there on what you think of this approach. :-)

Update

I got a suggestion from Ryan Nowakowski to make the server object as context manager and close the socket in __exit__ as it can't be guaranteed that __del__ will be called for objects which exists during interpreter exits. So I slightly modified the class to add __enter__ and __exit__ method like below and removed __del__

...

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        self.sock.close()

Usage pattern is slightly modified because of this and we need to use with statement while creating object.

with BroadCastReceiver(2000) as r:
    # use server object as you wish
    ...

It is also possible to cleanly close socket without adding context manager that is adding finally statement to our try and except block in __next__. The modified code without adding context manager looks like below.

def __next__(self):
    try:
        addr, data = self.sock.recvfrom(self.msg_len)
        return addr, data
    except Exception as e:
        print("Got exception trying to recv %s" % e)
        raise StopIteration
    finally:
        self.sock.close()

When we raise StopIteration again from except block, it will be temporarily saved and finally block is executed which will now close the socket.

Overriding version information from setup.py with pbr

2017-07-16T20:53:00+05:30

I recently raised a pull request on zfec for converting its python packaging from pure setup.py to pbr based. Today I got review from Brian Warner and one of the issue mentioned was python setup.py --version is not giving same output as previous version of setup.py.

Previous version used versioneer which extracts version information needed from VCS tags. Versioneer also provides flexibility of specifying type of VCS used, style of version, tag prefix (for VCS) etc. pbr also does extract version information from git tag but it expects git tag to be of format tags/refs/x.y.z format but zfec used a zfec- prefix to tag (example zfec-1.4.24) and pbr does not process this. End result, I get a version in format 0.0.devNN where NN is number of commits in the repository from its inception.

Me and Brian spent few hours trying to figure out a way to tell pbr that we would like to override version information it auto deduces, but there was none other than putting version string in PBR_VERSION environment variable. That documentation was contributed by me 3 years back to pbr project.

So finally I used versioneer to create a version string and put it in the environment variable PBR_VERSION.

import os
import versioneer

os.environ['PBR_VERSION'] = versioneer.get_version()

...
setup(
    setup_requires=['pbr'],
    pbr=True,
    ext_modules=extensions
)

And added below snippet to setup.cfg which is how versioneer can be configured with various information including tag prefixes.

[versioneer]
VCS = git
style = pep440
versionfile_source = zfec/_version.py
versionfile_build = zfec/_version.py
tag_prefix = zfec-
parentdir_prefix = zfec-

Though this work around gets the work done, it does not feel correct to set environment variable to change the logic of other part of same program. If you guys know the better way do let me know!. Also probably I should consider filing an feature request against pbr to provide a way to pass tag prefix for version calculation logic.

debcargo: Replacing subprocess crate with git2 crate

2017-07-15T15:35:00+05:30

In my previous post I talked about using subprocess crate to extract beginning and ending year from git repository for generating debian/copyright file. In this post I'm going to talk on how I replaced subprocess with native git2 crate and achieved the same result in much cleaner and safer way.

git2 is a native Rust crate which provides access to Git repository internals. git2 does not involve any unsafe invocation as it is built against libgit2-sys which is actually using Rust FFI to directly bind to underlying libgit library. Below is the new copyright_fromgit function with git2 crate.

fn copyright_fromgit(repo_url: &str) -> Result<String> {
   let tempdir = TempDir::new_in(".", "debcargo")?;
   let repo = Repository::clone(repo_url, tempdir.path())?;

   let mut revwalker = repo.revwalk()?;
   revwalker.push_head()?;

   // Get the latest and first commit id. This is bit ugly
   let latest_id = revwalker.next().unwrap()?;
   let first_id = revwalker.last().unwrap()?; // revwalker ends here is consumed by last

   let first_commit = repo.find_commit(first_id)?;
   let latest_commit = repo.find_commit(latest_id)?;

   let first_year =
       DateTime::<Utc>::from_utc(
               NaiveDateTime::from_timestamp(first_commit.time().seconds(), 0),
               Utc).year();

   let latest_year =
       DateTime::<Utc>::from_utc(
             NaiveDateTime::from_timestamp(latest_commit.time().seconds(), 0),
             Utc).year();

   let notice = match first_year.cmp(&latest_year) {
       Ordering::Equal => format!("{}", first_year),
       _ => format!("{}-{},", first_year, latest_year),
   };

   Ok(notice)
}

So here is what I'm doing

Use git2::Repository::clone to clone the given URL. We are thus avoiding exec of git clone command.
Get a revision walker object. git2::RevWalk implements Iterator trait and allows walking through the git history. This is what we are using to avoid exec of git log command.
revwalker.push_head() is important because we want to tell revwalker from where we want to walk the history. In this case we are asking it to walk history from repository HEAD. Without this line next line will not work. (Learned it in hard way :-) ).
Then we extract git2::Oid which is we can say similar to commit hash and can be used to lookup a particular commit. We take latest commit hash using RevWalk::next call and the first commit using Revwalk::last, note the order this is because Revwalk::last consumes the revwalker so doing it in reverse order will make borrow checker unhappy :-). This replaces exec of head -n1 command.
Look up the git2::Commit objects using git2::Repository::find_commit
Then convert the git2::Time to chrono::DateTime and take out the years.

After this change I found an obvious error which went unnoticed in previous version, that is if there was no repository key in Cargo.toml. When there was no repo URL git clone exec did not error out and our shell commands happily extracted year from the debcargo repository!. Well since I was testing code from debcargo repository It never failed, but when I executed from non-git repository folder git threw an error but that was git log and not git clone. This error was spotted right away because git2 threw me an error that I gave it empty URL.

When it comes to performance I see that debcargo is faster compared to previous version. This makes sense because previously it was doing 5 fork and exec system calls and now that is avoided.

Update: - Shell pipelines with subprocess crate and use of Exec::shell function

2017-06-19T20:48:00+05:30

In my previous post I used Exec::shell function from subprocess crate and passed it string generated by interpolating --author argument. This string was then run by the shell via Exec::shell. After publishing post I got ping on IRC by Jonas Smedegaard and Paul Wise that I should replace Exec::shell, as it might be prone to errors or vulnerabilities of shell injection attack. Indeed they were right, in hurry I did not completely read the function documentation which clearly mentions this fact.

When invoking this function, be careful not to interpolate arguments into the string run by the shell, such as Exec::shell(format!("sort {}", filename)). Such code is prone to errors and, if filename comes from an untrusted source, to shell injection attacks. Instead, use Exec::cmd("sort").arg(filename).

Though I'm not directly taking input from untrusted source, its still possible that the string I got back from git log command might contain some oddly formatted string with characters of different encoding which could possibly break the Exec::shell , as I'm not sanitizing the shell command. When we use Exec::cmd and pass argument using .args chaining, the library takes care of creating safe command line. So I went in and modified the function to use Exec::cmd instead of Exec::shell.

Below is updated function.

fn copyright_fromgit(repo: &str) -> Result<Vec<String>> {
    let tempdir = TempDir::new_in(".", "debcargo")?;
    Exec::cmd("git")
     .args(&["clone", "--bare", repo, tempdir.path().to_str().unwrap()])
     .stdout(subprocess::NullFile)
     .stderr(subprocess::NullFile)
     .popen()?;

    let author_process = {
        Exec::shell(OsStr::new("git log --format=\"%an <%ae>\"")).cwd(tempdir.path()) |
        Exec::shell(OsStr::new("sort -u"))
    }.capture()?;
    let authors = author_process.stdout_str().trim().to_string();
    let authors: Vec<&str> = authors.split('\n').collect();
    let mut notices: Vec<String> = Vec::new();
    for author in &authors {
        let author_string = format!("--author={}", author);
        let first = {
            Exec::cmd("/usr/bin/git")
             .args(&["log", "--format=%ad",
                    "--date=format:%Y",
                    "--reverse",
                    &author_string])
             .cwd(tempdir.path()) | Exec::shell(OsStr::new("head -n1"))
        }.capture()?;

        let latest = {
            Exec::cmd("/usr/bin/git")
             .args(&["log", "--format=%ad", "--date=format:%Y", &author_string])
             .cwd(tempdir.path()) | Exec::shell("head -n1")
        }.capture()?;

        let start = i32::from_str(first.stdout_str().trim())?;
        let end = i32::from_str(latest.stdout_str().trim())?;
        let cnotice = match start.cmp(&end) {
            Ordering::Equal => format!("{}, {}", start, author),
            _ => format!("{}-{}, {}", start, end, author),
        };

        notices.push(cnotice);
    }

    Ok(notices)
}

I still use Exec::shell for generating author list, this is not problematic as I'm not interpolating arguments to create command string.

Rust - Shell like Process pipelines using subprocess crate

2017-06-18T20:59:00+05:30

I had to extract copyright information from the git repository of the crate upstream. The need aroused as part of updating debcargo, tool to create Debian package source from the Rust crate.

General idea behind taking copyright information from git is to extract starting and latest contribution year for every author/committer. This can be easily achieved using following shell snippet

for author in $(git log --format="%an" | sort -u); do
   author_email=$(git log --format="%an <%ae>" --author="$author" | head -n1)
   first=$(git \
   log --author="$author" --date=format:%Y --format="%ad" --reverse \
             | head -n1)
   latest=$(git log --author="$author" --date=format:%Y --format="%ad" \
             | head -n1)
   if [ $first -eq $latest ]; then
       echo "$first, $author_email"
   else
       echo "$first-$latest, $author_email"
   fi
done

Now challenge was to execute these command in Rust and get the required answer. So first step was I looked at std::process, default standard library support for executing shell commands.

My idea was to execute first command to extract authors into a Rust vectors or array and then have 2 remaining command to extract years in a loop. (Yes I do not need additional author_email command in Rust as I can easily get both in the first command which is used in for loop of shell snippet and use it inside another loop). So I setup to 3 commands outside the loop with input and output redirected, following is snippet should give you some idea of what I tried to do.

let authors_command = Command::new("/usr/bin/git")
             .arg("log")
             .arg("--format=\"%an <%ae>\"")
             .spawn()?;
let output = authors_command.wait()?;
let authors: Vec<String> = String::from_utf8(output.stdout).split('\n').collect();
let head_n1 = Command::new("/usr/bin/head")
             .arg("-n1")
             .stdin(Stdio::piped())
             .stdout(Stdio::piped())
             .spwn()?;
for author in &authors {
             ...
}

And inside the loop I would create additional 2 git commands read their output via pipe and feed it to head command. This is where I learned that it is not straight forward as it looks :-). std::process::Command type does not implement Copy nor Clone traits which means one use of it I will give up the ownership!. And here I started fighting with borrow checker. I need to duplicate declarations to make sure I've required commands available all the time. Additionally I needed to handle error output at every point which created too many nested statements there by complicating the program and reducing its readability

When all started getting out of control I gave a second thought and wondered if it would be good to write down this in shell script ship it along with debcargo and use the script Rust program. This would satisfy my need but I would need to ship additional script along with debcargo which I was not really happy with.

Then a search on crates.io revealed subprocess, a crate designed to be similar with subprocess module from Python!. Though crate is not highly downloaded it still looked promising, especially the trait implements a trait called BitOr which allows use of | operator to chain the commands. Additionally it allows executing full shell commands without need of additional chaining of argument which was done above snippet. End result a much simplified easy to read and correct function which does what was needed. Below is the function I wrote to extract copyright information from git repo.

fn copyright_fromgit(repo: &str) -> Result<Vec<String>> {
    let tempdir = TempDir::new_in(".", "debcargo")?;
    Exec::shell(OsStr::new(format!("git clone --bare {} {}",
                                repo,
                                tempdir.path().to_str().unwrap())
                              .as_str())).stdout(subprocess::NullFile)
                              .stderr(subprocess::NullFile)
                              .popen()?;

    let author_process = {
         Exec::shell(OsStr::new("git log --format=\"%an <%ae>\"")).cwd(tempdir.path()) |
         Exec::shell(OsStr::new("sort -u"))
     }.capture()?;
    let authors = author_process.stdout_str().trim().to_string();
    let authors: Vec<&str> = authors.split('\n').collect();
    let mut notices: Vec<String> = Vec::new();
    for author in &authors {
        let reverse_command = format!("git log --author=\"{}\" --format=%ad --date=format:%Y \
                                    --reverse",
                                   author);
        let command = format!("git log --author=\"{}\" --format=%ad --date=format:%Y",
                           author);
        let first = {
             Exec::shell(OsStr::new(&reverse_command)).cwd(tempdir.path()) |
             Exec::shell(OsStr::new("head -n1"))
         }.capture()?;

         let latest = {
             Exec::shell(OsStr::new(&command)).cwd(tempdir.path()) | Exec::shell("head -n1")
         }.capture()?;

        let start = i32::from_str(first.stdout_str().trim())?;
        let end = i32::from_str(latest.stdout_str().trim())?;
        let cnotice = match start.cmp(&end) {
            Ordering::Equal => format!("{}, {}", start, author),
            _ => format!("{}-{}, {}", start, end, author),
        };

        notices.push(cnotice);
    }

    Ok(notices)
}

Of course it is not as short as the shell or probably Python code, but that is fine as Rust is system level programming language (which is intended to replace C/C++) and doing complex Shell code (complex due to need of shell pipelines) in approximately 50 lines of code in safe and secure way is very much acceptable. Besides code is as much readable as a plain shell snippet thanks to the | operator implemented by subprocess crate.

Tips for fuzzing network programs with AFL

2017-05-07T15:00:00+05:30

Fuzzing is method of producing random malformed inputs for a software and observe the software behavior. If a software crashes then there is a bug and it can have security implications. Fuzzing has gained a lot of interest now a days, especially with automated tools like American Fuzzy Lop (AFL) which can easily help you to fuzz the program and record inputs which causes crash in the software.

American Fuzzy Lop is a file based fuzzer which feeds input to program via standard input. Using it with network program like server's or clients is not possible in the original state. There is a version of AFL with patches to allow it fuzz network programs, but this patch is not merged upstream and I do not know if it ever makes into upstream or not. Also the above repository contains version 1.9 which is older compared to currently released versions.

There is another method for fuzzing network program using AFL with help of LD_PRELOAD tricks. preeny is a project which provides library which when used with LD_PRELOAD can desocket the network program and make it read from stdin.

There is this best tutorial from LoLWare which talks about fuzzing Nginx with preeny and AFL. There is a best AFL workflow by Foxglove Security which gives start to finish details about how to use AFL and its companion tool to do fuzzing. So I'm not going to talk about any steps of fuzzing in this post instead I'm going to list down my observations on changes that needs to be done to get clean fuzzing with AFL and preeny.

desock.so provided by preeny works only with read and write (or rather other system call does not work with stdin) system calls and hence you need to make sure you replace any reference to send, sendto, recv and recvfrom with read and write system calls respectively. Without this change program will not read input provided by AFL on standard input.
If your network program is using forking or threading model make sure to remove all those and make it plain simple program which receives request and sends out response. Basically you are testing the ability of program to handle malformed input so we need very minimum logic to make program do what it is supposed to do when AFL runs it.
If you are using infinite loop like all normal programs replace the infinite loop with below mentioned AFL macro and use afl-clang-fast to compile it. This speeds up the testing as AFL will run the job n times before discarding the current fork and doing a fresh fork. After all fork is costly affair.

while(__AFL_LOOP(1000)) { // Change 1000 with iteration count
             // your logic here
}

With above modification I could fuzz a server program talking binary protocol and another one talking textual protocol. In both case I used Wireshark capture to get the packet extract raw content and feed it as input to AFL.

I was successful in finding crashes which are exploitable in case of textual protocol program than in binary protocol case. In case of binary protocol AFL could not easily find new paths which probably is because of bad inputs I provided. I will continue to do more experiment with binary protocol case and provide my findings as new updates here.

If you have anything more to add do share with me :-). Happy fuzzing!.

Magic: Attach a HTML file to mail and get different on other side

2017-05-06T13:19:00+05:30

It's been a long time I did not write any new blog posts, and finally I got something which is interesting enough for a new post and so here it is.

I actually wanted some bills from my ISP for some work and I could not find mail from the ISP which had bills for some specific months in my mailbox. Problem with my ISP is bills are accessible in their account which can be only accessed from their own network. They do have a mobile app and that does not seem to work especially for the billing section. I used mobile app and selected month for which I did not have bill and clicked Send Mail button. App happily showed message saying it sent the bill to my registered mail address but that was a fancy lie. After trying several time I gave up and decided I will do it once I get back home.

Fast forward few hours, I'm back home from office and then I sat in front of my laptop and connected to ISP site, promptly downloaded the bills, then used notmuch-emacs and fire up a mail composer, attach those HTML file (yeah they produce bill in HTML file :-/) send it to my gmail and forget about it.

Fast forward few more days, I just remember I need those bills. I got hold of mail I sent earlier in gmail inbox and clicked on attachment and when browser opened the attachment I was shocked/surprised . Did you ask why?. See what I saw when I opened attachment.

Well I was confused for moment on what happened, I tried downloading the file and opened it an editor, and I see Chinese characters here also. After reaching home I checked the Sent folder in my laptop where I keep copy of mails sent using notmuch-emacs's notmuch-fcc-dirs variable. I open the file and open the attachement and I see same character as I saw in browser!. I open the original bills I downloaded and it looks fine. To check if I really attached the correct files I again drafted a mail this time to my personal email and sent it. Now I open the file from Sent folder and voila again there is different content inside my attachment!. Same happens when I receive the mail and open the attachment everything inside is Chinese characters!. Totally lost I finally opened the original file using emacs, since I use spacemacs it asked me for downloading of html layer and after which I was surprised because everything inside the editor is in Chinese!. OK finally I've something now so I opened same file at same time in nano from terminal and there it looks fine!. OK that is weird again I read carefully content and saw this line in beginning of file

<?xml version="1.0" encoding="UTF-16" ?>
<html xmlns="http://www.w3.org/1999/xhtml>....

OK that is new XML tag had encoding declared as UTF-16, really?. And just for checking I changed it to UTF-8 and voila file is back to normal in Emacs window!. To confirm this behavior I created a sample file with following content

<?xml version="1.0" encoding="utf-8"?><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /><title> Something weird </title><body>blablablablablalabkabakba</body></html>

Yeah with long line because ISP had similar case and now I opened the same file in nano using terminal and changed encoding="UTF-8" to encoding="UTF-16" and the behavior repeated I see Chinese character in the emacs buffer which has also opened the same file.

Below is the small gif showing this happening in real time, see what happens in emacs buffer when I change the encoding in terminal below. Fun right?.

I made following observation from above experiments.

When I open original file in browser or my sample file with encoding="UTF-16" its normal, no Chinese characters.
When I attach the mail and send it out the attachment some how gets converted into HTML file with Chinese characters and viewing source from browser shows header containing xml declarations and original <html> declarations get ripped off and new tags show up which I've pasted below.
If I download the same file and open in editor only Chinese characters are present and no HTML tags inside it. So definitely the new tags which I saw by viewing source in browser is added by Firefox.
I create similar HTML file and change encoding I can see the characters changing back and forth in emacs web-mode when I change encoding of the file.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head><META http-equiv="Content-Type"  content="text/html; charset=utf-8">

So if I'm understanding this correctly, Emacs due to encoding declaration interprets the actual contents differently. To see how Emacs really interpreted the content I opened the sent mail I had in raw format and saw following header lines for attachment.

Content-Type: text/html; charset=utf-8
Content-Disposition: attachment;
             filename=SOA-102012059675-FEBRUARY-2017.html
Content-Transfer-Encoding: base64

This was followed with base64 encoded data. So does this mean emacs interpreted the content as UTF-16 and encoded the content using UTF-8?. Again I've no clue, so I changed the encoding in both the files to be as UTF-8 and sent the mail by attaching these files again to see what happens. And my guess was right I could get the attachment as is on the receiving side. And inspecting raw mail the attachment headers now different than before.

Content-Type: text/html
Content-Disposition: attachment;
             filename=SOA-102012059675-DECEMBER-2016.html
Content-Transfer-Encoding: quoted-printable

See how Content-Type its different now also see the Content-Transfer-Encoding its now quoted-printable as opposed to base64 earlier. Additionally I can see HTML content below the header. When I opened the attachment from mail I get the actual bill.

As far as I understand base64 encoding is used when the data to be attached is base64. So I guess basically due to wrong encoding declared inside the file Emacs interpreted the content as a binary data and encoded it differently than what it really should be. Phew that was a lot of investigation to understand the magic but it was worth it.

Do let me know your thoughts on this.

Switching from approx to apt-cacher-ng

2016-07-17T16:30:00+05:30

After a long ~5 years (from 2011) journey with approx I finally wanted to switch to something new like apt-cacher-ng. And after a bit of changes I finally managed to get apt-cacher-ng into my work flow.

Bit of History

I should first give you a brief on how I started using approx. It all started in MiniDebconf 2011 which I organized at my Alma-mater. I met Jonas Smedegaard here and from him I learned about approx. Jonas has a bunch of machines at his home and he was active user of approx and he showed it to me while explaining the Boxer project. I was quite impressed with approx. Back then I was using a 230kbps slow INTERNET connection and I was also maintaining a couple of packages in Debian. Updating the pbuilder chroots was time consuming task for me as I had to download multiple times over slow net. And approx largely solved this problem and I started using it.

5 years fast forward I now have quite fast INTERNET with good FUP. (About 50GB a month), but I still tend to use approx which makes building packages quite faster. I also use couple of containers on my laptop which all use my laptop as approx cache.

Why switch?

So why change to apt-cacher-ng?. Approx is a simple tool, it runs mainly with inetd and sits between apt and the repository on INTERNET. Where as apt-cacher-ng provides a lot of features. Below are some listed from the apt-cacher-ng manual.

use of TLS/SSL repositories (may be possible with approx but I'm notsure how to do it)
Access control of who can access caching server
Integration with debdelta (I've not tried, approx also supports debdelta)
Avoiding use of apt-cacher-ng for some hosts
Avoiding caching of some file types
Partial mirroring for offline usage.
Selection of ipv4 or ipv6 for connections.

The biggest change I see is the speed difference between approx and apt-cacher-ng. I think this is mainly because apt-cacher-ng is threaded where as approx runs using inetd.

I do not want all features of apt-cacher-ng at the moment, but who knows in future I might need some features and hence I decided to switch to apt-cacher-ng over approx.

Transition

Transition from approx to apt-cacher-ng was smoother than I expected. There are 2 approaches you can use one is explicit routing another is transparent routing. I prefer transparent routing and I only had to change my /etc/apt/sources.list to use the actual repository URL.

deb http://deb.debian.org/debian unstable main contrib non-free
deb-src http://deb.debian.org/debian unstable main

deb http://deb.debian.org/debian experimental main contrib non-free
deb-src http://deb.debian.org/debian experimental main

After above change I had to add a 01proxy configuration file to /etc/apt/apt.conf.d/ with following content.

Acquire::http::Proxy "http://localhost:3142/"

I use explicit routing only when using apt-cacher-ng with pbuilder and debootstrap. Following snippet shows explicit routing through /etc/apt/sources.list.

deb http://localhost:3142/deb.debian.org/debian unstable main

Usage with pbuilder and friends

To use apt-cacher-ng with pbuilder you need to modify /etc/pbuilderrc to contain following line

MIRRORSITE=http://localhost:3142/deb.debian.org/debian

Usage with debootstrap

To use apt-cacher-ng with debootstrap, pass MIRROR argument of debootstrap as http://localhost:3142/deb.debian.org/debian.

Conclusion

I've now completed full transition of my work flow to apt-cacher-ng and purged approx and its cache.

Though it works fine I feel that there will be 2 caches created when you use transparent and explicit proxy using localhost:3142 URL. I'm sure it is possible to configure this to avoid duplication, but I've not yet figured it. If you know how to fix this do let me know.

Update

Jonas told me that its not 2 caches but 2 routing paths, one for transparent routing and another for explicit routing. So I guess there is nothing here to fix :-).

Integrating Cython extension with setuptools and unit testing

2016-06-26T19:54:00+05:30

I was reviewing changes for indic-trans as part of GSoC 2016. The module is an improvisation for our original transliteration module which was doing its job by substitution.

This new module uses machine learning of some sort and utilizes Cython, numpy and scipy. Student had kept pre-compiled shared library in the git tree to make sure it builds and passes the test. But this was not correct way. I started looking at way to build these files and remove it from the code base.

There is a cython documentation for distutils but none for setuptools. Probably it is similar to other Python extension integration into setuptools, but this was first time for me so after a bit of searching and trial and error below is what I did.

We need to use Extensions class from setuptools and give it path to modules we want to build. In my case beamsearch and viterbi are 2 modules. So I added following lines to setup.py

from setuptools.extension import Extension
from Cython.Build import cythonize

extensions = [
 Extension(
     "indictrans._decode.beamsearch",
     [
         "indictrans/_decode/beamsearch.pyx"
     ],
     include_dirs=[numpy.get_include()]
 ),
 Extension(
     "indictrans._decode.viterbi",
     [
         "indictrans/_decode/viterbi.pyx"
     ],
     include_dirs=[numpy.get_include()]
 )

]

First argument to Extensions is the module name and second argument is a list of files to be used in building the module. The additional inculde_dirs argument is not normally necessary unless you are working in virtualenv. In my system the build used to work without this but it was failing in Travis CI, so added it to fix the CI builds. OTOH it did work without this on Circle CI.

Next is provide this extensions to ext_modules argument to setup as shown below

setup(
 setup_requires=['pbr'],
 pbr=True,
 ext_modules=cythonize(extensions)
)

And for the reference here is full setup.py after modifications.

#!/usr/bin/env python

from setuptools import setup
from setuptools.extension import Extension
from Cython.Build import cythonize

import numpy


extensions = [
  Extension(
     "indictrans._decode.beamsearch",
     [
         "indictrans/_decode/beamsearch.pyx"
     ],
     include_dirs=[numpy.get_include()]
  ),
  Extension(
     "indictrans._decode.viterbi",
     [
         "indictrans/_decode/viterbi.pyx"
     ],
     include_dirs=[numpy.get_include()]
  )
]

setup(
  setup_requires=['pbr'],
  pbr=True,
  ext_modules=cythonize(extensions)
)

So now we can build the extensions (shared library) using following command.

python setup.py build_ext

Another challenge I faced was missing extension when running test. We use pbr in above project and testrepository with subunit for running tests. Looks like it does not build extensions by default so I modified the Makefile to build the extension in place before running test. The travis target of my Makefile is as follows.

travis:
     [ ! -d .testrepository ] || \
             find .testrepository -name "times.dbm*" -delete
     python setup.py build_ext -i
     python setup.py test --coverage \
             --coverage-package-name=indictrans
     flake8 --max-complexity 10 indictrans

I had to build the extension in place using -i switch. This is because other wise the tests won't find the indictrans._decode.beamsearch and indictrans._decode.viterbi modules. What basically -i switch does is after building shared library symlinks it to the module directory, in ourcase indictrans._decode

The test for existence of .testrepository folder is over come this bug in testrepository which results in test failure when running tests using tox.

Managing Virtual Network Devices with systemd-networkd

2016-01-10T22:26:00+05:30

I've been using bridge networking and tap networking for containers and virtual machines on my system. Configuration for bridge network which I use to connect containers was configured using /etc/network/interfaces file as shown below.

auto natbr0
iface natbr0 inet static
   address 172.16.10.1
   netmask 255.255.255.0
   pre-up brctl addbr natbr0
   post-down brctl delbr natbr0
   post-down sysctl net.ipv4.ip_forward=0
   post-down sysctl net.ipv6.conf.all.forwarding=0
   post-up sysctl net.ipv4.ip_forward=1
   post-up sysctl net.ipv6.conf.all.forwarding=1
   post-up iptables -A POSTROUTING -t mangle -p udp --dport bootpc -s 172.16.0.0/16 -j CHECKSUM --checksum-fill
   pre-down iptables -D POSTROUTING -t mangle -p udp --dport bootpc -s 172.16.0.0/16 -j CHECKSUM --checksum-fill

Basically I setup masquerading and IP forwarding when network comes up using this, so all my containers and virtual machines can access internet.

This can be simply done using systemd-networkd with couple of lines, yes couple of lines. For this to work first you need to enable systemd-networkd.

systemctl enable systemd-networkd.service

Now I need to write 2 configuration file for the above bridge interface under /etc/systemd/network. One file is natbr0.netdev which configures the bridge and the natbr0.network which configures IP address and other stuff for the bridge interface.

#natbr0.netdev
[NetDev]
Description=Bridge interface for containers/vms
Name=natbr0
Kind=bridge

#natbr0.network
[Match]
Name=natbr0

[Network]
Description=IP configuration for natbr0
Address=172.16.10.1/16
IPForward=yes
IPMasquerade=yes

The IPForward in above configuration is actually redundant, when I set IPMasquerade it automatically enables IPForward. So these configuration is equivalent of what I did in my interfaces file. It also avoids me doing additional iptables usage to add masquerading rules. This pretty much simplifies handling of virtual network devices.

There are many other things which can you do with systemd-networkd, like running a DHCPServer on the interface and many other things. I suggest you to read manual pages on systemd.network(5) and systemd.netdev(5).

systemd-networkd allows you configure all type of virtual networking devices and actual network interfaces. I've not myself used it to handle actual network interfaces yet.

Persisting resource control options for systemd-nspawn containers

2015-12-15T20:30:00+05:30

In my previous post on systemd-nspawn I mentioned, I was unclear on how to persist resource control options for a container. Today I accidentally discovered how can the property be persisted across boot without modifying service file or writing custom service file for container. It is done using systemctl set-property

To set CPUAccounting and CPUShares for a container we need to run following command.

systemctl set-property systemd-nspawn@container.service  CPUACCounting=1 CPUShares=200

This actually persists these settings at location, /etc/systemd/systemd-nspawn@container.service.d/ folder. So in our case there will be 2 files created under above location by name 50-CPUAccounting.conf and 50-CPUShares.conf with following contents.

# 50-CPUAccounting.conf
[Service]
CPUAccounting=yes

# 50-CPUShares.conf
[Service]
CPUShares=200

Today when I discovered this folder and saw the file contents, I became curious and started to wonder who created this folder. The look at systemctl man page made showed me this.

set-property NAME ASSIGNMENT...

Set the specified unit properties at runtime where this is supported. This allows changing configuration parameter properties such as resource control settings at runtime. Not all properties may be changed at runtime, but many resource control settings (primarily those in systemd.resource-control(5)) may. The changes are applied instantly, and stored on disk for future boots, unless --runtime is passed, in which case the settings only apply until the next reboot. The syntax of the property assignment follows closely the syntax of assignments in unit files.

Example: systemctl set-property foobar.service CPUShares=777

Note that this command allows changing multiple properties at the same time, which is preferable over setting them individually. Like unit file configuration settings, assigning the empty list to list parameters will reset the list.

I did remember doing this for my container and hence it became clear these files are actually written by systemctl set-property.

In case you don't want to persist the properties across boot you can simply pass --runtime switch.

Basically this is not just for container, resource control can be thus applied to any running service on the system. This is actually cool.

Ordering mount points in Jessie (systemd < 220)

2015-12-14T22:26:00+05:30

In the previous post I had mentioned that I didn't figure out how to add dependency on mount points so as to achieve correct ordering of mount points. After a lot of search today I finally figured it out thanks to the bug report and the patch which adds x-systemd.requires and other option to systemd.

So basically you can add additional ordering information to the .mount files generated by systemd-fstab-generator by adding a ordering.conf file to path /etc/systemd/system/usr-local-bin.mount.d/ path with following contents.

[Unit]
After=home.mount

And that is it, now usr-local-bin.mount will be mounted after home.mount is activated.

I couldn't find this information directly in related manpages like systemd.mount or systemd.automount. Also this information is not directly available in search results!.

Mount ordering and automount options in systemd

2015-12-13T15:13:00+05:30

Just to give a brief recap of why I'm writing this post, I've to describe a brief encounter I had with systemd which rendered my system unbootable. I first felt its systemd's problem but later figured out it was my mistake. So below is brief story divided into 2 sections Problem and Solution. Here Problem describes issue I faced with systemd and Solution discusses the .mount and .automount suffix files used by systemd.

Problem

I have several local bin folders and I don't want to modify PATH environment variable adding every other folder path. I first started using aufs as alternative to symlinks or modifying PATH variable. Recently I learnt about much more light weight union fs which is in kernel, overlayfs. And Google led me to Arch wiki article which showed me a fstab entry like below

overlay        /usr/local/bin        overlay noauto,x-systemd.automount,lowerdir=/home/vasudev/Documents/utilities/bin,upperdir=/home/vasudev/Documents/utilities/jonas-bin,workdir=/home/vasudev/Documents/utilities/bin-overlay    0       0

And after adding this entry, on next reboot systemd is not able to mount my LVM home and swap partition. It does however mount root partition. It gives me emergency target but login never returns. So without any alternative I had to reinstall the system. But funnily enough I started encountering the same issue (yeah after I had added above entry into fstab). Yeah that time I never thought that its culprit. My friend Ritesh finally got my system booting after the weird x-systemd.automount option. I never investigated further that time on why this problem occurred.

Solution

After re-encounter with similar problem and some other project I read the manual on systemd.mount, systemd-fstab-generator and systemd.automount and I had some understanding of what really went wrong in my case above.

So now let us see what really is happening. All the above happened because, systemd translates the /etc/fstab into .mount units at run time using systemd-fstab-generator. Every entry in fstab translates into a file named after the mount point. The / in the path of the mount point is replaced by a -. So / mount point is named as -.mount and /home is named as home.mount and /boot becomes boot.mount. All these files can be seen in directory /run/systemd/generator. And all these mount points are needed by local-fs.target , if any of these mount points fail, local-fs.target fails. And if local-fs.target fails it will invoke emergency.target.

systemd-fstab-generator manual suggests the ordering information in /etc/fstab is discarded. That means if you have union mounts, bind mounts or fuse mounts in the fstab which is normally at end of fstab and uses path from /home or some other device mount points it will fail to get mounted. This happens because systemd-fstab-generator didn't consider the ordering in fstab. This is what happened in my case my overlay mount point which is usr-local-bin.mount was some how happened to be tried before home.mount because there is no explicit dependency like Requires= or After= was declared. When systemd tried to mount it the path required under /home were not really present hence failed which in return invoked emergency.target as usr-local-bin.mount is Requires= dependency for local-fs.target.

Now what I didn't understand is why emregency.target never gave me the root shell after entering login information. I feel that this part is unrelated to the problem I described above and is some other bug.

To over come this what we can do is provided systemd-fstab-generator some information on dependency of each mount point. systemd.mount manual page suggests several such options. One which I used in my case is x-systemd.requires which should be placed in options column of fstab and specify the mount point which is needed before it has to be mounted. So my overlay fs entry translates to something like below

overlay        /usr/local/bin        overlay noauto,x-systemd.requires=/home,x-systemd.automount,lowerdir=/home/vasudev/Documents/utilities/bin,upperdir=/home/vasudev/Documents/utilities/jonas-bin,workdir=/home/vasudev/Documents/utilities/bin-overlay   0       0

There is another special option called x-systemd.automount this will make systemd-fstab-generator to create a .automount file for this mount point. What does systemd.automount do? It achieves on-demand file mounting and parallelized file system mounting. Something similar to systemd.socket the file system will be mounted when you access the mount point for first time.

So now if you try to see dependency of usr-local-bin.mount you will see following.

systemctl show -p After usr-local-bin.mount
After=-.mount systemd-journald.socket local-fs-pre.target system.slice usr-local-bin.automount

This means now usr-local-bin.mount depends on usr-local-bin.automount. And let us see what usr-local-bin.automount needs.

systemctl show -p Requires usr-local-bin.automount
Requires=-.mount home.mount

systemctl show -p After usr-local-bin.automount
After=-.mount home.mount

So clearly usr-local-bin.automount is activated after -.mount and home.mount is activated. Similar can be done for any bind mounts or fuse mounts which require other mount points before it can be mounted. Also note that x-systemd.autmount is not mandatory option for declaring dependencies, I used it just to make sure /usr/local/bin is mounted only when its really needed.

Conclusion

A lot of traditional way has been changed by systemd. I never understood why my system failed to boot in first place this happened because I was not really aware of how systemd works and was trying to debug the problem with traditional approach. So there is really a learning curve involved for every sysadmin out there who is going to use systemd. Most of them will read documentation before hand but others like me will learn after having a situation like above. :-).

So systemd interfering into /etc/fstab is good?. I don't know but since systemd parallelizes the boot procedure something like this is really needed. Is there a way to make systemd not touch /etc/fstab?. Yes there is you need to pass fstab=0 option in kernel command line and systemd-fstab-generator doesn't create any .mount or .swap files from your /etc/fstab.

!NB Also it looks like x-systemd.requires option was introduced recently and is not available in systemd <= 215 which is default in Jessie. So how do you declare dependencies in Jessie system?. I don't have any answer!. I did read that x-systemd.automount which is available in those versions of systemd can be used, but I'm yet to experiment this. If it succeeds I will write a post on it.

Forwarding host port to service in container

2015-11-09T17:06:00+05:30

Problem

I've lxc container running distcc daemon and I would like to forward the distcc traffic coming to my host system to container.

Solution

Following simple script which uses iptable did the job.

 #!/bin/sh
 set -e


usage() {
    cat <<EOF
    $(basename $0) [options] <in-interface> <out-interface> <port> <destination>

    --clear           Clear the previous rules before inserting new
                      ones

    --protocol        Protocol for the rules to use.

    in-interface      Interface on which incoming traffic is expected
    out-interface     Interface to which incoming traffic is to be
                      forwarded.

    port              Port to be forwarded. Can be integer or string
                      from /etc/services.
    destination       IP and port of the destination system to which
                      traffic needs to be forwarded. This should be in
                      form <destination_ip:port>

(C) 2015 Vasudev Kamath - This program comes with ABSOLUTELY NO
WARRANTY. This is free software, and you are welcome to redistribute
it under the GNU GPL Version 3 (or later) License

EOF
}

setup_portforwarding () {
    local protocol="$1"
    iptables -t nat -A PREROUTING -i $IN_INTERFACE -p "$protocol" --dport $PORT \
           -j DNAT --to $DESTINATION
    iptables -A FORWARD -p "$protocol" -d ${DESTINATION%%\:*} --dport $PORT -j ACCEPT

    # Returning packet should have gateway IP
    iptables -t nat -A POSTROUTING -s ${DESTINATION%%\:*} -o \
           $IN_INTERFACE -j SNAT --to ${IN_IP%%\/*}

}

if [ $(id -u) -ne 0 ]; then
    echo "You need to be root to run this script"
    exit 1
fi

while true; do
    case "$1" in
      --clear)
          CLEAR_RULES=1
          shift
          ;;
      --protocol|--protocol=*?)
      if [ "$1" = "--protocol" -a -n "$2" ];then
          PROTOCOL="$2"
          shift 2
      elif [ "${1#--protocol=}" != "$1" ]; then
          PROTOCOL="${1$--protocol=}"
          shift 1
      else
          echo "You need to specify protocl (tcp|udp)"
          exit 2
      fi
      ;;

      *)
          break
          ;;
    esac
done

if [ $# -ne 4 ]; then
    usage $0
    exit 2
fi

IN_INTERFACE="$1"
OUT_INTERFACE="$2"
PORT="$3"
DESTINATION="$4"

# Get the incoming interface IP. This is used for SNAT.
IN_IP=$(ip addr show $IN_INTERFACE|\
             perl -nE '/inet\s(.*)\sbrd/ and print $1')


if [ -n "$CLEAR_RULES" ]; then
    iptables -t nat -X
    iptables -t nat -F
    iptables -F
fi

if [ -n "$PROTOCOL" ]; then
    setup_portforwarding $PROTOCOL
else
    setup_portforwarding tcp
    setup_portforwarding udp
fi

Coming to systemd-nspawn I see there is --port option which takes argument in form proto:hostport:destport where proto can be either tcp or udp, hostport and destport are number from 1-65535. This option assumes private networking enabled in the container. I've not tried this option yet but that simplifies quite a lot of thing, its like -p switch used by docker.

Taming systemd-nspawn for running containers

2015-11-09T12:52:00+05:30

I've been trying to run containers using systemd-nspawn for quite some time. I was always bumping to one or other dead end. This is not systemd-nspawn's fault, rather my impatience stopping me from reading manual pages properly lack of good tutorial like article available online. Compared to this LXC has a quite a lot of good tutorials and howto's available online.

This article is my effort to create a notes putting all required information in one place.

Creating a Debian Base Install

First step is to have a minimal Debian system some where on your hard disk. This can be easily done using debootsrap. I wrote a custom script to avoid reading manual every time I want to run debootstrap. Parts of this script (mostly packages and the root password generation) is stolen from lxc-debian template provided by lxc package.

#!/bin/sh

set -e
set -x

usage () {
    echo "${0##/*} [options] <suite> <target> [<mirror>]"
    echo "Bootstrap rootfs for Debian"
    echo
    cat <<EOF
    --arch         set the architecture to install
    --root-passwd  set the root password for bootstrapped rootfs
    EOF
}

# copied from the lxc-debian template
packages=ifupdown,\
locales,\
libui-dialog-perl,\
dialog,\
isc-dhcp-client,\
netbase,\
net-tools,\
iproute,\
openssh-server,\
dbus

if [ $(id -u) -ne 0 ]; then
    echo "You must be root to execute this command"
    exit 2
fi

if [ $# -lt 2 ]; then
   usage $0
fi

while true; do
    case "$1" in
        --root-passwd|--root-passwd=?*)
            if [ "$1" = "--root-passwd" -a -n "$2" ]; then
                ROOT_PASSWD="$2"
                shift 2
            elif [ "$1" != "${1#--root-passwd=}" ]; then
                ROOT_PASSWD="${1#--root-passwd=}"
                shift 1
            else
                # copied from lxc-debian template
                ROOT_PASSWD="$(dd if=/dev/urandom bs=6 count=1 2>/dev/null|base64)"
                ECHO_PASSWD="yes"
            fi
            ;;
        --arch|--arch=?*)
            if [ "$1" = "--arch" -a -n "$2" ]; then
                ARCHITECTURE="$2"
                shift 2
            elif [ "$1" != "${1#--arch=}" ]; then
                ARCHITECTURE="${1#--arch=}"
                shift 1
            else
                ARCHITECTURE="$(dpkg-architecture -q DEB_HOST_ARCH)"
            fi
            ;;
        *)
            break
            ;;
    esac
done



release="$1"
target="$2"

if [ -z "$1" ] || [ -z "$2" ]; then
    echo "You must specify suite and target"
    exit 1
fi

if [ -n "$3" ]; then
    MIRROR="$3"
fi

MIRROR=${MIRROR:-http://httpredir.debian.org/debian}

echo "Downloading Debian $release ..."
debootstrap --verbose --variant=minbase --arch=$ARCHITECTURE \
             --include=$packages \
             "$release" "$target" "$MIRROR"

if [ -n "$ROOT_PASSWD" ]; then
    echo "root:$ROOT_PASSWD" | chroot "$target" chpasswd
    echo "Root password is '$ROOT_PASSWRD', please change!"
fi

It just gets my needs done, if you don't like it feel free to modify or use debootstrap directly.

!NB Please install dbus package in the minimal base install, otherwise you will not be able to control the container using machinectl

Manually Running Container and then persisting it

Next we need to run the container manually. This is done by using following command.

systemd-nspawn -bD   /path/to/container --network-veth \
     --network-bridge=natbr0 --machine=Machinename

--machine option is not mandatory, if not specified systemd-nspawn will take the directory name as machine name, and if you have characters like - in the directory name it translates to hexcode x2d and controlling container with name becomes difficult.

--network-veth specifies the systemd-nspawn to enable virtual ethernet based networking and --network-bridge tells the bridge interface on host system to be used by systemd-nspawn. These options together constitutes private networking for container. If not specified container can use host systems interface there by removing network isolation of container.

Once you run this command container comes up. You can now run machinectl to control the container. Container can be persisted using following command

machinectl enable container-name

This will create a symbolic link of /lib/systemd/system/systemd-nspawn@service to /etc/systemd/system/machine.target.wants/. This allows you to start or stop container using machinectl or systemctl command. Only catch here is your base install should be in /var/lib/machines/. What I do in my case is create a symbolic link from my base container to /var/lib/machines/container-name.

!NB Note that symbolic link name under /var/lib/machines should be same as the container name you gave using --machine switch or the directory name if you didn't specify --machine

Persisting Container Networking

We did persist the container in above step, but this doesn't persist the networking options we provided in command line. systemd-nspawn@.service provides following command to invoke container.

ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth --settings=override --machine=%I

To persist the bridge networking configuration we did in command line, we need the help of systemd-networkd. So first we need to enable the systemd-networkd.service on both container and the host system.

systemctl enable systemd-networkd.service

Now inside the container, interfaces will be named as hostN. Depending on how many interfaces we have N increments. In our example case we had single interface, hence it will named as host0. By default network interfaces will be down inside container, hence systemd-networkd is needed to put it up.

We put the following in /etc/systemd/network/host0.network file inside the container.

[Match]
Name=host0

[Network]
Description=Container wired interface host0
DHCP=yes

And in the host system we just configure the bridge interface using systemd-nspawn. I put following in natbr0.netdev in /etc/systemd/network/

[NetDev]
Description=Bridge natbr0
Name=natbr0
Kind=bridge

In my case I already had configured the bridge using /etc/network/interfaces file for lxc. I think its not really needed to use systemd-networkd in this case. Since systemd-networkd doesn't do anything if network / virtual device is already present I safely put above configuration and enabled systemd-networkd.

Just for the notes here is my natbr0 configuration in interfaces file.

auto natbr0
iface natbr0 inet static
   address 172.16.10.1
   netmask 255.255.255.0
   pre-up brctl addbr natbr0
   post-down brctl delbr natbr0
   post-down sysctl net.ipv4.ip_forward=0
   post-down sysctl net.ipv6.conf.all.forwarding=0
   post-up sysctl net.ipv4.ip_forward=1
   post-up sysctl net.ipv6.conf.all.forwarding=1
   post-up iptables -A POSTROUTING -t mangle -p udp --dport bootpc -s 172.16.0.0/16 -j CHECKSUM --checksum-fill
   pre-down iptables -D POSTROUTING -t mangle -p udp --dport bootpc -s 172.16.0.0/16 -j CHECKSUM --checksum-fill

Once this is done just reload the systemd-networkd and make sure you have dnsmasq or any other DHCP server running in your system.

Now the last part is to tell systemd-nspawn to use the bridge networking interface we have defined. This is done using container-name.nspawn file. Put this file under /etc/systemd/nspawn folder.

[Exec]
Boot=on

[Files]
Bind=/home/vasudev/Documents/Python/upstream/apt-offline/

[Network]
VirtualEthernet=yes
Bridge=natbr0

Here you can specify networking, and files mounting section of the container. For full list please refer the systemd.nspawn manual page.

Now all this is done you can happily do

machinectl start container-name
#or
systemctl start systemd-nspawn@container-name

Resource Control

Now all things said and done, one last part remains. Yes what is the point if we can't control how much resource does the container use. Atleast it is more important for me, because I use old and bit low powered laptop.

Systemd provides way to control the resource using Control interface. To see all the the interfaces exposed by systemd please refer systemd.resource-control manual page.

The way to control the resource is using systemctl. Once container starts running we can run following command.

systemctl set-property container-name CPUShares=200 CPUQuota=30% MemoryLimit=500M

The manual page does say that these things can be put under [Slice] section of unit files. Now I don't have clear idea if this can be put under .nspawn files or not. For the sake of persisting the container I manually wrote the service file for container by copying systemd-nspawn@.service and adding [Slice] section. But if I don't know how to find out if this had any effect or not.

If some one knows about this please share your suggestions to me and I will update this section with your provided information.

Conclusion

All in all I like systemd-nspawn a lot. I use it to run container for development of apt-offline. I previously used lxc where all can be controlled using a single config file. But I feel systemd-nspawn is more tightly integrated with system than lxc.

There is definitely more in systemd-nspawn than I've currently figured out. Only thing is its not as popular as other alternatives and definitely lacks good howto documentation.For now only way out is dig the manual pages, scratch your head, pull your hair out and figure out new possibilities in systemd-nspawn. ;-)

Running Plan9 using 9vx - using vx32 sandboxing library

2015-02-21T12:32:00+05:30

Now a days I'm more and more attracted towards Plan9, an Operating System meant to be the successor of UNIX and created by same people who created original UNIX. I'm always baffled by the simplicity of Plan9. Sadly Plan9 never took off for whatever reasons.

I've been for a while trying to run Plan9, I ran Plan9 on Raspberry Pi model B using 9pi, but I couldn't experiment with it more due to some restrictions in my home setup.

I installed original Plan9 4th Edition from Bell labs (now part of Alcatel-Lucent), I will write about it in on different post. But running virtual machine on my system is again PITA as system is already old (3 and half year). I came across the 9vx which is port of Plan9 for FreeBSD, Linux and Mac OSX by Russ Cox.

I downloaded original 9vx version 0.9.12 from Russ's page linked above. The archive contains a Plan9 rootfs along with precompiled 9vx binaries for Linux, FreeBSD and Mac OS X. I ran the Linux binary but it crashed.

./9vx.Linux -u glenda

I was seeing some illegal instruction error in dmesg. I didn't bother to do more investigation.

A bit of googling showed me Arch Linux's wiki page on 9vx. I got errors trying to compile the original vx32 from rsc's repository but later saw that AUR 9vx package is built from different repository forked from rsc's found here.

I cloned the repository to local and compiled it, I don't really remember if I had installed any additional packages. But if you get error you will know what additional thing is required. After compilation the 9vx binary is found inside src/9vx/9vx. I used this newly compiled 9vx to run the the rootfs I downloaded from Russ's website.

9vx -u glenda -r /path/to/extracted/9vx-0.9.12/

This launches Plan9 and allows you to work inside Plan9. The good part is its not resource hungry and still looks like you have a VM running with Plan9 on it.

But there seems to be a better way to do this directly from plan9 iso from bell labs. It can be found on 9fans list. Now I'm going to try that out too :-). And in next post I will share my experience of using Plan9 on Qemu.

Notes: LXC How-To

2014-12-26T08:11:00+05:30

LXC - Linux Containers allows us to run multiple isolated Linux system under same control host. This will be useful for testing application without changing our existing system.

To create an LXC container we use the lxc-create command, it can accepts the template option, with which we can choose the OS we would like to run under the virtual isolated environment. On a Debian system I see following templates supported

[vasudev@rudra: ~/ ]% ls /usr/share/lxc/templates
lxc-alpine*    lxc-archlinux*  lxc-centos*  lxc-debian*    lxc-fedora*  lxc-openmandriva*  lxc-oracle*  lxc-sshd*    lxc-ubuntu-cloud*
lxc-altlinux*  lxc-busybox*    lxc-cirros*  lxc-download*  lxc-gentoo*  lxc-opensuse*      lxc-plamo*   lxc-ubuntu*

For my application testing I wanted to create a Debian container for my By default the template provided by lxc package creates Debian stable container. This can be changed by passing the option to debootstrap after -- as shown below.

sudo MIRROR=http://localhost:9999/debian lxc-create -t debian \
     -f   container.conf -n container-name -- -r sid

-r switch is used to specify the release, MIRROR environment variable is used to choose the required Debian mirror. I wanted to use my own local approx installation, so I can save some bandwidth.

container.conf is the configuration file used for creating the LXC, in my case it contains basic information on how container networking should b setup. The configuration is basically taken from LXC Debian wiki

lxc.utsname = aptoffline-dev
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0.1
lxc.network.name = eth0
lxc.network.ipv4 = 192.168.3.2/24
lxc.network.veth.pair = vethvm1

I'm using VLAN setup described in Debian wiki: LXC VLAN Networking page. Below is my interfaces file.

iface eth0.1 inet manual

iface br0.1  inet manual
   bridge_ports eth0.1
   bridge_fd 0
   bridge_maxwait 0

Before launching the LXC make sure you run below command

sudo ifup eth0.1
sudo ifup br0.1

# Also give ip to bridge in same subnet as lxc.network.ipv4
sudo ip addr add 192.168.3.1/24 dev br0.1

I'm giving ip address to bridge so that I can communicate with container from my control host, once it comes up.

Now start the container using below command

sudo lxc-start -n container -d -c tty8

We are starting lxc in daemon mode and attaching it to console tty8. If you want, you can drop -d and -c option to start lxc in foreground. But its better you start it in background and attach using lxc-console command shown below.

sudo lxc-console -n container -t tty8

You can detach from console using Ctrl+a q combination and let lxc execute in background.

Its also possible to simply ssh into the running container since we have enabled networking.

Stopping the container should be done using lxc-stop command, but without -k switch (kill) this command never returned. Even with timeout container is not stopped.

sudo lxc-stop -n container

-r can be used for reboot of container. Since I couldn't get clean shutdown I normally attach the console and issue a halt command in container itself. Not sure if this is the right way, but it gets the thing done.

I consider Linux container as a better alternative for spawning a virtual Linux environment instead of running a full blown VM like Virtualbox or VMware.

Note to Self: LVM Shrink Resize HowTo

2014-10-05T12:00:00+05:30

Recently I had to reinstall a system at office with Debian Wheezy and I thought I should use this opportunity to experiment with LVM. Yeah I've not used LVM till date, even though I'm using Linux for more than 5 years now. I know many DD friends who use LVM with LUKS encryption and I always wanted to experiment, but since my laptop is only thing I've and its currently perfectly in shape I didn't dare to experiment it there. This reinstall was golden opportunity for me to experiment and learn something new.

I used Wheezy CD ISO downloaded using jigdo for installation. Now I will just go bit off topic and want to share the USB stick preparation. I have to say this because I had not done installation for quite a while now. Last I did was during Squeeze time so like usual I blindly executed following command.

cat debian-wheezy.iso > /dev/sdb

Surprisingly USB stick didn't boot! I was getting Corrupt or missing ISO.bin. So next I tried using dd for preparing.

dd if=debian-wheezy.iso of=/dev/sdb

Surprisingly this also didn't work and I get same error message as above. This is when I went back to debian manual and looked for installation step and there I found new way!

cp debian-wheezy.iso /dev/sdb

Look at destination, its a device and voilà this worked! This is something new I learnt and I'm surprised how easy it is now to prepare USB stick. But I still didn't get why first 2 methods failed!. If you guys know please do share.

Now coming back to LVM. I used default LVM when disk partitioning was asked, and I used guided partitioning method provided by debian-installer and ended up with following layout

$ lvs
  LV     VG        Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
home   system-disk -wi-ao-- 62.34g
root   system-disk -wi-ao--  9.31g
swap_1 system-disk -wi-ao--  2.64g

So guided partitioning of debian-installer allocates 10G for root and rest to home and swap. This is not a problem but when I started installing required software, I could see root running out of space quickly so I wanted to resize root and give it 10G more, for this I need to reduce the home by 10G for which I need to first unmount the home partition. Unmounting home from running system isn't possible so I booted into recovery assuming I can unmount home there but I couldn't. lsof didn't show any one using /home after searching a bit I found fuser command and it looks like kernel is using /home which is mounted by it.

$ fuser -vm /home
                     USER        PID ACCESS COMMAND
/home:               root     kernel mount /home

So it isn't possible to unmount /home in recovery mode also. Online materials told me to use live-cd for doing this but I didn't have patience to do that so I just went ahead commented /home mounting in /etc/fstab and rebooted!. This time it worked and /home is not mounted on recovery mode. Now comes the hard part resizing home, thanks to TLDP doc on reducing I coud do this with following step

# e2fsck -f /dev/volume-name/home
# resize2fs /dev/volume-name/home 52G
# lvreduce -L-10G /dev/volume-name/home

And now the next part live extending the root partition again thanks to TLDP doc on extending following command did it.

# lvextend -L+10G /dev/volume-name/root
# resize2fs /dev/volumne-name/root

And now important part! Uncomment /home line in /etc/fstab so it will be mounted normally in next boot and reboot! On login I can see my partitions updated.

# lvs
  LV     VG        Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
home   system-disk -wi-ao-- 52.34g
root   system-disk -wi-ao-- 19.31g
swap_1 system-disk -wi-ao--  2.64g

I've started liking LVM more now! :)

Stop messing with my settings Network Manager

2014-07-20T00:39:00+05:30

I use a laptop with Atheros wifi card with ath9k driver. I use hostapd to convert my laptop wifi into AP (Access point) so I can share network with my Nexus 7 and Kindle. This has been working fine for quite some time till my recent update.

After recent system update (I use Debian Sid), I couldn't for some reason convert my wifi into AP so my device can connect. I can't find anything in log nor in hostapd debug messages which is useful to trouble shoot the issue. Every time I start the laptop my wifi card will be blocked by RF-KILL and I have manually unblock (both hard and soft). The script which I use to convert my Wifi into AP is below

#Initial wifi interface configuration
ifconfig "$1" up 192.168.2.1 netmask 255.255.255.0
sleep 2

# start dhcp
sudo systemctl restart dnsmasq.service

iptables --flush
iptables --table nat --flush
iptables --delete-chain
iptables -t nat -A POSTROUTING -o "$2" -j MASQUERADE
iptables -A FORWARD -i "$1" -j ACCEPT

sysctl -w net.ipv4.ip_forward=1

#start hostapd
hostapd /etc/hostapd/hostapd.conf  &> /dev/null &

I tried rebooting the laptop and for some time I managed to convert my wifi into AP, I noticed at same time that Network Manager is not started once laptop is booted, yeah this also started happening after recent upgrade which I guess is the black magic of systemd. After some time I noticed wifi has went down and now I can't bring it up because its blocked by RF-KILL. After checking the syslog I noticed following lines.

Jul 18 23:09:30 rudra kernel: [ 1754.891060] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Jul 18 23:09:30 rudra NetworkManager[5485]: <info> (mon.wlan0): using nl80211 for WiFi device control
Jul 18 23:09:30 rudra NetworkManager[5485]: <info> (mon.wlan0): driver supports Access Point (AP) mode
Jul 18 23:09:30 rudra NetworkManager[5485]: <info> (mon.wlan0): new 802.11 WiFi device (driver: 'ath9k' ifindex: 10)
Jul 18 23:09:30 rudra NetworkManager[5485]: <info> (mon.wlan0): exported as /org/freedesktop/NetworkManager/Devices/8
Jul 18 23:09:30 rudra NetworkManager[5485]: <info> (mon.wlan0): device state change: unmanaged -> unavailable (reason 'managed') [10 20 2]
Jul 18 23:09:30 rudra NetworkManager[5485]: <info> (mon.wlan0): preparing device
Jul 18 23:09:30 rudra NetworkManager[5485]: <info> devices added (path: /sys/devices/pci0000:00/0000:00:1c.1/0000:04:00.0/net/mon.wlan0, iface: mon.wlan0)
Jul 18 23:09:30 rudra NetworkManager[5485]: <info> device added (path: /sys/devices/pci0000:00/0000:00:1c.1/0000:04:00.0/net/mon.wlan0, iface: mon.wlan0): no ifupdown configuration found.
Jul 18 23:09:33 rudra ModemManager[891]: <warn>  Couldn't find support for device at '/sys/devices/pci0000:00/0000:00:1c.1/0000:04:00.0': not supported by any plugin

Well I couldn't figure out much but it looks like NetworkManager has come up and after seeing interface mon.wlan0, a monitoring interface created by hostapd to monitor the AP goes mad and tries to do something with it. I've no clue what it is doing and don't have enough patience to debug that. Probably some expert can give me hints on this.

So as a last resort I purged the NetworkManager completely from the system and settled back to good old wicd and rebooted the system. After reboot wifi card is happy and is not blocked by RF-KILL and now I can convert it AP and use it as long as I wish without any problems. Wicd is not a great tool but its good enough to get the job done and does only what is asked to it unlike the NetworkManager.

So in short

NetworkManager please stop f***ing with my settings and stop acting oversmart.

Exposing function in python module using entry_points, WSME in a Flask webapp

2014-06-07T18:10:00+05:30

The heading might be ambiguous, but I couldn't figure out better heading so let me start by explaining what I'm trying to solve here.

Problem

I have a python module which contains a function which I want to expose as a REST web service in a Flask application. I use WSME for Flask application, which actually needs signature of function in question and problem comes to picture because function to be exposed is foreign to Flask application, it resides in separate python module.

Solution

While reading Julien Danjou's Hackers Guide To Python book I came across the setuptools entry_points concept which can be used to extend existing feature of a tool like plug-ins. So here I'm going to use this entry_points feature from setuptools to provide function in the module which can expose the signature of function[s] to be exposed through REST. Of course this means I need to modify module in question to write entry_points and function for giving out signature of function to be exposed.

I will explain this with small example. I have a dummy module which provides a add function and a function which exposes the add functions signature.

def add(a, b):
    return a + b

def expose_rest_func():
    return [add, int, int, int]

This is stored in dummy/__init__.py file. I use pbr tool to package my python module. Below is content of setup.cfg file.

[metadata]
name = dummy
author = Vasudev Kamath
author-email = kamathvasudev@gmail.com
summary = Dummy module for testing purpose
version = 0.1
license = MIT
description-file =
  README.rst
requires-python = >= 2.7

[files]
packages =
  dummy

[entry_points] =
myapp.api.rest =
  rest = dummy:expose_rest_func

The special thing in above file is entry_points section, which defines function to be hooked into entry_point. In our case entry_point myapp.api.rest is used by our Flask application to interact with modules which expose them. The function which will be got accessing the entry_point is expose_rest_func which gives the function to be exposed its arg types and return types as a list.

If we are looking at only supporting python3 it was sufficient to know function name only and use function annotations in function definition. Since I want to support both python2 and python3 this is out of question.

Now, just run the following command in virtualenv to get the module installed.

PBR_VERSION=0.1 python setup.py sdist
pip install dist/dummy_module-0.1.tar.gz

Now if you want to see if the module is exposing entry_point or not just use entry_point_inspector tool after installing you will get a command called epi if you run it as follows you should note the dummy_module in its output

epi group list
+-----------------------------+
| Name                        |
+-----------------------------+
| cliff.formatter.completion  |
| cliff.formatter.list        |
| cliff.formatter.show        |
| console_scripts             |
| distutils.commands          |
| distutils.setup_keywords    |
| egg_info.writers            |
| epi.commands                |
| flake8.extension            |
| setuptools.file_finders     |
| setuptools.installation     |
| myapp.api.rest              |
| stevedore.example.formatter |
| stevedore.test.extension    |
| wsme.protocols              |
+-----------------------------+

So our entry_point is exposed now, we need to access it in our Flask application and expose the function using WSME. It is done by below code.

from wsmeext.flask import signature

import flask
import pkg_resources


def main():
   app = flask.Flask(__name__)
   app.config['DEBUG'] = True
   for entrypoint in pkg_resources.iter_entry_points('myapp.api.rest'):
       # Ugly but fix is only supporting python3
       func_signature = entrypoint.load()()
       app.route('/' + func_signature[0].__name__, methods=['POST'])(
           signature(func_signature[-1],
               *func_signature[1:-1])(func_signature[0]))
   app.run()

if __name__ == '__main__':
    main()

entry_point myapp.api.rest are iterated using the pkg_resources package provided by setuptools, when I load the entry_point I get back the function to be used which is called in same place to get function signature. Then I'm calling Flask and WSME decorator functions (yeah instead of decorating I'm using them directly over function to be exposed).

The code looks bit ugly at the place where I'm accessing list using slices but I can't help it due to limitation of python2 with python3 there is new packing and unpacking stuff which makes code look bit more cooler see below.

from wsmeext.flask import signature

import flask
import pkg_resources


def main():
    app = flask.Flask(__name__)
    app.config['DEBUG'] = True
    for entrypoint in pkg_resources.iter_entry_points('silpa.api.rest'):
        func, *args, rettype = entrypoint.load()()
        app.route('/' + func.__name__, methods=['POST'])(
        signature(rettype, *args)(func_signature[0]))
    app.run()

if __name__ == '__main__':
    main()

You can access the service at http://localhost:5000/add depending on Accept header of HTTP you will get either XML or JSON response. If you access it from browser you will get XML response.

Usecase

Now if you are wondering what is the reason behind this experience, this is for SILPA Project. I'm trying to implement REST service for all Indic language computing module. Since all these module are independent of SILPA which is a Flask web app I had to find a way to achieve this, and this is what I came up with.

Conclusion

I'm not sure if there is any other approaches to achieve this, if there I would love to hear about them. You can write your comments and suggestion over email

Using WSME with Flask microframework

2014-06-03T20:15:00+05:30

After reading Julien Danjou's I found out WSME (Web Service Made Easy) a Python framework which allows us to easily create Web Services in Python. For SILPA we needed a REST like interface and I thought of giving it try as WSME readily advertised the Flask integration, and this post was born when I read the documentation for Flask integration.

First of all Flask is a nice framework which will right way allow development of REST api for simple purposes, but my requirement was bit complicated where I had to expose function in a separate python modules through SILPA. I think detailed requirement can be part of another post, so let me explain how to use WSME with Flask app.

WSME integration with Flask is done via decorator function wsmeext.flask.signature which expects you to provide it with signature of function to expose. And here is its documentation, basically signature of signature function is

wsmeext.flask.signature(return_type, *arg_types, **options)

Yeah thats all docs have sadly.

So basically exposing is the only thing WSME handles for us here, routing and other stuffs need to be done by Flask itself. So lets consider a example, simple function to add as shown below.

def add(a, b):
    return a + b

For providing REST like service, all you need below code.

from flask import Flask
from wsmeext.flask import signature

app = Flask(__name__)

@app.route('/add')
@signature(int, int, int)
def add(a, b):
    return a + b

if __name__ == '__main__':
       app.run()

So first argument to signature is return type of function and rest arguments are the argument to function to be exposed. Now you can access the newly exposed service by visiting http://localhost:5000/add but don't forget to pass the arguments either via query string or through post. You can restrict methods of access via Flask's route.

So what's the big deal of not having docs right?.. Well fun part began when we use bit more complex return type like dictionaries or lists . Below is modified code I'm using to demonstrate problem I faced during using dict as return type.

from flask import Flask
from wsmeext.flask import signature

app = Flask(__name__)

@app.route('/add')
@signature(dict, int, int)
def add(a, b):
    return {"result": a + b}

if __name__ == '__main__':
    app.run()

Basically I'm returning a dictionary containing result now, for demonstration purpose. When I run the application boom python barked at me with following message.

Traceback (most recent call last):
File "wsme_dummy.py", line 7, in <module>
 @signature(dict, int, int)
File "c:\Users\invakam2\.virtualenvs\wsmetest\lib\site-packages\wsmeext\flask.py", line 48, in decorator
 funcdef.resolve_types(wsme.types.registry)
File "c:\Users\invakam2\.virtualenvs\wsmetest\lib\site-packages\wsme\api.py", line 109, in resolve_types
 self.return_type = registry.resolve_type(self.return_type)
File "c:\Users\invakam2\.virtualenvs\wsmetest\lib\site-packages\wsme\types.py", line 739, in resolve_type
 type_ = self.register(type_)
File "c:\Users\invakam2\.virtualenvs\wsmetest\lib\site-packages\wsme\types.py", line 668, in register
 class_._wsme_attributes = None
TypeError: can't set attributes of built-in/extension type 'dict'

After going through code from files involved in above traces this is what I found

wsmeext.flask.signature inturn uses wsme.signature which is just alias of wsme.api.signature.
Link in documentation in sentence See @signature for parameter documentation is broken and should actually link to wsme.signature in docs.
wsme.signature actually calls resolve_type to check on types of return and arguments. This function checks if types are instance of dict or list in such cases it creates instances of wsme.type.DictType and wsme.type.ArrayType respectively with values from the argument.
When I just passed built-in type dict the control went to else part which just passed the type to wsme.type.Register.registry function which tries to set the attribute _wsme_attribute which actually raises TypeError as we can't set attribute for built-in types.

So by inspecting code of wsme.type.Registry.resolve_type and wsme.type.Registry.register its clear that what signature expects when arguments or return type is dictionary/list is instance of dictionary/list with type of value in it. May be sentence is bit vague but I'm not sure how to put it more clearly, as an example in our case add function returns dictioanry with key as string and value as int, so return type argument for signature will be {str: int}. Similarly if you return array with int values it will be [int].

With above understanding our add function now looks like below.

@signature({str: int}, int, int)
def add(a, b):
   return {'result': a + b}

and now code worked just fine!. What I couldn't figure out here is there is no way to have tuple as return value or argument, but I guess that is not big deal.

So immidiate task for me after finding this is fix the link in documentation to point to wsme.signature and probably put some note some where in documentation about above finding.

Enabling DNSSEC for copyninja.info

2014-05-13T20:30:00+05:30

Recently I've been seeing lot of posts about DNSSEC on Internet and I thought I should configure my domain to be secured by DNSSEC.

copyninja.info domain is now secured with DNSSEC you can verify this by DNSSEC analyzer by Verisign and DNSViz online tool or by installing DNSSEC validator addon for your browser.

There are good amount of tutorials and guides available to enable DNSSEC for your domain, still I want to note down steps I followed to here for the record (of course it will be helpful for me if I forget it ;-))

First step will be installing bind9 and dnssec-tools package, if you use aptitude installing dnssec-tools will pull down the bind9 unless you have configured aptitude to not install the Recommends.

Next setting up the zone file for your domain, for this first make a copy of /etc/bind/db.local as /etc/bind/db.example.com, replace example.com with your domain name. Now you need to add your zone records to the zone file.

Next edit the /etc/bind/named.conf.local file and add following lines

zone "example.com" {
    type master;
    file "/etc/bind/db.copyninja.info";
    allow-transfer {secondary;};
};

Here replace secondary with your secondary DNS servers, if you don't have one you can ommit this but its always recommended to have secondary DNS servers for a zone, in cases when primary fails. After this we need to enable DNSSEC on bind, this is done by editing the file /etc/bind/named.conf.options. Add following lines into options section.

dnssec-validation yes;
dnssec-enable yes;
dnssec-lookaside auto;

A more explanation on this can be found on Linux Journal article.

Now its time to create DNSSEC keys and sign your zone, more about different DNSSEC keys and records can be found in the Linux Journal Article on implementation.

I used zonesigner utility from dnssec-tools which does job of signing and including KSK and ZSK keys into bind configuration which otherwise should be done manually. Here is the command line I used for generating keys, thanks to Jonas for this.

mkdir -p /etc/bind/keys
zonesigner -algorithm RSASHA256 -keydirectory /etc/bind/keys\
       -dsdir /etc/bind/keys -archivedir /etc/bind/keys/archive \
       /etc/bind/db.example.com

Here we store our keys into /etc/bind/keys directory, and use RSASHA256 algorithm for key generation which is more stronger than the default used RSASHA1 (atleast thats what Jonas told me). This will create ZSK and KSK for the signing zone and creates a signed zone file db.example.com.signed in same directory as original zone file. Now all you need to do is replace the zone file from db.example.com to db.example.com.signed in file directive with your named.conf.local file.

Note that this keys expire after 30 days so you need to resign your zone before 30 days. For resigning just run zonesigner from /etc/bind/keys. You can setup cron job to do this periodically.

zonesigner -zone example.com /path/to/db.example.com

Our signed zone is ready but we are not done yet! For DNSSEC to work others should trust your signed record for this you need to register your public keys with registrar for your domain and this can be done via your domain provider (in my case this is Gandi).

You need to check your domain name providers documentation on how to do this. For Gandi users there is a nice documentation.

Reference

Note to Self: How to use url_for in Flask application

2014-05-02T11:50:00+05:30

url_for is normally used to avoid hard coding of URL in Flask based webapps. I've been using it in SILPA port written using Flask by. But this was written long back and written as POC to show for Santhosh author of SILPA app which even managed to go into production :-). Recently I started organizing the code and started using flask.Blueprint which was previously written using flask.views.MethodView class and here I started facing problem with old templates. I'm just gonna explain what is the difference when using url_for with MethodView and Blueprint.

Lets have some sample code for MethodView first, this code is from Flask documentation.

from flask.views import MethodView

class UserAPI(MethodView):

       def get(self):
           users = User.query.all()
           ...

       def post(self):
           user = User.from_form_data(request.form)
           ...

app.add_url_rule('/users/', view_func=UserAPI.as_view('users'))

Here we have a view which is derived from MethodView which has logic on how to handle the requests. We use add_url_rule to register rule to handle /users/ url end point and we pass the class as view function which is done by as_view class method and we can refer this method in our Jinja2 templates using name users. So a statement like

url_for('users')

in our templates will be converted to /users/ URL when page is rendered to client by Flask.

Now when I replaced the MethodView in favor of Blueprint I started getting up werkzeug.routing.BuildError thrown on my face and I had no clue why!. Yeah I know I'm bad at reading documentation but even after reading documentation I was still thinking that

url_for('/')

should return me a proper URL and I was wondering why its failing. Finally after re-reading documentation for url_for it was becoming clear to me, url_for definition looks like below

flask.url_for(endpoint, **values)

Here endpoint is actually a function which is supposed to be serving the URL and **values is the arguments for this function. The URL in question should be defined in python code using decorator. In my case following is the new function serving the web pages for SILPA.

bp = Blueprint('frontend', __name__)


@bp.route(_BASE_URL, defaults={'page': 'index.html'})
@bp.route(_BASE_URL + '<page>')
def serve_pages(page):
       if page == "index.html":
          return render_template('index.html', title='SILPA',
                               main_page=_BASE_URL,
                               modules=_modulename_to_display)
       elif page == "License":
          return render_template('license.html', title='SILPA License',
                                main_page=_BASE_URL,
                                modules=_modulename_to_display)
       elif page == "Credits":
           return render_template('credits.html', title='Credits',
                                main_page=_BASE_URL,
                                modules=_modulename_to_display)
       elif page == "Contact":
           return render_template('contact.html', title='Contact SILPA Team',
                                main_page=_BASE_URL,
                                modules=_modulename_to_display)
       else:
           # modules requested!.
           if page in _display_module_map:
               return render_template(_display_module_map[page] + '.html',
                                    title=page, main_page=_BASE_URL,
                                    modules=_modulename_to_display)
           else:
               # Did we encounter something which is not registered by us?
               return abort(404)

You can ignore function code but just note the decorators, here I'm registering the function serve_pages with page as argument for URL patterns / and /<page>, _BASE_URL here is mount point of application it can be just / or /mountpoint depending on that URL registered changes. Now I need to modify code for all url_for in my template to look like below

url_for('.serve_pages', page='/License') # for /License
url_for('.serve_pages') # which will turn in to /index.html

The . in front of function is for referring current Blueprint, in my case the Flask will consider it as frontend.serve_pages as function name and generates appropriate URL at run time.

So my fault was misunderstanding endpoint argument as URL endpoint but where as its actually function name supposed to serve the page. But when using MethodViews I can simply convert class to a function with my preferred name just like UserAPI.as_view('/') so url_for('/') just works.

Loading Python modules/plug-ins at runtime

2014-04-07T21:55:00+05:30

Some times it is desired to load arbitrary python files or pre- installed python modules during application run time.I had encountered 2 such usecases, one is in SILPA application and other is dictionary-bot which I was refactoring recently.

Case 1: Loading installed python module

In case of SILPA I need to load pre-installed modules and here is the old code , that is a bit hacky code I copied from Active State Python recipies. I found a bit better way to do it using importlib module as shown below.

from __future__ import print_function
import sys
import importlib

def load_module(modulename):
    mod = None
    try:
        mod = importlib.import_module(modulename)
    except ImportError:
        print("Failed to load {module}".format(module=modulename),
                     file=sys.stderr)
    return mod

Here importlib itself takes care of checking if modulename is already loaded by checking sys.modules[modulename], if loaded it returns that value, otherwise it loads the module and sets it to sys.modules[modulename] before returning module itself.

Case 2: Loading python files from arbitrary location

In case of dictionary bot my requirement was bit different, I had some python files lying around in a directory, which I wanted to plug into the bot during run time and use them depending on some conditions. So basic structure which I was looking was as follows.

      pluginloader.py
      plugins
      |
      |__ aplugin.py
      |
      |__ bplugin.py

pluginloader.py is the file which needs to load python files under plugins directory. This was again done using importlib module as shown below.

import os
import sys
import re
import importlib

def load_plugins():
    pysearchre = re.compile('.py$', re.IGNORECASE)
    pluginfiles = filter(pysearchre.search,
                           os.listdir(os.path.join(os.path.dirname(__file__),
                                                 'plugins')))
    form_module = lambda fp: '.' + os.path.splitext(fp)[0]
    plugins = map(form_module, pluginfiles)
    # import parent module / namespace
    importlib.import_module('plugins')
    modules = []
    for plugin in plugins:
             if not plugin.startswith('__'):
                 modules.append(importlib.import_module(plugin, package="plugins"))

    return modules

Above code first searches for all python file under specified directory and creates a relative module name from it. For eg. file aplugin.py will become .aplugin. Before loading modules itself we will load the parent module in our case plugins, this is because relative imports in python expects parent module to be already loaded. Finally for relative imports to work with importlib.import_module we need to specify parent module name in package argument. Note that we ignore files begining with __, or specifically we don't want to import __init__.py, this will be done when we import parent module.

The above code was inspired from a answer on StackOverflow, which uses imp module, I avoided imp because its been deprecated from Python 3.4 in favor of importlib module.

Working around type system of Go with unsafe

2014-03-12T22:22:00+05:30

Go language has a strong type system unlike C, and some times this will be head ache when we want to interact with C data types with Cgo or just to convert a Go type to lets say byte slice. I recently faced the same problem and after poking around things, I learned the Go provides unsafe package which can be used to work around the Go's type system.

Problem

Recently I started using Cgo to use some C library I had to write some tools for development and testing. The reason I chose Go for this was prototyping and writing some quick tools is much easier in Go than done in C.

The C library had some function which takes pointer to array types and fills it with some values. The problem I was facing here was how to create a C array in Go. Go does have array but its largely used as internal representation for much efficient type called slice and I can't directly cast a byte slice into an C array.

Second problem I had was I had to store arbitrary Go types like float (float32,float64) and int (int32, int64) etc. into C array.

So in brief, the problems that needed to solve are

Find a way to convert byte slice from Go into C array and vice versa.
Find a way to convert and store Go types into a C array.

Solution

Basically C array are sequence of memory location which can be statically or dynamically allocated. In Cgo its possible to access C standard library functions for memory allocation, so why not use it. The memory allocation function then returns the pointer to starting of allocated memory, we can use this pointer to write Go's bytes into the memory location and bytes from memory location into Go's slice.

The pointer returned by C allocation functions are not directly usable for memory dereferencing in Go, here is where unsafe package kicks in. We will cast the return of C allocation function as unsafe.Pointer type and from the documentation of unsafe package,

A pointer value of any type can be converted to a Pointer.

A Pointer can be converted to a pointer value of any type.

A uintptr can be converted to a Pointer.

A Pointer can be converted to a uintptr.

So we can then cast unsafe.Pointer to uintptr which is the Go type which is large enough to hold any memory address in Go and can be used for pointer arithmetic just like we do in C (of course with some more castings).

Below I'm pasting a simplified code in C which I wrote for this post.

#ifndef __BYTETEST_H__
#define __BYTETEST_H__

typedef unsigned char UBYTE;

extern void ArrayReadFunc(UBYTE *arrayout);
extern void ArrayWriteFunc(UBYTE *arrayin);

#endif

#include "bytetest.h"
#include <stdio.h>
#include <string.h>

void ArrayReadFunc(UBYTE *arrayout)
{
     UBYTE array[20] = {1, 2, 3, 4,5, 6, 7, 8, 9, 10,
                        11, 12, 13, 14, 15, 16, 17,
                        18, 19, 20};
     memcpy(arrayout, array, 20);
}

void ArrayWriteFunc(UBYTE *arrayin)
{
     UBYTE array[20];

     memcpy(array, arrayin, 20);

     printf("Byte slice array received from Go:\n");
     for(int i = 0; i < 20; i ++){
             printf("%d ", array[i]);
     }

     printf("\n");
}

Functions are written just for this post and they don't really do anything. As you can see ArrayReadFunc takes a pointer to array and fills it with content of another array using memcpy. Function ArrayWriteFunc on other hand takes pointer to array and copies its content to internal array. I've added print logic to ArrayWriteFunc just to show that values passed from Go are making it here.

Below is the Go code which uses the above C files passes byte slice to get value out of C code and array made of byte slice to C function to send values in.

package main

/*
#cgo CFLAGS: -std=c99

#include "bytetest.h"
#include <stdlib.h>
*/
import "C"

import (
     "fmt"
     "unsafe"
)

func ReadArray() unsafe.Pointer {
       var outArray = unsafe.Pointer (C.calloc(20,1))
       C.ArrayReadFunc((*C.UBYTE)(outArray))

       return outArray
}

func WriteArray(inArray unsafe.Pointer) {
       C.ArrayWriteFunc((*C.UBYTE)(inArray))
}

func CArrayToByteSlice(array unsafe.Pointer, size int) []byte {
       var arrayptr = uintptr(array)
       var byteSlice = make([]byte, size)

       for i := 0; i < len(byteSlice); i ++ {
               byteSlice[i] = byte(*(*C.UBYTE)(unsafe.Pointer(arrayptr)))
               arrayptr ++
       }

       return byteSlice
}

func ByteSliceToCArray (byteSlice []byte) unsafe.Pointer {
       var array = unsafe.Pointer(C.calloc(C.size_t(len(byteSlice)), 1))
       var arrayptr = uintptr(array)

       for i := 0; i < len(byteSlice); i ++ {
              *(*C.UBYTE)(unsafe.Pointer(arrayptr)) = C.UBYTE(byteSlice[i])
              arrayptr ++
       }

       return array
}

func main(){
        carray := ReadArray()
        defer C.free(carray)

        carraybytes := CArrayToByteSlice(carray, 20)

        fmt.Println("C array converted to byte slice:")
        for i := 0; i < len(carraybytes); i ++ {
                fmt.Printf("%d ", carraybytes[i])
        }

        fmt.Println()

        gobytes := []byte{21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
                31, 32, 33, 34, 35, 36, 37, 38, 39, 40}
        gobytesarray := ByteSliceToCArray(gobytes)
        defer C.free(gobytesarray)

        WriteArray(gobytesarray)
}

Functions ReadArray and WriteArray are just wrapper to the calls to C counter parts ArrayReadFunc and ArrayWriteFunc. ReadArray returns unsafe.Pointer which is allocated C array and should be freed by caller. WriteArray takes unsafe.Pointer which is pointing to memory location containing C array.

Now the functions of interest are CArrayToByteSlice and ByteSliceToCArray. It should be pretty clear from the above code to understand what is happening in these functions. Still I will just put explain them briefly.

ByteSliceToCArray allocates a C array using calloc from C standard library. It then creates a uintptr, a pointer type in Go which is used to dereference the each memory location and store bytes from the input byte slice in them.

CArrayToByteSlice on other hands creates a uintptr type by casting input unsafe.Pointer type and then uses this pointer type to dereference values from memoy and store it in byte slice with suitable casting.

So lets build the code and run it and see the output:

C array converted to byte slice:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Byte slice array received from Go:
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

So yes it actually works and values are moving across C and Go. This solves first problem in hand next is converting arbitrary Go types into byte slices.

There are many cases where we would like to convert an arbitrary Go types like (int, float) into bytes. One such use case I found was when writing a TCP client for communicating with a Server written using C speaking custom protocol. Here I'm just going to show how to convert types like float, int to byte slice, I've not tried converting structures but it is certainly possible.

Below is the function which can convert int32,float32 into byte slice it can also be extended for other types.

func CopyValueToByte(value interface{}) []byte {
     var valptr uintptr
     var slice []byte

     switch t := value.(type) {
     case int32:
             i := value.(int32)
             valptr = uintptr(unsafe.Pointer(&i))
             slice = make([]byte, unsafe.Sizeof(i))
     case float32:
             f := value.(float32)
             valptr = uintptr(unsafe.Pointer(&f))
             slice = make([]byte, unsafe.Sizeof(f))
     default:
             fmt.Fprintf(os.Stderr,"Unsupported type: %T\n", t)
             os.Exit(1)
     }

     for i := 0; i < len(slice); i++ {
             slice[i] = byte(*(*byte)(unsafe.Pointer(valptr)))
             valptr++
     }

     return slice
}

This function is generic which can take various types of value. First we will use Go's type assertion to determine the type and creates a uintptr pointer for the value and allocates byte slice depending on the size of the value as calculated using unsafe.Sizeof. Later it uses the pointer to dereference value from memory location and copies each byte into byte slice. The idea used here is every type is represented as certain number of bytes in the memory. Below is the entire program.

package main

import (
     "fmt"
     "unsafe"
     "os"
)

func CopyValueToByte(value interface{}) []byte {
     var valptr uintptr
     var slice []byte

     switch t := value.(type) {
     case int32:
             i := value.(int32)
             valptr = uintptr(unsafe.Pointer(&i))
             slice = make([]byte, unsafe.Sizeof(i))
     case float32:
             f := value.(float32)
             valptr = uintptr(unsafe.Pointer(&f))
             slice = make([]byte, unsafe.Sizeof(f))
     default:
             fmt.Fprintf(os.Stderr,"Unsupported type: %T\n", t)
             os.Exit(1)
     }

     for i := 0; i < len(slice); i++ {
             slice[i] = byte(*(*byte)(unsafe.Pointer(valptr)))
             valptr++
     }

     return slice
}

func main() {
     a := float32(-10.3)

     floatbytes := CopyValueToByte(a)

     fmt.Println("Float value as byte slice:")
     for i := 0; i < len(floatbytes); i++ {
             fmt.Printf("%x ", floatbytes[i])
     }

     fmt.Println()

     b := new(float32)
     bptr := uintptr(unsafe.Pointer(b))

     for i := 0; i < len(floatbytes); i++ {
             *(*byte)(unsafe.Pointer(bptr)) = floatbytes[i]
             bptr++
     }

     fmt.Printf("Byte value copied to float var: %f\n", *b)

}

The above conversion can also be achieved using encoding/binary package provided by Go. But its been told to me that it makes things pretty slow.

Conclusion

So goes unsafe.Pointer is really powerful thing which allows us to work around the Go's type system but as package documentation says it should be used with care

PS: I'm not really sure if its recommended to use allocation functions from C standard library, I will wait for expert gophers to comment on that.

Installing and Configuring NetBSD on Qemu in Debian

2014-03-01T22:10:00+05:30

First step will be getting a NetBSD ISO image for installation purpose. It can be downloaded from here.

Next step will be creating a disk image for installing NetBSD. This is done using qemu-img tool like below

$ qemu-img create -f raw netbsd-disk.img 10G

Image format used is raw and size is specified as last arugment. Tune the size as per your need.

To start the installation run the following command

$ qemu-system-x86_64 -m 256M -hda netbsd-disk.img -cdrom \
             NetBSD-6.1.3-amd64.iso -display curses -boot d \
             -net nic -net user

So this is using user mode networking so you won't be able to have internet access during installation. I couldn't figure out on how to get network working during installation so I configured network after installation.

Once you run above command you will be given with 4 options as follows.

install netbsd
install netbsd with ACPI disabled
install netbsd with ACPI and SMP disabled
drop to boot shell

Even though I first installed using the option 1 I couldn't get it boot after installation so had to reinstall with option 2 and it works fine. I'm not gonna explain each step of installation here because the installer is really simple and straight forward! I guess the NetBSD installer is the first simplest installer I have encountered from the day I started with Linux. Its simple but powerful and gets job done very easily, and I didn't read manual for installation before using it.

Enabling Networking

This section involves mixture of configuration in Debian host and inside NetBSD to get the network working. The wiki page of Debian on Qemu. helped me here.

To share the network with Qemu there are 2 possiblities

Bridged networking between host and guest using bridge-utils
Using VDE (Virtual Distributed Ethernet)

The option 1 which is explained in wiki linked above didn't work for me as I use CDC Ether based datacard for connecting to Internet which gets detected as eth1 on my machine. When bridging happens between tap0 and eth1 I end up loosing Internet on my host machine. So I selected to use VDE instead.

First install packages vde2 and uml-utilities once done edit the /etc/network/interfaces file and add following lines:

auto vdetap
iface vdetap inet static
   address 192.168.2.1
   netmask 255.255.255.0
   vde2-switch -t vdetap

We can use dnsmasq as a DHCP server for the vdetap interface, but I'm not gonna explain the configuration of dnsmasq here. Run the below command to get vdetap up

modprobe tun
ifup vdetap
/etc/init.d/dnsmasq restart
newgrp vde2-net # run as user starting Qemu VM's

I couldn't get successful output for newgrp command, I was getting some crypt: Invalid argument output. But I could still get network working on NetBSD so I considered to ignore that for now.

Now start the NetBSD qemu instance running using following command

$ qemu-system-x86_64 -m 256M -hda \
             /mnt/kailash/BSDWorld/netbsd-disk.img \
             -net nic -net vde,sock=/var/run/vde2/vdetap.ctl -display curses

Once the system is up login using root user, NetBSD will warn you for this and suggest to create another user but for now ignore it. To find the network interface in NetBSD just run the usual ifconfig command. In my case interface is named wm0.

First step will be configuring the IP address for your interface and setting up the gateway route. Run the below command for this purpose

# ifconfig wm0 192.168.2.2 netmask 255.255.255.0
# route add default 192.168.2.1

Note that I added gateway as IP address of vdetap on my host machine. Now try pinging the host and even you can try ssh to host system.

But note that this is not persistent over the reboots and for some reason I didn't yet figure out how to make NetBSD get address over DHCP from my host machine. I will update once I figure it out. Now to make the connection address persistent over reboots you need to create a file by name /etc/ifconfg.<interface>. Replace interface with a proper interface on your running NetBSD. In my case this file is /etc/ifconfig.wm0 and has following content.:

192.168.2.2 netmask 0xffffff00 media autoselect

Set the DNS server as host by adding file /etc/resolv.conf with following content.:

nameserver 192.168.2.1

After this you need to do NAT/masquerading using iptables. Just copy following script to a file and execute it as root

#!/bin/sh

# Restart the dnsmasq
/etc/init.d/dnsmasq restart

# Set nat rules in iptables
iptables --flush
iptables --table nat --flush
iptables --delete-chain
iptables --table nat --delete-chain

# Replace accordingly usb0 with ppp0 for 3G
iptables --table nat --append POSTROUTING --out-interface eth1 -j MASQUERADE
iptables --append FORWARD --in-interface vdetap -j ACCEPT

# Enable IP forwarding in Kernel
sysctl -w net.ipv4.ip_forward=1

With the above setup you will be able to get DNS resolution even after you reboot the Qemu instance but Internet connection will not work untill you run the route command I mentioned above. I still didn't figure out how to persist route but I will update it here once I figure it out.

Note that you won't be able to SSH as root to NetBSD (may be its configured to not allow by default), so you would need to create a normal user before trying to SSH from host to guest. Also make sure you add the user to wheel group to allow him execute su command.

So now I've NetBSD running on my laptop with mere 256M ram and its amazingly fast even at such low RAM. I've created a new user and can SSH into it from host machine and use it just like I use a server!. I will put up the notes of my BSD adventure here periodically.

The feeling of using a BSD is amazing :-)

Update: I forgot to add masquerading step, I've added it now.

Gzipped response for CSS/JS with Apache and mod_uwsgi

2014-03-01T09:00:00+05:30

So it may sound normal to receive compressed response for CSS/JS file from Apache2 but what I faced is completely different behavior, here I saw not just body of response but also header are getting compressed and send to browser and browser was unable to interpret the response there by leading page rendered without CSS and JS.

That was long story short, so let me explain in detail the case and what I found out.

I have hosted SILPA on my VPS using uWSGI. I had used till date using libapache2-mod-proxy-uwsgi plugin for which uWSGI should be using network socket and in Apache2 config I need ProxyPass directive pointing to uwsgi://host:port and everything used to work out of box. There is a drawback for using network socket in uWSGI that is limited amount of ports (65535) and possible clash with other services if we use wrong port number, additionally network sockets will be slower compared to the file socket so yesterday I thought of changing it to file socket and removed socket directive in the ini file for uWSGI application container for SILPA and here is where all problems started. My updated configuration can be seen in our documentation page that is if you are interested to look my configuration file :-).

Problem Reporting

We got selected to Gsoc again this year and we have lot of students jumping in our channel #silpa on irc.freenode.net and yesterday night some students reported they are facing 500 internal server error and subsequently others mentioned there is no theme being rendered. I checked and found everything was fine for me only to notice that my browser was using cached content, and when I clear cache there I go everything vanished, no theme no css no javascript just plain html!!!.

Investigation

So I thought I should investigate this and fired up firebug on my iceweasel to observe the network traffic. And below is what I observed in firebug network console!.

So you can see no response header in the above image but there is a response and below how is response content visible on firebug.

Still weird part was when using wget/curl directly to get the CSS I was getting correct reply and the file, which is puzzling.

So I went ahead opened the CSS url directly using browser and saved the resulting file. When I used file command on it, output suggested the saved file in gzip compressed!. I went ahead uncompressed it and opened the file and inside it I found response header and the response body! So what just happened? Did apache some how manage to compress entire respone not just response body?

The more puzzling question was why did it work when I was using mod_proxy_uwsgi and failed when I switched to mod_uwsgi. I had no clue at this point why this behavior is coming up.

Possible Solution

I was still not sure how to resolve this, and started searching in the network but nothing was coming up in search result which can explain me this. And finally I stumbled on a link which is totally unrelated but I saw the word mod_deflate in the content and wandered on my system to see if its enabled. Yes mod_deflate was enabled I just opened the conf file and saw following.

<IfModule mod_deflate.c>
       # these are known to be safe with MSIE 6
       AddOutputFilterByType DEFLATE text/html text/plain text/xml

       # everything else may cause problems with MSIE 6
       AddOutputFilterByType DEFLATE text/css
       AddOutputFilterByType DEFLATE application/x-javascript application/javascript application/ecmascript
       AddOutputFilterByType DEFLATE application/rss+xml
</IfModule>

Interesting so it suggests to compress css, javascript files etc. So I thought possibly its compressing the result of mod_uwsgi, yeah possibly so why not try disabling mod_deflate and check if it works well and good if not I'm not gonna loose anything. So I disabled mod_deflate and voila! Everything is working fine!

What might be the reason for this behavior?

Well I'm not exactly sure but here is my assumption. The whole behavior depends on internal implementation of mod_uwsgi and mod_proxy_uwsgi module. To confirm this I switched back to use mod_proxy_uwsgi module and enabled mod_deflate and observed request response in Firebug network console and below is what I observed.

So response is still gziped but this time only response body was gziped and Content-Encoding field was set properly which makes browser to properly uncompress the response body and use it.

So it seems like there is a difference between mod_uwsgi and mod_proxy_uwsgi implementation. mod_uwsgi was sending the response along with response headers which was compressed by mod_deflate there by rendering browser helpless to interpret the response. But mod_proxy_uwsgi seems to be only sending the response content without response body and this content was later compressed by mod_deflate and proper response headers were set by apache before sending it to browser.

So now whose fault is this? Is it the bug in mod_uwsgi or, mod_deflate not being able to exclude response headers from compression? I've no clue! If you have a clue and you want to share it with me, please consider writing to me over email. My details are available in Contacts

First experience of driving in Bangalore

2014-02-22T17:24:00+05:30

So finally I did it, I managed to ride the 2-wheeler in Bangalore roads!. Officially I've got a license to drive 2 and 4 wheelers back in 2011 but I was never confident enough to ride the vehicle on road and in traffic. Partially this is because I don't own a vehicle and I've never practiced driving, and partially because of road and traffic conditions in India I was bit afraid to do so.

So today morning my roommate and me planned to go to temple in the early morning. We got up at 530 in the morning and we were ready to drive to Mahalaxmi Layout at 615. Since during this time traffic is very less I asked my roommate shall I drive and he agreed!.

So I drove the bike from Malleshwaram to Mahalaxmi Layout and it was fun. I did had hick up in uphill driving as my mind was not co-ordinating well with breaks and gear. I did have hickups in first gear too but I guess this will be alright once I get hold of things.

I'm still not having enough confidence to drive during the peak hours because people are not at all following traffic rules. They just change the lane without any hints or just come to middle of road from side roads without looking at other side to see if there is a vehicle is coming from other side of the road (yes they don't have courtesy to wait). And finally they can shout at persons who is actually following rules. In general very less people have traffic discipline in India. But I will adapt to this once I start driving so I'm now all set to get my 2 wheeler soon and ride on!

Kontalk FLOSS alternative for Whatsapp and Co.

2014-02-21T19:51:00+05:30

So Whatsapp has been acquired by Facebook and this news is still hot and people are discussing it all over the twitterverse. So I took this opportunity to stop using Whatsapp and removed it from my phone. Possibly I could have deleted my account but who cares.

Anyway I've been searching for a secure and FLOSS alternative for Whatsapp for quite some time now, few days ago I found about Telegram but after reading post by Tincho on planet Debian, I decided not to use it. Recently while going through the talk list for fossmeet.in I found link for Kontalk in a privacy awareness talk proposal by Praveen and thought I should give it a try. So below is my first feelings about Kontalk.

Installation and Activation

Kontalk can be installed from playstore for verification purpose it requests your phone number and country code and request for verification. This should send a SMS with a code which should be entered in the text box given and app is ready to use. There is also a possibility to use pre-existing verification code (if you got one from developer directly, read below for details).

I did see some glitches like I never got the SMS delivered to my phone after 2 attempts and a days wait. Then I went ahead and reported a bug and the main developer was quick to respond. After a discussion it was noticed that SMS was blocked by spam filters. He also mentioned its tough to get SMS delivered to India. He was kind enough to provide me with a verification code and I used the Use existing code option to enter it and get Kontalk activated.

The SMS delivery inconsistency is still present for India (and may be other nations too) some people get code immidiately others may be after couple of days and some might not. Upstream is already working on a possible workaround.

User Experience

Now coming to usage part, the UI is neat and clean, I won't say super polished as Whatsapp or other popular apps but its really neat and easy to use. Some points which I like are

Ability to hide presence, so others won't know you are online or offline. (unlike Whatsapp which advertises last seen)
Encrypted messages and ability to optin or optout.
Encrypted status messages! Only user with your phone number can see your status. (Cool isn't it).
Manually requesting to find contacts who already uses kontalk!. Right it doesn't read your contact list without your permission you need to refresh to check who in your contacts is using Kontalk.
Attach and smiley options in the top right corner of Chat window which allows easy accessing unlike keyboard - smiley switching of Whatsapp.
No automatic download of media contents which is shared. Yes by default it doesn't download any picture or video automatically if you want to see something click on it and select download.
Running your own server for Kontalk! Now thats something which is interesting for people who doesn't want to host their data on some other peoples infrastructure. Code for server is available at xmppserver repo.

But there are some rough edges also but I'm sure this can be improved. But some points which I noticed are

Contact name disappears and only number is displayed. This is something which happened with one of my contact so I'm not really sure its a bug.
My friend noticed all his existing contacts suddenly vanished when he refreshed contacts list. Again this is possible bugs and we are considering reporting it upstream.
No group chats yet. I don't see a option to do that yet.
Attachment at the moment is restricted only to pictures (and video? never tried) and uploading takes quite sometime and sometime hangs forever.

So I'm considering forwarding these to upstream and help them by providing enough data so these can be fixed.

Technical Side

All code for client server and protocol specs are available under GPL-v3 at the Kontalk project site. Server software is written in Python and I guess uses XMPP (but I've not cross verified). Server also uses MySQL as database. These can be hosted on our own servers but possibly needs more than that like SMS sending options etc.

Conclusion

In my views Kontalk can become a great alternative for Whatsapp and co from Free Software world and I encourage every one to give it a try which will be the first step to help improving it.

Disclaimer: I'm not a privacy or security expert so whatever I shared above are what I noticed and experts may see something different than this. In any case I welcome comments and suggestions.

Friendica instance on my VPS is down

2014-02-08T20:31:00+05:30

I started running Friendica instance on my VPS. With help of Jonas Smedegaard I managed to run Friendica in a uWSGI container. The site was running at samsargika.copyninja.info and is no longer accessible.

Since VPS itself was running Debian Wheezy I couldn't run uWSGI with PHP support on it. (support for PHP in uWSGI landed after Wheezy). But Jonas was kind enough for me to provide a backported version.

Recently Wheezy got a security update for PHP and thats where all the problem started. The backported uwsgi-plugin-php was not recompiled to use security updated PHP and I couldn't upgrade the things. After few days I noticed first freeze on my VPS and I had to reboot the VPS to get it online again. The fact I noticed was uWSGI being killed due to a OOM in syslog but I didn't explore much and consulted Jonas to get a updated uWSGI. But that didn't happen as Jonas himself is facing some problem with uWSGI builds. While again checking with aptitude for upgrade I accidentally confirmed removal of uwsgi-plugin-php for getting security updates :-/. But nothing happened to my running service as upgrade of libs in Debian doesn't cause the restart of all services which are using that lib (desired effect is restart of service but I don't know the side affects involved).

Second freeze happened yesterday and on the reboot uwsgi-plugin-php was missing there by taking my Friendica instance down. More closer investigation showed the same OOM but this time I noticed each OOM occurred just after cronjob was running poller.php a script which is actually causing all federation in Friendica. So it was clear there is something wrong either in poller.php or in my setup which was making it eat memory and freeze my VPS.

But I also found some stupidity I did during configuration which Jonas also pointed me out.

Installing cronjob inside crontab rather than cron.d
Installing poller.php crontab for root user :-/

I basically violated the basic rule by running a unsafe script as root user, good that script didn't do some crazy stuff. So even though my instance went down I learnt my lessons

Don't ever ever ever run a unsafe script as root that too through cron
Sandbox the unsafe script so it can be killed on time rather than it taking the whole system down.
PHP is not really secure, if it was secure there wouldn't be security updates and atleast my site would be still running :-D

So I now need to wait till Jonas get me new shiny backported uWSGI linked against new PHP on Wheezy and till that time I need to explore on how I can sandbox the poller.php script.

Moving weblog from Jekyll to Pelican

2014-02-02T08:55:00+05:30

Its been a while and I was not actively blogging. I wanted to start blogging again but I was for some reason not happy with Jekyll static site generator which was powering my previous site. So I took this chance to explore other static site generators and redesign my blog.

I explored a bit of Pelican, Nikola and Frog. I didn't feel comfortable with Nikola. Frog is a static site generator written using *Racket* and since I'm learning Scheme using racket I was leaning towards using it but at the last moment I decided to settle with Pelican. The main reason for not using Frog was that I just wanted to get my site up rather than waiting for me to complete learning racket and use Frog.

My feeling about Pelican is that its a nice tool and is simple and gives all basic things like generating RSS/ATOM feeds and nice theming support. In case of Jekyll I had to write custom page for creating RSS feed and had pain in creating my first theme (yeah I'm not a designer). But now things might have changed in Jekyll land but anyway I don't care anymore.

New design uses pelican-bootstrap3 theme and there is no comment facility available. If you want to comment or suggesting things on my post consider writing mail to me :-).

So what happened to older entries? Moving the old entries from Jekyll format to Pelican was real PITA so I just renamed my old site to blog-archive. So this is a from scratch site and will contain only new entries. So how do you like my new blog design? :-)