Posts on Yejun Su

My 4K monitor setup

Wed, 12 Nov 2025 17:17:00 +0800

I recently purchased an LG 27UP850K monitor with a 4K resolution (3840x2160). At its native resolution, everything is tiny. When scaled to the 1080p resolution (1920x1080), it displays the same content as a 1080p screen but with much sharper clarity. Then I scaled to the 2K resoluition (2560x1440), it warns “Using a scaled resolution may affect performance”. I finally switched back to the native 4K resolution because it displays much more content, which was my main reason for buying this monitor.

I adjusted the settings for the apps I use most often. Here’s a summary:

Google Chrome: Settings -> Appearance -> Set “Page zoom” to 150%.
Slack: Preferences -> Accessibility -> Set “Zoom” to 150%.

Emacs: Create a “large” fontaine preset.

(use-package fontaine
  :custom
  (fontaine-presets
   '((regular
      :default-height 160)
     (large
      :default-height 240)
     (t
      :default-family "Aporetic Sans Mono"
      :default-weight regular
      :default-slant normal
      :default-width normal
      :default-height 100
      :fixed-pitch-family "Aporetic Sans Mono"
      :variable-pitch-family "Aporetic Serif")
     ))
  :config
  (fontaine-set-preset (or (fontaine-restore-latest-preset) 'regular)))

Zed: Create a “large” profile and set up the font sizes.

{
  "profiles": {
    "large": {
      "ui_font_size": 24,
      "buffer_font_size": 24,
      "agent_buffer_font_size": 24
    }
  }
}

Ghostty: By default, the new windows and tabs will inherit the font size of the previously focused window. So I just set font size to 24px using a keybinding in any window, and create new windows and tabs from that.
```
keybind = ctrl+shift+l=set_font_size:24
```

Update on 2025-11-15

I’ve been using the above settings for three days, and the biggest problem is the overall user interface appearing very small. I switched the display resolution to 2K scaling, and everything is readable now. The warning mentioned at the beginning is now a thing of the past, as explained in “4K Scaling” Is Not a Problem on Modern Macs:

The way around this is to have macOS “scale” the display to a more ideal lower resolution, but choosing that option in display preferences presents a warning: “Using a scaled resolution may affect performance.” What the OS does here is to scale up the chosen resolution to double height and double width (4x the pixels displayed) and then scale them back down to the display’s native resolution — 60 times per second. Indeed, this can be too much for certain older systems out there. But, as you will see, modern Macs should be able to handle the task just fine.

Update on 2025-11-19

I found the text on the 4K monitor is really sharp, so that I could easily read 12px font when scaled down to 1080p, whereas I previously needed 16px on a regular 1080p monitor. So I configured editors and the terminal to use 12px font, and set the Google Chrome’s page zoom to 75% to read more content.

Update on 2025-12-14

I use 16px font in editor and 100% page zoom in Google Chrome, it means the same configuration as before, but my eyes are very comfortable so that I could work in front of the monitor for longer periods of time without eye strain.

Blogging using Denote and Hugo

Tue, 04 Mar 2025 17:17:00 +0800

I’ve been thinking of writing a post about my current blogging workflow for quite some time. After reading Blogging using Emacs Org Roam and Hugo, I noticed we have a similar approach: we both use Nix, Hugo, and ox-hugo. The main difference is that my workflow is based on Denote instead of Org Roam.

Denote

Denote is a simple note-taking tool created by Protesilaos Stavrou (known as Prot), based on the idea that notes should follow a predictable and descriptive file-naming scheme. The default format is DATE==SIGNATURE--TITLE__KEYWORDS.EXTENSION, such as 20231007T104700--static-website-with-hugo-and-nix__hugo_nix.org. This format is both URL-friendly and easy to search.

Attachments

My note-taking system is inspired by Taking Notes With the Emacs Denote Package. I’m impressed by the “Attachments” section:

An attachment in Denote is any file that is not recognised by Denote as a note, but with a compatible filename. Any file stored in the Denote directory that follows the Denote file naming convention will be recognised as an attachment and can be linked from inside a Denote file.

I save all attachments in the attachments directory alongside my notes, and I insert links using [[file:]] syntax rather than denote-link. The benefits are:

Since most attachments are pictures, I can view them directly in an Org file using org-toggle-inline-images. They’re also visible on GitHub.
Attachments in the form of [[file:]] are automatically copied to the <HUGO_WEBSITE>/static/attachments/¹ directory, with no manual work needed.

I use a function to insert attachments from any location on my computer to the current note by inserting a link if the attachment is already in the attachments directory, or renaming and moving the attachment to the attachments directory first, and then inserting a link.

(setq my-notes-attachments-directory (expand-file-name "attachments/" (denote-directory)))

(defun my/denote-org-extras-insert-attachment (file)
    "Process FILE to use as an attachment in the current buffer.

If FILE is already in the attachments directory, simply insert a link to it.
Otherwise, rename it using `denote-rename-file' with a title derived from
the filename, move it to the attachments directory, and insert a link.

The link format used is '[[file:attachments/filename]]', following Org syntax.
This function is ideal for managing referenced files in note-taking workflows."
    (interactive (list (read-file-name "File: " my-notes-attachments-directory)))
    (let* ((orig-buffer (current-buffer))
           (attachments-dir my-notes-attachments-directory))

      ;; Check if the file is already in the attachments directory
      (if (string-prefix-p
           (file-name-as-directory attachments-dir)
           (expand-file-name file))

          ;; If already in attachments, just insert the link
          (with-current-buffer orig-buffer
            (insert (format "[[file:attachments/%s]]" (file-name-nondirectory file))))

        ;; Otherwise, rename and move the file
        (let ((title (denote-sluggify-title (file-name-base file))))
          (when-let* ((renamed-file (denote-rename-file file title))
                      (renamed-name (file-name-nondirectory renamed-file))
                      (final-path (expand-file-name renamed-name attachments-dir)))
            (rename-file renamed-file final-path t)
            (with-current-buffer orig-buffer
              (insert (format "[[file:attachments/%s]]" renamed-name))))))))

Here’s a screencast of inserting an image from the Desktop directory into the current note using this function:

Reference links

I create references to my notes using denote-link. The problem is that after exporting, these references are converted into Markdown links that point to the original Org files, which are inaccessible on the published website:

[Static website with Hugo and Nix](20231007T104700--static-website-with-hugo-and-nix__hugo_nix.org)

I add an denote-link-ol-export advice to solve the problem:

(advice-add 'denote-link-ol-export :around
            (lambda (orig-fun link description format)
              (if (and (eq format 'md)
                       (eq org-export-current-backend 'hugo))
                  (let* ((path (denote-get-path-by-id link))
                         (export-file-name
                          (or
                           ;; Use export_file_name if it exists
                           (when (file-exists-p path)
                             (with-temp-buffer
                               (insert-file-contents path)
                               (goto-char (point-min))
                               (when (re-search-forward "^#\\+export_file_name: \\(.+\\)" nil t)
                                 (match-string 1))))
                           ;; Otherwise, use the original file's base name
                           (file-name-nondirectory path))))
                    (format "[%s]({{< relref \"%s\" >}})"
                            description
                            export-file-name))
                (funcall orig-fun link description format))))

Now the references use the relref shortcode, which is a Hugo feature to create relative links to documents:

[Static website with Hugo and Nix]({{< relref "static-website-with-hugo-and-nix" >}})

You can visit the article here: Static website with Hugo and Nix.

Hugo

Hugo is a fast static site generator that’s easy to install with just an executable binary.

Nix

I use Nix for my development environment because I need go for Hugo Modules:

{
  description = "My personal website";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs =
    {
      self,
      nixpkgs,
      flake-utils,
    }:
    flake-utils.lib.eachDefaultSystem (
      system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
      in
      {
        devShells.default = pkgs.mkShell {
          packages = with pkgs; [
            hugo
            go
          ];
        };
      }
    );
}

Theme

I created hugo-modus theme specifically for this blog. It uses the colour palette of the Modus themes, and supports both light and dark modes.

Since I use hugo-modus for my blog and also work on its development, I need to switch between the local and remote versions. Fortunately, Hugo supports split configuration by environment:

config/
├── _default
│   └── hugo.toml
└── development
    └── hugo.toml

In config/_default/hugo.toml, set the theme to be used in the production environment:

[module]
  [[module.imports]]
    path = "github.com/goofansu/hugo-modus"

In config/development/hugo.toml, override the theme to use the local hugo-modus directory:

[module]
  replacements = "github.com/goofansu/hugo-modus -> ../../hugo-modus"

Check the active modules by running hugo mod graph:

$ hugo mod graph -e development
project ../../hugo-modus

$ hugo mod graph -e production
github.com/goofansu/yejun.dev github.com/goofansu/hugo-modus@v0.0.0-20250423135050-b8cf9a1e9268

Denote + Hugo

I use ox-hugo to export Org-mode files to Markdown for Hugo. It offers two options for organizing posts: “One post per Org subtree” and “One post per Org file”. In my blogging workflow, I choose the “One post per Org file” method because notes are just posts.

Using this post as an example, after creating the Org-mode note using the denote command, it contains the following content.

#+title:      Blogging using Denote and Hugo
#+date:       [2025-03-04 Tue 17:17]
#+filetags:   :blogging:denote:hugo:
#+identifier: 20250304T171750

The note is not Hugo-exportable at the moment.

Make note Hugo-exportable

Set #+hugo_base_dir, and that’s the only necessary configuration:

#+hugo_base_dir: ~/code/yejun.dev

By default, M-x org-hugo-export-to-md exports the note to ~/code/yejun.dev/content/posts/20250304T171750--blogging-using-denote-and-hugo__blogging_denote_hugo.md.

Thanks to Denote’s file-naming scheme, which creates URL-friendly names, I can export my TILs and Links using their original file names.

But I prefer a shorter file name

Set #+export_file_name to define the name of the exported file:

#+export_file_name: blogging-using-denote-and-hugo

This time, M-x org-hugo-export-to-md exports the note to ~/code/yejun.dev/content/posts/blogging-using-denote-and-hugo.md.

How does the Markdown file look?

Looking at the Markdown file, it contains a YAML front-matter²:

---
title: "Blogging using Denote and Hugo"
author: ["Yejun Su"]
date: 2025-03-04T17:17:00+08:00
tags: ["blogging", "denote", "hugo"]
---

Where does the front-matter come from?

ox-hugo converts #+title, #+date, and #+filetags into Hugo front-matter and automatically includes the author which reads user-full-name. Org meta-data to Hugo front-matter lists all Hugo front-matter translations for file-based exports.

Interestingly, #filetags isn’t included in that list; instead, there is #+hugo_tags. I found #+filetags wasn’t supported until this pull request, which was made to support Org Roam’s parsing of tags from the #+filetags keyword starting with Org Roam v2.

I want to export a TIL note

By default, ox-hugo exports notes as posts, you can change the behaviour by:

Setting #+hugo_section in the note
```
#+hugo_section: til
```
Setting the org-hugo-default-section-directory variable globally
```
(setq org-hugo-default-section-directory "til")
```

See Transform ox-hugo anchors to links for an example.

Can I automatically export a note every time I save it?

Sure. ox-hugo offers a guide on automatically exporting when saving. Just add the following snippet at the end of the note:

* Footnotes
* COMMENT Local Variables :ARCHIVE:
# Local Variables:
# eval: (org-hugo-auto-export-mode)
# End:

Can I search all Hugo-exportable notes?

Yes. The approach is simple - I just search ripgrep the Org files in my denote-directory for #+hugo_base_dir:

(defun my/org-hugo-denote-files ()
    "Return a list of Hugo-compatible files in `denote-directory'."
    (let ((default-directory (denote-directory)))
      (process-lines "rg" "-l" "^#\\+hugo_base_dir" "--glob" "*.org")))

(defun my/org-hugo-denote-files-find-file ()
    "Search Hugo-compatible files in `denote-directory' and visit the result."
    (interactive)
    (let* ((default-directory (denote-directory))
           (prompt (format "Select FILE in %s: "  default-directory))
           (selected-file (consult--read
                           (my/org-hugo-denote-files)
                           :state (consult--file-preview)
                           :history 'denote-file-history
                           :require-match t
                           :prompt prompt)))
      (find-file selected-file)))

Can I export all Hugo-exportable notes?

Absolutely yes! Just loop my/org-hugo-denote-files and execute org-hugo-export-to-md:

(defun my/org-hugo-export-all-denote-files ()
    "Export all Hugo-compatible files in `denote-directory'."
    (interactive)
    (let ((org-export-use-babel nil))
      (dolist (file (my/org-hugo-denote-files))
        (with-current-buffer (find-file-noselect file)
          (org-hugo-export-to-md)))))

Conclusion

I can publish any note to any Hugo website by setting #+hugo_base_dir.
I can publish any note to any Hugo section by setting #+hugo_base_section.
I can insert any file from my computer into any note using denote-rename-file.

Together, these advantages create an efficient publishing workflow.

PS: Emacs code used in this post can be found here.

ox-hugo by default copies files to static/ox-hugo directory, I changed the behaviour by customizing org-hugo-default-static-subdirectory-for-externals to "attachments". ↩︎
ox-hugo supports exporting the front-matter in TOML (default) or YAML. I prefer YAML and have customized org-hugo-front-matter-format to use "yaml". ↩︎

Static website with Hugo and Nix

Sat, 07 Oct 2023 10:47:00 +0800

This website is built with Hugo, a static site generator, and built with Nix¹. This approach ensures the build won’t break in the future because the content, generator and builder are all unchanged.

The configuration

Nix flakes is still an experimental feature but has been available for a long time.

A flake is simply a source tree (such as a Git repository) containing a file named flake.nix that provides a standardized interface to Nix artifacts such as packages or NixOS modules. Flakes can have dependencies on other flakes, with a “lock file” pinning those dependencies to exact revisions to ensure reproducible evaluation.

– https://www.tweag.io/blog/2020-05-25-flakes/

Here is the flake.nix of this website:

{
  description = "My personal website";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let pkgs = nixpkgs.legacyPackages.${system};
      in {
        packages.website = pkgs.stdenv.mkDerivation {
          name = "blog";
          src = self;
          buildPhase = "${pkgs.hugo}/bin/hugo";
          installPhase = "cp -r public $out";
        };

        defaultPackage = self.packages.${system}.website;
        devShells.default = pkgs.mkShell { packages = [ pkgs.hugo ]; };
      });
}

Everytime I clone the repo, there are two options:

run nix develop to enter a pure environment with the hugo program.
run nix build to get the static website in the ./result directory.

It eliminates the need to worry about the development will be broken in the future.

Auto deployment

I use sourcehut to host my website. When I pushed to the git repository, it will be deployed automatically using the sourcehut build service. In the build, nix build uses flake.nix to create exactly same environment as in development:

image: nixos/latest
oauth: pages.sr.ht/PAGES:RW
packages:
- nixos.hut
environment:
  NIX_CONFIG: experimental-features = nix-command flakes
  site: yejun.dev
tasks:
- package: |
    cd $site
    nix build
    tar -C result -cvz . > ../site.tar.gz
- upload: |
    hut pages publish -d $site site.tar.gz

Nix is a tool focused on reproducible package management and system configuration. ↩︎

Send Nix code from Emacs to Kitty

Tue, 03 Oct 2023 02:38:00 +0800

I’m learning Nix by following this tutorial and taking notes using Org Mode. I organized the notes with my thoughts and some code examples. Doom Emacs provides a function +eval/send-region-to-repl to run code in a language’s REPL buffer. It’s useful until multiple-line input comes, which prints like so:

nix-repl> let
  first_name = "Yejun";
  last_name = "Su";
in
  {
    full_name = "${first_name} ${last_name}";
  }

let
            first_name = "Yejun";
            last_name = "Su";
          in
            {
              full_name = "${first_name} ${last_name}";
            }
{ full_name = "Yejun Su"; }

It appears that the input is also getting printed along with the result, which isn’t expected. However, when I tried with nix repl in Kitty, I didn’t encounter this problem. A viable solution that emerged is to send the Nix code from the org-babel source block directly to the Nix REPL in Kitty.

Thanks to the Kitty’s remote control, it’s easy to advise the +eval/send-region-to-repl function to send the Nix code to Kitty:

(defun kitty--ensure-nix-repl-tab ()
  (unless (zerop (shell-command "kitty @ ls | grep -q '\"title\": \"nix-repl\"'"))
    (shell-command "kitty @ launch --type tab --tab-title nix-repl nix repl")))

(defun kitty--send-region-to-nix-repl-tab ()
  (shell-command-on-region
   (if (use-region-p) (region-beginning) (point-min))
   (if (use-region-p) (region-end) (point-max))
   "kitty @ send-text --match-tab title:nix-repl --stdin"))

(defun org-babel-src-block-language-p (language)
  (let ((block-info (org-element-at-point)))
    (and (eq (car block-info) 'src-block)
         (string= language (org-element-property :language block-info)))))

(defadvice +eval/send-region-to-repl (around my-send-region-to-repl activate)
  (if (and (eq major-mode 'org-mode)
           (org-babel-src-block-language-p "nix"))
      (progn
        (kitty--ensure-nix-repl-tab)
        (kitty--send-region-to-nix-repl-tab))
    ad-do-it))

When calling the +eval/send-region-to-repl function (SPC c s), if current major mode is org-mode and the language of the org-babel source block is nix, it sends the selected code to Kitty’s nix-repl tab, which runs the nix repl so that the code is evaluated in the Nix REPL. The tab will be created if not exist.

Backup GnuPG private key

Mon, 18 Sep 2023 22:12:00 +0800

gpg-toolkit is inspired by this article.

I’m using GNU Privacy Guard (GnuPG or GPG) in various ways, such as encrypting passwords, decrypting emails, signing git commits, and more. So it’s time to find a way to backup the GPG private key.

I asked GPT-4 for a method to keep the private key safe, and it tells me to convert it to QR code and print it on the paper. That’s a good idea! I’ll start from backup to a printable text, and then the QR code.

Preparation

Before backup, you need to know your GPG key ID. Run this command:

gpg --list-keys --keyid-format long

It will list all the public keys in your system. Search your own key according to this format pub rsa2048/{YOUR KEY ID}, the {YOUR KEY ID} part is your GPG key ID.

Then export both the public and private key:

gpg --export-secret-keys {YOUR KEY ID} > private-key.gpg
gpg --export {YOUR KEY ID} > public-key.gpg

Backup to printable text

One of the program to be used is paperkey, it transforms the GPG private key to a printable format. The usage is straightforward:

# backup
paperkey --secret-key private-key.gpg --output printable.txt

Keep in mind that you need both the public key and the printable text to restore the private key:

# restore
paperkey --pubring public-key.gpg --secrets printable.txt --output restored-private-key.gpg

Backup to QR code

The process is similar to the previous method, but it requires two more programs: qrencode to create QR code and zbar to read QR code. Generally speaking, the more programs you rely on, the more friction you run into. But I just tried it for fun:

# backup
paperkey --output-type raw --secret-key private-key.gpg | base64 | qrencode -o qrcode.png

With the public key and QR code in hand, you can restore the private key:

# restore
zbarimg qrcode.png | cut -d':' -f2 | base64 --decode | paperkey --pubring public-key.gpg --output restored-private-key.gpg

Simplify TOTP management in Emacs

Wed, 13 Sep 2023 02:54:00 +0800

Time-based one-time password (TOTP) is a computer algorithm that generates a one-time password (OTP) that uses the current time as a source of uniqueness.

– Wikipedia

password-store-otp is an Emacs package which provides functions to interact with the pass-otp extension for pass. There are two functions that insert or append the OTP key URI to a selected pass entry. But it’s not that easy because websites generally provide QR Code or secret key, which requires me to compose the OTP key URI manually.

To simplify the process, I created two corresponding functions based on those in password-store-otp:

(defun yejun/password-store-otp-append (entry issuer secret)
  "Append to ENTRY the OTP-URI consisting of issuer and secret."
  (interactive (list (password-store-otp-completing-read)
                     (read-string "Issuer: ")
                     (read-passwd "Secret: " t)))
  (let* ((secret (replace-regexp-in-string "\\s-" "" secret))
         (otp-uri (format "otpauth://totp/totp-secret?secret=%s&issuer=%s" secret issuer)))
    (password-store-otp-add-uri 'append entry otp-uri)))

(defun yejun/password-store-otp-insert (entry issuer secret)
  "Insert a new ENTRY containing OTP-URI consisting of issuer and secret."
  (interactive (list (password-store-otp-completing-read)
                     (read-string "Issuer: ")
                     (read-passwd "Secret: " t)))
  (let* ((secret (replace-regexp-in-string "\\s-" "" secret))
         (otp-uri (format "otpauth://totp/totp-secret?secret=%s&issuer=%s" secret issuer)))
    (password-store-otp-add-uri 'insert entry otp-uri)))

When calling either function, it will prompt for the issuer and secret, then compose an OTP key URI with the values in this format: otpauth://totp/totp-secret?secret=<secret>&issuer=<issuer>, which saves manual effort.

To keep it simpler, I extracted a function to create OTP key URI, so it can be applied in more scenarios:

(defun yejun/otp-key-uri (issuer secret)
  "Create and copy the OTP key URI consisting of issuer and secret."
  (interactive (list (read-string "Issuer: ")
                     (read-passwd "Secret: " t)))
  (let* ((secret (replace-regexp-in-string "\\s-" "" secret))
         (otp-uri (format "otpauth://totp/totp-secret?secret=%s&issuer=%s" secret issuer)))
    (kill-new otp-uri)
    (message "OTP key URI created and copied.")))

By calling the function, an OTP key URI is created and copied to the clipboard for later use.

Skip sourcehut build in Emacs

Tue, 05 Sep 2023 10:23:00 +0800

builds.sr.ht is the GitHub Actions counterpart in sourcehut, it can run jobs when you push to a git repository that contains a build.yml file. According to the manual, you can skip submitting a build by using git push -o skip-ci.

In Emacs, you can achieve this by adding an infix argument:

(transient-append-suffix 'magit-push "-n"
  '("-s" "Skip CI" "--push-option=skip-ci"))

This inserts a new infix argument to toggle the --push-option=skip-ci argument after the infix argument that toggles --dry-run in magit-push.

However, it is strange that neither argument -o skip-ci nor -o=skip-ci will take effect:

(transient-append-suffix 'magit-push "-n"
  '("-s" "Skip CI" "-o=skip-ci"))

(transient-append-suffix 'magit-push "-n"
  '("-s" "Skip CI" "-o skip-ci"))

Create sourcehut paste in Emacs

Tue, 05 Sep 2023 00:27:00 +0800

Paste is a sourcehut service like GitHub’s Gist. I wrote a function utilizing hut to create a paste in Emacs:

(defun yejun/paste-region-or-buffer (&optional p)
  (interactive "P")
  (let ((filename (read-string "Enter filename: " (buffer-name)))
        (output-buffer " *paste-output*")
        (public (if p " --visibility public" "")))
    (shell-command-on-region
     (if (use-region-p) (region-beginning) (point-min))
     (if (use-region-p) (region-end) (point-max))
     (concat "hut paste create --name \"" filename "\"" public)
     output-buffer)
    (with-current-buffer output-buffer
      (goto-char (point-max))
      (forward-line -1)
      (kill-new (thing-at-point 'line)))
    (kill-buffer output-buffer)))

When calling the function, you will be prompted for a filename, by default, buffer-name is pre-filled for convenience. After creating the paste, the URL is copied to the clipboard.

There are two advices on giving a filename:

Give a meaningful filename: currently you cannot search paste by content, so a meaningful filename makes search easier.
Give a filename extension for code highlighting: see this paste for details.

The default visiblity is unlisted, using C-u to create a public paste, for Doom Emacs users, that’s SPC u.

Problem loading Rails configuration

Sat, 14 Jan 2023 00:00:00 +0800

After upgrading our application to Rails 5.2, InvalidAuthenticityToken errors are found in our public APIs. The error is caused by CSRF protection is enabled by default since Rails 5.2 while it is not required in public APIs. I updated the config to disable the default behavior, but it doesn’t take effect.

# config/initializers/new_framework_defaults_5_2.rb
Rails.application.config.action_controller.default_protect_from_forgery = false

The problem

According to the source code, the config is used to toggle CSRF protection after ActionController::Base is loaded:

initializer "action_controller.request_forgery_protection" do |app|
  ActiveSupport.on_load(:action_controller_base) do
    puts "default_protect_from_forgery: #{app.config.action_controller.default_protect_from_forgery}"

    if app.config.action_controller.default_protect_from_forgery
      protect_from_forgery with: :exception
    end
  end
end

I use this gist to debug how configurations are loaded:

before_configuration
application.rb loaded
before_initialize
default_protect_from_forgery is true, expected: false
config/initializers loaded
ActiveSupport.on_load(:before_initialize) runs at the end of before_initialize
after_initialize
default_protect_from_forgery is false, expected: false
ActiveSupport.on_load(:after_initialize) runs at the end of after_initialize

The highlighted line shows the value of default_protect_from_forgery is true while it should be false. So the problem is ActionController::Base is loaded before loading config/initializers.

Investigation

I commented all gems except the rails gem, then uncommented them one by one and debug configuration loading with rails c to see the prints like above, finally found it is caused by the wechat gem:

Solution

The problem can be fixed if:

Gems load files with Zeitwerk (e.g. https://github.com/Eric-Guo/wechat/pull/293).
gems load files in after_initialize with Railtie.

I created a repo for showing the problem, see the following commits for details:

A tour of functional thinking

Sun, 15 May 2022 00:00:00 +0800

There were several server downtime on May 4th and 5th, my colleagues found IOPS burst in database when school clicking the Pending Emails button, which will show the incoming emails one by one for school users to match them with specific students.

As the requests are very slow to response, the users keep clicking the button again and again, looking forward to get the results, which in turn degraded the database performance and causing the downtime.

Background

After reading the book Grokking Simplicity, I want to use a skill from the book to learn how Pending Emails works without running the program.

The skill is called Distinguishing Actions, Calculations and Data (ACD). Let me start with a brief introduction:

Actions are functions with side-effects, they are affected by when and how many times the functions are called, so they need our most attention. They are normally the real business we want to do, e.g., sending an email, calling an API, etc.
Calculations are pure functions without any side-effects, they are decisions in our system, e.g., decide the user to send an email, calculate cart’s total price, etc.
Data is facts about events, it can be immutable and safely serialized to be transmitted on a wire or stored to disk, e.g. user struct, phone number, etc.

It can be used in all the phases of coding:

Thinking about a problem before coding:
- Clarifying problems needs attention (Actions)
- What decisions we will need to make (Calculations)
- What data we will need to capture (Data)
Coding a solution:
- Writing code with immutability in mind. Actions are mutable, Calculations and Data are immutable.
- A program should have less Actions and more Calculations and Data, in short, Data > Calculations > Actions.
Reading code:
- Refactor the code to better separate from Actions, Calculations and Data.

Explore existing code

With the weapon in hand, I’m going on with confident. After reading code, a diagram with above categories appears:

There are several problems:

There is no Calculations (pure functions without any side-effect).
There are 6 Actions for DB read operations, it reveals some of the reasons for database performance degrading.
In the UI, previous_pending, next_pending only requires id for pagination, but they return an object with all attributes.

Notice we spot problems without running any program yet!

Coding a solution

We can fix above problems by:

Selecting only the necessary columns for pending_email.
Calculating previous_pending, next_pending and total_count without DB operations.

Three Actions are changed to Calculations using the data from the newly added Action “Select only id for pending emails from DB”. Remember Calculations are pure functions, they are safe to call and easy to test, we should prefer Calculations to Actions. Thus, we can put more attention on Actions as they will change according to when and how many times they are called, they are not stable and not easy to test.

After deploying the fix, we verified it works by clicking the button in the same school, the response is fast and IOPS never burst. See the comparison for your reference:

Before
After

Conclusion

By reviewing the performance improvement, I introduced a practical skill (shared on Speaker Deck) to spot problems in existing code and evolve a solution by migrating Actions to Calculations. It is really powerful and easy to use.

A tour of scientist gem

Sat, 16 Apr 2022 00:00:00 +0800

Schools with large amounts of families complain that when visiting the Families roster, the request is too slow to use and even times out. This is a tour about how we made a performance boost while keeping the functionality work as normal.

Bottleneck

As the problem is repeatable, the bottleneck is easily found by checking logs. It is worth mentioning that, Flame Graph is a better way for the job¹.

The bottleneck is a method fetching ancestor ids of all families to support the “Select all families” functionality, which allows users to select families from all pages.

@families.map { |family| family.valid_ordered_ancestors.map(&:id).join(',') }

Notice valid_ordered_ancestors is an association, it means 1000 SQLs for 1000 families, causing the duration increasing linearly with the families count.

First thought

As the method returns ids only, after talking with my team leader, we decided to add a column to cache the ids of valid_ordered_ancestors, named cached_valid_ordered_ancestors_ids, and update it in the after_touch callback:

serialize :cached_valid_ordered_ancestors_ids, Array

after_touch :update_cached_valid_ordered_ancestors_ids

def update_cached_valid_ordered_ancestors_ids
  self.cached_valid_ordered_ancestors_ids = valid_ordered_ancestors.ids
  self.save!
end

Then the solution comes out:

@families.map { |family| family.cached_valid_ordered_ancestors_ids.join(',') }

Great! It reduced SQLs from 1000 to 1. Problem resolved!

But there is a risk: if the column is not updated in all cases, the method will return the wrong result, which is never expected. Because of this, we want to try the solution on production before switching to it while breaking the original functionality.

Be a “scientist”

The gem scientist comes to rescue. Here is the description:

🔬 A Ruby library for carefully refactoring critical paths.

It provides the ability to run experiment code on production while keeping the original functionality.

As a scientist, one of the ways to improve an existing solution is to make an controlled experiment²:

Set two groups for the experiment: existing solution as the control group (control), new solution as the experimental group (candidate);
Use control group’s result as baseline, then compare the candidate’s result with it to see whether the new solution is expected and better.

This concept can be seen in the following code examples.

Solution 1

First, include the Scientist module, it provides DSLs for defining experiments:

include Scientist

science 'family-roster-collect-ids' do |e|
  # control group, old way
  e.use { @families.map { |family| family.valid_ordered_ancestors.map(&:id).join(',') } }

  # candidate group, new way
  e.try { @families.map { |family| family.cached_valid_ordered_ancestors_ids.join(',') } }
end

To enable the experiment, it also requires us to provide an implementation of Scientist::Experiment. I put it in config/initializers/scientist.rb:

require "scientist/experiment"

class ScientistExperiment
  include Scientist::Experiment

  attr_accessor :name

  def initialize(name)
    @name = name
  end

  def enabled?
    true
  end

  def raised(operation, error)
    super
  end

  def publish(result)
    puts "science.{name}" # science.family-roster-collect-ids
    puts result.matched? # whether results equal in both ways
    puts result.control.duration # old way duration
    puts result.candidates.first.duration # new way duration
  end
end

Two methods need to notice:

enabled? returns true to enable the experiment;
publish exposes the experiment results so you can send them to console, files or error monitors.

We deploy the code to production to collect the results for comparison and see whether the new solution is usable.

The results for a school having 4000 families:

Another school having 8000 families:

This solution is 3-4x faster but still not enough.

Solution 2

I assume it can be further improved if we return the joined ids directly, so another column named cached_valid_ordered_ancestors_ids_str is added to store the joined string.

def update_cached_valid_ordered_ancestors_ids
  cache_ids = valid_ordered_ancestors.ids.dup
  self.cached_valid_ordered_ancestors_ids = cache_ids
  self.cached_valid_ordered_ancestors_ids_str = cache_ids.join(',')
  self.save!
end

To confirm the assumption, I made a benchmark with benchmark-ips:

require 'benchmark/ips'

Benchmark.ips do |x|
  x.config(:time => 3, :warmup => 0)
  @collection = Family.all

  # solution 1
  x.report("map") do
    @collection.map { |f| f.cached_valid_ordered_ancestors_ids.join(',') }
  end

  # solution 2
  x.report("collect") do
    @collection.collect(&:cached_valid_ordered_ancestors_ids_str)
  end

  x.compare!
end

and executed it with 600 families and the result is:

Calculating -------------------------------------
                 map    980.811  (±21.7%) i/s -      2.618k in   2.997818s
             collect      4.545k (±10.5%) i/s -     13.288k in   2.992205s

Comparison:
             collect:     4545.4 i/s
                 map:      980.8 i/s - 4.63x  (± 0.00) slower

collect is 4.63 times faster than the map, it is really exciting!

Then the solution is changed to the following and deployed to production:

include Scientist

science 'family-roster-collect-ids' do |e|
  # control group, old way
  e.use { @families.map { |family| family.valid_ordered_ancestors.map(&:id).join(',') } }

  # candidate group, new way
  e.try { @collection.collect(&:cached_valid_ordered_ancestors_ids_str) }
end

The results for the 8000 families school, from 30 seconds to 0.01, it’s a 3000x boost! 💪

The school having 4000 families:

By using this solution, the method time comes to almost the same level regardless of the families count.

Observation

After running the experiment for two weeks, we can see performance improvement without result mismatch, so we switched to the new solution.

Conclusion

In this case we benefit much from scientist. It also has its limitation, for example, it may be not suitable if the results are not comparable, or you can not make decision just from the results. But when it is suitable, you can enjoy writing code with calm and confidence.

Manage runtime versions with asdf

Tue, 08 Mar 2022 00:00:00 +0800

To manage various runtime versions, almost all languages has their own version managers, for example, here is a short list:

Ruby: rvm and rbenv
Python: pyenv and virtualenv
NodeJS: nvm
Erlang: kerl
Elixir: kiex
Go: gvm and goenv

If you are using multiple runtimes for development, you may face the pitfalls to install and maintain different kinds of version managers. Although it is not cumbersome to use any of them, there is an overhead of context switching in your brain. So asdf comes to the rescue.

What is asdf?

asdf is a tool version manager. All tool version definitions are contained within one file (.tool-versions) which you can check in to your project’s Git repository to share with your team, ensuring everyone is using the exact same versions of tools.

– https://asdf-vm.com/guide/introduction.html#introduction

The target of asdf target is to “Manage all your runtime versions with one tool!”. It achieves the target through plugins, internally, it uses the version managers mentioned above, but as a unified command line.

How to use asdf?

First, install asdf on your machine. Then, it comes to the real cases, here we use Ruby and NodeJS for examples:

# List all plugins
asdf plugin list all

# Install plugins for runtimes
asdf plugin add ruby
asdf plugin add nodejs

# List installed plugins
asdf plugin list

# List all verions
asdf list all ruby
asdf list all nodejs

# Install a version
asdf install ruby 2.7.5
asdf install nodejs 16.13.0

# View installed version
asdf list ruby
asdf list nodejs

# Use runtimes in current project
asdf local ruby 2.7.5
asdf local nodejs 16.13.0

# Use runtimes globally
asdf global ruby 2.7.5
asdf global nodejs 16.13.0

# Use runtimes only in current shell
asdf shell ruby 2.7.5
asdf shell nodejs 16.13.0

# Check runtimes that are currently using
asdf current

It absolutely reduces the complexity of managing runtime versions with asdf, I recommend to have a try and you’ll love it.

Workflows in my daily work

Wed, 01 Sep 2021 00:00:00 +0800

In this post, I mainly talked about how I handle Jira tickets.

Principles

Focus on one thing at a time
Avoid starting from scratch
Don’t repeat yourself

Focus on one thing at a time

Notion¹ and Workona² are the cornerstones in my workflows.

Every time I’m going to process a Jira ticket, I save it to Notion with the Notion Web Clipper extension, then apply a pre-defined template to the Notion page. For example, there are four basic templates for Jira tickets: Bug, Task, Story and Epic, I apply the template according to the type of ticket, then snippets or scripts are auto filled, it really saves me large amounts of time.

The most interesting part is for Story and Epic. I create a Workona workspace for them when I begin to develop, it helps me focus on one thing at a time. When things are done, I archive the workspace for later reference, I can recover all tabs, notes and tasks at any time.

For Epic, I also create a Board View in Notion, then link all related pages to the Epic, it helps me to quickly get a glance on how things are going.

For each In Progress ticket, Zapier helps to create a GitHub issue automatically. Later I either convert it to a pull request or copy-paste the contents from Notion to help others know how to handle such tickets in the future.

Avoid starting from scratch

There are always similar tasks that have been done before, so take advantage of existing solutions is better than start from scratch.

Search before going

I search Jira or GitHub before I start to process, it can reduce much time on investigating, debugging and developing.

I created some custom queries to ease the search with Alfred’s Web Search functionality:

Search Jira tickets: https://JIRA_CLOUD_DOMAIN/secure/QuickSearch.jspa?searchString={query}
Search my own issue: https://github.com/issues?q=is%3Aissue+is%3Aopen+assignee%3Agoofansu+archived%3Afalse+{query}
Search the whole GitHub: https://github.com/search?q={query}&ref=opensearch

Utilize the command line tools

Required softwares: gh and fzf
Checkout any pull request: gh pr list | fzf | awk '{print $1}' | read -l result; and gh pr checkout $result
Checkout any git branch: git branch | fzf | awk '{print $1}' | read -l result; and git checkout $result

Don’t repeat yourself

If some scripts are always used, I create a rake task and submit to the repo.
If some scripts need to run among several servers, I create ansible playbooks, execute a command and waiting for results coming to my machine.

Notion is an all-in-one note-taking app. ↩︎
Workona is a browser extension that manages your tabs into projects. ↩︎

Thoughts on note-taking apps

Mon, 23 Aug 2021 00:00:00 +0800

Today I tidy up my digital things into three apps: Notion, Roam Research and Obsidian. In my opinion, they are not mutually exclusive as they have their own focuses:

Notion: write and publish, focus on sharing and coordinating.
Roam Research: my second brain, focus on summarizing and thinking.
Obsidian: save code snippets, focus on private things.

Notion

I use Notion every day although my company uses Basecamp and Jira for project management. It has several features that I feel grateful:

Database template. I set various templates for different kinds of Jira tickets, for example, bug, story or task. For some repeatable tickets, I set a template for it and write some pre-defined code blocks to speed up the workflow.
Powerful embeds. Notion supports many popular websites by embed them in a page, such as GitHub Gist, CodePen, Excalidraw, Figma and Sketch, etc. It helps me to avoid jumping among kinds of web services.
Easy-to-use coordination. In the three apps, Notion is the easiest one to share and coordinate with others. Even in the free plan, you can invite 5 guests to your workspace. I often share the solution for a Jira ticket to my colleagues and talk with them, as it is a part of workflow, all things I did before are just not wasted.
Full export. In Notion, you can choose to export a page with its files and images, even sub-pages, while in Roam Research, you can only export a single markdown file.

Roam Research

Roam Research is date-centered, backlink-powered and bullet-styled.I use Roam to summarize for things I read in Kindle or Instapaper, and write notes as described in Tiago Forte’s review of the book How to Take Smart Notes.

Roam’s backlink is very powerful, it is bottom up, so I can find the root page from any leaf page. It also supports showing Unlinked References in the page, any other page that contains the words will appear in the section, it really helps on finding potential relative things that you may not considered. This feature is more intuitive and useful than Notion’s backlink feature.

For summarize books and articles, it works best with Readwise as

Roam enables some powerful use cases. For example, each new highlight includes a backlink to your Daily Note of the day they were synced.

– How does the Readwise to Roam export integration work?

I like this feature as some things are time-relative, they are useful in the context of that time, I can easily know whether they are still useful at the moment.

Obsidian

Obsidian is based on markdown files, in other words, the file format is not proprietary, you own the files. It best suits the need for private things. I use it to save code snippets, as company code are not sharable.

Obsidian is extensible as it has a plugin system and an active community, so there is no need to worry about the ecosystem. My frequent used plugins are git, gist and excalidraw.

Livebook-driven development

Tue, 08 Jun 2021 00:00:00 +0800

Last week I published ogp which is a simple wrapper for Open Graph protocol. Coding is easy, I want to share the progress of developing it in Livebook.

What is Livebook?

Livebook is a notebook for writing Elixir and Markdown in an interactive way. We can evaluate Elixir code blocks and see the results immediately.

There are several methods to run Livebook, I choose the escript way.

$ mix escript.install hex livebook
$ livebook server
[Livebook] Application running at http://localhost:8080/?token=xxxxx

Livebook is running now! Just open it in a browser. Click the New Notebook button and start to write Elixir. With Elixir 1.12, you get the ability to run libraries after Mix.install/1 them.

How does Livebook benefit the development process?

Before Livebook, I write code in IEx, which is a REPL. It has some helpers to ease the way to explore code, but in my opinion, Livebook exceeds in two factors:

Code history

In fact, IEx can enable code history by setting export ERL_AFLAGS="-kernel shell_history enabled" in the shell profile file. You can also search the IEx code history with Ctrl-r and apply it. But as Livebook is essentially a notebook, you can see all texts and evaluation results without the need to set anything.

Visualization

Livebook has a clean UI. You can write documents in Markdown and evaluate Elixir code blocks. It is more continuous, you can review every step of your thought by scrolling the page.

How to develop in Livebook?

Use ogp as an example.

First, I explore the idea with code blocks.

Install floki¹ with Mix.install/1.
Explore floki with Open Graph protocol.
The variables in a code block can be referenced by blocks below it.
As going deeper, a simple parser comes out.

Then I create an Elixir project and run the Livebook in the project.

Create an Elixir project with mix new ogp --module OpenGraph --sup.
Save the Livebook file in the project.
Move the parser into the project and run Livebook in Mix standalone mode.

Connection timed out occurs, to resolve it, run Livebook with a short name: livebook server --sname notebook. See https://github.com/elixir-nx/livebook/issues/275 for detail.
Set up Mix standalone mode again successfully, uses the module from the ogp project.

Conclusion

Developing in Livebook is really delightful. It helps to explore the code and writing documents. Give it a try and you will love it!

PS: The Livebook file created in this story can be found here.

Update on 2021-06-09

Livebook supports URL input since this PR, which is a perfect use case for ogp. See how it works:

Enjoy!

Floki is an HTML parser in Elixir. ↩︎