Post 5 - Emacs and Elisp

I started using Emacs around 2020. At the beginning, I really only wanted to use it for its superior support of LSP and Org-mode, which in my opinion, is a far more complete mark-up language compared to Markdown. It is simple to use, but it can be used to do complex things too, if that’s your preference. I am able to have a directory with multiple Org files, TODO items in various files, and all of these can be presented in a familiar daily, weekly, monthly task-list format using org-agenda. I can schedule things, set deadlines, set reminders based on those deadlines, and repeat tasks periodically. These are all things that Org’s design allows: the native integration into Emacs makes it easy to get started using all the functionality provided by Org, without having to depend on external packages. Systems like this probably exist for Markdown too. I never looked for one because my problems with Markdown began well before I found out about Org-mode. Writing Elisp and extending Emacs was something that I started doing because I had no choice. I have not gotten far enough with it yet, but I have been able to implement some useful functions, for which ready-made packages don’t already exist.

unwrap-all remains the most useful of the functions that I put together with fairly rudimentary knowledge of Elisp. The function itself is probably written in a way that is very non-idiomatic Elisp. This function takes a file¹, assuming that it is wrapped at a set text width (say 80 characters) and paragraphs are separated by a newline, it will unwrap all the text so that each paragraph is on a single line, with empty lines separating them. The resulting text works well with most web applications that don’t support Markdown. The tool that was used to fill in self-assessments and peer reviews at work did not support Markdown; people would regularly start writing their peer reviews within the tool and lose it because their browser crashed or the tab refreshed or something. I never considered writing a long block of text within a web browser to be a real option anyway. Without features provided by keyboard shortcuts-based editing, such as being able to jump to the beginning or end of the line, word; or to delete until some character, or the remaining part of the current line, one has to constantly jump between the keyboard and the mouse, which leads to quite a bit of inefficiency. (The tool eventually implemented auto-save to prevent this, but not Markdown support.) After manually doing the wrapping and unwrapping manually for 2 quarters, and then searching online for a local tool or script which could do this², I decided to finally write something that would solve this problem.

The function was surprisingly easy to write. Elisp is esoteric, with terminology such as car and cdr, which refer to [0] and [1:] in the usual Golang notation. In a handful of iterations, I had something that actually worked. As I was writing in Org-mode most of the time, the function handles Org-mode text and does not unwrap things like code-blocks or tables. (Of course, these were not part of my self-assessment. They came in later when I realized that unwrapping was useful even before pasting text into Google Docs. This was before Google Docs had native support for Markdown.

Showing or hiding the word count in the editor’s mode line (bottom bar) was also something that was primarily useful for me during the performance review cycle, when there were guidelines about how long peer reviews generally should be. After having written a bit, being able to see how much I have written and whether I am being too verbose was a useful gauge. Once again, due to Org-mode’s magic, I was able to open a single sub-heading from a much larger file in a separate buffer. This buffer’s modeline would show me only the word count of the text within this sub-heading! This is what I mean when I say that Markdown is not a complete system. This is something that is obvious to me: A single file may have a lot of information but headings separate it, and it should be very easy to treat those headings as distinct files.

One thing that I have started doing recently is to move all of my periodic TODOs into my existing notes directory. These usually include things like logging into a website, and downloading an invoice or a CSV file and storing it somewhere. These actions are often manual and monthly, because there is no API which can be called and the systems that provided these files are well behind the usual trends. Although quite slowly, this system has started to show some value: Lately, I have forgotten about most of the periodic TODO items and I am trusting the Org-Agenda weekly view to show them to me whenever I open that view. This system works well at work, where I am opening my computer on every workday, and checking the agenda. On the personal side of things, since I don’t use my computer every day, or even if I use it, I might not open my agenda, this might need to be tweaked to allow notifications: A systemd unit which runs the org-agenda command and sends a notification using notify-send when there is a TODO which needs to be done today or is overdue, perhaps? These are ideas that I have been exploring.

One of the limitations of Emacs is that it is a single-threaded application. No other modern editor is single-threaded. Editors from JetBrains like RubyMine can run spawn the process which will run tests in the background, bring that output into the editor’s test “visualization” buffer, while keeping the panes that show code browsable. This is not possible with Emacs currently: The unwrap function, for instance, blocks the editor from doing anything until it is complete. This is not a problem with most functions. It is an issue with Org-agenda though: When you run Org-agenda on a directory with 100s of Org files, it takes a lot of time. If you want to measure it, you can do so using Org:

#+begin_src elisp
  (benchmark-elapse
    (org-agenda-list))
#+end_src

When you run C-c C-c with your cursor at end_src above, Org-mode will offer up a prompt to run the Elisp in that block, and if you agree, it will print the results of that Elisp into the same Org file. The result of the benchmark-elapse function is the amount of time that it takes to run the code within that function:

#+RESULTS:
: 146.782507877

This is the number of seconds that org-agenda-list takes when run on a directory with a few 100 Org files. Yes, that’s a lot. And during that whole time, the editor is blocked because (you guessed it!) the main thread is executing this function and is not able to do anything else.

One saving grace is that its only the first execution after Emacs is started which takes so long. Executions after the first one finish in under 10 seconds. I guess there is some caching or tracking of what file is changing, or something. I don’t know why that caching does not work across Emacs sessions; maybe the cache results are stored in the memory of the current Emacs process. 150 seconds (and even 10 seconds for reloading the agenda) is unacceptable.

My guess is that it takes too long for Emacs to figure out which of these 100s of files have TODO items in the first place because it is single threaded. The difference between single-threaded (and battle-tested) Grep and multithreaded ripgrep is well-known. For instance, this is the difference in runtimes:

# Single-threaded
$ time find . -type f | xargs -I{} egrep '(TODO|WAITING|DONE|CANCELED)' {} -l >/dev/null
noglob find . -type f  0.00s user 0.00s system 90% cpu 0.003 total
xargs -I{} egrep '(TODO|WAITING|DONE|CANCELED)' {} -l > /dev/null  1.08s user 0.42s system 103% cpu 1.446 total

# Multi-threaded
$ time rg -w '(TODO|WAITING|DONE|CANCELED)' -l >/dev/null
rg -w '(TODO|WAITING|DONE|CANCELED)' -l > /dev/null  0.04s user 0.01s system 288% cpu 0.017 total

1.446 seconds for grep; 0.017 seconds for ripgrep: roughly 85 times faster.

org-agenda-list uses org-agenda-files as the list of files or directories which it traverses to find all the TODOs. What if I gave it a list that came from rg to start with? Will that speed things up? How much?

I tried it. This oneliner converts the list of files output by Ripgrep into a list for Elisp:

$ rg -w '(TODO|WAITING|DONE|CANCELED)' -l | \
    sed -e 's#^#"~/notes/#g' -e 's#$#"#g' | \
    paste -sd' ' | \
    sed -e "s#^#(setq org-agenda-files '(#g" -e 's#$#))#g'
(setq org-agenda-files '("~/notes/some-file.org" "~/notes/some-other-file.org"))

I put this list manually into the Emacs configuration, for testing. And voila: The time taken to run org-agenda-list and present the current week’s drops to about 5 seconds! The time taken to reload the agenda is almost 0, at about 0.28 seconds. The list found by ripgrep contains only 100 files, and each of them contains a TODO item. As ripgrep itself takes an insignificant amount of time to run, I think I am going to wire up ripgrep to always run and over-write the value of org-agenda-files right before org-agenda is called. I believe that this is possible using advice-add, which allows users to add code that executes before, after, or around existing Elisp functions!