l3kn / Org Zettelkasten

• Org Zettelkasten This package should be considered work in progress. I use it on a daily basis but the API regularly changes in breaking ways.

I still need to write proper setup instructions. For now, feel free to look around for pieces of code that might be useful to you.

#+CAPTION: User Interface [[file:images/interface.png]]

** Motivation When managing large collections of org-files (> 1000), many of the default functions / modes (i.e. org-agenda) become very slow.

This package is an attempt at alleviating this problem by introducing a caching layer that is persisted to disk, and a set of command for interacting with / viewing this data.

The code for this cache can be found at [[https://github.com/l3kn/org-el-cache][l3kn/org-el-cache]].

(Planned) features include:

• A command for finding files by their titles
• Commands for renaming files updating all links to them
• Commands for extracting a headline into a separate file (or doing the inverse)
• A hydra with a bunch of useful hotkeys
• A project view for listing files marked as GTD projects
• A graph view (prototype stage)
• A sidebar showing backlinks for the current file (prototype stage)
• A TODO list that can show tasks collected from hundreds of files (not working at the moment)
• Full-text search (not working at the moment) ** Inspiration *** David Allen - Getting Things Done 📖 *** Lion Kimbro - [[https://users.speakeasy.net/~lion/nb/html/][How to Make a Complete Map of Every Thought You Think]] :website: *** Niklas Luhman - Zettelkasten ** Similar Projects
• [[https://jblevins.org/projects/deft/][deft]], Emacs mode for browsing, filtering & editing directories of files
• [[https://github.com/EFLS/zetteldeft][EFLS/zetteldeft]], Zettelkasten system for deft
• [[https://github.com/jethrokuan/org-roam/][jethrokuan/org-roam]], Extensions to deft, graph view
• [[https://github.com/Kungsgeten/org-brain][Kungsgeten/org-brain]], concept mapping tool for org-mode, graph view
• [[https://github.com/alphapapa/org-graph-view][alphapapa/org-graph-view]], Link-graph view for org-mode
• [[https://github.com/alphapapa/org-ql][alphapapa/org-ql]], Query-language & views for org-mode

If you are not working with > 1000 org-mode files or if you have no yet run into any performance problems, one (or a combination of multiple) of the packages listed above might fit your needs better. ** Limitations org-zettelkasten uses a cache to enable fast searching of files and headlines.

This cache is updated each time a file is saved, so to ensure the information displayed in the views is up to date, files should be saved often.

I'm using [[https://github.com/bbatsov/super-save/][bbatsov/super-save]] for that.

A big limitation when working with cached headlines is that we can't use markers (like org-agenda does) to jump to a headline in a file, as these can't be persisted to disk.

Instead, headline IDs are used for that.

This means that every headline that should be accessible through one of the views must have an ID.

There are two functions to make this easier:

org-zk-add-ids-to-buffer adds IDs to all task-headlines in the current buffer.

I'm using a hook to do this each time a file is saved.

#+begin_src emacs-lisp (add-hook 'org-mode-hook (lambda () (add-hook 'before-save-hook 'org-zk-add-ids-to-buffer nil 'local))) #+end_src

If you want to use org-zettelkasten for an existing collection of files, you can use M-x org-zk-refactor-add-ids to add IDs to all files managed by org-zettelkasten. ** Dependencies

• [[https://github.com/abo-abo/swiper][abo-abo/swiper]] For searching / selecting files
• [[https://github.com/abo-abo/ace-link][abo-abo/ace-link]] For following links
• [[https://github.com/abo-abo/hydra][abo-abo/hydra]] ** Collections org-zk-collections is used to customize org-zk settings per directory. Entries should be of the form (:path ... :other-settings ...).

This is most useful for configuring the :setup-fn for a file.

Due to the way this list is searched when determining the category of a file, sub-directories should be listed before their parent directory.

*** Example Here is the configuration I'm using for my personal setup:

#+begin_src emacs-lisp (defun org-zk-website-setup-fn (title) (insert "#+SETUPFILE: ~/org/website/setup.org\n") (org-zk-default-setup-fn title))

(setq org-zk-collections '((:path "/org/flashcards/" :name "fc") (:path "/org/sandbox/" :name "sandbox") (:path "/org/archives/" :name "archives" :ignore t) (:path "/org/website/pages/" :name "pages" :setup-fn org-zk-website-setup-fn) (:path "/org/website/" :name "website" :setup-fn org-zk-website-setup-fn) (:path "/org/deft/" :name "zk") (:path "/org/paper/" :name "paper") (:path "/org/" :name "main"))) #+end_src *** Alternative Implementations This could be implemented using [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Directory-Variables.html][Directory Variables]], however in that case, there is no one way to see the configuration for all categories and each variable that is set has to be declared non-risky. ** TODO Backlink View :PROPERTIES: :ID: e91516d4-efe2-43ee-a5d4-2106b8e2d7c0 :END: A sidebar displaying all files linking to the file currently being viewed. ** File View The file view shows a list of all files included in org-zk. It can be used to quickly edit file metadata or find all files matching some query.

M-x org-zk-file-view or =F= from the org-zk-hydra.

*** Hotkeys

• =RET= to open a file
• =k= to edit a files keywords *** TODO Screenshot :PROPERTIES: :ID: 08bd91bc-f39b-43c3-bae1-e5353f7c1b59 :END: *** Queries
• keyword:my-keyword or k:my-keyword
• category:my-category or c:my-keyword (uses org-zk categories, not the org-mode =#+CATEGORY= keyword) ** Project View Variant of the file view for managing GTD projects using the =#+GTD_STATE= keyword.

Based on this keyword, a view of all (active) projects can be created and it's easy to mark a whole project as =on_hold= or =someday= to remove it's tasks from the task view. ** TODO Task View :PROPERTIES: :ID: 2e8af26d-fd38-4a31-a0fb-fa84bfc24ce5 :END: Simplified analog to the =org-agenda= that uses cached headlines. ** TODO Calendar View :PROPERTIES: :ID: 6d5ce334-d249-4fd1-9ebb-d0f029b0e196 :END: Variant of the task view that takes into account repeated tasks and limits the display to tasks scheduled or due in the next n days. ** git I'm willing to sacrifice a bit of memory and meaningful commit messages to make sure no information / history is lost.

Files are committed to version control every hour. ** Cache There are a few other packages that provide more advanced / performant queries on org-mode files:

• [[https://github.com/alphapapa/org-ql][org-ql]]
• [[https://github.com/ndwarshuis/org-sql][org-sql]]

This package implements a cache for files and headlines using a hash table directly in Emacs, no external database is needed.

To avoid recreating this table on each startup, it can be written to disk at regular intervals and when Emacs is closed.

Each file entry has a hash value that is checked against the files hash on startup. With this, updating the cache for my collection of files takes around 5s.

Cache entries are updated when a =.org= file is saved, created, moved or deleted. ** Derived Tasks Some tasks can be fully derived from the current state of the rest of the system.

Examples:

• Reviewing due flashcards
• Processing the Inbox
• Reading mail

This could be implemented by skipping the detour through .org files on disk and adding derived tasks to the list when opening the task view.

The downside of this approach is that these tasks would not show up in the default org-agenda an keeping track of their time-tracking information would require an additional database.

Instead, derived tasks are implemented by checking for some condition in regular intervals, then adding a headline to a predefined file.

To avoid cluttering this file, if it already includes a headline with the same title that is not marked as "DONE", no new entry is added.

Derived tasks are stored in a hash-table to simplify changing the predicate of an existing derived task.

Therefore, each derived task should have a unique task title.

Derived tasks can be registered using the (def-org-zk-derived-task title priority tags predicate) macro.

The example below adds a new task "Process Inbox" if there are at least five entries in the inbox.

#+begin_src emacs-lisp (def-org-zk-derived-task "Process Inbox" "A" '("gtd") (>= (org-zk-inbox-count) 5)) #+end_src ** Components *** org-zk-keywords File-level attributes are stored as =#+KEY: value= org mode keywords. These should be placed at the start of the file, not containing any newlines before or between keywords.

• (org-zk-keywords-set-or-add (key value))
• (org-zk-keywords-add (key value))
• (org-zk-keywords-delete (key))

The macro org-zk-def-keyword can be used to create commands to set keywords to one of a list of predefined values using ivy-read. When generating the functions name, the keyword is converted to lowercase and "_" are replaced by "-".

**** Example #+begin_src emacs-lisp (org-zk-def-keyword "GTD_STATE" '("active" "someday" "planning" "cancelled" "done")) #+end_src

Generated function: org-zk-set-gtd-state *** org-zk-cache By default, emacs is not fast enough to efficiently search large collections (>1k files) for TODO keywords, tags, dates etc.

To get around this limitation, a cache is introduced.

This cache works by running org-element-process-buffer each time a file is saved or the buffer moves out of focus (e.g. when switching to another window or buffer).

Sub-modules can register on element types to compute data on a file or headline level that can then be used to implement fast views on all the data in the zettelkasten, e.g. for listing open projects, tasks or calendar entries, generating clocking reports and statistics or for use with the integrated spaced repetition system.

A query language is implemented on this cache for building custom views on the data. *** org-zk-titlecase When creating a note using the org-zk commands, the title that is entered is automatically converted to title-case.

This only works for English text and not all rules are implemented. Multi-word conjunctions are not supported yet. ** Testing Integration testing is done by providing an input file, a set of operations and an file with the expected output.

The output is written to a third =_got= file, which can be diffed with the expected output or used to replace the =_expected= file if the output generated was valid. ** Credits

• The file view uses the same font faces as [[https://github.com/skeeto/elfeed][skeeto/elfeed]]. ** License Copyright © Leon Rische and contributors. Distributed under the GNU General Public License, Version 3