transclusion minor-minor mode

work in progress in progress.

made to work with org-mode files

what it’s supposed to do

quickstart

after cloning this repo

git submodule init  # pulls ext--json-rpc-request
git submodule update

set up and run the transclusion server somewhere

start emacs and load xcl-transclude.el

open this file, M-x xcl-transclude-mode, cursor over a transclusion directive, and C-c E

(functions and key chords are subject to change)

development setup

assuming you have the nix package manager, clone this repository and run nix-shell.

emacs

from the nix-shell, running emacs should give you a development-ready emacs instance

webserver

the transclusion engine is a nodejs webserver; see transclusion server for install + setup; launch it from another terminal

node build/webserver.js

using org-mode MACRO syntax instead of INCLUDE

with org macros this seems to be a better option.

see https://github.com/fniessen/org-macros/blob/master/org-macros.setup

To test this, run the org-mode export process (e.g. C-c C-e then h H or t A), then find the text from LICENSE exported as well.

TODO: figure out how to make this org-compatible. Right now it is trivially easy to cause export-time conflicts due to eval

org-mode export interop test

using org’s native macro facility in normal org export

partial transclusion

{{{transclude(LICENSE::60,500)}}}

the whole thing

{{{transclude(LICENSE)}}}

transclusion addressing syntax

primary consideration is edit-time user friendliness;

for accuracy, fuzzy text anchors should be used

non-resolution should lead to an empty resource, i.e. an empty string at render time

point vs range resource resolution

There is 1 goal of the transclusion target notation: to establish a range. However, conventional address methods do not distinguish between point and range resources. For transclusion, we need to ensure that we operate, code-wise, in range space. Hence, when given a point resource, we must convert it into a range resource.

In addition to the straightforward range methods covered in sections below, some examples of org internal links:

• textual link (see documentation) like [[My Target]] can resolve, in this order, to:
  1. a text label <<My Target>> aka dedicated target. This defines a point resource.
  2. an element, #+NAME: My Target. This needs to be interpreted as the range encompassing that element.
  3. a headline, * My Target. This is the range encompassing the section.
• CUSTOM_ID a la [[#my-custom-id]], resolves exactly to an entry with CUSTOM_ID: my-custom-id. This needs to be interpreted as the range encompassing the section bearing that id.

the textual link method logic may involve back-tracking, so it may be sensible to simply disallow providing a single textual link for transclusion.

non-semantic, non-structural ranges

full file

{{{transclude(file:tiny.org)}}}

lines

{{{transclude(file:tiny.org::5)}}}

line range

{{{transclude(file:tiny.org::2-3)}}}

from start

{{{transclude(file:tiny.org::-2)}}}

until end

{{{transclude(file:tiny.org::7-)}}}

character index range

character range

{{{transclude(file:tiny.org::0,100)}}}

from start

{{{transclude(file:tiny.org::,100)}}}

until end

{{{transclude(file:tiny.org::50,)}}}

percent range

convert to line first

{{{transclude(file:moozorgjs.org::1%-3%)}}}

structural ranges

contextual target

native org links

{{{transclude(file:tiny.org::*decoy 1)}}}

exact match with range

{{{transclude(file:tiny.org::in 2101…for great justice)}}}

exact match with range

these should all resolve to

huh

glob file name

{{{transclude(file:t*ny.org::*huh)}}}

fuzzy find file by content

{{{transclude(grep:gg+you::*huh)}}}

{{{transclude(grep:gg you::*huh)}}}

{{{transclude(grep:gg%20you::*huh)}}}

reconcile org internal link syntax with structural ranges

constriction syntax / semantic range

a better term than constriction / constrictor probably exists

the $constrictor should fallback to resolution logic of the colon-based content address when no constriction method is found

content based range constriction

address like:

• file:$RESOLVER#$constrictor
• https://$RESOLVER#constrictor
• http://example.com/#dummy target#ignore again#This domain...

where $constrictor …

anchor based constriction

not yet a user friendly way to create anchors in edit phase

examples using moozorgjs.org

nearest token set (fallback)

$RESOLVER#handling todo

resolve to Handling TODO

via first + best sequential match

token set range (fallback)

$RESOLVER#in 2101…for great justice

• [X] js poc
• [ ] emacs poc

line: nearest statement / line

possibly just L

$RESOLVER#line/support+scheduled

resolve to the line,

“Support attributes like SCHEDULED:.”

Implement todo attributes

• [X] js poc
• [ ] emacs poc

para: nearest paragraph

possibly just P

$RESOLVER#para/org.js parser converter

intro paragraph

intro paragraph

• [X] js poc
• [ ] emacs poc

section nearest hierarchical section

need org context

$RESOLVER#section/web browser

Web Browser

Web Browser

??? nearest conceptual unit

hardest – need broader semantic analysis

$RESOLVER#unit/require org

require(“org”)

the src block with “require” within

Node.js

post-transclusion processor syntax

this only makes sense in read-only transclusion

the pipe is an obvious choice of “filter” designation, but since it conflicts with org-mode table syntax, the post-processor parsing should also recognize unicode BROKEN BAR ¦ as an alternative

• file:$RESOLVER#$constrictor|post-processor
• file:$RESOLVER#$constrictor¦post-processor

collapse newlines

this currently makes sense for something transcluded into an org doc where only the text is wanted, but the transclusion target’s (host document) location has significant whitespace, such as in an org table. Linebreaks in the source document cause the table to break.

strip heading

when the transclusion target is defined by an org headline section, one may want to keep the text only. The address may look like

file:somefile.org::*infamous muse of famous mouse¦no-headline

testing decoy headers

real thing

decoy 1

text in the real

fake thing

decoy 1

text in the fake

what does org do?

(org picks the first occurrence)

decoy 1 (real)

decoy 1 (fake)

testing org-mode compatibility

• transcluding-org-elements.org? # opens dired
• transcluding-org-elements.org?s=1 # opens dired

transclusion macro syntax interaction in emacs

this relies on running the webserver from xcl/src/xcl/node_webserver.cljs

see xcl/README.org for details

playground

{{{transclude(LICENSE::2,10)}}}

{{{transclude(file:transcluding-org-elements.org)}}}

{{{transclude(file:./transcluding-org-elements.org::2-10)}}}

{{{transclude(file:./transcluding-org-elements.org::20)}}}

{{{transclude(xcl:./public/tracemonkey.pdf?p=3&s=Monkey observes that…so TraceMonkey attempts)}}}

{{{transclude(calibre:quick start?s=The real advantage…on your computer.)}}}

{{{transclude(zotero:just-in-time?s=Monkey observes…TraceMonkey attempts)}}}

{{{transclude(xcl:./public/alice.epub?p=2&s=Would you tell me, please…walk long enough)}}}

{{{transclude(zotero:faulty reward functions?s=We assumed…measure the performance)}}}

also see

using org dynamic blocks

http://stackoverflow.com/questions/15328515/iso-transclusion-in-emacs-org-mode

http://stackoverflow.com/a/15352203

(defun org-dblock-write:transclusion (params)
  (progn
    (with-temp-buffer
      (insert-file-contents (plist-get params :filename))
      (let ((range-start (or (plist-get params :min) (line-number-at-pos (point-min))))
            (range-end (or (plist-get params :max) (line-number-at-pos (point-max)))))
        (copy-region-as-kill (line-beginning-position range-start)
                             (line-end-position range-end))))
    (yank)))

Then to include a line range from a given file, you can create a dynamic block like so:

C-c C-x C-u (org-dblock-update) will on the BEGIN/END block updates that block; org-update-all-dblocks does all in the buffer.

This is an unsatisfactory solution because it is actually just unidirectional quasi-dynamic inclusion; quasi, because the update step has no knowledge of the target until the update is invoked. It is possible to add action hooks to run updates automatically, but that is no different from export-time inclusion, which by nature assumes a unidirectional flow of information. Org already has a number of ways to achieve this, such as a sh src block that runs cat.

Transclusion is different in practice: it is the actual content, appearing at the location of transclusion.

John Kitchin’s version (requires org-mode 9)

see transcluding-org-elements.org