[O] Proposal: references from code to text.

[ZHUO QL (KDr2)]

Hi all, I just post a proposal for orgmode on my website(generated by orgmode):
http://kdr2.com/tech/emacs/1805-proposal-org-ref-code-to-text.html .
Here is the source of the proposal:
# -*- mode: org; mode: auto-fill; -*-
#+TITLE: Proposal for Orgmode: references from code to text.
#+AUTHOR: KDr2

#+OPTIONS: toc:nil
#+OPTIONS: num:nil

#+BEGIN: inc-file :file "common.inc.org"
#+END:
#+CALL: dynamic-header() :results raw
#+CALL: meta-keywords(kws='("orgmode" "literate programming")) :results raw

# - DATE
<2018-05-12 Sat>


Literate programming is writing out the program logic in a human
language with included (separated by a primitive markup) code snippets
and macros. A preprocessor is used to substitute arbitrary
hierarchies, or rather "interconnected 'webs' of macros", to produce
the compilable source code with one command ("tangle"), and
documentation with another ("weave").

So in a literate programming source file, any chunk is either a block
of code or a block of document in a human language, I'll call them
*code* and *text* below.

In my opinion, code and text should be able to reference each other:

- *Text refers to text* \\
In the exported/woven documentation, these references act as links,
make interconnections between relevant topics.
- *Text refers to code* \\
The links in the exported/woven documentation can also link to a
line of code if the code blocks are also included in the
exported/woven documentation.
- *Code refers to code* \\
That's the most important part of the literate programming. This
kind of references act as placeholders or something like macro in
the C-programming language. With it, literate programming provides
an ability to change the order of the source code from a
machine-imposed sequence to one convenient to the human mind. [[https://en.wikipedia.org/wiki/Literate_programming#Example][Here
is a simple Example depicts this concept]].
- *Code refers to text* \\
So far as I know, no literate programming tool has this kind of
references. However, this kind of references could be very benifical
too. Many programming languages have a mechanism to support
embedding documentation(usually as comments) in the source code, AKA
[[https://en.wikipedia.org/wiki/Docstring][Docstring]]. These embedded documentations are widely and conveniently
used in editor, REPL and IDE. For instance, the docs of any elisp
function that we can read in Emacs are from the elisp source.

But in literate programming we seldom embed comments into the code
blocks, as an impliaction the extracted/tangled code has no docs
embeded. If we can refer to chunks of text in the code blocks, we
will have the chance to include these chunks of text as embeded docs
in the code in the tangling process, which results in
well-documented source code. This is useful especially while the
programmer are authoring reusable source code, like libraries or
frameworks.

Here is the implementation status of these kinds of references of
noweb and Org mode (~->~ means 'has the ability to make reference
to'):

| tool | text -> text | code -> code | text -> code | code -> text |
|----------+--------------+------------------+----------------+--------------|
| noweb | No | Yes | No | No |
| Org mode | Yes, links | Yes, noweb style | Yes, ~(ref:*)~ | No |


As we saw, both noweb and Org mode do not implement the reference from
code to text. Could we provide this feature in Org mode -- the best
and widely used literate programming tool?

As a proposal, in Org mode, we can mimic the labels in
code blocks(i.e. text -> code reference), use something like
~(ref:text:<REF-NAME>)~ as the reference from code to text:

#+BEGIN_SRC org
,#+NAME: DOC-OF-ADD
We use the function add to calculate the sum of two numbers.

,#+BEGIN_SRC elisp
(defun add (x y)
"(ref:text:DOC-OF-ADD)"
(+ x y))
,#+END_SRC
#+END_SRC

The Org file above will produce the code below:

#+BEGIN_SRC elisp
(defun add (x y)
"We use the function add to calculate the sum of two numbers."
(+ x y))
#+END_SRC

- Beside the ~NAME~, we could also support reference via the
~CUSTOM_ID~ property or anything else that can locate a text chunk.
- In the tangling process, we should do some kind of transformation of
the target text, for example, escape the quote mark. A hook may be
needed to let the user customize the transformation too.

What do you think?

* References
- https://en.wikipedia.org/wiki/Literate_programming
- https://en.wikipedia.org/wiki/Noweb
- https://en.wikipedia.org/wiki/Docstring
- http://orgmode.org/manual/Extracting-source-code.html#Extracting-source-code
- http://orgmode.org/manual/Noweb-reference-syntax.html#Noweb-reference-syntax
- http://orgmode.org/manual/Literal-examples.html#Literal-examples
- https://orgmode.org/worg/dev/org-syntax.html

* Discuss and Comment
#+BEGIN: inc-file :file "comment.inc.org"
#+END:

[Eric S Fraga]

Post by ZHUO QL (KDr2)
http://kdr2.com/tech/emacs/1805-proposal-org-ref-code-to-text.html .
# -*- mode: org; mode: auto-fill; -*-
#+TITLE: Proposal for Orgmode: references from code to text.
#+AUTHOR: KDr2

[...]

Post by ZHUO QL (KDr2)
- Beside the ~NAME~, we could also support reference via the
~CUSTOM_ID~ property or anything else that can locate a text chunk.
- In the tangling process, we should do some kind of transformation of
the target text, for example, escape the quote mark. A hook may be
needed to let the user customize the transformation too.
What do you think?

I like the symmetry of having src reference text. I guess, in what you
propose above, that the text that would be incorporated would be a
single element (in org parsing parlance) or else we would have to
specify an extent in the link information.

Are you able to implement something? I would definitely help test it.

[John Kitchin]

It seems like this is nearly all possible already (see below). It is a
little wonky that you have to put text in a src txt block to achieve it,
but otherwise what is missing?

* Main document

See [[id:BAD97113-3561-4A4A-BA07-0CD5BF6BA35F][There is a reason we only
support two args]] (text to text) for notes about this function. The
addition is done in line [[(add)]] (text to code).

#+NAME: DOC-OF-ADD
#+BEGIN_SRC txt
"We use the function add to calculate the sum of two numbers."
#+END_SRC

#+NAME: ADD
#+BEGIN_SRC emacs-lisp -n -r :noweb yes :tangle test.el
(defun add (x y)
<<DOC-OF-ADD>> ;; code to code
;; [[id:BAD97113-3561-4A4A-BA07-0CD5BF6BA35F][There is a reason we only
support two args]] code to text
(+ x y) (ref:add)
)
#+END_SRC

#+RESULTS: ADD
: add


** Notes

*** There is a reason we only support two args
:PROPERTIES:
:ID: BAD97113-3561-4A4A-BA07-0CD5BF6BA35F
:END:


John

-----------------------------------
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Post by Eric S Fraga

Post by ZHUO QL (KDr2)
Hi all, I just post a proposal for orgmode on my website(generated by
http://kdr2.com/tech/emacs/1805-proposal-org-ref-code-to-text.html .
# -*- mode: org; mode: auto-fill; -*-
#+TITLE: Proposal for Orgmode: references from code to text.
#+AUTHOR: KDr2

[...]

Post by ZHUO QL (KDr2)
- Beside the ~NAME~, we could also support reference via the
~CUSTOM_ID~ property or anything else that can locate a text chunk.
- In the tangling process, we should do some kind of transformation of
the target text, for example, escape the quote mark. A hook may be
needed to let the user customize the transformation too.
What do you think?

I like the symmetry of having src reference text. I guess, in what you
propose above, that the text that would be incorporated would be a
single element (in org parsing parlance) or else we would have to
specify an extent in the link information.
Are you able to implement something? I would definitely help test it.
--
Eric S Fraga via Emacs 27.0.50, Org release_9.1.6-591-gee336b

[Eric S Fraga]

... continued ...

although I found a strange bug in the tangling and noweb syntax. I
didn't want to have quotes in the text so I removed them and put them in
the lisp src block instead:

#+begin_src org
,#+NAME: ADD
,#+BEGIN_SRC emacs-lisp -n -r :noweb yes :tangle test.el

(defun add (x y)
"<<DOC-OF-ADD>>" ;; code to code
(+ x y) (ref:add)
)
,#+END_SRC
#+end_src

which when exported or tangled resulted in some strange quoting:

#+begin_src emacs-lisp
(defun add (x y)
"
"We use the function add to calculate the sum of two numbers." ;; code to code
(+ x y)
)
#+end_src

I'm probably not supposed to put a <<...>> link within quotes but the
output is surprising in any case.

[Eric S Fraga]

Post by John Kitchin
It seems like this is nearly all possible already (see below). It is a
little wonky that you have to put text in a src txt block to achieve it,
but otherwise what is missing?

Ah, yes, quite nice. Thanks!

[ZHUO QL (KDr2)]

Well, it's a nice approach.
But it has a few cons:
1. Putting text in a source block is not a very natural way. It may disturb the fluidity, both for writing the Org source and reading the exported document(e.g. html);2. The bug Eric just found while putting a <<...>> link within quotes, although, it may be easy to fix. 

Greetings.

ZHUO QL (KDr2, http://kdr2.com)



On Monday, May 14, 2018, 11:06:34 PM GMT+8, John Kitchin <***@andrew.cmu.edu> wrote:

It seems like this is nearly all possible already (see below). It is a little wonky that you have to put text in a src txt block to achieve it, but otherwise what is missing?
* Main document
See [[id:BAD97113-3561-4A4A-BA07-0CD5BF6BA35F][There is a reason we only support two args]] (text to text) for notes about this function. The addition is done in line [[(add)]] (text to code).
#+NAME: DOC-OF-ADD#+BEGIN_SRC txt"We use the function add to calculate the sum of two numbers."#+END_SRC
#+NAME: ADD#+BEGIN_SRC emacs-lisp -n -r :noweb yes :tangle test.el(defun add (x y)  <<DOC-OF-ADD>> ;; code to code  ;; [[id:BAD97113-3561-4A4A-BA07-0CD5BF6BA35F][There is a reason we only support two args]] code to text  (+ x y) (ref:add)  )#+END_SRC
#+RESULTS: ADD: add

** Notes
*** There is a reason we only support two args    :PROPERTIES:    :ID:       BAD97113-3561-4A4A-BA07-0CD5BF6BA35F    :END:

John

-----------------------------------
Professor John Kitchin 
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-***@johnkitchin
http://kitchingroup.cheme.cmu.edu

[Eric S Fraga]

2. The bug Eric just found while putting a <> link within
quotes, although, it may be easy to fix.

I realised afterwards that it is not a bug but is a feature: org assumes
that anything before the << start of the link should be repeated on each
line of the incorporated src block. This allows for easy incorporation
of code into a comment block, e.g. for C or sh etc.

[ZHUO QL (KDr2)]

Oh, that makes sense.
So in this way, we must put the text into quotes, and then into a txt code block. And also we don't have a chance to do any transformation of the text(e.g. org-mode list to javadoc format).  
I think the direct reference from code to text is better. I will try to implement it, but I'm not very familiar with the code base and don't have much time, so I can't give any commitment. But once I have anything to show, I'll be back here  :)
Greetings.

ZHUO QL (KDr2, http://kdr2.com)

2. The bug Eric just found while putting a <> link within
quotes, although, it may be easy to fix. 

I realised afterwards that it is not a bug but is a feature: org assumes
that anything before the << start of the link should be repeated on each
line of the incorporated src block.  This allows for easy incorporation
of code into a comment block, e.g. for C or sh etc.

[John Kitchin]

* Main document

See [[id:BAD97113-3561-4A4A-BA07-0CD5BF6BA35F][There is a reason we only
support two args]] (text to text) for notes about this function. The
addition is done in line [[(add)]] (text to code).

Here we put names on different kinds of elements so we can put them into a
tangled file later.

#+NAME: DOC-OF-ADD
We use the function add to calculate the sum of two numbers.

#+NAME: add-options
- one
- two
- three
- and of course "optional things"


We use a block like this to get the contents of an org-element by name as a
string, and possibly transform it some how, e.g. in this case I escape
quotes. I guess you could also use an exporter to convert it to what ever
form you want. You might bury this block at the end in an appendix so it
isn't in the middle of your document like this.

#+name: get-string
#+BEGIN_SRC emacs-lisp :var name="add-options"
(let ((el (org-element-map (org-element-parse-buffer)
org-element-all-elements
(lambda (el)
(when (string= (org-element-property :name el) name)
el))
nil t)))
(let ((s (buffer-substring (org-element-property :contents-begin el)
(org-element-property :contents-end el))))
(replace-regexp-in-string "\\\"" "\\\\\"" s)))
#+END_SRC


Now, we can use those elements in a src-block like this.

#+NAME: ADD
#+BEGIN_SRC emacs-lisp -n -r :noweb yes :tangle test.el
(defun add (x y)
"One line description of adding X and Y.
<<get-string("DOC-OF-ADD")>> ;; code to code
<<get-string("add-options")>>"
;; [[id:BAD97113-3561-4A4A-BA07-0CD5BF6BA35F][There is a reason we only
support two args]] code to text
(+ x y) (ref:add)
;; it appears the coderef needs to be on it's own line, otherwise you get
a org-link-search: No match for coderef: add when you click on the link.
)
#+END_SRC

#+RESULTS: ADD
: add

The above block tangles to this for me:

#+BEGIN_SRC emacs-lisp
(defun add (x y)
"One line description of adding X and Y.
We use the function add to calculate the sum of two numbers.
;; code to code
- one
- two
- three
- and of course \"optional things\"
"
;; [[id:BAD97113-3561-4A4A-BA07-0CD5BF6BA35F][There is a reason we only
support two args]] code to text
(+ x y)
;; it appears the coderef needs to be on it's own line, otherwise you get
a org-link-search: No match for coderef: add when you click on the link.
)

#+END_SRC


** Notes

*** There is a reason we only support two args
:PROPERTIES:
:ID: BAD97113-3561-4A4A-BA07-0CD5BF6BA35F
:END:


John

-----------------------------------
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Post by ZHUO QL (KDr2)
Oh, that makes sense.
So in this way, we must put the text into quotes, and then into a txt code
block. And also we don't have a chance to do any transformation of the
text(e.g. org-mode list to javadoc format).
I think the direct reference from code to text is better. I will try to
implement it, but I'm not very familiar with the code base and don't have
much time, so I can't give any commitment. But once I have anything to
show, I'll be back here :)
Greetings.
ZHUO QL (KDr2, http://kdr2.com)
On Tuesday, May 15, 2018, 6:05:31 PM GMT+8, Eric S Fraga <

2. The bug Eric just found while putting a <> link within
quotes, although, it may be easy to fix.

I realised afterwards that it is not a bug but is a feature: org assumes
that anything before the << start of the link should be repeated on each
line of the incorporated src block. This allows for easy incorporation
of code into a comment block, e.g. for C or sh etc.
--
Eric S Fraga via Emacs 27.0.50, Org release_9.1.6-591-gee336b

[Samuel Wales]

i'm not sure if i understand the goal. is it babel-only?

this email might be off-topic.

what i want is links, using something like org-id,

- from non-org [for example, a link in file.el in a comment]
- to non-org [for example, a named or id'ed location in the same file]
- or to org [e.g. an org-id'ed org entry, or a named or id'ed
location in a babel block]

that can be, if needed, effectively, bidirectional. thus, you could
document a function in file.el using a subtree in main.org and bounce
between the function and the documentation because there is a link for
each.

non-org [e.g. file.el] can be an original file [i.e. no babel
anywhere] or the result of tangling from babel.

i find that org-link-minor-mode is /highly/ useful for the non-org
part of this, but it doesn't support all of the above functionality
automatically.

i'd want a general package that is like org-link-minor-mode, but takes
care of supporting the user by creating good ids and links as needed.