CMSM

Common markup state machine.

Together, the parsing rules described below define what is referred to as a Common Markup parser.

This document is currently in progress. It is developed jointly with a reference parser: micromark. Contributions are welcome.

Some parts that are still in progress:

• Adapters
• Define the regular constructs
• Adapter for rich text to check whether emphasis, strong, resource, or reference sequences make up syntax or text
• Tokenizing the input stream in reverse (GFM allows [email protected], so it seems we need to somehow allow to match the @ and parse backwards)
• Add an appendix of extensions

Table of contents

• 1 Background
• 2 Overview
• 3 Infra
• 4 Characters
  • 4.1 Character groups
  • 4.2 Conceptual characters
  • 4.3 Tabs
• 5 Input stream
  • 5.1 Preprocessing the input stream
• 6 Parsing
  • 6.1 Tokenization
  • 6.2 State
  • 6.3 Constructs
  • 6.4 Effects
• 7 Flow state machine
  • 7.1 Flow prefix start state
  • 7.2 Flow start state
  • 7.3 Flow prefix initial state
  • 7.4 Flow initial state
  • 7.5 Blank line state
  • 7.6 ATX heading start state
  • 7.7 ATX heading fence open inside state
  • 7.8 ATX heading inside state
  • 7.9 Thematic break asterisk start state
  • 7.10 Thematic break asterisk inside state
  • 7.11 Setext heading underline dash start state
  • 7.12 Setext heading underline dash inside state
  • 7.13 Setext heading underline dash after state
  • 7.14 Thematic break dash start state
  • 7.15 Thematic break dash inside state
  • 7.16 Flow HTML start state
  • 7.17 Flow HTML tag open state
  • 7.18 Flow HTML markup declaration open state
  • 7.19 Flow HTML end tag open state
  • 7.20 Flow HTML tag name state
  • 7.21 Flow HTML basic self closing state
  • 7.22 Flow HTML complete attribute name before state
  • 7.23 Flow HTML complete attribute name state
  • 7.24 Flow HTML complete attribute name after state
  • 7.25 Flow HTML complete attribute value before state
  • 7.26 Flow HTML complete attribute value double quoted state
  • 7.27 Flow HTML complete attribute value single quoted state
  • 7.28 Flow HTML complete attribute value unquoted state
  • 7.29 Flow HTML complete self closing state
  • 7.30 Flow HTML complete tag after state
  • 7.31 Flow HTML continuation state
  • 7.32 Flow HTML continuation comment inside state
  • 7.33 Flow HTML continuation raw tag open state
  • 7.34 Flow HTML continuation raw end tag state
  • 7.35 Flow HTML continuation character data inside state
  • 7.36 Flow HTML continuation declaration before state
  • 7.37 Flow HTML continuation close state
  • 7.38 Setext heading underline equals to start state
  • 7.39 Setext heading underline equals to inside state
  • 7.40 Setext heading underline equals to after state
  • 7.41 Thematic break underscore start state
  • 7.42 Thematic break underscore inside state
  • 7.43 Fenced code grave accent start state
  • 7.44 Fenced code grave accent open fence inside state
  • 7.45 Fenced code grave accent open fence after state
  • 7.46 Fenced code grave accent continuation state
  • 7.47 Fenced code grave accent close fence inside state
  • 7.48 Fenced code grave accent close fence after state
  • 7.49 Fenced code grave accent continuation inside state
  • 7.50 Fenced code tilde start state
  • 7.51 Fenced code tilde open fence inside state
  • 7.52 Fenced code tilde open fence after state
  • 7.53 Fenced code tilde continuation state
  • 7.54 Fenced code tilde close fence inside state
  • 7.55 Fenced code tilde close fence after state
  • 7.56 Fenced code tilde continuation inside state
  • 7.57 Flow content state
• 8 Content state machine
  • 8.1 Content start state
  • 8.2 Content initial state
  • 8.3 Definition label start state
  • 8.4 Definition label before state
  • 8.5 Definition label inside state
  • 8.6 Definition label between state
  • 8.7 Definition label escape state
  • 8.8 Definition label after state
  • 8.9 Definition destination before state
  • 8.10 Definition destination quoted inside state
  • 8.11 Definition destination quoted escape state
  • 8.12 Definition destination unquoted inside state
  • 8.13 Definition destination unquoted escape state
  • 8.14 Definition destination after state
  • 8.15 Definition title before state
  • 8.16 Definition title or label before state
  • 8.17 Definition title double quoted state
  • 8.18 Definition title double quoted between state
  • 8.19 Definition title double quoted escape state
  • 8.20 Definition title single quoted state
  • 8.21 Definition title single quoted between state
  • 8.22 Definition title single quoted escape state
  • 8.23 Definition title paren quoted state
  • 8.24 Definition title paren quoted between state
  • 8.25 Definition title paren quoted escape state
  • 8.26 Definition title after state
  • 8.27 Phrasing content state
• 9 Text state machine
  • 9.1 Text start state
  • 9.2 Text initial state
  • 9.3 Text state
  • 9.4 Plain text state
  • 9.5 End-of-line state
  • 9.6 Plain end-of-line state
  • 9.7 Image label start state
  • 9.8 Image label start after state
  • 9.9 Character reference state
  • 9.10 Character reference start after state
  • 9.11 Character reference named state
  • 9.12 Character reference numeric state
  • 9.13 Character reference hexadecimal start state
  • 9.14 Character reference hexadecimal state
  • 9.15 Character reference decimal state
  • 9.16 Delimiter run asterisk start state
  • 9.17 Delimiter run asterisk state
  • 9.18 Autolink state
  • 9.19 Autolink open state
  • 9.20 Autolink email atext state
  • 9.21 Autolink email label state
  • 9.22 Autolink email at sign or dot state
  • 9.23 Autolink email dash state
  • 9.24 Autolink scheme or email atext state
  • 9.25 Autolink scheme inside or email atext state
  • 9.26 Autolink URI inside state
  • 9.27 HTML state
  • 9.28 HTML open state
  • 9.29 HTML declaration start state
  • 9.30 HTML comment open inside state
  • 9.31 HTML comment inside state
  • 9.32 HTML comment close inside state
  • 9.33 HTML comment close state
  • 9.34 HTML CDATA inside state
  • 9.35 HTML declaration inside state
  • 9.36 HTML instruction inside state
  • 9.37 HTML instruction close state
  • 9.38 HTML tag close start state
  • 9.39 HTML tag close inside state
  • 9.40 HTML tag close between state
  • 9.41 HTML tag open inside state
  • 9.42 HTML tag open between state
  • 9.43 HTML tag open self closing state
  • 9.44 HTML tag open attribute name state
  • 9.45 HTML tag open attribute name after state
  • 9.46 HTML tag open attribute before state
  • 9.47 HTML tag open double quoted attribute state
  • 9.48 HTML tag open single quoted attribute state
  • 9.49 HTML tag open unquoted attribute state
  • 9.50 Link label start state
  • 9.51 Character escape state
  • 9.52 Character escape after state
  • 9.53 Break escape state
  • 9.54 Break escape after state
  • 9.55 Label resource close state
  • 9.56 Label resource end after state
  • 9.57 Resource information open state
  • 9.58 Resource information destination quoted inside state
  • 9.59 Resource information destination quoted escape state
  • 9.60 Resource information destination quoted after state
  • 9.61 Resource information destination unquoted inside state
  • 9.62 Resource information destination unquoted escape state
  • 9.63 Resource information between state
  • 9.64 Resource information title double quoted inside state
  • 9.65 Resource information title double quoted escape state
  • 9.66 Resource information title single quoted inside state
  • 9.67 Resource information title single quoted escape state
  • 9.68 Resource information title paren quoted inside state
  • 9.69 Resource information title paren quoted escape state
  • 9.70 Resource information title after state
  • 9.71 Label reference close state
  • 9.72 Label reference end after state
  • 9.73 Reference label open state
  • 9.74 Reference label inside state
  • 9.75 Reference label between state
  • 9.76 Reference label escape state
  • 9.77 Label reference shortcut close state
  • 9.78 Delimiter run underscore start state
  • 9.79 Delimiter run underscore state
  • 9.80 Code start state
  • 9.81 Code open fence inside state
  • 9.82 Code between state
  • 9.83 Code inside state
  • 9.84 Code close fence inside state
• 10 Labels
  • 10.1 NOK label
  • 10.2 Blank line end label
  • 10.3 ATX heading start label
  • 10.4 ATX heading fence start label
  • 10.5 ATX heading fence end label
  • 10.6 ATX heading end label
  • 10.7 Thematic break start label
  • 10.8 Thematic break end label
  • 10.9 Setext heading underline start label
  • 10.10 Setext heading underline end label
  • 10.11 Fenced code start label
  • 10.12 Fenced code fence start label
  • 10.13 Fenced code fence sequence start label
  • 10.14 Fenced code fence sequence end label
  • 10.15 Fenced code fence end label
  • 10.16 Fenced code end label
  • 10.17 Content definition partial label
  • 10.18 Content definition start label
  • 10.19 Content definition label start label
  • 10.20 Content definition label open label
  • 10.21 Content definition label close label
  • 10.22 Content definition label end label
  • 10.23 Content definition destination start label
  • 10.24 Content definition destination quoted open label
  • 10.25 Content definition destination quoted close label
  • 10.26 Content definition destination unquoted open label
  • 10.27 Content definition destination unquoted close label
  • 10.28 Content definition destination end label
  • 10.29 Content definition title start label
  • 10.30 Content definition title open label
  • 10.31 Content definition title close label
  • 10.32 Content definition title end label
  • 10.33 Content definition end label
  • 10.34 Hard break label
  • 10.35 Soft break label
  • 10.36 Image label start label
  • 10.37 Image label open label
  • 10.38 Character reference start label
  • 10.39 Character reference end label
  • 10.40 Delimiter run start label
  • 10.41 Delimiter run end label
  • 10.42 Autolink start label
  • 10.43 Autolink open label
  • 10.44 Autolink email close label
  • 10.45 Autolink email end label
  • 10.46 Autolink uri close label
  • 10.47 Autolink uri end label
  • 10.48 HTML start label
  • 10.49 HTML end label
  • 10.50 Link label start label
  • 10.51 Link label open label
  • 10.52 Character escape start label
  • 10.53 Character escape end label
  • 10.54 Break escape start label
  • 10.55 Break escape end label
  • 10.56 Label close label
  • 10.57 Label end label
  • 10.58 Resource information start label
  • 10.59 Resource information open label
  • 10.60 Resource information destination quoted start label
  • 10.61 Resource information destination quoted open label
  • 10.62 Resource information destination quoted close label
  • 10.63 Resource information destination quoted end label
  • 10.64 Resource information destination unquoted start label
  • 10.65 Resource information destination unquoted open label
  • 10.66 Resource information destination unquoted close label
  • 10.67 Resource information destination unquoted end label
  • 10.68 Resource information title start label
  • 10.69 Resource information title open label
  • 10.70 Resource information title close label
  • 10.71 Resource information title end label
  • 10.72 Resource information close label
  • 10.73 Resource information end label
  • 10.74 Reference label start label
  • 10.75 Reference label open label
  • 10.76 Reference label collapsed close label
  • 10.77 Reference label full close label
  • 10.78 Reference label end label
  • 10.79 Code start label
  • 10.80 Code fence start label
  • 10.81 Code fence end label
  • 10.82 Code end label
• 11 Tokens
  • 11.1 Whitespace token
  • 11.2 Line terminator token
  • 11.3 End-of-file token
  • 11.4 End-of-line token
  • 11.5 Marker token
  • 11.6 Sequence token
  • 11.7 Content token
• 12 Appendix
  • 12.1 Raw tags
  • 12.2 Basic tags
  • 12.3 Named character references
• 13 References
• 14 Acknowledgments
• 15 License

1 Background

The common markup parser parses a markup language that is commonly known as Markdown.

The first definition of this format gave several examples of how it worked, showing input Markdown and output HTML, and came with a reference implementation (known as Markdown.pl). When new implementations followed, they mostly followed the first definition, but deviated from the first implementation, thus making the format a family of formats.

Some years later, an attempt was made to standardize the differences between implementations, by specifying how several edge cases should be handled, through more input and output examples. This attempt is known as CommonMark, and many implementations now follow it.

This document defines a more formal format, based on CommonMark, by documenting how to parse it, instead of documenting input and output examples. This format is:

• strict, as it defines a state machine, which leaves significantly less room for interpretation
• agnostic of HTML, as it does not show examples of output, which lets the format be used in new ways
• streaming, because coupling with HTML is what requires a whole stream to be buffered as references can resolve to later definitions
• complete, as it defines different types of tokens and how they are grouped, which allows the format to be represented as a concrete syntax tree
• extensible, because the format is often used in combination with new and custom constructs

The origin story of Markdown is similar to that of HTML, which at a time was also a family of formats. Through incredible efforts of the WHATWG, a Living Standard was created on how to parse the format, by defining a state machine.

2 Overview

The common markup parser receives input, typically coming over the network or from the local file system. This input is represented as characters in the input stream. Depending on a character, certain effects occur, such as that a new token is created, one state is switched to another, or something is labelled. Each line is made up of tokens, such as whitespace, markers, sequences, and content, and labels, that are both enqueued. At a certain point, it is known what to do with the queue, whether to discard it or to use it, in which case it is adapted.

The parser parses in three stages: flow, content, and text, respectively coming with their own state machines (flow state machine, content state machine, text state machine), and their own adapters.

3 Infra

This section defines the fundamental concepts upon which this document is built.

A variable is declared in the shared state with let, cleared with unset, or changed with set, increment, decrement, append or prepend.

4 Characters

A character is a Unicode code point and is represented as a four to six digit hexadecimal number, prefixed with U+ ([UNICODE]).

4.1 Character groups

An ASCII digit is a character in the inclusive range U+0030 (0) to U+0039 (9).

An ASCII upper hex digit a character in the inclusive range U+0041 (A) to U+0046 (F).

An ASCII lower hex digit a character in the inclusive range U+0061 (a) to U+0066 (f).

An ASCII hex digit is an ASCII digit, ASCII upper hex digit, or an ASCII lower hex digit

An ASCII upper alpha is a character in the inclusive range U+0041 (A) to U+005A (Z).

An ASCII lower alpha is a character in the inclusive range U+0061 (a) to U+007A (z).

An ASCII alpha is an ASCII upper alpha or ASCII lower alpha.

An ASCII alphanumeric is an ASCII digit or ASCII alpha.

An ASCII punctuation is a character in the inclusive ranges U+0021 EXCLAMATION MARK (!) to U+002F SLASH (/), U+003A COLON (:) to U+0040 AT SIGN (@), U+005B LEFT SQUARE BRACKET ([) to U+0060 GRAVE ACCENT (`), or U+007B LEFT CURLY BRACE ({) to U+007E TILDE (~).

An ASCII control is a character in the inclusive range U+0000 NULL (NUL) to U+001F (US), or U+007F (DEL).

A Unicode whitespace is a character in the Unicode Zs (Separator, Space) category, or U+0009 CHARACTER TABULATION (HT), U+000A LINE FEED (LF), U+000C (FF), or U+000D CARRIAGE RETURN (CR) ([UNICODE]).

A Unicode punctuation is a character in the Unicode Pc (Punctuation, Connector), Pd (Punctuation, Dash), Pe (Punctuation, Close), Pf (Punctuation, Final quote), Pi (Punctuation, Initial quote), Po (Punctuation, Other), or Ps (Punctuation, Open) categories, or an ASCII punctuation ([UNICODE]).

An atext is an ASCII alphanumeric, or a character in the inclusive ranges U+0023 NUMBER SIGN (#) to U+0027 APOSTROPHE ('), U+002A ASTERISK (*), U+002B PLUS SIGN (+), U+002D DASH (-), U+002F SLASH (/), U+003D EQUALS TO (=), U+003F QUESTION MARK (?), U+005E CARET (^) to U+0060 GRAVE ACCENT (`), or U+007B LEFT CURLY BRACE ({) to U+007E TILDE (~) ([RFC5322]).

To ASCII-lowercase a character, is to increase it by 0x20, if it an ASCII upper alpha.

To digitize a character, is to decrease it by 0x30, 0x37, or 0x57, if it is an ASCII digit, ASCII upper hex digit, or ASCII lower hex digit, respectively.

4.2 Conceptual characters

A VIRTUAL SPACE character is a conceptual character representing an expanded column size of a U+0009 CHARACTER TABULATION (HT).

An EOL character is a conceptual character representing a break between two lines.

An EOF character is a conceptual character representing the end of the input.

VIRTUAL SPACE, EOL, and EOF are not real characters, but rather represent a character increase the size of a character, a break between characters, or the lack of any further characters.

4.3 Tabs

Tabs (U+0009 CHARACTER TABULATION (HT)) are typically not expanded into spaces, but do behave as if they were replaced by spaces with a tab stop of 4 characters. These character increments are represented by VIRTUAL SPACE characters.

For the following markup (where ␉ represent a tab):

>␉␉a

We have the characters: U+003E GREATER THAN (>), U+0009 CHARACTER TABULATION (HT), VIRTUAL SPACE, VIRTUAL SPACE, U+0009 CHARACTER TABULATION (HT), VIRTUAL SPACE, VIRTUAL SPACE, VIRTUAL SPACE, and U+0061 (a).

When transforming to an output format, tab characters that are not part of syntax should be present in the output format. When the tab itself (and zero or more VIRTUAL SPACE characters) are part of syntax, but some VIRTUAL SPACE characters are not, the remaining VIRTUAL SPACE characters should be considered a prefix of the content.

5 Input stream

The input stream consists of the characters pushed into it.

The input character is the first character in the input stream that has not yet been consumed. Initially, the input character is the first character in the input. When the last character in a line is consumed, the input character is an EOL. Finally, when all character are consumed, the input character is an EOF.

Any occurrences of U+0009 CHARACTER TABULATION (HT) in the input stream is represented by that character and 0-3 VIRTUAL SPACE characters.

5.1 Preprocessing the input stream

The input stream consists of the characters pushed into it as the input is decoded.

The input, when decoded, is preprocessed and pushed into the input stream as described in the following algorithm:

• Let tabSize be 4

• Let line be 1

• Let column be 1

• Let offset be 0

• Check:

  If offset is equal to the length of the document, push an EOF into the input stream representing the lack of any further characters, and return

  Otherwise, if the current character is:

  • ↪ U+0000 NULL (NUL)

    Increment offset by 1, increment column by 1, push a U+FFFD REPLACEMENT CHARACTER (�) into the input stream, and go to the step labelled check

  • ↪ U+0009 CHARACTER TABULATION (HT)

    Set count to the result of calculating (tabSize - 1) - (column % tabSize). Increment offset by 1, increment column by 1, and push the character into the input stream.

    Perform the following steps count times: increment column by 1 and push a VIRTUAL SPACE into the input stream representing the size increase.

    Finally, go to the step labelled check

  • ↪ U+000A LINE FEED (LF)

    Increment offset by 1, increment line by 1, and set column to 1, push an EOL into the input stream representing the character, and go to the step labelled check

  • ↪ U+000D CARRIAGE RETURN (CR)

    Increment offset by 1, increment line by 1, set column to 1, and go to the step labelled carriage return check

  • ↪ Anything else

    Increment offset by 1, increment column by 1, push the character into the input stream, and go to the step labelled check

• Carriage return check: if the current character is:

  • ↪ U+000A LINE FEED (LF)

    Increment offset by 1 and push an EOL into the input stream representing the previous and current characters

  • ↪ Anything else

    Push an EOL into the input stream representing the previous character and perform the step labelled check on the current character

6 Parsing

The states of state machines have certain effects, such as that they create items in the queue (tokens and labels). The queue is used by tree adapters, in case a valid construct is found. After using the queue, or when in a bogus construct is found, the queue is discarded.

The shared space is accessed and mutated by both the tree adapter and the states of the state machine.

Constructs are registered by hooking a case (one or more characters or character groups) into certain states. Upon registration, they define the states used to parse a construct, and the adapter used to handle the construct.

6.1 Tokenization

Implementations must act as if they use several state machines to tokenize common markup. The flow state machine is used to tokenize the line constructs that make up the structure of the document. The content state machine is used to tokenize the inline constructs part of content blocks. The text state machine is used to tokenize the inline constructs part of rich or plain text.

Most states consume the input character, and either remain in the state to consume the next character, reconsume the input character in a different state, or switch to a different state to consume the next character. States enqueue tokens and labels.

6.2 State

The shared space is a map of key/value pairs.

The queue is a list of tokens and labels that are enqueued. The current token is the last token in the queue.

6.3 Constructs

Markup is parsed per construct. Some constructs are considered regular (those from CommonMark, such as ATX headings) and other constructs are extensions (such as YAML frontmatter or MDX).

❗️ Define constructs.

6.4 Effects

6.4.1 Switch

To switch to a state is to wait for the next character in the given state.

6.4.2 Consume

To consume the input character affects the current token. Due to the nature of the state machine, it is not possible to consume if there is no current token.

6.4.3 Reconsume

To reconsume is to switch to the given state, and consume the input character there.

6.4.4 Enqueue

To enqueue a label is to mark a point between two tokens with a semantic name, at which point there is no current token.

To enqueue a token is to add a new token of the given type to the queue, making it the new current token.

6.4.5 Ensure

To ensure a token is to enqueue that token if the current token is not of the given type, and otherwise do nothing.

7 Flow state machine

The flow state machine is used to tokenize the line constructs that make up the structure of the document (such as headings or thematic breaks) and must start in the Flow prefix start state.

7.1 Flow prefix start state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Flow start state

7.2 Flow start state

Hookable, there are no regular hooks

• ↪ Anything else

  Reconsume in the Flow initial state

7.3 Flow prefix initial state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Flow initial state

7.4 Flow initial state

❗️ Todo: Indented code v.s. content

Hookable, the regular hooks are:

• EOL: Blank line state
• U+0023 NUMBER SIGN (#): ATX heading start state
• U+002A ASTERISK (*): Thematic break asterisk start state
• U+002A ASTERISK (*):
• U+002B PLUS SIGN (+):
• U+002D DASH (-): Setext heading underline dash start state
• U+002D DASH (-): Thematic break dash start state
• U+002D DASH (-):
• U+003C LESS THAN (<): Flow HTML start state
• U+003D EQUALS TO (=): Setext heading underline equals to start state
• U+003E GREATER THAN (>):
• U+005F UNDERSCORE (_): Thematic break underscore start state
• U+0060 GRAVE ACCENT (`): Fenced code grave accent start state
• U+007E TILDE (~): Fenced code tilde start state
• ASCII digit:

❗️ Todo, continuation:

• Flow HTML continuation state
• Fenced code grave accent continuation state
• Fenced code tilde continuation state

• ↪ EOF

  Enqueue an End-of-file token

• ↪ Anything else

  Reconsume in the Flow content state

7.5 Blank line state

• ↪ EOL

  Enqueue a Blank line end label, enqueue an End-of-line token, and consume

• ↪ Anything else

  Enqueue a NOK label

7.6 ATX heading start state

• ↪ U+0023 NUMBER SIGN (#)

  Let sizeFence be 1, enqueue an ATX heading start label, enqueue an ATX heading fence start label, enqueue a Sequence token, consume, and switch to the ATX heading fence open inside state

• ↪ Anything else

  Enqueue a NOK label

7.7 ATX heading fence open inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Unset sizeFence, enqueue an ATX heading fence end label and reconsume in the ATX heading inside state

• ↪ U+0023 NUMBER SIGN (#)

  If sizeFence is not 6, increment sizeScheme by 1 and consume

  Otherwise, treat it as per the “anything else” entry below

• ↪ Anything else

  Unset sizeFence and enqueue a NOK label

7.8 ATX heading inside state

• ↪ EOF

  Enqueue an ATX heading end label and enqueue an End-of-file token

• ↪ EOL

  Enqueue an ATX heading end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+0023 NUMBER SIGN (#)

  Ensure a Sequence token and consume

• ↪ Anything else

  Ensure a Content token and consume

7.9 Thematic break asterisk start state

• ↪ U+002A ASTERISK (*)

  Let sizeTotalSequence be 1, enqueue a Thematic break start label, enqueue a Sequence token, consume, and switch to the Thematic break asterisk inside state

• ↪ Anything else

  Enqueue a NOK label

7.10 Thematic break asterisk inside state

• ↪ EOF

  If sizeTotalSequence is greater than or equal to 3, unset sizeTotalSequence, enqueue a Thematic break end label, and enqueue an End-of-file token

  Otherwise, treat it as per the “anything else” entry below

• ↪ EOL

  If sizeTotalSequence is greater than or equal to 3, unset sizeTotalSequence, enqueue a Thematic break end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

  Otherwise, treat it as per the “anything else” entry below

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+002A ASTERISK (*)

  Increment sizeTotalSequence by 1, ensure a Sequence token, and consume

• ↪ Anything else

  Unset sizeTotalSequence and enqueue a NOK label

7.11 Setext heading underline dash start state

❗️ Todo: exit if not preceded by content

• ↪ U+002D DASH (-)

  Enqueue a Setext heading underline start label, enqueue a Sequence token, consume, and switch to the Setext heading underline dash inside state

• ↪ Anything else

  Enqueue a NOK label

7.12 Setext heading underline dash inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Setext heading underline dash after state

• ↪ U+002D DASH (-)

  Consume

• ↪ Anything else

  Enqueue a NOK label

7.13 Setext heading underline dash after state

❗️ Todo: Close content if ok, create a new content if nok

• ↪ EOF

  Enqueue a Setext heading underline end label and enqueue an End-of-file token

• ↪ EOL

  Enqueue a Setext heading underline end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Enqueue a NOK label

7.14 Thematic break dash start state

• ↪ U+002D DASH (-)

  Let sizeTotalSequence be 1, enqueue a Thematic break start label, enqueue a Sequence token, consume, and switch to the Thematic break dash inside state

• ↪ Anything else

  Enqueue a NOK label

7.15 Thematic break dash inside state

• ↪ EOF

  If sizeTotalSequence is greater than or equal to 3, unset sizeTotalSequence, enqueue a Thematic break end label, and enqueue an End-of-file token

  Otherwise, treat it as per the “anything else” entry below

• ↪ EOL

  If sizeTotalSequence is greater than or equal to 3, unset sizeTotalSequence, enqueue a Thematic break end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

  Otherwise, treat it as per the “anything else” entry below

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+002D DASH (-)

  Increment sizeTotalSequence by 1, ensure a Sequence token, and consume

• ↪ Anything else

  Unset sizeTotalSequence and enqueue a NOK label

7.16 Flow HTML start state

• ↪ U+003C LESS THAN (<)

  Let kind be 0, let endTag be null, let tagName be the empty string, enqueue a Content token, consume, and switch to the Flow HTML tag open state

• ↪ Anything else

  Enqueue a NOK label

7.17 Flow HTML tag open state

• ↪ U+0021 EXCLAMATION MARK (!)

  Consume and switch to the Flow HTML markup declaration open state

• ↪ U+002F SLASH (/)

  Set endTag to true, consume, and switch to the Flow HTML end tag open state

• ↪ U+003F QUESTION MARK (?)

  Set kind to 3, unset endTag, consume, and switch to the Flow HTML continuation declaration before state

• ↪ ASCII alpha

  Append the ASCII-lowercased character to tagName, consume, and switch to the Flow HTML tag name state

• ↪ Anything else

  Unset kind, endTag, and tagName, and enqueue a NOK label

7.18 Flow HTML markup declaration open state

• ↪ -- (two U+002D DASH (-) characters)

  Set kind to 2, unset endTag, consume, and switch to the Flow HTML continuation declaration before state

• ↪ [CDATA[ (the five upper letters “CDATA” with a U+005B LEFT SQUARE BRACKET ([) before and after)

  Set kind to 5, unset endTag, consume, and switch to the Flow HTML continuation state

• ↪ ASCII alpha

  Set kind to 4, unset endTag, consume, and switch to the Flow HTML continuation state

• ↪ Anything else

  Unset kind, endTag, and tagName, and enqueue a NOK label

7.19 Flow HTML end tag open state

• ↪ ASCII alpha

  Append the ASCII-lowercased character to tagName, consume, and switch to the Flow HTML tag name state

• ↪ Anything else

  Unset kind, endTag, and tagName, and enqueue a NOK label

7.20 Flow HTML tag name state

• ↪ EOF
  ↪ EOL

  If tagName is a raw tag and endTag is not true, set kind to 1, unset tagName, unset endTag, and reconsume in the Flow HTML continuation state

  Otherwise, if tagName is a basic tag, set kind to 6, unset tagName, unset endTag, and reconsume in the Flow HTML continuation state

  Otherwise, treat it as per the “anything else” entry below

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  If tagName is a raw tag and endTag is not true, set kind to 1, unset tagName, unset endTag, and switch to the Flow HTML continuation state

  ❗️ Todo: ignore this check if interrupting content.

  Otherwise, if tagName is not a raw tag, unset tagName, consume, and switch to the Flow HTML complete attribute name before state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+002D DASH (-)

  Append the character to tagName and consume

• ↪ U+002F SLASH (/)

  If tagName is a basic tag, unset tagName, consume, and switch to the Flow HTML basic self closing state

  ❗️ Todo: ignore this check if interrupting content.

  Otherwise, if tagName is not a raw tag and endTag is not true, unset tagName, consume, and switch to the Flow HTML complete self closing state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+003E GREATER THAN (>)

  If tagName is a raw tag and endTag is not true, set kind to 1, unset tagName, unset endTag, consume, and switch to the Flow HTML continuation state

  Otherwise, if tagName is a basic tag, set kind to 6, unset tagName, unset endTag, and reconsume in the Flow HTML continuation state

  ❗️ Todo: ignore this check if interrupting content.

  Otherwise, if tagName is not a raw tag, unset tagName, consume, and switch to the Flow HTML complete tag after state

  Otherwise, treat it as per the “anything else” entry below

• ↪ ASCII alphanumeric

  Append the ASCII-lowercased character to tagName and consume

• ↪ Anything else

  Unset kind, endTag, and tagName, and enqueue a NOK label

7.21 Flow HTML basic self closing state

• ↪ U+003E GREATER THAN (>)

  Set kind to 6, unset endTag, consume, and switch to the Flow HTML continuation state

• ↪ Anything else

  Unset kind and endTag and enqueue a NOK label

7.22 Flow HTML complete attribute name before state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Consume

• ↪ U+002F SLASH (/)

  If endTag is not true, consume and switch to the Flow HTML complete self closing state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+003A COLON (:)
  ↪ U+005F UNDERSCORE (_)
  ↪ ASCII alpha

  If endTag is not true, consume and switch to the Flow HTML complete attribute name state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+003E GREATER THAN (>)

  Consume and switch to the Flow HTML complete tag after state

• ↪ Anything else

  Unset kind and endTag and enqueue a NOK label

7.23 Flow HTML complete attribute name state

• ↪ U+002D DASH (-)
  ↪ U+002E DOT (.)
  ↪ U+003A COLON (:)
  ↪ U+005F UNDERSCORE (_)
  ↪ ASCII alphanumeric

  Consume

• ↪ Anything else

  Reconsume in the Flow HTML complete attribute name after state

7.24 Flow HTML complete attribute name after state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Consume

• ↪ U+002F SLASH (/)

  If endTag is not true, consume and switch to the Flow HTML complete self closing state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+003D EQUALS TO (=)

  Consume and switch to the Flow HTML complete attribute value before state

• ↪ U+003E GREATER THAN (>)

  Consume and switch to the Flow HTML complete tag after state

• ↪ Anything else

  Unset kind and endTag and enqueue a NOK label

7.25 Flow HTML complete attribute value before state

• ↪ EOF
  ↪ EOL
  ↪ U+003C LESS THAN (<)
  ↪ U+003D EQUALS TO (=)
  ↪ U+003E GREATER THAN (>)
  ↪ U+0060 GRAVE ACCENT (`)

  Unset kind and endTag and enqueue a NOK label

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Consume

• ↪ U+0022 QUOTATION MARK (")

  Consume and switch to the Flow HTML complete attribute value double quoted state

• ↪ U+0027 APOSTROPHE (')

  Consume and switch to the Flow HTML complete attribute value single quoted state

• ↪ Anything else

  Consume and switch to the Flow HTML complete attribute value unquoted state

7.26 Flow HTML complete attribute value double quoted state

• ↪ EOF
  ↪ EOL

  Unset kind and endTag and enqueue a NOK label

• ↪ U+0022 QUOTATION MARK (")

  Consume and switch to the Flow HTML complete attribute name before state

• ↪ Anything else

  Consume

7.27 Flow HTML complete attribute value single quoted state

• ↪ EOF
  ↪ EOL

  Unset kind and endTag and enqueue a NOK label

• ↪ U+0027 APOSTROPHE (')

  Consume and switch to the Flow HTML complete attribute name before state

• ↪ Anything else

  Consume

7.28 Flow HTML complete attribute value unquoted state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)
  ↪ U+0022 QUOTATION MARK (")
  ↪ U+0027 APOSTROPHE (')
  ↪ U+003C LESS THAN (<)
  ↪ U+003D EQUALS TO (=)
  ↪ U+003E GREATER THAN (>)
  ↪ U+0060 GRAVE ACCENT (`)

  Reconsume in the Flow HTML complete attribute name after state

• ↪ Anything else

  Consume

7.29 Flow HTML complete self closing state

• ↪ U+003E GREATER THAN (>)

  Consume and switch to the Flow HTML complete tag after state

• ↪ Anything else

  Unset kind and endTag and enqueue a NOK label

7.30 Flow HTML complete tag after state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Set kind to 7, unset endTag, and reconsume in the Flow HTML continuation state

• ↪ Anything else

  Unset kind and endTag and enqueue a NOK label

7.31 Flow HTML continuation state

• ↪ EOF

  Unset kind, enqueue an HTML end label, and enqueue an End-of-file token

• ↪ EOL

  Enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ U+002D DASH (-)

  If kind is 7, consume and switch to the Flow HTML continuation comment inside state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+003C LESS THAN (<)

  If kind is 1, consume and switch to the Flow HTML continuation raw tag open state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+003E GREATER THAN (>)

  If kind is 4, consume and switch to the Flow HTML continuation close state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+003F QUESTION MARK (?)

  If kind is 3, consume and switch to the Flow HTML continuation declaration before state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+005D RIGHT SQUARE BRACKET (])

  If kind is 5, consume and switch to the Flow HTML continuation character data inside state.

  Otherwise, treat it as per the “anything else” entry below

• ↪ Anything else

  Consume

7.32 Flow HTML continuation comment inside state

• ↪ U+002D DASH (-)

  Consume and switch to the Flow HTML continuation declaration before state

• ↪ Anything else

  Reconsume in the Flow HTML continuation state

7.33 Flow HTML continuation raw tag open state

• ↪ U+002F SLASH (/)

  Let tagName be the empty string, consume, and switch to the Flow HTML continuation raw end tag state

• ↪ Anything else

  Reconsume in the Flow HTML continuation state

7.34 Flow HTML continuation raw end tag state

Note: This state can be optimized by either imposing a maximum size (the size of the longest possible raw tag name) or by using a trie of the possible raw tag names.

• ↪ ASCII alphanumeric

  Append the ASCII-lowercased character to tagName and consume

• ↪ U+003E GREATER THAN (>)

  If tagName is a raw tag, unset `tagName, consume, and switch to the Flow HTML continuation close state

  Otherwise, treat it as per the “anything else” entry below

• ↪ Anything else

  Unset `tagName and reconsume in the Flow HTML continuation state

7.35 Flow HTML continuation character data inside state

• ↪ U+005D RIGHT SQUARE BRACKET (])

  Consume and switch to the Flow HTML continuation declaration before state

• ↪ Anything else

  Reconsume in the Flow HTML continuation state

7.36 Flow HTML continuation declaration before state

• ↪ U+003E GREATER THAN (>)

  Consume and switch to the Flow HTML continuation close state

• ↪ Anything else

  Reconsume in the Flow HTML continuation state

7.37 Flow HTML continuation close state

• ↪ EOF

  Unset kind, enqueue an HTML end label, and enqueue an End-of-file token

• ↪ EOL

  Unset kind, enqueue an HTML end label, and enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ Anything else

  Consume

7.38 Setext heading underline equals to start state

❗️ Todo: exit if not preceded by content

• ↪ U+003D EQUALS TO (=)

  Enqueue a Setext heading underline start label, enqueue a Sequence token, consume, and switch to the Setext heading underline equals to inside state

• ↪ Anything else

  Enqueue a NOK label

7.39 Setext heading underline equals to inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Setext heading underline equals to after state

• ↪ U+003D EQUALS TO (=)

  Consume

• ↪ Anything else

  Enqueue a NOK label

7.40 Setext heading underline equals to after state

❗️ Todo: Close content if ok, create a new content if nok

• ↪ EOF

  Enqueue a Setext heading underline end label and enqueue an End-of-file token

• ↪ EOL

  Enqueue a Setext heading underline end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Enqueue a NOK label

7.41 Thematic break underscore start state

• ↪ U+005F UNDERSCORE (_)

  Let sizeTotalSequence be 1, enqueue a Thematic break start label, enqueue a Sequence token, consume, and switch to the Thematic break underscore inside state

• ↪ Anything else

  Enqueue a NOK label

7.42 Thematic break underscore inside state

• ↪ EOF

  If sizeTotalSequence is greater than or equal to 3, unset sizeTotalSequence, enqueue a Thematic break end label, and enqueue an End-of-file token

  Otherwise, treat it as per the “anything else” entry below

• ↪ EOL

  If sizeTotalSequence is greater than or equal to 3, unset sizeTotalSequence, enqueue a Thematic break end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

  Otherwise, treat it as per the “anything else” entry below

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+005F UNDERSCORE (_)

  Increment sizeTotalSequence by 1, ensure a Sequence token, and consume

• ↪ Anything else

  Unset sizeTotalSequence and enqueue a NOK label

7.43 Fenced code grave accent start state

• ↪ U+0060 GRAVE ACCENT (`)

  Let sizeOpen be 1, enqueue a Fenced code start label, enqueue a Fenced code fence start label, enqueue a Fenced code fence sequence start label, enqueue a Sequence token, consume, and switch to the Fenced code grave accent open fence inside state

• ↪ Anything else

  Enqueue a NOK label

7.44 Fenced code grave accent open fence inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  If sizeOpen is greater than or equal to 3, enqueue a Fenced code fence sequence end label, and reconsume in the Fenced code grave accent open fence after state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+0060 GRAVE ACCENT (`)

  Increment sizeOpen by 1 and consume

• ↪ Anything else

  Unset sizeOpen and enqueue a NOK label

7.45 Fenced code grave accent open fence after state

• ↪ EOF

  Enqueue a Fenced code fence end label and enqueue an End-of-file token

• ↪ EOL

  Enqueue a Fenced code fence end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+0060 GRAVE ACCENT (`)

  Unset sizeOpen and enqueue a NOK label

• ↪ Anything else

  Ensure a Content token and consume

7.46 Fenced code grave accent continuation state

• ↪ U+0060 GRAVE ACCENT (`)

  Let sizeClose be 1, enqueue a Fenced code fence sequence start label, enqueue a Sequence token, consume, and switch to the Fenced code grave accent close fence inside state

• ↪ Anything else

  Reconsume in the Fenced code grave accent continuation inside state

7.47 Fenced code grave accent close fence inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  If sizeClose is greater than or equal to sizeOpen, enqueue a Fenced code fence sequence end label and reconsume in the Fenced code grave accent close fence after state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+0060 GRAVE ACCENT (`)

  Increment sizeClose by 1 and consume

• ↪ Anything else

  Unset sizeClose and reconsume in the Fenced code grave accent continuation inside state

7.48 Fenced code grave accent close fence after state

• ↪ EOF

  Unset sizeOpen, unset sizeClose, enqueue a Fenced code fence end label, enqueue a Fenced code end label, and enqueue an End-of-file token

• ↪ EOL

  Unset sizeOpen, unset sizeClose, enqueue a Fenced code fence end label, enqueue a Fenced code end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Fenced code grave accent continuation inside state

7.49 Fenced code grave accent continuation inside state

• ↪ EOF

  Unset sizeOpen, unset sizeClose, enqueue a Fenced code fence end label, enqueue a Fenced code end label, and enqueue an End-of-file token

• ↪ EOL

  Enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ Anything else

  Ensure a Content token and consume

7.50 Fenced code tilde start state

• ↪ U+007E TILDE (~)

  Let sizeOpen be 1, enqueue a Fenced code start label, enqueue a Fenced code fence start label, enqueue a Fenced code fence sequence start label, enqueue a Sequence token, consume, and switch to the Fenced code tilde open fence inside state

• ↪ Anything else

  Enqueue a NOK label

7.51 Fenced code tilde open fence inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  If sizeOpen is greater than or equal to 3, enqueue a Fenced code fence sequence end label, and reconsume in the Fenced code tilde open fence after state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+007E TILDE (~)

  Increment sizeOpen by 1 and consume

• ↪ Anything else

  Unset sizeOpen and enqueue a NOK label

7.52 Fenced code tilde open fence after state

• ↪ EOF

  Enqueue a Fenced code fence end label and enqueue an End-of-file token

• ↪ EOL

  Enqueue a Fenced code fence end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Ensure a Content token and consume

7.53 Fenced code tilde continuation state

• ↪ U+007E TILDE (~)

  Let sizeClose be 1, enqueue a Fenced code fence sequence start label, enqueue a Sequence token, consume, and switch to the Fenced code tilde close fence inside state

• ↪ Anything else

  Reconsume in the Fenced code tilde continuation inside state

7.54 Fenced code tilde close fence inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  If sizeClose is greater than or equal to sizeOpen, enqueue a Fenced code fence sequence end label, and reconsume in the Fenced code tilde close fence after state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+007E TILDE (~)

  Increment sizeClose by 1 and consume

• ↪ Anything else

  Unset sizeClose and reconsume in the Fenced code tilde continuation inside state

7.55 Fenced code tilde close fence after state

• ↪ EOF

  Unset sizeOpen, unset sizeClose, enqueue a Fenced code fence end label, enqueue a Fenced code end label, and enqueue an End-of-file token

• ↪ EOL

  Unset sizeOpen, unset sizeClose, enqueue a Fenced code fence end label, enqueue a Fenced code end label, enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Fenced code tilde continuation inside state

7.56 Fenced code tilde continuation inside state

• ↪ EOF

  Unset sizeOpen, unset sizeClose, enqueue a Fenced code fence end label, enqueue a Fenced code end label, and enqueue an End-of-file token

• ↪ EOL

  Enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ Anything else

  Ensure a Content token and consume

7.57 Flow content state

• ↪ EOF

  Enqueue an End-of-file token

• ↪ EOL

  Enqueue an End-of-line token, consume, and switch to the Flow prefix initial state

• ↪ Anything else

  Consume

8 Content state machine

The content state machine is used to tokenize the inline constructs part of content blocks in a document (such as regular definitions and phrasing) and must start in the Content start state.

8.1 Content start state

Hookable, the regular hooks are:

• U+005B LEFT SQUARE BRACKET ([): Definition label start state

• ↪ Anything else

  Reconsume in the Content initial state

8.2 Content initial state

Hookable, there are no regular hooks.

• ↪ Anything else

  Reconsume in the Phrasing content state

8.3 Definition label start state

• ↪ U+005B LEFT SQUARE BRACKET ([)

  Enqueue a Content definition start label, enqueue a Content definition label start label, enqueue a Marker token, consume, enqueue a Content definition label open label, and switch to the Definition label before state

• ↪ Anything else

  Enqueue a NOK label

8.4 Definition label before state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue an End-of-line token and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+005D RIGHT SQUARE BRACKET (])

  Enqueue a NOK label

• ↪ Anything else

  Enqueue a Content token, consume, and switch to the Definition label inside state

8.5 Definition label inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Definition label between state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Definition label escape state

• ↪ U+005D RIGHT SQUARE BRACKET (])

  Enqueue a Content definition label close label, enqueue a Marker token, consume, enqueue a Content definition label end label, and switch to the Definition label after state

• ↪ Anything else

  Ensure a Content token and consume

8.6 Definition label between state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue an End-of-line token and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Definition label inside state

8.7 Definition label escape state

• ↪ U+005C BACKSLASH (\)
  ↪ U+005D RIGHT SQUARE BRACKET (])

  Consume and switch to the Definition label inside state

• ↪ Anything else

  Reconsume in the Definition label inside state

8.8 Definition label after state

• ↪ U+003A COLON (:)

  Enqueue a Marker token, consume, and switch to the Definition destination before state

• ↪ Anything else

  Enqueue a NOK label

8.9 Definition destination before state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue an End-of-line token and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+003C LESS THAN (<)

  Enqueue a Content definition destination start label, enqueue a Marker token, enqueue a Content definition destination quoted open label, consume, and switch to the Definition destination quoted inside state

• ↪ ASCII control

  Enqueue a NOK label

• ↪ Anything else

  Let balance be 0, enqueue a Content definition destination start label, enqueue a Content definition destination unquoted open label, enqueue a Content token, and reconsume in the Definition destination unquoted inside state

8.10 Definition destination quoted inside state

• ↪ EOF
  ↪ EOL
  ↪ U+003C LESS THAN (<)

  Enqueue a NOK label

• ↪ U+003E GREATER THAN (>)

  Enqueue a Content definition destination quoted close label, enqueue a Marker token, consume, enqueue a Content definition destination end label, and switch to the Definition destination after state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Definition destination quoted escape state

• ↪ Anything else

  Ensure a Content token and consume

8.11 Definition destination quoted escape state

• ↪ U+003C LESS THAN (<)
  ↪ U+003E GREATER THAN (>)
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Definition destination quoted inside state

• ↪ Anything else

  Reconsume in the Definition destination quoted inside state

8.12 Definition destination unquoted inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Unset balance, enqueue a Content definition destination unquoted close label, enqueue a Content definition destination end label, and reconsume in the Definition title before state

• ↪ U+0028 LEFT PARENTHESIS (()

  Increment balance by 1, ensure a Content token, and consume

• ↪ U+0029 RIGHT PARENTHESIS ())

  If balance is 0, treat it as per the “ASCII control” entry below

  Otherwise, decrement balance by 1, ensure a Content token, and consume

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Definition destination unquoted escape state

• ↪ ASCII control

  Unset balance and enqueue a NOK label

• ↪ Anything else

  Ensure a Content token and consume

8.13 Definition destination unquoted escape state

• ↪ U+0028 LEFT PARENTHESIS (()
  ↪ U+0029 RIGHT PARENTHESIS ())
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Definition destination unquoted inside state

• ↪ Anything else

  Reconsume in the Definition destination unquoted inside state

8.14 Definition destination after state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Definition title before state

• ↪ Anything else

  Enqueue a NOK label

8.15 Definition title before state

• ↪ EOL

  Enqueue a Content definition partial label, enqueue an End-of-line token, consume, and switch to the Definition title or label before state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+005B LEFT SQUARE BRACKET ([)

  Enqueue a NOK label

• ↪ Anything else

  Reconsume in the Definition title or label before state

8.16 Definition title or label before state

• ↪ EOF

  Enqueue a Content definition end label and enqueue an End-of-file token

• ↪ EOL

  Enqueue an End-of-line token and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+0022 QUOTATION MARK (")

  Enqueue a Content definition title start label, enqueue a Marker token, consume, enqueue a Content definition title open label, and switch to the Definition title double quoted state

• ↪ U+0027 APOSTROPHE (')

  Enqueue a Content definition title start label, enqueue a Marker token, consume, enqueue a Content definition title open label, and switch to the Definition title single quoted state

• ↪ U+0028 LEFT PARENTHESIS (()

  Enqueue a Content definition title start label, enqueue a Marker token, consume, enqueue a Content definition title open label, and switch to the Definition title paren quoted state

• ↪ Anything else

  Reconsume in the Content start state

8.17 Definition title double quoted state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Definition title double quoted between state

• ↪ U+0022 QUOTATION MARK (")

  Enqueue a Content definition title close label, enqueue a Marker token, consume, enqueue a Content definition title end label, and switch to the Definition title after state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Definition title double quoted escape state

• ↪ Anything else

  Ensure a Content token and consume

8.18 Definition title double quoted between state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue an End-of-line token and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Definition title double quoted state

8.19 Definition title double quoted escape state

• ↪ U+0022 QUOTATION MARK (")
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Definition title double quoted state

• ↪ Anything else

  Reconsume in the Definition title double quoted state

8.20 Definition title single quoted state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Definition title single quoted between state

• ↪ U+0027 APOSTROPHE (')

  Enqueue a Content definition title close label, enqueue a Marker token, consume, enqueue a Content definition title end label, and switch to the Definition title after state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Definition title single quoted escape state

• ↪ Anything else

  Ensure a Content token and consume

8.21 Definition title single quoted between state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue an End-of-line token and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Definition title single quoted state

8.22 Definition title single quoted escape state

• ↪ U+0027 APOSTROPHE (')
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Definition title single quoted state

• ↪ Anything else

  Reconsume in the Definition title single quoted state

8.23 Definition title paren quoted state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Definition title paren quoted between state

• ↪ U+0029 RIGHT PARENTHESIS ())

  Enqueue a Content definition title close label, enqueue a Marker token, consume, enqueue a Content definition title end label, and switch to the Definition title after state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Definition title paren quoted escape state

• ↪ Anything else

  Ensure a Content token and consume

8.24 Definition title paren quoted between state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue an End-of-line token and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Definition title paren quoted state

8.25 Definition title paren quoted escape state

• ↪ U+0029 RIGHT PARENTHESIS ())
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Definition title paren quoted state

• ↪ Anything else

  Reconsume in the Definition title paren quoted state

8.26 Definition title after state

• ↪ EOF

  Enqueue a Content definition end label and enqueue an End-of-file token

• ↪ EOL

  Enqueue a Content definition end label, enqueue an End-of-line token, consume, and switch to the Content start state

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Enqueue a NOK label

8.27 Phrasing content state

• ↪ EOF

  Enqueue an End-of-file token

• ↪ EOL

  Enqueue an End-of-line token, consume, and switch to the Content initial state

• ↪ Anything else

  Consume

9 Text state machine

The text state machine is used to tokenize the inline constructs part of rich text (such as regular resources and emphasis) or plain text (such as regular character escapes or character references) in a document and must start in the Text start state.

If text is parsed as plain text, the Text start state, Text initial state, and Text state all forward to the Plain text state.

If text is parsed as rich text, an additional variable prev must be tracked. Initial set to EOF, it must be set to the input character right before a character is consumed.

9.1 Text start state

Hookable, there are no regular hooks.

• ↪ Anything else

  Reconsume in the Text initial state

9.2 Text initial state

Hookable, there are no regular hooks.

• ↪ Anything else

  Reconsume in the Text state

9.3 Text state

Hookable, the regular hooks are:

• EOL: End-of-line state
• U+0021 EXCLAMATION MARK (!): Image label start state
• U+0026 AMPERSAND (&): Character reference state
• U+002A ASTERISK (*): Delimiter run asterisk start state
• U+003C LESS THAN (<): Autolink state
• U+003C LESS THAN (<): HTML state
• U+005B LEFT SQUARE BRACKET ([): Link label start state
• U+005C BACKSLASH (\): Character escape state
• U+005C BACKSLASH (\): Break escape state
• U+005D RIGHT SQUARE BRACKET (]): Label resource close state
• U+005D RIGHT SQUARE BRACKET (]): Label reference close state
• U+005D RIGHT SQUARE BRACKET (]): Label reference shortcut close state
• U+005F UNDERSCORE (_): Delimiter run underscore start state
• U+0060 GRAVE ACCENT (`): Code start state

• ↪ EOF

  Enqueue an End-of-file token

• ↪ Anything else

  Ensure a Content token and consume

9.4 Plain text state

Hookable, the regular hooks are:

• EOL: Plain end-of-line state
• U+0026 AMPERSAND (&): Character reference state
• U+005C BACKSLASH (\): Character escape state

• ↪ EOF

  Enqueue an End-of-file token

• ↪ Anything else

  Ensure a Content token and consume

9.5 End-of-line state

• ↪ EOL

  If the break represented by the character starts with two or more of VIRTUAL SPACE, U+0009 CHARACTER TABULATION (HT), or U+0020 SPACE (SP), enqueue a Hard break label, enqueue an End-of-line token, consume, and switch to the Text initial state

  Otherwise, enqueue a Soft break label, enqueue an End-of-line token, consume, and switch to the Text initial state

• ↪ Anything else

  Enqueue a NOK label

9.6 Plain end-of-line state

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and switch to the Text initial state

• ↪ Anything else

  Enqueue a NOK label

9.7 Image label start state

• ↪ U+0021 EXCLAMATION MARK (!)

  Enqueue an Image label start label, enqueue a Marker token, consume, and switch to the Image label start after state

• ↪ Anything else

  Enqueue a NOK label

9.8 Image label start after state

• ↪ U+005B LEFT SQUARE BRACKET ([)

  Enqueue a Marker token, consume, enqueue an Image label open label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.9 Character reference state

• ↪ U+0026 AMPERSAND (&)

  Enqueue a Character reference start label, enqueue a Marker token, consume, and switch to the Character reference start after state

• ↪ Anything else

  Enqueue a NOK label

9.10 Character reference start after state

• ↪ U+0023 NUMBER SIGN (#)

  Enqueue a Marker token, consume, and switch to the Character reference numeric state

• ↪ ASCII alphanumeric

  Let entityName be the empty string, append the character to entityName, enqueue a Content token, consume, and switch to the Character reference named state

• ↪ Anything else

  Enqueue a NOK label

9.11 Character reference named state

Note: This state can be optimized by either imposing a maximum size (the size of the longest possible named character reference) or by using a trie of the possible named character references.

• ↪ U+003B SEMICOLON (;)

  If entityName is a character reference name, unset entityName, enqueue a Marker token, consume, enqueue a Character reference end label, and switch to the Text state

  Otherwise, treat it as per the “anything else” entry below

• ↪ ASCII alphanumeric

  Append the character to entityName and consume

• ↪ Anything else

  Unset entityName and enqueue a NOK label

9.12 Character reference numeric state

• ↪ U+0058 (X)
  ↪ U+0078 (x)

  Let characterReferenceCode be 0, enqueue a Marker token, consume, and switch to the Character reference hexadecimal start state

• ↪ ASCII digit

  Let characterReferenceCode be 0, enqueue a Content token and reconsume in the Character reference decimal state

• ↪ Anything else

  Enqueue a NOK label

9.13 Character reference hexadecimal start state

• ↪ ASCII hex digit

  Enqueue a Content token and reconsume in the Character reference hexadecimal state

• ↪ Anything else

  Unset characterReferenceCode and enqueue a NOK label

9.14 Character reference hexadecimal state

Note: This state can be optimized by imposing a maximum size (the size of the longest possible valid hexadecimal character reference, 6).

• ↪ U+003B SEMICOLON (;)

  Unset characterReferenceCode, enqueue a Marker token, consume, enqueue a Character reference end label, and switch to the Text state

• ↪ ASCII digit

  Multiply characterReferenceCode by 0x10, add the digitized input character to characterReferenceCode, and consume

• ↪ ASCII upper hex digit

  Multiply characterReferenceCode by 0x10, add the digitized input character to characterReferenceCode, and consume

• ↪ ASCII lower hex digit

  Multiply characterReferenceCode by 0x10, add the digitized input character to characterReferenceCode, and consume

• ↪ Anything else

  Unset characterReferenceCode and enqueue a NOK label

9.15 Character reference decimal state

Note: This state can be optimized by imposing a maximum size (the size of the longest possible valid decimal character reference, 7).

• ↪ U+003B SEMICOLON (;)

  Unset characterReferenceCode, enqueue a Marker token, consume, enqueue a Character reference end label, and switch to the Text state

• ↪ ASCII digit

  Multiply characterReferenceCode by 10, add the digitized input character to characterReferenceCode, and consume

• ↪ Anything else

  Unset characterReferenceCode and enqueue a NOK label

9.16 Delimiter run asterisk start state

• ↪ U+002A ASTERISK (*)

  Let delimiterRunAfter be null and let delimiterRunBefore be 'whitespace' if prev is EOF, EOL, or Unicode whitespace, 'punctuation' if it is Unicode punctuation, or null otherwise

  Enqueue a Delimiter run start label, enqueue a Sequence token, consume, and switch to the Delimiter run asterisk state

• ↪ Anything else

  Enqueue a NOK label

9.17 Delimiter run asterisk state

• ↪ EOF
  ↪ EOL
  ↪ Unicode whitespace

  Let delimiterRunAfter be 'whitespace' and treat it as per the “anything else” entry below

• ↪ U+002A ASTERISK (*)

  Consume

• ↪ Unicode punctuation

  Let delimiterRunAfter be 'punctuation' and treat it as per the “anything else” entry below

• ↪ Anything else

  Let leftFlanking be whether both delimiterRunAfter is not 'whitespace', and that either delimiterRunAfter is not 'punctuation' or that delimiterRunBefore is not null

  Let rightFlanking be whether both delimiterRunBefore is not 'whitespace', and that either delimiterRunBefore is not 'punctuation' or that delimiterRunAfter is not null

  Unset delimiterRunBefore, unset delimiterRunAfter, enqueue a Delimiter run end label, and reconsume in the Text state

9.18 Autolink state

• ↪ U+003C LESS THAN (<)

  Enqueue an Autolink start label, enqueue a Marker token, consume, enqueue an Autolink open label, and switch to the Autolink open state

• ↪ Anything else

  Enqueue a NOK label

9.19 Autolink open state

• ↪ ASCII alpha

  Consume, let sizeScheme be 1, and switch to the Autolink scheme or email atext state

• ↪ atext
  ↪ U+002E DOT (.)

  Consume and switch to the Autolink email atext state

• ↪ Anything else

  Enqueue a NOK label

9.20 Autolink email atext state

• ↪ U+0040 AT SIGN (@)

  Enqueue a Marker token, consume, let sizeLabel be 1, enqueue a Content token, and switch to the Autolink email at sign or dot state

• ↪ atext
  ↪ U+002E DOT (.)

  Consume

• ↪ Anything else

  Enqueue a NOK label

9.21 Autolink email label state

• ↪ U+002D DASH (-)

  If sizeLabel is not 63, increment sizeLabel by 1, consume, and switch to the Autolink email dash state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+002E DOT (.)

  If sizeLabel is not 63, increment sizeLabel by 1, consume, and switch to the Autolink email at sign or dot state

  Otherwise, treat it as per the “anything else” entry below

• ↪ U+003E GREATER THAN (>)

  Unset sizeLabel, enqueue an Autolink email close label, enqueue a Marker token, consume, enqueue an Autolink email end label, and switch to the Text state

• ↪ ASCII alphanumeric

  If sizeLabel is not 63, increment sizeLabel by 1 and consume

  Otherwise, treat it as per the “anything else” entry below

• ↪ Anything else

  Unset sizeLabel and enqueue a NOK label

9.22 Autolink email at sign or dot state

• ↪ ASCII alphanumeric

  If sizeLabel is not 63, increment sizeLabel by 1, consume, and switch to the Autolink email label state

  Otherwise, treat it as per the “anything else” entry below

• ↪ Anything else

  Unset sizeLabel and enqueue a NOK label

9.23 Autolink email dash state

• ↪ U+002D DASH (-)

  If sizeLabel is not 63, increment sizeLabel by 1 and consume

  Otherwise, treat it as per the “anything else” entry below

• ↪ ASCII alphanumeric

  If sizeLabel is not 63, increment sizeLabel by 1, consume, and switch to the Autolink email label state

  Otherwise, treat it as per the “anything else” entry below

• ↪ Anything else

  Unset sizeLabel and enqueue a NOK label

9.24 Autolink scheme or email atext state

• ↪ U+002B PLUS SIGN (+)
  ↪ U+002E DOT (.)
  ↪ U+002D DASH (-)
  ↪ ASCII alphanumeric

  Increment sizeScheme by 1, consume, and switch to the Autolink scheme inside or email atext state

• ↪ atext

  Unset sizeScheme, consume, and switch to the Autolink email atext state

• ↪ Anything else

  Unset sizeScheme and enqueue a NOK label

9.25 Autolink scheme inside or email atext state

• ↪ U+003A COLON (:)

  Unset sizeScheme, enqueue a Marker token, consume, enqueue a Content token, and switch to the Autolink URI inside state

• ↪ U+0040 AT SIGN (@)

  Unset sizeScheme, enqueue a Marker token, consume, let sizeLabel be 1, enqueue a Content token, and switch to the Autolink email at sign or dot state

• ↪ U+002B PLUS SIGN (+)
  ↪ U+002E DOT (.)
  ↪ U+002D DASH (-)
  ↪ ASCII alphanumeric

  If sizeScheme is not 32, increment sizeScheme by 1 and consume

  Otherwise, treat it as per the “atext” entry below

• ↪ atext

  Unset sizeScheme, consume, and switch to the Autolink email atext state

• ↪ Anything else

  Unset sizeScheme and enqueue a NOK label

9.26 Autolink URI inside state

• ↪ EOF
  ↪ EOL
  ↪ ASCII control
  ↪ U+0020 SPACE (SP)
  ↪ U+003C LESS THAN (<)

  Enqueue a NOK label

• ↪ U+003E GREATER THAN (>)

  Enqueue an Autolink uri close label, enqueue a Marker token, consume, enqueue an Autolink uri end label, and switch to the Text state

• ↪ Anything else

  Consume

9.27 HTML state

• ↪ U+003C LESS THAN (<)

  Enqueue an HTML start label, enqueue a Content token, consume, and switch to the HTML open state

• ↪ Anything else

  Enqueue a NOK label

9.28 HTML open state

• ↪ U+0021 EXCLAMATION MARK (!)

  Consume and switch to the HTML declaration start state

• ↪ U+002F SLASH (/)

  Consume and switch to the HTML tag close start state

• ↪ U+003F QUESTION MARK (?)

  Consume and switch to the HTML instruction inside state

• ↪ ASCII alpha

  Consume and switch to the HTML tag open inside state

• ↪ Anything else

  Enqueue a NOK label

9.29 HTML declaration start state

• ↪ U+002D DASH (-)

  Consume and switch to the HTML comment open inside state

• ↪ [CDATA[ (the five upper letters “CDATA” with a U+005B LEFT SQUARE BRACKET ([) before and after)

  Consume and switch to the HTML CDATA inside state

• ↪ ASCII alpha

  Reconsume in the HTML declaration inside state

• ↪ Anything else

  Enqueue a NOK label

9.30 HTML comment open inside state

• ↪ U+002D DASH (-)

  Consume and switch to the HTML comment inside state

• ↪ Anything else

  Enqueue a NOK label

9.31 HTML comment inside state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ U+002D DASH (-)

  Consume and switch to the HTML comment close inside state

• ↪ Anything else

  Consume

9.32 HTML comment close inside state

• ↪ U+002D DASH (-)

  Consume and switch to the HTML comment close state

• ↪ Anything else

  Reconsume in the HTML comment inside state

9.33 HTML comment close state

Note: a CM comment may not contain two dashes (--), and may not end in a dash (which would result in --->). Here we have seen two dashes, so we can either be at the end of a comment, or no longer in a comment.

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.34 HTML CDATA inside state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ ]]> (two of U+005D RIGHT SQUARE BRACKET (]), with a U+003E GREATER THAN (>) after)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ Anything else

  Consume

9.35 HTML declaration inside state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ Anything else

  Consume

9.36 HTML instruction inside state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ U+003F QUESTION MARK (?)

  Consume and switch to the HTML instruction close state

• ↪ Anything else

  Consume

9.37 HTML instruction close state

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ Anything else

  Reconsume in the HTML instruction inside state

9.38 HTML tag close start state

• ↪ ASCII alpha

  Consume and switch to the HTML tag close inside state

• ↪ Anything else

  Enqueue a NOK label

9.39 HTML tag close inside state

• ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the HTML tag close between state

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ ASCII alphanumeric
  ↪ U+002D DASH (-)

  Consume

• ↪ Anything else

  Enqueue a NOK label

9.40 HTML tag close between state

Note: an EOL is technically allowed here, but as a > after an EOL would start a blockquote, practically it’s not possible.

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Consume

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.41 HTML tag open inside state

• ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the HTML tag open between state

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ ASCII alphanumeric
  ↪ U+002D DASH (-)

  Consume

• ↪ Anything else

  Enqueue a NOK label

9.42 HTML tag open between state

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Consume

• ↪ U+002F SLASH (/)

  Consume and switch to the HTML tag open self closing state

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ ASCII alpha
  ↪ U+003A COLON (:)
  ↪ U+005F UNDERSCORE (_)

  Consume and switch to the HTML tag open attribute name state

• ↪ Anything else

  Enqueue a NOK label

9.43 HTML tag open self closing state

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.44 HTML tag open attribute name state

• ↪ ASCII alphanumeric
  ↪ U+002D DASH (-)
  ↪ U+002E DOT (.)
  ↪ U+003A COLON (:)
  ↪ U+005F UNDERSCORE (_)

  Consume

• ↪ Anything else

  Reconsume in the HTML tag open attribute name after state

9.45 HTML tag open attribute name after state

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Consume

• ↪ U+002F SLASH (/)

  Consume and switch to the HTML tag open self closing state

• ↪ U+003D EQUALS TO (=)

  Consume and switch to the HTML tag open attribute before state

• ↪ U+003E GREATER THAN (>)

  Consume, enqueue an HTML end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.46 HTML tag open attribute before state

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Consume

• ↪ U+0022 QUOTATION MARK (")

  Consume and switch to the HTML tag open double quoted attribute state

• ↪ U+0027 APOSTROPHE (')

  Consume and switch to the HTML tag open single quoted attribute state

• ↪ U+003C LESS THAN (<)
  ↪ U+003D EQUALS TO (=)
  ↪ U+003E GREATER THAN (>)
  ↪ U+0060 GRAVE ACCENT (`)

  Enqueue a NOK label

• ↪ Anything else

  Consume and switch to the HTML tag open unquoted attribute state

9.47 HTML tag open double quoted attribute state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ U+0022 QUOTATION MARK (")

  Consume and switch to the HTML tag open between state

• ↪ Anything else

  Consume

9.48 HTML tag open single quoted attribute state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, consume, and enqueue a Content token

• ↪ U+0027 APOSTROPHE (')

  Consume and switch to the HTML tag open between state

• ↪ Anything else

  Consume

9.49 HTML tag open unquoted attribute state

• ↪ EOF
  ↪ U+0022 QUOTATION MARK (")
  ↪ U+0027 APOSTROPHE (')
  ↪ U+003C LESS THAN (<)
  ↪ U+003D EQUALS TO (=)
  ↪ U+0060 GRAVE ACCENT (`)

  Enqueue a NOK label

• ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)
  ↪ U+003E GREATER THAN (>)

  Reconsume in the HTML tag open between state

• ↪ Anything else

  Consume

9.50 Link label start state

• ↪ U+005B LEFT SQUARE BRACKET ([)

  Enqueue a Link label start label, enqueue a Marker token, consume, enqueue a Link label open label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.51 Character escape state

• ↪ U+005C BACKSLASH (\)

  Enqueue a Character escape start label, enqueue a Marker token, consume, and switch to the Character escape after state

• ↪ Anything else

  Enqueue a NOK label

9.52 Character escape after state

• ↪ ASCII punctuation

  Enqueue a Content token, consume, enqueue a Character escape end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.53 Break escape state

• ↪ U+005C BACKSLASH (\)

  Enqueue a Break escape start label, enqueue a Hard break label, enqueue a Marker token, consume, and switch to the Break escape after state

• ↪ Anything else

  Enqueue a NOK label

9.54 Break escape after state

• ↪ EOL

  If the break represented by the character does not start with a VIRTUAL SPACE, U+0009 CHARACTER TABULATION (HT), or U+0020 SPACE (SP), enqueue an End-of-line token, consume, enqueue a Break escape end label, and switch to the Text initial state

  Otherwise, treat it as per the “anything else” entry below

• ↪ Anything else

  Enqueue a NOK label

9.55 Label resource close state

• ↪ U+005D RIGHT SQUARE BRACKET (])

  Enqueue a Label close label, enqueue a Marker token, consume, enqueue a Label end label, and switch to the Label resource end after state

• ↪ Anything else

  Enqueue a NOK label

9.56 Label resource end after state

• ↪ U+0028 LEFT PARENTHESIS (()

  Enqueue a Resource information start label, enqueue a Marker token, consume, enqueue a Resource information open label, and switch to the Resource information open state

• ↪ Anything else

  Enqueue a NOK label

9.57 Resource information open state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+0029 RIGHT PARENTHESIS ())

  Enqueue a Resource information close label, enqueue a Marker token, consume, enqueue a Resource information end label, and switch to the Text state

• ↪ U+003C LESS THAN (<)

  Enqueue a Resource information destination quoted start label, enqueue a Marker token, consume, enqueue a Resource information destination quoted open label, and switch to the Resource information destination quoted inside state

• ↪ ASCII control

  Enqueue a NOK label

• ↪ Anything else

  Enqueue a Resource information destination unquoted start label, enqueue a Content token, consume, enqueue a Resource information destination unquoted open label, and switch to the Resource information destination unquoted inside state

9.58 Resource information destination quoted inside state

• ↪ EOF
  ↪ EOL
  ↪ U+003C LESS THAN (<)

  Enqueue a NOK label

• ↪ U+003E GREATER THAN (>)

  Enqueue a Resource information destination quoted close label, enqueue a Marker token, consume, enqueue a Resource information destination quoted end label, and switch to the Resource information destination quoted after state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Resource information destination quoted escape state

• ↪ Anything else

  Ensure a Content token and consume

9.59 Resource information destination quoted escape state

• ↪ U+003C LESS THAN (<)
  ↪ U+003E GREATER THAN (>)
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Resource information destination quoted inside state

• ↪ Anything else

  Reconsume in the Resource information destination quoted inside state

9.60 Resource information destination quoted after state

• ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)
  ↪ U+003E GREATER THAN (>)

  Reconsume in the Resource information between state

• ↪ U+0029 RIGHT PARENTHESIS ())

  Enqueue a Resource information close label, enqueue a Marker token, consume, enqueue a Resource information end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.61 Resource information destination unquoted inside state

• ↪ EOF

  Unset balance and enqueue a NOK label

• ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)
  ↪ U+003E GREATER THAN (>)

  Enqueue a Resource information destination unquoted close label, enqueue a Resource information destination unquoted end label, and reconsume in the Resource information between state

• ↪ U+0028 LEFT PARENTHESIS (()

  Increment balance by 1 and consume

• ↪ U+0029 RIGHT PARENTHESIS ())

  If balance is 0, unset balance, enqueue a Resource information destination unquoted close label, enqueue a Resource information destination unquoted end label, and reconsume in the Resource information between state

  Otherwise, decrement balance by 1, and consume

• ↪ U+005C BACKSLASH (\)

  Consume and switch to the Resource information destination unquoted escape state

• ↪ ASCII control

  Unset balance and enqueue a NOK label

• ↪ Anything else

  Consume

9.62 Resource information destination unquoted escape state

• ↪ U+0028 LEFT PARENTHESIS (()
  ↪ U+0029 RIGHT PARENTHESIS ())
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Resource information destination unquoted inside state

• ↪ Anything else

  Reconsume in the Resource information destination unquoted inside state

9.63 Resource information between state

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+0022 QUOTATION MARK (")

  Enqueue a Resource information title start label, enqueue a Marker token, consume, enqueue a Resource information title open label, and switch to the Resource information title double quoted inside state

• ↪ U+0027 APOSTROPHE (')

  Enqueue a Resource information title start label, enqueue a Marker token, consume, enqueue a Resource information title open label, and switch to the Resource information title single quoted inside state

• ↪ U+0028 LEFT PARENTHESIS (()

  Enqueue a Resource information title start label, enqueue a Marker token, consume, enqueue a Resource information title open label, and switch to the Resource information title paren quoted inside state

• ↪ U+0029 RIGHT PARENTHESIS ())

  Enqueue a Resource information close label, enqueue a Marker token, consume, enqueue a Resource information end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.64 Resource information title double quoted inside state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ U+0022 QUOTATION MARK (")

  Enqueue a Resource information title close label, enqueue a Marker token, consume, enqueue a Resource information title end label, and switch to the Resource information title after state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Resource information title double quoted escape state

• ↪ Anything else

  Ensure a Content token and consume

9.65 Resource information title double quoted escape state

• ↪ U+0022 QUOTATION MARK (")
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Resource information title double quoted inside state

• ↪ Anything else

  Reconsume in the Resource information title double quoted inside state

9.66 Resource information title single quoted inside state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ U+0027 APOSTROPHE (')

  Enqueue a Resource information title close label, enqueue a Marker token, consume, enqueue a Resource information title end label, and switch to the Resource information title after state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Resource information title single quoted escape state

• ↪ Anything else

  Ensure a Content token and consume

9.67 Resource information title single quoted escape state

• ↪ U+0027 APOSTROPHE (')
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Resource information title single quoted inside state

• ↪ Anything else

  Reconsume in the Resource information title single quoted inside state

9.68 Resource information title paren quoted inside state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ U+0029 RIGHT PARENTHESIS ())

  Enqueue a Resource information title close label, enqueue a Marker token, consume, enqueue a Resource information title end label, and switch to the Resource information title after state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Resource information title paren quoted escape state

• ↪ Anything else

  Ensure a Content token and consume

9.69 Resource information title paren quoted escape state

• ↪ U+0029 RIGHT PARENTHESIS ())
  ↪ U+005C BACKSLASH (\)

  Consume and switch to the Resource information title paren quoted inside state

• ↪ Anything else

  Reconsume in the Resource information title paren quoted inside state

9.70 Resource information title after state

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+0029 RIGHT PARENTHESIS ())

  Enqueue a Resource information close label, enqueue a Marker token, consume, enqueue a Resource information end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.71 Label reference close state

• ↪ U+005D RIGHT SQUARE BRACKET (])

  Enqueue a Label close label, enqueue a Marker token, consume, enqueue a Label end label, and switch to the Label reference end after state

• ↪ Anything else

  Enqueue a NOK label

9.72 Label reference end after state

• ↪ U+005B LEFT SQUARE BRACKET ([)

  Enqueue a Reference label start label, enqueue a Marker token, consume, enqueue a Reference label open label, and switch to the Reference label open state

• ↪ Anything else

  Enqueue a NOK label

9.73 Reference label open state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ U+005D RIGHT SQUARE BRACKET (])

  Enqueue a Reference label collapsed close label, enqueue a Marker token, consume, enqueue a Reference label end label, and switch to the Text state

• ↪ Anything else

  Enqueue a Content token, consume, and switch to the Reference label inside state

9.74 Reference label inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Reference label between state

• ↪ U+005C BACKSLASH (\)

  Ensure a Content token, consume, and switch to the Reference label escape state

• ↪ U+005D RIGHT SQUARE BRACKET (])

  Enqueue a Reference label full close label, enqueue a Marker token, consume, enqueue a Reference label end label, and switch to the Text state

• ↪ Anything else

  Ensure a Content token and consume

9.75 Reference label between state

• ↪ EOF

  Enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Reference label inside state

9.76 Reference label escape state

• ↪ U+005C BACKSLASH (\)
  ↪ U+005D RIGHT SQUARE BRACKET (])

  Consume and switch to the Reference label inside state

• ↪ Anything else

  Reconsume in the Reference label inside state

9.77 Label reference shortcut close state

• ↪ U+005D RIGHT SQUARE BRACKET (])

  Enqueue a Label close label, enqueue a Marker token, consume, enqueue a Label end label, and switch to the Text state

• ↪ Anything else

  Enqueue a NOK label

9.78 Delimiter run underscore start state

• ↪ U+005F UNDERSCORE (_)

  Let delimiterRunAfter be null and let delimiterRunBefore be 'whitespace' if prev is EOF, EOL, or Unicode whitespace, 'punctuation' if it is Unicode punctuation, or null otherwise

  Enqueue a Delimiter run start label, enqueue a Sequence token, consume, and switch to the Delimiter run underscore state

• ↪ Anything else

  Enqueue a NOK label

9.79 Delimiter run underscore state

• ↪ EOF
  ↪ EOL
  ↪ Unicode whitespace

  Let delimiterRunAfter be 'whitespace' and treat it as per the “anything else” entry below

• ↪ U+005F UNDERSCORE (_)

  Consume

• ↪ Unicode punctuation

  Let delimiterRunAfter be 'punctuation' and treat it as per the “anything else” entry below

• ↪ Anything else

  Let leftFlanking be whether both delimiterRunAfter is not 'whitespace', and that either delimiterRunAfter is not 'punctuation' or that delimiterRunBefore is not null

  Let rightFlanking be whether both delimiterRunBefore is not 'whitespace', and that either delimiterRunBefore is not 'punctuation' or that delimiterRunAfter is not null

  Unset delimiterRunBefore, unset delimiterRunAfter, enqueue a Delimiter run end label, and reconsume in the Text state

9.80 Code start state

• ↪ U+0060 GRAVE ACCENT (`)

  Let sizeOpen be 1, enqueue a Code start label, enqueue a Code fence start label, enqueue a Marker token, consume, and switch to the Code open fence inside state

• ↪ Anything else

  Enqueue a NOK label

9.81 Code open fence inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Enqueue a Code fence end label and reconsume in the Code between state

• ↪ U+0060 GRAVE ACCENT (`)

  Increment sizeOpen by 1 and consume

• ↪ Anything else

  Enqueue a Code fence end label, enqueue a Content token, consume, and switch to the Code inside state

9.82 Code between state

• ↪ EOF

  Unset sizeOpen and enqueue a NOK label

• ↪ EOL

  Enqueue a Soft break label, enqueue an End-of-line token, and consume

• ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Ensure a Whitespace token and consume

• ↪ Anything else

  Reconsume in the Code inside state

9.83 Code inside state

• ↪ EOF
  ↪ EOL
  ↪ VIRTUAL SPACE
  ↪ U+0009 CHARACTER TABULATION (HT)
  ↪ U+0020 SPACE (SP)

  Reconsume in the Code between state

• ↪ U+0060 GRAVE ACCENT (`)

  Let sizeClose be 1, enqueue a Code fence start label, enqueue a Marker token, consume, and switch to the Code close fence inside state

• ↪ Anything else

  Ensure a Content token and consume