The preceding sections have demonstrated the use of AFL in writing audio rendering rules for conveying complex structures. However, oral communication takes more time than written communication, and the listener has to retain a lot more information than a person perusing printed text. This disadvantage is most felt when listening to complex mathematics. It takes time to speak complicated expressions and sometimes the listener has forgotten the beginning of an expression by the time it has been fully rendered.

Conjecture 1 Top-level structure: An experienced reader of mathematical formulae first looks at the top-level structure of a complex equation, and then progressively reads the subexpressions.

Thus, given Faa De Bruno’s formula:

| (4.5) |

We see it as an equation with a derivative on the LHS and a double summation on the RHS. We then see that the inner summation has a complicated constraint and that the summand is a fraction. Finally, we read the entire expression.

The steps enumerated above are carried out implicitly by the eye, making it difficult to identify the atomic actions involved. Yet, it is clear that we rely on this type of breaking up or “chunking” of complex expressions when understanding them. In fact, most of visual mathematical notation is an attempt at aiding this process of grouping subexpressions together in a meaningful manner —even in the visual setting, writing out Faa De Bruno’s formula in a fully linearized manner (e.g., the TEX encoding) makes comprehension difficult, if not impossible.

In the audio setting, the listener does not have the luxury of being able to view both the top level structure as well as the leaves of the formula when listening to a straight rendering of the expression. This means that AS TE R needs to take over part of the work that was described as being implicit in the visual context —AS TE R needs to recognize and convey the same kind of grouping that the experienced reader perceives in the visual notation.

We call this process rendering with variable substitution. Thus, given a complex
expression of the form , where the e_{i} are complex math expressions, AS TE R recognizes this
top-level structure and produces the rendering, “Fraction x over y, where x is ⋯ and y is ⋯”. In
the following subsections, we enumerate the conditions under which such variable
substitution is performed. Based on these, we have implemented a variable substitution
rendering style. The listener can activate this style and have AS TE R perform variable
substitution where appropriate.

To motivate the discussion, here are some examples of how variable substitution can produce renderings that are easier to understand.

| (4.6) |

would be spoken as

i = integral with respect to x from 0 to infinity of f dx,

where f is …

This technique is particularly useful when presenting very complex equations. Variable substitution transforms equation 4.5 on page 134 to:

| (4.7) |

which can now be rendered as:

Enneth derivative with respect to x of w equals

Summation over 0 less than or equal to j less than or equal to n

Summation over lower constraint 1 of

jayth derivative of u with respect to x times the fraction

numerator 1 over denominator 1. where lower constraint 1 is …,

numerator 1 is …,

denominator 1 is ….

It takes 68 seconds to speak equation 4.5 on page 134, making it difficult to perceive the top-level structure from listening to a straight rendering of the expression. Using style variable substitution, the top-level expression is rendered in 23 seconds, and it takes a further 57 seconds to render the substitutions. Thus, though the total time taken to speak the entire expression is more, the listener can understand the top-level structure in about a third of the time it would take to listen to the entire expression.

Variable substitution should be used sparingly, since renderings using this style take a longer time to convey entire expressions, in this example, 80 seconds against 68 seconds—AS TE R uses variable substitution only if an expression is sufficiently complex.

The following enumerates a few of the points to be considered when designing a variable substitution scheme.

- Minimize the number of levels of substitution. Ideally this should not be more than 1.
- Avoid unnecessary substitutions. In equation 4.6 on page 135, substituting a variable for the entire right-hand side is redundant.
- Use a complexity measure that determines when an expression is sufficiently complex to warrant variable substitution. This measure should capture the following properties of an expression:

The first requirement says that any variable substitution scheme we apply should result in a simple top-level expression. The second requirement ensures that the top-level expression conveys as much information as possible. In addition, it ensures that the renderings resulting from variable substitution do not end up being more complicated than plain renderings. Thus, in equation 4.6 on page 135, substituting identifier x for the entire right-hand side to produce

i equals x where x equals …

does not simplify the rendering. This is because the top-level expression is a relation, and substituting for one of the sides of a relation only produces a new relation which is as complex as the original. The third requirement ensures that all expressions are compared using the same weighting scheme. Measure weight described in Section 4.4 is used.

We introduce the notion of relative complexity below:

Definition 3 Relative Complexity:

Given expression e with child c_{i} with weight(e) = w and weight(c_{i}) = w_{i},

| (4.8) |

Here is a sketch of the variable-substitution algorithm. It uses three user-specified complexity thresholds, whose purpose will become clear in the following description.

Given an expression e, compute its weight w. Do not perform substitutions if w < *absolute-complexity-threshold*. Otherwise, e is a good candidate for variable substitution.

First try to substitute for the children of e. Given children c_{i} {1 ≤ i ≤ n},
compute their weights w_{i}. Substitute for a child c_{i} if and only if its relative
complexity is greater than *proportional-complexity-threshold*. Thus, for each i, if
w_{i} ≥ w ⋅ *proportional-complexity-threshold*, apply the algorithm recursively to c_{i}. If no
substitution can be performed on the descendants of c_{i}, replace c_{i} itself.

If no substitution can be performed on any of the c_{i} or their subexpressions, then
substitute for e provided that e is not a top-level expression. Also do not substitute if e is one
of the sides of a relational.

Given a top level expression e having weight w, this algorithm is called as follows:

(collect-substitutions e (complexity-threshold e))

where (complexity-threshold e) is defined as:

(defun complexity-threshold (object)

"Compute the threshold value for this object"

(let ((proportional

(+ 1 (truncate

(⋆ (weight object )

⋆proportional-complexity-threshold⋆ )))))

(max proportional ⋆absolute-complexity-threshold⋆)))

Instead of computing < *proportional-complexity-threshold*, we initialize the algorithm with the threshold w ⋅ *proportional-complexity-threshold*, since w remains constant throughout the algorithm.

We now introduce the following refinements to the algorithm:

- If the parent expression is a relation, then do not perform substitutions on the children themselves. Proper subexpressions of the children are still considered for substitution.
- Consider attributes as well as the children of an expression for substitution.
- Use a different weighting scheme for deciding when to substitute for attributes by introducing a third constant, *attribute-complexity-threshold*. This constant is used to scale *proportional-complexity-threshold* when considering attributes.

The following values were arrived at experimentally:

AS TE R chooses identifier names that convey some information about the object being replaced. This has two advantages:

- An identifier like “summand” when referring to the expression appearing as the summand in a summation conveys more information than the identifier x.
- When substitutions are spoken after rendering the top-level expression, the listener finds it easier to relate them to the top-level expression.

Names for the substituted expressions are chosen using the following information:

- Type information. The objects returned by the recognizer are all typed. So at the very least, we have type information for all objects, e.g., mathematical function, parenthesized expression, etc.
- Contextual information. The first child of a fraction is called its “numerator”. The left-hand side of an implication is called the “premise” and its right-hand side the “conclusion”. We have built in this information for standard mathematical objects and provided a flexible mechanism for the user to add or modify such information.
- Special patterns: The special patterns presented in Section 4.4 are also used in synthesizing meaningful names for the substitutions.

Thus, when substituting for the subscript to a summation operator, AS TE R uses the name “lower constraint”. Since more than one such “lower constraint” may be substituted in a general expression, such names are appended with an integer to make them unique. This is how the rendering shown earlier for Faa De Bruno’s formula is produced.

The following illustrate the effect of applying variable substitution.

The above expression has been annotated with underbraces to indicate the variable substitution that is performed. Notice that applying variable substitution reveals the recursive nature of the expression. It is interesting to note that if we now run our algorithm on this annotated expression, the subexpressions marked by underbraces are chosen for substitution.

We had to resolve some interesting implementation issues in order to provide the functionality described above. All the previously active rendering rules and styles work in conjunction with the variable substitution rendering style. This means that the expressions resulting from variable substitution are rendered exactly as they would be if they occurred by themselves. To achieve this, AS TE R first applies variable substitution and then applies the currently active rendering rules to the result. This added level of complexity is completely transparent to a user of AS TE R, who can continue to add rendering rules and modify styles in the presence of variable substitution.