Visualization Theory
====================

Inner Product, Norm, and Angle
-----------------------------------

Fixing some :math:`d` and :math:`M`, we can define the inner product
of two tensors by

.. math::

   \langle T, U \rangle = \sum_{i_1=1}^M \sum_{i_2=1}^M\cdots
   \sum_{i_d=1}^M T(i_1,i_2,\ldots,i_d)\bar{U}(i_1,i_2,\ldots,i_d) \,,

where :math:`\bar{U}` indicates complex conjugate, if appropriate. 
If :math:`T` and :math:`U` are represented with separation ranks
:math:`r` and :math:`s` using vectors :math:`V^l_k` and :math:`W^l_k`,
then their inner product can also be computed via

.. math::

   \langle T, U \rangle 
   =\sum_{l=1}^r \sum_{m=1}^s \prod_{k=1}^d  \sum_{i=1}^MV^l_k(i)\bar{W}^m_k(i)
   =\sum_{l=1}^r \sum_{m=1}^s \prod_{k=1}^d  \langle V^l_k, W^m_k
   \rangle\,,

which costs :math:`\mathcal{O}(rsdM)` operations.


With this
inner product we can define length with the norm

.. math::

   \|T\|= \sqrt{\langle T, T \rangle}
   = \left(\sum_{i_1=1}^M \sum_{i_2=1}^M\cdots
   \sum_{i_d=1}^M | T(i_1,i_2,\ldots,i_d)|^2\right)^{1/2}\,. 

We can also define the angle between :math:`T` and :math:`U` relative
to the origin with

.. math::

	\angle(T,U)= \arccos\left(\frac{\langle T, U \rangle}{\|T\|\,\|U\|}\right)\,,

realizing that this may be a complex number if :math:`T` or :math:`U`
is complex.

The operations necessary for the visualization will be based on
this inner product, and so are essentially coordinate-free. In
particular, a separable change of coordinates does not effect the
vizualization. 


Intersections
---------------------------------

We now consider how to visualize a set :math:`S` of tensors.
Since :math:`S` is defined as a subset of :math:`R^{M^d}` we cannot
visualize it all at once. Instead we can intersect it with a simpler
object and visualize this intersection.
Any two distinct tensors :math:`T_1` and :math:`T_2` define a line,
which can be represented by

.. math::

   a_1 T_1+(1-a_1)T_2 \,.

Similarly, any three tensors :math:`T_1`, :math:`T_2`, and :math:`T_2`
that do not lie on a line define a plane 

.. math::

   a_1 T_1+a_2T_2+(1-a_1-a_2)T_3 \,.

By intersecting :math:`S` with one of these simpler objects we obtain
something that we can visualize.

.. and any three tensors not on a plane define a volume. 


Including Nearby Points
^^^^^^^^^^^^^^^^^^^^^^^

Unfortunately, in practice the straightforward intersection described
above yields too few points to plot. If we proceed by generating
points in :math:`S`, the probability that any of them lie in the
given line/plane is virtually zero. If we proceed by generating points in
the line/plane then we need to test their membership in :math:`S`. For the
sets we are interested in this membership test can fail, and at
best can produce a tensor in :math:`S` within some small distance
of the tensor we are checking.

A *star-shaped set* is a set :math:`S` such that :math:`T\in S
\Rightarrow cT \in S` for any scalar :math:`c`. Such a :math:`S` is
connected and simply connected.  Instead of asking if :math:`T\in S`
is on our plotting line/plane, we can ask if :math:`cT` is on our plotting
line/plane for some :math:`c`, and if so, plot that :math:`cT`. Thus we can
increase the number of plotted points and remove the uninteresting
scalar structure. Formally, we consider tensors that are positive
scalar multiples of one another to be equivalent. Locally, this is
similar to restricting to tensors of norm one. (Identifying negative
scalar multiples as well does not seem to be worthwhile.)


We will add further information to the visualization by including
points that are in :math:`S` but not on the plotting line/plane and
coloring them to indicate how far from the plotting plane they
are. Since we identify  :math:`T` and :math:`cT` we measure "how far"
using an angle instead of a distance.

We will describe the the case for a plane, since the case for a line
is a simple modification of it.  
From the three points :math:`T_1`, :math:`T_2`, and :math:`T_3` that
define our plane, we define the matrix

.. math::

   G=
   \left[\begin{array}{ccc}
   \langle T_1,T_1\rangle & \langle T_2,T_1\rangle& \langle T_3,T_1\rangle\\
   \langle T_1,T_2\rangle & \langle T_2,T_2\rangle& \langle T_3,T_2\rangle\\
   \langle T_1,T_3\rangle & \langle T_2,T_3\rangle& \langle T_3,T_3\rangle
   \end{array}\right]	
   \,.

Given :math:`T\in S`, its projection onto the span of :math:`T_1`,
:math:`T_2`, and :math:`T_3` is given by

.. math::
   \hat{T}=c_1 T_1 + c_2 T_2+ c_3 T_3

with

.. math::
  
   \left[\begin{array}{c}
   c_1\\
   c_2\\
   c_3
   \end{array}\right]	
   =G^{-1}
   \left[\begin{array}{c}
   \langle T,T_1\rangle\\
   \langle T,T_2\rangle\\
   \langle T,T_3\rangle
   \end{array}\right]	
   \,.

Dividing by the scalar :math:`c_1+c_2+c_3`, we obtain the tensor 

.. math::
   U=b_1 T_1 + b_2 T_2+ b_3 T_3
   =\frac{c_1}{c_1+c_2+c_3} T_1 + \frac{c_2}{c_1+c_2+c_3} T_2
   + \frac{c_3}{c_1+c_2+c_3} T_3\,,

which lies on the plotting plane. To measure how much :math:`T` failed
to be on the plane, we compute the angle

.. math::
   \angle(T,U)=\angle(T,\hat{T})
   =\arccos\left(\frac{\langle T, \hat{T}\rangle}{\|T\|\,\|\hat{T}\|}\right)
   =\arccos\left(\frac{\langle \hat{T},\hat{T}\rangle}{\|T\|\,\|\hat{T}\|}\right)
   =\arccos\left(\frac{\|\hat{T}\|}{\|T\|}\right)\,.

An angle of zero indicates that some scalar multiple of :math:`T` is
on the plane and an angle of :math:`\pi/2` indicates that :math:`T` is
orthogonal to the span of :math:`T_1`, :math:`T_2`, and
:math:`T_3`. This projection process (neglecting :math:`T_3`) is
illustrated below:

.. math::
   \setlength{\unitlength}{2mm}
   \begin{picture}(50,50)(-5,-5)
   \put(0,0){\circle*{1}}
   \put(-5,-3){\mbox{Origin}}
   \put(28,0){\circle*{1}}
   \put(25.8,-4){\mbox{$T_1$}}
   \put(5,15){\circle*{1}}	
   \put(5,18){\mbox{$T_2$}}
   \put(-1,19){\line(3,-2){35}}
   \put(-5,25){\mbox{Plotting Plane}}
   \put(0,24){\vector(0,-1){4}}
   \put(0,0){\vector(2,1){30}}
   \put(0,0){\vector(1,1){30}}
   \put(24,24){\circle*{1}}	
   \put(20,24){\mbox{$T$}}
   \put(15.8,8){\circle*{1}}	
   \put(14,3){\mbox{$U$}}
   \put(24,24){\line(0,-1){12}}
   \put(22,13){\line(0,-1){2}}
   \put(22,13){\line(2,1){2}}
   \put(24,12){\circle*{1}}		
   \put(23,8){\mbox{$\hat{T}$}}
   \put(-5,6){\mbox{$\angle(T,U)$}}
   \put(1,5){\vector(1,-1){2.5}}
   \end{picture}


To indicate this angle in the visualization, we color the plotted
point with black indicting angle nearly zero, then through a spectrum
of ever-lightening colors until light yellow indicates angles near
:math:`\pi/2`.  We use the colors

.. image:: spectrum.*

and bin the angles logarithmicly by

.. math::
   [0,\pi/2048],(\pi/2048,\pi/1024],\ldots,(\pi/8,\pi/4],(\pi/4,\pi/2].

In cases where more than one tensor projects to the same plotting
point, we plot the darker of the two. Since we plot points as small
discs, in cases where the discs overlap we plot the darker disc on
top. 

If :math:`T` is orthogonal to the span of :math:`T_1`, :math:`T_2`,
and :math:`T_3` we will have :math:`c_1=c_2=c_3=0` and cannot consruct
:math:`U`. Thus :math:`T` should not appear in the
visualization. (Numerically one produces a white point at a location
determined by roundoff errors.)


Real Tensors, plotted on a real plane
--------------------------------------

For tensors over the real numbers, our tool uses three tensors
yielding a plane, so we now consider that case in more detail. We
arbitrarily choose to orient the visualization so that :math:`T_3` is
at :math:`(0,0)` and :math:`T_1` is in the direction
:math:`(1,0)`. The angle from :math:`T_1` to :math:`T_3` to
:math:`T_2` is given by

.. math::
   \theta=\arccos\left(\frac{\langle T_1-T_3, T_2-T_3\rangle}
   {\|T_1-T_3\|\,\|T_2-T_3\|} \right)\,

so :math:`T_2` is in the direction :math:`(\cos(\theta),\sin(\theta))`.
A tensor :math:`T=a_1 T_1+a_2T_2+(1-a_1-a_2)T_3` is at the point 

.. math::
   (a_1 \|T_1-T_3\| + \cos(\theta) a_2 \|T_2-T_3\|, 
   \sin(\theta) a_2 \|T_2-T_3\|).

This plotting geometry is illustrated below:

.. math::
   \setlength{\unitlength}{1.7mm}
   \begin{picture}(50,50)(-5,-5)
   \put(-5,0){\vector(1,0){50}}
   \put(0,-5){\vector(0,1){50}}
   \put(40,-5){\mbox{x}}
   \put(-5,40){\mbox{y}}
   \put(0,0){\circle*{1}}	
   \put(30,0){\circle*{1}}	
   \put(30,40){\circle*{1}}	
   \put(-4,-4){\mbox{$T_3$}}
   \put(28,-4){\mbox{$T_1$}}
   \put(25,40){\mbox{$T_2$}}
   \put(0,0){\line(3,4){30}}
   \put(5,2){\mbox{$\theta$}}
   \put(25,2){\mbox{$\theta$}}
   \put(0,0){\vector(1,0){20}}
   \put(4,-3){\mbox{$a_1\|T_1-T_3\|$}}
   \put(20,0){\vector(3,4){15}}
   \put(30,10){\mbox{$a_2\|T_2-T_3\|$}}
   \put(35,20){\circle*{1}}	
   \put(30,20){\mbox{$T$}}
   \end{picture}


Complex Tensors, plotted on a complex line
-------------------------------------------

For tensors over the complex numbers, our tool uses two tensors
yielding a complex line. We
arbitrarily choose to orient the visualization so that :math:`T_2` is
at :math:`0+0i` and :math:`T_1` is in the direction
:math:`1+0i`. 
A tensor :math:`T=a_1 T_1+(1-a_1)T_2` is at the complex point :math:`a_1`.
This plotting geometry is illustrated below:

.. math::
   \setlength{\unitlength}{1.7mm}
   \begin{picture}(50,40)(-5,-5)
   \put(-5,0){\vector(1,0){50}}
   \put(0,-5){\vector(0,1){40}}
   \put(40,-5){\mbox{real}}
   \put(2,30){\mbox{imaginary}}
   \put(0,0){\circle*{1}}	
   \put(30,0){\circle*{1}}	
   \put(-4,-4){\mbox{$T_2$}}
   \put(28,-4){\mbox{$T_1$}}
   \put(0,0){\vector(3,4){15}}
   \put(9,8){\mbox{$a_1$}}
   \put(15,20){\circle*{1}}	
   \put(13,21){\mbox{$T$}}
   \end{picture}