2 to the 9th Tensors
====================

We now study the case of tensors with :math:`d=9` and :math:`M=2`.
At worst, the maximum rank is :math:`2^8=256`. By unfolding such a
tensor to a :math:`2^4\times 2^5` matrix, one can see that rank
:math:`2^4=16` is definitely achievable.


Visualizations over the Real numbers
------------------------------------


A Specific Example
^^^^^^^^^^^^^^^^^^^

We are interested in analyzing the tensor

.. math::
   L_9= \sum_{j=1}^d 
   \left(\bigotimes_{k=1}^{j-1}[1,0]\right)
   \otimes [0,1] \otimes 
   \left(\bigotimes_{k=j+1}^{d}[1,0]\right)\,,

which has nominal rank 9. Consider the tensor-valued function

.. math::
   G(t) =\bigotimes_{k=1}^{d}[1,t]

and notice that the derivative of :math:`G(t)` at 0 is :math:`L_9`, i.e.
:math:`G'(0)=L_9`.
Since 

.. math::
   G'(0) &=\lim_{h\rightarrow \infty} \frac{G(h)-G(0)}{h}\\
   &=\frac{1}{h}\bigotimes_{k=1}^{d}[1,h] 
   -\frac{1}{h}\bigotimes_{k=1}^{d}[1,0]\,,

we see that :math:`L_9` is a limit of rank 2 tensors. 
(To avoid loss-of-precision errors due to cancellation, larger rank is
needed; see [BEY-MOH2005]_.)

To visualize :math:`L_9`, we choose 

.. math::
   T_1&=\sum_{j=1}^3 
   \left(\bigotimes_{k=1}^{j-1}[1,0]\right)
   \otimes [0,1]  \otimes 
   \left(\bigotimes_{k=j+1}^{d}[1,0]\right),\\
   T_2&=\sum_{j=4}^6 
   \left(\bigotimes_{k=1}^{j-1}[1,0]\right)
   \otimes [0,1]  \otimes 
   \left(\bigotimes_{k=j+1}^{d}[1,0]\right),\quad\text{and}\\
   T_3&=\sum_{j=7}^9 
   \left(\bigotimes_{k=1}^{j-1}[1,0]\right)
   \otimes [0,1]  \otimes 
   \left(\bigotimes_{k=j+1}^{d}[1,0]\right)\,,

so that :math:`L_9` will be in the center of these three tensors.
Since these three tensors are in :math:`S_3`, any tensor
on the line connecting any two of them is in :math:`S_6`, and a tensor
in the plane is generically in :math:`S_9`.


We first produce a visualization of :math:`S_2` on
:math:`[-1,2]\times[-1,2]`, and obtain

.. image:: 2to9pr2L9.*

[data file `<2to9pr2L9.dat>`_] 
Black points are scattered sparsely on
the visualization, but without apparent voids. It appears that the ALS
routine had some difficulty (with its fixed number of
iterations). There is a light horizontal streak along the line
connecting :math:`T_1` to :math:`T_3` of unknown cause, probably a
numerical artifact.

Next, we produce a visualization of :math:`S_3`, and obtain

.. image:: 2to9pr3L9.*

[data file `<2to9pr3L9.dat>`_] 

Next, we produce a visualization of :math:`S_6`, which includes the
lines connecting any two of our base tensors, and obtain

.. image:: 2to9pr6L9.*

[data file `<2to9pr6L9.dat>`_] 

A Random Example
^^^^^^^^^^^^^^^^^

To complement the specific example above, 
we choose :math:`T_1`, :math:`T_2`, and :math:`T_3` randomly in
:math:`S_3`. Since these three tensors are in :math:`S_3`, any tensor
on the line connecting any two of them is in :math:`S_6`, and a tensor
in the plane is generically in :math:`S_9`.


We first produce a visualization of :math:`S_3` on
:math:`[-1,2]\times[-1,2]`, and obtain

.. image:: 2to9pr3random.*

[data file `<2to9pr3random.dat>`_]
Next, using the same set of random :math:`T_1`, :math:`T_2`, and
:math:`T_3`, we produce a visualization of :math:`S_6`, and obtain

.. image:: 2to9pr6random.*

[data file `<2to9pr6random.dat>`_]
Since in neither case do we find any unexpected black points, we
conclude that the specific example above is not typical.


.. _2to9SecondExample-section-label:

A Second Random Example
^^^^^^^^^^^^^^^^^^^^^^^

To complement the specific example above, 
we choose :math:`T_1`, :math:`T_2`, and :math:`T_3` randomly in
:math:`S_{16}`. Since these three tensors are in :math:`S_{16}`, any tensor
on the line connecting any two of them is in :math:`S_{32}`, and a tensor
in the plane is generically in :math:`S_{48}`.


We first produce a visualization of :math:`S_{16}` on
:math:`[-1,2]\times[-1,2]`, and obtain

.. image:: 2to9pr16random2.*

[data file `<2to9pr16random2.dat>`_]
Next, using the same set of random :math:`T_1`, :math:`T_2`, and
:math:`T_3`, we produce a visualization of :math:`S_{32}`, and obtain

.. image:: 2to9pr32random2.*

[data file `<2to9pr32random2.dat>`_]

Next, using the same set of random :math:`T_1`, :math:`T_2`, and
:math:`T_3`, we produce a visualization of :math:`S_{40}`, and obtain

.. image:: 2to9pr40random2.*

[data file `<2to9pr40random2.dat>`_]

 
Finally, we produce a visualization of :math:`S_{48}`, which should be
black, and obtain

.. image:: 2to9pr48random2.*

[data file `<2to9pr48random2.dat>`_] Although we do not obtain black,
we do obtain dark blue, indicating that the fitting is working fairly well.

The first time we ran the tests for this example, we used the ALS
algorithm at fixed rank rather than growing the rank (see
:ref:`ALSperform-section-label`). That algorithm performed much
worse, as illustrated in the Section :ref:`ALSfailed-section-label`.