Download The Role of Tensor Rank in the Complexity Analysis of Bilinear Forms

Preliminaries
Properties of tensor rank
Open problems
The role of tensor rank in the complexity analysis
of bilinear forms
Dario A. Bini
Dipartimento di Matematica, Università di Pisa
www.dm.unipi.it/ebini
ICIAM07, Zürich, 16-20 July 2007
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
1
Preliminaries
Tensors
Tensor rank
Bilinear forms
2
Properties of tensor rank
Border rank
Lower bounds
3
Open problems
The direct sum conjecture
Some tensors of unknown rank
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Tensors
Let F be a number field, say, R, C; tensors of the kind
A = (ai1 ,i2 ,...,ih ) ∈ Fn1 ×n2 ×···×nh ,
that is, h-way arrays, are encountered in many problems of very
different nature [Comon 2001], [Comon, Golub, Lim, Mourrain,
2006], [De Silva, Lim 2006]
Blind source separation
High order factor analysis
Independent component analysis
Candecomp/Parafac model
Complexity analysis
Psycometric, Chemometric, Economy,...
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Tensors
It is surprising that little interplay occurred among these
different research areas
Some properties have been rediscovered in different contexts
Apparently, results obtained in one field have not migrated to
the other fields
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Tensors
It is surprising that little interplay occurred among these
different research areas
Some properties have been rediscovered in different contexts
Apparently, results obtained in one field have not migrated to
the other fields
In this talk I wish to provide an overview of the main results
concerning tensors obtained in the research field of computational
complexity (starting from 1969) with the aim of
creating a synergic exchange of information between these
research areas
presenting problems which might be solved with the more
recent tools
presenting old results that might be adapted and extended to
the new needs
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Tensor rank
Definition (Hitchcock 1927)
A tensor T = (ti1 ,...,ih ) has rank 1 if there exist vectors
(k)
(1) (2)
(h)
u(k) = (ui ) ∈ Fnk , k = 1 : h such that ti1 ,...,ih = ui1 ui2 · · · uih ,
T = u(1) ◦ u(2) ◦ · · · ◦ u(h)
Definition (Hitchcock 1927)
The tensor rank rk(A) of A = (ai1 ,...,ih ) is the minimum number r
of rank-1 tensors Ti ∈ Fn1 ×···×nh such that
A = T1 + T2 + · · · + Tr
Dario A. Bini
canonical decomposition
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Some remarks
For A ∈ Fm×n×p a canonical decomposition
A = u(1) ◦ v(1) ◦ w(1) + u(2) ◦ v(2) ◦ w(2) + · · · + u(r ) ◦ v(r ) ◦ w(r )
is defined by three matrices
U = (ui,j ) ∈ Fm×r , V = (vi,j ) ∈ Fn×r , W = (wi,j ) ∈ Fp×r ,
whose columns are the vectors u(i) , v(i) , w(i) , i = 1 : r ,
respectively.
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Some remarks
A tensor A = (ai,j,k ) ∈ Fm×n×p , can be represented by means of
the set of the 3-slabs Ak = (ai,j,k )i,j ∈ Fm×n of A or by a single
matrix of variables
p
X
A=
sk Ak
k=1
A=
1 0
0 −1
0 1
;
1 0
Dario A. Bini
↔
s1
s2
The role of tensor rank
s2
−s1
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Some remarks
A tensor A = (ai,j,k ) ∈ Fm×n×p , can be represented by means of
the set of the 3-slabs Ak = (ai,j,k )i,j ∈ Fm×n of A or by a single
matrix of variables
p
X
A=
sk Ak
k=1
A=
1 0
0 −1
0 1
;
1 0
↔
s1
s2
s2
−s1
This suggests a different point of view for tensor rank:
rk(A) is the minimum set of rank one matrices which span the linear
space generated by the 3-slabs A1 , . . . , Ap [Gastinel 71, Fiduccia 72].
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Bilinear forms
Problem: Given matrices Ak = (ai,j,k )i=1:m,j=1:n , k = 1 : p
compute the set of bilinear forms
fk (x, y) = xT Ak y, k = 1 : p
with the minimum number of nonscalar multiplications with no use
of commutativity (noncommutative bilinear complexity)
A nonscalar multiplication is a multiplication of the kind
m
n
X
X
s=(
αi xi )(
βj yj ), αi , βj ∈ F
i=1
j=1
Remark: The set of bilinear forms is uniquely determined by the
tensor A = (ai,j,k ).
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
Bilinear forms
A canonical decomposition of the tensor A associated with the set
of bilinear forms provides an algorithm of complexity r . In fact
A=
r
X
u
(`)
◦v
(`)
◦w
(`)
→
Ak =
r
X
wk,` u(`) ◦ v(`)
`=1
`=1
→
xT Ak y =
r
X
wk,` (xT u(`) )(v(`)T y)
`=1
Theorem (Strassen 1975)
The noncommutative bilinear complexity of a set of bilinear forms
fk (x, y) = xT Ak y, k = 1 : p, Ak = (ai,j,k ), is given by the tensor
rank of the associated tensor A = (ai,j,k ).
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
An example
Multiplication of complex numbers
(x1 + ix2 )(y1 + iy2 ) = (x1 y1 − x2 y2 ) + i(x1 y2 + x2 y1 ) = f1 + if2
Apparently, 4 multiplications are needed
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
An example
Multiplication of complex numbers
(x1 + ix2 )(y1 + iy2 ) = (x1 y1 − x2 y2 ) + i(x1 y2 + x2 y1 ) = f1 + if2
Apparently, 4 multiplications are needed
1 0
0 1
Tensor:
A=
;
0 −1
1 0
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Tensors
Tensor rank
Bilinear forms
An example
Multiplication of complex numbers
(x1 + ix2 )(y1 + iy2 ) = (x1 y1 − x2 y2 ) + i(x1 y2 + x2 y1 ) = f1 + if2
Apparently, 4 multiplications are needed
1 0
0 1
Tensor:
A=
;
0 −1
1 0
The tensor rank
1
0
0
1
is at most 3:
0
=
−1
1
1 1
=
−
0
1 1
1
0
1
0
0
−
0
0
−
0
0
0
0
0
0
1
0
1
Algorithm:
s1 = (x1 + x2 )(y1 + y2 ),
s 2 = x1 y1 ,
s 3 = x2 y2
Dario A. Bini
f1 = s2 − s3 ,
f2 = s1 − s2 − s3
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Tensor rank
Main problem: For a given tensor, compute its rank and a canonical
decomposition.
Two ways of attacking the problem
looking for lower bounds to the tensor rank
looking for upper bounds to the tensor rank
Things get more complicate: unlike the case of m × n matrices
The rank depends on the ground field F
High rank tensors can be approximated by low rank tensors
Rational algorithms for tensor rank, like Gaussian elimination, cannot exist
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Topological properties of tensors: Border Rank
For m × n matrices, any sequence {Ak } of m × n matrices of rank
r with limit A = limk Ak is such that rank(A) ≤ r .
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Topological properties of tensors: Border Rank
For m × n matrices, any sequence {Ak } of m × n matrices of rank
r with limit A = limk Ak is such that rank(A) ≤ r .
For higher order tensor this property does not hold anymore
Example (Bini, Capovani, Lotti, Romani 1980)
The tensor
1 0
0 1
0 1
;
0 0
1 0
1
0 1
;
0 0
A=
has rank 3. The tensor
A =
has rank 2 for any 6= 0
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Canonical decomposition
1 0
1
0 1
0 0
=
1 −1
1
=
Dario A. Bini
−
−1
0 1
0 0
0 1
0 0
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Border rank
Definition (Bini, Capovani, Lotti, Romani 1980)
The border rank of A ∈ Fm×n×p (F = R, C) is
brk(A) = min{r : ∀ > 0 ∃E ∈ Fm×n×p : ||E|| < , rk(A+E) = r }
where || · || is any norm
Some properties:
brk(A) ≤ rk(A)
brk(A) is the minimum number of nonscalar multiplications
sufficient to approximate the set of bilinear forms associated
with A with arbitrarily small nonzero error
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
More on rank and border rank
Let A = A + E be such that
rk(A ) = brk(A)
the entries of E are polynomials of degree d
Then
rk(A) ≤ (d + 1)brk(A)
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
More on rank and border rank
Let A = A + E be such that
rk(A ) = brk(A)
the entries of E are polynomials of degree d
Then
rk(A) ≤ (d + 1)brk(A)
Proof:
Write d + 1 copies of the canonical decomposition of length
rk(A ) obtained with (d + 1) pairwise different values of Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
More on rank and border rank
Let A = A + E be such that
rk(A ) = brk(A)
the entries of E are polynomials of degree d
Then
rk(A) ≤ (d + 1)brk(A)
Proof:
Write d + 1 copies of the canonical decomposition of length
rk(A ) obtained with (d + 1) pairwise different values of Take linear combinations of these decompositions with
coefficients γj , j = 1 : P
d + 1 in order to kill the terms in i ,
i = 1 : d and to have
γj = 1
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
More on rank and border rank
Let A = A + E be such that
rk(A ) = brk(A)
the entries of E are polynomials of degree d
Then
rk(A) ≤ (d + 1)brk(A)
Proof:
Write d + 1 copies of the canonical decomposition of length
rk(A ) obtained with (d + 1) pairwise different values of Take linear combinations of these decompositions with
coefficients γj , j = 1 : P
d + 1 in order to kill the terms in i ,
i = 1 : d and to have
γj = 1
Obtain a decomposition of length (d + 1)brk(A) with no error
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Lower bounds and linear algebra
Simple criteria for providing lower bounds on tensor rank and
border rank can be given
Trivial bounds
brk(A) ≥ dim(span(A1 , . . . , Ap ))
Similar inequalities are valid w.r.t. the other coordinates
Assume for simplicity p = dim(span(A1 , . . . , Ap ))
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Lower bounds and linear algebra
Let U, V , W be the matrices defining a canonical factorization of
A of length rk(A)
A=
rk
(A)
X
u(`) ◦ v(`) ◦ w(`)
`=1
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Lower bounds and linear algebra
Let U, V , W be the matrices defining a canonical factorization of
A of length rk(A)
A=
rk
(A)
X
u(`) ◦ v(`) ◦ w(`)
`=1
Assume w.l.o.g. that the first p columns of W are linearly
independent, that is
W = W1 W2 , W1 ∈ Fp×p , det W1 6= 0
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Lower bounds and linear algebra
Let U, V , W be the matrices defining a canonical factorization of
A of length rk(A)
A=
rk
(A)
X
u(`) ◦ v(`) ◦ w(`)
`=1
Assume w.l.o.g. that the first p columns of W are linearly
independent, that is
W = W1 W2 , W1 ∈ Fp×p , det W1 6= 0
P
(−1)
Let A0k = pj=1 wk,j Aj , define
A0 = [A01 , A02 , . . . , A0p ] = A •3 W1−1 ,
i.e., choose a different basis to represent the space spanned by the
slabs A1 , . . . , Ap
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Evidently, A0 has the same rank of A
decomposition is given by U 0 = U, V 0

1

0
W 0 = I W1−1 W2 = 
 0
0
and a canonical
= V , W 0 = W1−1 W , where

0 0 0 ∗ ∗ ∗
1 0 0 ∗ ∗ ∗ 

0 1 0 ∗ ∗ ∗ 
0 0 1 ∗ ∗ ∗
Remark
Any subtensor Ab formed by k 3-slabs [A0σ1 , . . . , A0σk ] is such that
b ≤ rk(A) − (p − k)
rk(A)
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
More generally,
Theorem
rk(A) ≥ max min(p − k + rk(A •3 TS))
T
T : k × p submatrix of Ip ,
S
S: p × p nonsingular
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
More generally,
Theorem
rk(A) ≥ max min(p − k + rk(A •3 TS))
T
T : k × p submatrix of Ip ,
S
S: p × p nonsingular
Corollary
If for any basis of span(A1 , . . . , Ap ) there exists a matrix of rank q
then
rk(A) ≥ p + q − 1
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Higher order generalization:
Corollary
Let A = [A1 , . . . , Anh ] ∈ Fn1 ×···×nh , Ak ∈ Fn1 ×···×nh−1 be such
that nh = dim(span(A1 , . . . , Anh )). If for any basis of
span(A1 , . . . , Anh ) there exists a tensor in the basis of rank q then
rk(A) ≥ nh + q − 1
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Higher order generalization:
Corollary
Let A = [A1 , . . . , Anh ] ∈ Fn1 ×···×nh , Ak ∈ Fn1 ×···×nh−1 be such
that nh = dim(span(A1 , . . . , Anh )). If for any basis of
span(A1 , . . . , Anh ) there exists a tensor in the basis of rank q then
rk(A) ≥ nh + q − 1
Example: For
A=
1 0
0 1
0 1
;
0 0
any basis must contain a nonsingular matrix, therefore
rk(A) ≥ 2 + 2 − 1 = 3
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Lower bounds to border rank
Unfortunately the same technique cannot be applied to border rank
W1 may be singular in the limit as → 0 so that E •3 W1−1 may not be
infinitesimal anymore
Remedy: compute the QR factorization of W and
which has Euclidean norm 1 for any . One has

∗ ∗ ∗ ∗ ∗

0 ∗ ∗ ∗ ∗
W 0 = QT W = 
 0 0 ∗ ∗ ∗
0 0 0 ∗ ∗
multiply W by Q T
∗
∗
∗
∗
Remark
There exists an m × n × k subtensor Ab such that
b ≤ brk(A) − (p − k)
brk(A)
Dario A. Bini
The role of tensor rank

∗
∗ 

∗ 
∗
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Theorem
brk(A) ≥ min min(p − k + brk(A •3 TS))
T
T : k × p submatrix of Ip ,
S
S: p × p nonsingular
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Theorem
brk(A) ≥ min min(p − k + brk(A •3 TS))
T
T : k × p submatrix of Ip ,
S
S: p × p nonsingular
Corollary
If any basis of the linear space spanned by A1 , . . . , Ap is made up
by matrices of rank at most q then brk(A) ≥ p + q − 1
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Theorem
brk(A) ≥ min min(p − k + brk(A •3 TS))
T
T : k × p submatrix of Ip ,
S
S: p × p nonsingular
Corollary
If any basis of the linear space spanned by A1 , . . . , Ap is made up
by matrices of rank at most q then brk(A) ≥ p + q − 1
Corollary (Generalization to higher dimension)
Let A = [A1 , . . . , Anh ]. If any basis of the linear space spanned by
A1 , . . . , Anh is made up by tensors of rank at most q then
brk(A) ≥ nh + q − 1
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Example
2
2
2
The tensor A ∈ Fn ×n ×n whose 3-slabs span the linear space




s1,1 . . . s1,n
S

 .

.. 
..
..
I ⊗S =
 , S =  ..
.
.
. 
S
sn,1 . . . sn,n
is associated with n × n matrix multiplication.
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Example
2
2
2
The tensor A ∈ Fn ×n ×n whose 3-slabs span the linear space




s1,1 . . . s1,n
S

 .

.. 
..
..
I ⊗S =
 , S =  ..
.
.
. 
S
sn,1 . . . sn,n
is associated with n × n matrix multiplication.
All the matrices in any basis of the linear space have rank at least
n. Therefore
brk(A) ≥ n2 + n − 1
The bound can be improved to
brk(A) ≥ n2 + 2n − 2
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Example
The tensor A ∈ F4×4×3 whose 3-slabs

s1 s2 0
 s3 s4 0
0 0 s1
span the linear space

0
0 
s2
is associated with the multiplication of a 2 × 2 triangular matrix
and a full 2 × 2 matrix
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
Border rank
Lower bounds
Example
The tensor A ∈ F4×4×3 whose 3-slabs

s1 s2 0
 s3 s4 0
0 0 s1
span the linear space

0
0 
s2
is associated with the multiplication of a 2 × 2 triangular matrix
and a full 2 × 2 matrix
For any basis of the linear space there exist two matrices which
form a tensor of rank at least 4.
rk(A) ≥ 4 − 2 + 4 = 6
Since there exists a canonical approximate decomposition of length
5 of A then brk(A) ≤ 5 < 6 = rk(A)
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
The direct sum conjecture
Some tensors of unknown rank
The direct sum conjecture
0
0
0
Let A ∈ Fm×n×p , B ∈ Fm ×n ×p . Consider the direct sum of A
and B
0
0
0
C = A ⊕ B ∈ F(m+m )×(n+n )×(p+p )
Direct sum conjecture, Strassen 1973
rk(C) = rk(A) + rk(B)
The complexity of two disjoint sets of bilinear forms is the sum of
the complexities of each set.
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
The direct sum conjecture
Some tensors of unknown rank
The direct sum conjecture
0
0
0
Let A ∈ Fm×n×p , B ∈ Fm ×n ×p . Consider the direct sum of A
and B
0
0
0
C = A ⊕ B ∈ F(m+m )×(n+n )×(p+p )
Direct sum conjecture, Strassen 1973
rk(C) = rk(A) + rk(B)
The complexity of two disjoint sets of bilinear forms is the sum of
the complexities of each set.
Fact
There are cases where brk(C) < brk(A) + brk(B)
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
The direct sum conjecture
Some tensors of unknown rank
Example (A. Schoenhage 81)
brk(A) = 10 < 9 + 4





A=





s11 s12 s13
s21 s22 s23
s21 s22 s23









t
t
t
t
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems




















1
1
1
The direct sum conjecture
Some tensors of unknown rank
. . .
. 1 1 . 1 1
2
2
2
2
1
.
.
−
1
.
.











− −
−
−


− − 






−
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
The direct sum conjecture
Some tensors of unknown rank
Matrix multiplication and open problems
2 × 2 matrix product:

A = I2 ⊗
s11 s12
s21 s22

s
s
11
12
 s21 s22


=

s11 s12 
s21 s22
rk(A) = brk(A) = 7 [Strassen 69, Landsberg 05]


s11 s12 s13
3 × 3 matrix product A = I3 ⊗  s21 s22 s23 
s31 s32 s33
19 ≤ rk(A) ≤ 23 [Laderman 76], [Bläser 03]
13 ≤ brk(A) ≤ 22 [Schönhage 81]
Dario A. Bini
The role of tensor rank
Preliminaries
Properties of tensor rank
Open problems
The direct sum conjecture
Some tensors of unknown rank
Matrix multiplication and open problems

s11
 s21
4 × 4 matrix product A = I4 ⊗ 
 s31
s41
s12
s22
s32
s42
s13
s23
s33
s43
34 ≤ rk(A) ≤ 49
22 ≤ brk(A) ≤ 49
Dario A. Bini
The role of tensor rank

s14
s24 

s34 
s44