Show that each of the following matrices is orthogonal and find the rotation and/or reflection it produces as an operator acting on vectors. If a rotation, find the axis and angle; if a reflection, find the reflecting plane and the rotation, if any, about the normal to that plane.

$\frac{\mathbf{1}}{\mathbf{11}}\left(\begin{array}{ccc}2& 6& 9\\ 6& 7& -6\\ 9& -6& 2\end{array}\right)$

$M=\frac{1}{11}\left(\begin{array}{ccc}2& 6& 9\\ 6& 7& -6\\ 9& -6& 2\end{array}\right)$

The inverse of a matrix is mathematically presented as${\text{M}}^{-1}=\frac{1}{\det \text{M}}{C}^{\text{T}}$

First, verify that the matrix is orthogonal by calculating its inverse.

${\text{M}}^{-1}=\frac{1}{\det \text{M}}{C}^{\text{T}}$

Here, $C$ is the matrix of co-factors.

The determinant of $M$is, $$\begin{gathered} M=\frac{1}{{11}^3}\left(2\right(14-36)-6(12+54)+9(-36-63\left)\right) \\ =-1 \end{gathered}$$

The matrix of cofactors is, $C=-\frac{1}{11}\left(\begin{array}{ccc}2& 6& 9\\ 6& 7& -6\\ 9& -6& 2\end{array}\right)$

Then , $M^{-1}=\left(\begin{array}{ccc}2& 6& 9\\ 6& 7& -6\\ 9& -6& 2\end{array}\right)$

Since the matrix is symmetric, $M=M^T$, which gives $M^T=M^{-1}$. This means that the matrix is orthogonal. Since the determinant of $M$is $-1$, it is a reflection. To calculate the plane of reflection, search for a vector that is perpendicular to this plane, by solving the equation $Mr=-r$. This is an eigenvalue problem and is solved as shown below.

$$\begin{gathered} \frac{1}{11}\left(\begin{array}{ccc}2& 6& 9\\ 6& 7& -6\\ 9& -6& 2\end{array}\right)\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=-\left(\begin{array}{c}x\\ y\\ z\end{array}\right) \\ \frac{1}{11}\left(\begin{array}{ccc}13& 6& 9\\ 6& 18& -6\\ 9& -6& 13\end{array}\right)\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=0. \end{gathered}$$

Multiply by $11$ and obtain the equations.

$$\begin{gathered} 13x+6y+9z=0, \\ 6x+18y-6z=0, \\ 9x-6y+13z=0. \end{gathered}$$

Add the first and the last equation and obtain $x+z=0$. Insert this into the last equation, and obtain $y=-2/3x$. Write the vector $r=3i-2j-3k$. The plane of reflection is $3x-2y-3z=0$. Now, study the rotation of this vector and solve the equation to diagonalize matrix $M$as shown below.

$$\begin{gathered} \frac{1}{11}\left|\begin{array}{ccc}2-11\lambda & 6& 9\\ 6& 7-11\lambda & -6\\ 9& -6& 2-11\lambda \end{array}\right|=0 \\ {(2-11\lambda )}^2(7-11\lambda )-648-81(7-11\lambda )-72(2-11\lambda )=0 \\ {\lambda }^3-{\lambda }^2-\lambda +1=0 \\ ({\lambda }^2-1)(\lambda -1)=0 \end{gathered}$$

The eigenvalues are $1,1,-1$. From this, we can conclude that the diagonal matrix

is $D=\left(\begin{array}{ccc}-1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right)$

If this were a reflection and a rotation around the z axis—

If this were a reflection and a rotation around the $z$ axis— $3i-2j-3k$, $\cos \theta =1$, and $cos\theta =-1$—it cannot be true. Therefore, there is no rotation.