If the planes x - cy - bz = 0, cx - y + az = 0 and bx + ay - z = 0 pass through a straight line, then the value of a² + b² + c² + 2abc is:

  $x-cy-bz=0$

$cx-y+az=0$

$bx+ay-z=0$

Equation of any plane passing through the line of intersection of planes (1) and (2)

$x-cy-bz+\lambda (cx-y+az)=0$

$(1+\lambda c)x-(c+\lambda )y+(-b+a\lambda )z=0$

If (3) &  (4) is same plane.

$\frac{1+c\lambda }{b}=\frac{-(c+\lambda )}{a}=\frac{-b+a\lambda }{-1}$

(i)

(ii) (iii)

By (i)  & (ii) $\lambda =-\frac{(a+bc)}{ac+b}$

By (ii) & (iii)

$\lambda =\frac{-(ab+c)}{1-a^2};a^2+b^2+c^2+2abc=1$

$\overrightarrow{c}=2\left(\overrightarrow{a}\times \overrightarrow{b}\right)-3\overrightarrow{b}$  

Dot product with  $\overrightarrow{a}$ on both sides

$\overrightarrow{c}\cdot \overrightarrow{a}=-6$ ... (1)

Dot product with  $\overrightarrow{b}$  on both sides

$\overrightarrow{b}\cdot \overrightarrow{c}=-48$ ... (2)

$\overrightarrow{c}\cdot \overrightarrow{c}=4{\left|\overrightarrow{a}\times \overrightarrow{b}\right|}^2+9{\left|\overrightarrow{b}\right|}^2$

${\left|\overrightarrow{c}\right|}^2=4\left[{\left|\overrightarrow{a}\right|}^2\cdot {\left|\overrightarrow{b}\right|}^2-{\left(\overrightarrow{a}\cdot \overrightarrow{b}\right)}^2\right]+9{\left|\overrightarrow{b}\right|}^2$

${\left|\overrightarrow{c}\right|}^2=4\left[\left(1\right){\left(4\right)}^2-\left(4\right)\right]+9\left(16\right)$

${\left|\overrightarrow{c}\right|}^2=4\left[12\right]+144$

${\left|\overrightarrow{c}\right|}^2=48+144$

${\left|\overrightarrow{c}\right|}^2=192$

$cos\theta =\frac{\overrightarrow{b}\cdot \overrightarrow{c}}{\left|\overrightarrow{b}\right|\left|\overrightarrow{c}\right|}$

$cos\theta =\frac{-48}{\sqrt[]{192}\cdot 4}$

$cos\theta =\frac{-48}{8\sqrt[]{3}\cdot 4}$

$cos\theta =\frac{-3}{2\sqrt[]{3}}$

$cos\theta =\frac{-\sqrt[]{3}}{2}$ $\theta =co{s}^{-1}\left(\frac{-\sqrt[]{3}}{2}\right)$