When a set has more vectors than entries in each vector, it is said to be linearly dependent.

Let \({v_1},{v_2}\,\),and \({v_3}\) be the three vectors. The linear dependence of these three vectors in the form of an augmented matrix is \(\left[ {\begin{array}{*{20}{c}}{{v_1}}&{{v_2}}&{{v_3}}&0\end{array}} \right]\).

Hence, the augmented matrix is:

\(\left[ {\begin{array}{*{20}{c}}1&{ - 5}&1&0\\{ - 1}&7&1&0\\{ - 3}&8&h&0\end{array}} \right]\)

Apply row operation \({R_2} \to {R_2} + {R_1}\) to the augmented matrix above.

\(\left[ {\begin{array}{*{20}{c}}1&{ - 5}&1&0\\0&2&2&0\\{ - 3}&8&h&0\end{array}} \right]\)

Apply row operation \({R_3} \to {R_3} + 3{R_1}.\)

\(\left[ {\begin{array}{*{20}{c}}1&{ - 5}&1&0\\0&2&2&0\\0&{ - 7}&{h + 3}&0\end{array}} \right]\)

Apply row operation \({R_3} \to {R_3} + \frac{7}{2}{R_2}.\)

\(\left[ {\begin{array}{*{20}{c}}1&{ - 5}&1&0\\0&2&2&0\\0&0&{h + 10}&0\end{array}} \right]\)

The pivots in the echelon matrix are represented as:

If the zero vector appears in a set \(S = \left\{ {{v_1},{v_2},.....{v_p}} \right\}\) in \({R^n}\) , the set is linearly dependent.

Thus, the equation can be written as \({x_1}{v_1} + {x_2}{v_2} + {x_3}{v_3} = 0\). The vector will have a nontrivial solution if the value of \(h\) is \(h + 10 = 0\).

Hence, the vectors are linearly dependent if and only if the value of \(h = - 10\).