$\mathbf{F}(x,y,z)=-xz\mathbf{i}-yz\mathbf{j}+z^2\mathbf{k}$, where S is the cone with equation $z=\sqrt{x^2+y^2}$between $z=2,4$, with n pointing outwards.

Consider the given question,

$$\begin{gathered} \mathbf{F}(x,y,z)=-xz\mathbf{i}-yz\mathbf{j}+z^2\mathbf{k}\mathbf{,} \\ z=\sqrt{x^2+y^2} \end{gathered}$$

If a surface S is the graph of $\mathit{F}\left(x,y,z\right)$, then the Flux of $\mathit{F}\left(x,y,z\right)$ through S is given below,

$\begin{array}{r}{\int }_S \mathbf{F}(x,y,z)\cdot \mathbf{n}dS={\iint }_D \left(\mathbf{F}\right(x,y,z)\cdot \mathbf{n})∥z_x\times z_y∥dA\\ ={\iint }_D \left(\mathbf{F}\right(x,y,z)\cdot \mathbf{n})\left(\sqrt{{\left(\frac{\mathrm{\partial }z}{\mathrm{\partial }x}\right)}^2+{\left(\frac{\mathrm{\partial }z}{\mathrm{\partial }y}\right)}^2+1}dA\dots ...\left(i\right)\right.\end{array}$

Now first find $\frac{\mathrm{\partial }z}{\mathrm{\partial }x},\frac{\mathrm{\partial }z}{\mathrm{\partial }\mathrm{y}}$. The first partial derivates of z are given below,

$\begin{array}{r}\frac{\mathrm{\partial }z}{\mathrm{\partial }x}=\frac{\mathrm{\partial }}{\mathrm{\partial }x}\left(\sqrt{x^2+y^2}\right)\\ =\frac{x}{\sqrt{x^2+y^2}}\\ \begin{array}{r}\frac{\mathrm{\partial }z}{\mathrm{\partial }y}=\frac{\mathrm{\partial }}{\mathrm{\partial }y}\left(\sqrt{x^2+y^2}\right)\\ =\frac{y}{\sqrt{x^2+y^2}}\end{array}\end{array}$

On calculating the value,

$\begin{array}{r}\sqrt{{\left(\frac{\mathrm{\partial }z}{\mathrm{\partial }x}\right)}^2+{\left(\frac{\mathrm{\partial }z}{\mathrm{\partial }y}\right)}^2+1}=\sqrt{{\left(\frac{x}{\sqrt{x^2+y^2}}\right)}^2+{\left(\frac{y}{\sqrt{x^2+y^2}}\right)}^2+1}\\ =\sqrt{\frac{x^2}{x^2+y^2}+\frac{y^2}{x^2+y^2}+1}\\ =\sqrt{2}\end{array}$

If $z=z(x,y)$, then the following vector,

is normal to the surface.

For the value of z, it gives the following vector perpendicular to this surface,

Then, the desired normal vector will be,

The value of $\mathbf{F}(x,y,z)\cdot \mathbf{n}$is given below,

On substituting the value of z in equation (ii),

$\begin{array}{r}\mathbf{F}(x,y,z)\cdot \mathbf{n}=\frac{1}{\sqrt{2}}\left(z\sqrt{x^2+y^2}+z^2\right)\\ =\sqrt{2}{z}^2\end{array}$

Substitute the values in equation (i),

$\begin{array}{r}{\int }_S \mathbf{F}(x,y,z)\cdot \mathbf{n}dS={\iint }_D \left(\mathbf{F}\right(x,y,z)\cdot \mathbf{n})\left(\sqrt{{\left(\frac{\mathrm{\partial }z}{\mathrm{\partial }x}\right)}^2+{\left(\frac{\mathrm{\partial }z}{\mathrm{\partial }y}\right)}^2+1}\right)dA\\ ={\iint }_D \left(\sqrt{2}{z}^2\right)\left(\sqrt{2}\right)dA\\ =2{\iint }_D z^{2}dA\end{array}$

The surface Sis the cone between $z=2,4$, that is, the surface Sis the cone between $$\begin{gathered} \sqrt{x^2+y^2}=2, \\ \sqrt{x^2+y^2}=4 \end{gathered}$$, so the region of integration D is annulus determined by $2\le z\le 4$ in the xy-plane.

Change it to polar coordinates and integrate with the region of integration,

$D=\left\{\right(r,\theta )\mid 2\le r\le 4,0\le \theta \le 2\pi \}$

In this case,

$\begin{array}{r}x=r\cos \theta \\ y=r\sin \theta \\ z=\sqrt{x^2+y^2}=r\end{array}$

Then, the flux of the vector field through the surface S is given below,

$\begin{array}{r}{\int }_S \mathbf{F}(x,y,z)\cdot \mathbf{n}dS=2{\iint }_D z^{2}dA\\ =2{\int }_0^{2\pi } {\int }_2^4 \left(r^2\right)rdrd\theta \\ \begin{array}{r}=2{\int }_0^{2\pi } \left[{\int }_2^4 r^{3}dr\right]d\theta \\ =2{\int }_0^{2\pi } {\left[\frac{r^4}{4}\right]}_2^{4}d\theta \end{array}\end{array}$

On solving the above equation,

$\begin{array}{r}{\int }_S \mathbf{F}(x,y,z)\cdot \mathbf{n}dS=2{\int }_0^{2\pi } [64-4]d\theta \\ \begin{array}{r}=120{\int }_0^{2\pi } d\theta \\ =120[\theta {]}_0^{2\pi }\\ =120[2\pi -0]\\ =120\left[2\pi \right]\end{array}\\ =240\pi \end{array}$