Step 1 :

Factor out common factor $ \color{blue}{ 5m^2 } $:

$$ 30m^3n+15m^3+40m^2n+20m^2 = 5m^2 ( 6mn+3m+8n+4 ) $$

Step 2 :

To factor $ 6mn+3m+8n+4 $ we can use factoring by grouping.

Group $ \color{blue}{ 6mn }$ with $ \color{blue}{ 3m }$ and $ \color{red}{ 8n }$ with $ \color{red}{ 4 }$ then factor each group.

$$ \begin{aligned} 6mn+3m+8n+4 &= ( \color{blue}{ 6mn+3m } ) + ( \color{red}{ 8n+4 }) = \\ &= \color{blue}{ 3m( 2n+1 )} + \color{red}{ 4( 2n+1 ) } = \\ &= (3m+4)(2n+1) \end{aligned} $$

Polynomial Factoring Calculator