In order to simplify this expression, we need to add the like terms.
Like terms are terms that have the same variables to the same power.
So we have:
[tex]\begin{gathered} 3a^2b+5ab-4ab+ab^2+8a^2b \\ =(3a^2b+8a^2b)+(5ab-4ab)+ab^2 \\ =11a^2b+ab+ab^2 \end{gathered}[/tex]
Therefore the correct option is the fourth one.
