Ok so , one thing to notice is that U - potential energy and the dimensional formula of any kind of energy is one and the same = [ML2T-2]
Now see that the term 'B' is being added to x2 - square of distance - with dimensional formula of [L2]
Only like terms can be added. That is distance2 can be added to distance2 only, neither can it be added to distance or distance3 , nor can it be added to some other qty altogether like pressure or force etc.
So we now know that B has D.F = [L2]
Now solving Ax1/2/x2 = Ax-3/2. - whic technically should be equal to energy D.F
So solving on both sides we get A = [ML7/2T-2]
AB = [ML7/2T-2][L2] = [ML11/2T-2] .
Hence option(2)
Hope this helped !:))