First, let us rewrite the cube of sine and cosine in the left-hand side of the above equation as a product of a square of the sine/cosine and the sine/cosine:
(sinx)^2*sinx*sin(3x) + (cosx)^2*cosx*cos(3x) = 0.
Now, group the product of sinx and sin(3x) together and apply the product sines formula. The same can be done with the product of cosx and cos(3x).
The general form of the formulas is as follows:
Product of sines—sina*sinb = 1/2(cos(a-b) - cos(a+b))
Product of cosines—cosa*cosb = 1/2(cos(a+b) + cos(a-b))
Accordingly, sinx*sin(3x) = 1/2(cos(2x) - cos(4x)), and cosx*cos(3x) = 1/2(cos2x + cos(4x)).
Plug these into the equation and multiply both sides by 2:
(sinx)^2*(cos(2x) - cos(4x)) + (cosx)^2*(cos(2x)+cos(4x)) = 0.
Now, open the parenthesis:
(sinx)^2*cos(2x) - (sinx)^2*cos(4x) + (cosx)^2*cos(2x) + (cosx)^2*cos(4x) = 0.
Group together the terms that contain cos(2x) and cos(4x):
cos(2x)*((sinx)^2 + (cosx)^2) + cos(4x)*((cosx)^2 - (sinx)^2) = 0.
Applying Pythagorean Theorem ((sinx)^2 +(cosx)^2 = 1) and the double-angle identity for cosine ((cosx)^2 - (sinx)^2 = 1) gives us the following:
cos(2x) +cos(4x)*cos(2x) = 0
Factoring out cos(2x) results in cos(2x)*(1 + cos4x) = 0.
We can now set each factor to zero and solve each of the resultant equations separately: cos(2x) = 0 and 1 + cos(4x) = 0.
Cosine is 0 when the angle is pi/2 or -pi/2 and for all coterminal angles:
2x = +-pi/2 + pi*k, where k is an integer (0, +-1, +-2, and so on).
Dividing this by 2 yields x = +-pi/4 +pi/2*k.
The second factor set to zero gives cos(4x) = -1. The solutions to this equation, however, are already included in the set above.
Therefore, the solution set of the above equation is x = +-pi/4 + pi/2*k, where k is an integer.