First of all remove the thought from your mind that English is tough. Take it as a language that you informally use with your friends and family. Read more of books and write down the difficult words and find their meaning.
The next step would be to use those words while framing sentences. More and more of speaking will help you get the fluency in the language.
Hope this helps you!!