According to Patanjali, there are quite a few obstacles towards reaching the promised nirvana of yoga. Sickness, apathy, doubt, heedlessness, sloth, (i love that one), dissipation, false vision, nonattainment, and instability.
The Buddhists speak of the three poisons…greed, aversion, and delusion.
Regardless of who you choose to listen to, the message is similar. Our human condition of wanting, worrying, and wondering clearly distracts us from the bliss of a higher consciousness.
What to do?
Just keep practicing.