Do you ever shift back and forth between moral views? Even just the understanding of both sides can confuse you more than you can even fathom. What happens when you begin seeing the positive side of something negative? What is negative anyway? What constitutes a "wrong" desire? I suppose something you've experienced first hand that has caused wretched repercussions in a personal view would suffice for not having irresistible urges towards certain things.
Certain energies gravitate entities to certain things. Putting yourself in a situation that will hold something you're attempting to avoid will inevitably induce feelings of desire towards it. Where do you draw the line between forces? How do we determine pros and cons? There are so many good aspects to specific situations and so many life scattering ones; how do we weigh it all out? Even regarding people that influence you in a way you wish not to gravitate, is it possible to attempt to highlight the progressively positive things about them and not shed energy on the bad parts so you can regulate their influence upon you?
If there is a possibility of engaging in activities that are not right in your mind, it is the wisest decision to avoid any chance of said situation. As righteous as some people are, it can be imperative to distance yourself from some in order to achieve your goal. Do what you must with what you have when you can and you will prosper greatly from it.