Clearly "good" people are not perfect. They make mistakes. Some are "honest" mistakes, but some are not - selfishness, lying, etc. They try to do the best they can and, being human, fail from time to time; but not from lack of effort.Also, "evil" people sometimes do good.
Sometimes it's because, momentarily, it's in their best interests to do so. Sometimes it's a rare moment of goodness. Back during Nazi Germany, Himmler gave one high ranking SS general a sword for his birthday. The general, stunned by this act, took it upon himself to discover the factory which made the sword, and ordered an even more ornate and luxurious one to present to Himmler on HIS birthday. These are the men responsible for the Final Solution and the Holocaust, outdoing each other on birthday presents (!).
This context provides other examples of "good" and "evil". While many of the SS and police men the Germans assigned to Einsatzgruppen (the units which took Jews out to the forest to shoot them down) were outright sadists or indifferently accepted this duty as "just doing their job", many others had serious moral problems with it. Some even went insane and shot themselves. Recognizing that these jobs were... unpleasant...the SS freely allowed men to transfer to other jobs. So even within the SS at least some of its personnel could recognize evil when they saw it and refuse to participate in it.
On the other side are local civilians and German Army personnel who not only cooperated with the Nazis efforts to weed out and exterminate local Jewish populations, they enthusiastically volunteered. In Poland, Ukraine, and Romania, local civilians sometimes mistreated and even killed some of the Jews before the SS even got their hands on them. Clearly, the Nazis had no monopoly on evil. I won't even get into Nanking, where the Japanese even took pictures of themselves raping Chinese women or bayonetting prisoners as sport, to the point where a Nazi businessman, John Rabe, complained to Hitler and the Japanese about this.
The story of Faust has been told dozens of times, and resurrected in various forms. Professor Faust makes a pact with Mephistopholes, the devil, whereby he gets renewed youth and vigor and almost unlimited power for 24 years, after which time the devil claims his soul. But throughout that time, and all the adventures he has with the devil at his side, he uses these powers of evil for good. His last act, before the 24 year period expires, is to have the devil save an elderly couple whose house, located on Faust's property, went up into flames. The power of evil, used for good. This alone prompts God to rescue Faust's soul from Hell and breaks the pact he had with the devil.
Do the evil perceive themselves as evil? Another complex question. One rationalization is that "everyone does it". It's a slippery slope: to the extent no one is perfect, and even so-called "good people" occasionally lapse into selfishness, everyone is evil; some are more honest about it than others, and those who claim to be good are nothing more than arrogant hypocrites. But it only takes a moment of thinking to dispel this. Clearly there are those who lie, cheat, steal and kill without any reluctance or remorse whatsoever; not as a "last resort", not "compelled by unique circumstances", but as easily as they walk, talk, eat, sleep or breathe. I know several people who lie without hesitation about anything. Maybe no one can claim to never have lied in their lives, but that doesn't mean they're not honest in their dealings and even tell the truth when it hurts their own interests. At some level, the evil do recognize that others do not act they way they do, and there is some definite moral distinction. But rather than admit to themselves, much less to others, their own evil, they prefer to rationalize it away and continue with their business of lies, deceit and evil. To them, the "good" are simply those too stupid to lie effectively, too lazy, too weak or just plain not ambitious or competent enough to assert their will over others. I'm not sure how this translates into modern times.
Can evil people change? I hate to quote religious sources, but three examples jump out. First is St Paul, formerly Saul. He started out persecuting Christians to the death, but experienced a dramatic vision on the road to Damascus and converted. St. Augustine had a similar conversion; he previously lived a wild and debauched life until his mother and Bishop Ambrose of Milan helped him to convert. Finally, there is the famous parable of the prodigal son, who spent his inheritance on gambling and whores, returned home a wretched beggar, but sincere in his remorse, and was welcomed home by his father.
In fact, a remarkable take on this is "My Name is Earl", a new TV show with Jason Lee, who is a frequent actor in many of the Kevin Smith films and the main role in "Mallrats". Earl is a redneck who leads a fairly decrepit lifestyle, lying, cheating, stealing, basically a bad person. Finally he gets hit by a car and figures that this major injury was well-deserved. He decides, from that point on, to turn his life around and only do good things. He makes a "list" of all the nasty, evil, dishonest things he's done over his entire life (the list is very long and can probably string the series over several seasons) and sets out to correct each and every evil deed, one by one. He invokes "karma", which he describes as "if I do good things, good things will happen to me." It's not completely selfless, as his motivation is to have a good life, not merely to do good for the sake of doing so. But he does recognize that certain things are "wrong" and that he shouldn't do them even if others around him are doing it.
More things to think about.