Chihuahua habits!?

I love my little chihuahua Tyson with all my heart he is affectionate....energetic...and a great joy....but he is dirty....He was trained to go on paper when we first got him, which is fair enough....but he toilets anywhere he feels like it! You can take him out and everything but he will still do it around the place! I have heard that allot of chi's are dirty like this and for some they never stop! Is this true? Has anyone else got this problem ?