Milk protein allergy in feeding son, can I eat lactose?

My son was congested from the time we brought him home from the hospital until 5 weeks, at which point I figured out that he had a milk protein allergy.He is exclusively fed. I stopped eating all milk products and his congestion went away. Then to test it, I ate some cheese, and that night he was congested again. It has been two weeks since I have stopped eating all milk products, and he is probably about 80% better. He still gets congested occasionally, but nowhere near as often or as bad. I have been avoiding all milk products. But some of the things I want to eat only have lactose (which is a sugar in milk) and not any whey or casein, or other milk proteins. Has anyone with a milk protein allergy been able to eat lactose without a problem, or fed their feeding infant without a problem? I would think that since it is a sugar it shouldn't be a problem, but I was curious what other people thought.