Lee spent his childhood in kitchens in Atlanta and Saint Simons Island, Georgia. During World War II, he served as a mess sergeant and instructor, for which he received a medal from President Franklin D. Roosevelt. He became the first Black executive chef in Harrisburg, Pennsylvania.