Robert and his wife Nancy were strict with only child Audrey, thrusting responsibility upon her from a young age. When their daughter fell pregnant at the age of sixteen in 1956, the Potters were horrified, and Robert accepted an offer from close friends and neighbours Joyce and Malcolm Reid to adopt the child and raise him as their own.
When their daughter became pregnant again two years later with Gail, she stood up to her father and was adamant she was keeping this child. Although Audrey was able to persuade her parents, her efforts were unnecessary as Robert suddenly died, shortly before she gave birth to Gail. Audrey later wondered if the stress she had caused her father as a teenager had been a contributing factor to his early death.