What you describe is exactly what my grandparents did with a 3 family home. They built up enough money to move to a two family home where we all lived. Bought it outright. That house was worth more than most single family homes in the area. By 1987 they sold it and moved to a single family house in a very nice neighborhood.

It’s definitely a solid plan. I always choose the path of least resistance when possible. I’m not allergic to resistance, or more work when necessary. Being a landlord is not nearly as easy as it sounds. I failed at it almost immediately.