If you don't smell or hear that their diaper is dirty, do you take off the swaddle or leave it for the middle of the night feedings?