When did you stop using the insert in your Ergo? Did you stop using the pillow first?

DS has decided he does not like any froggy leg position anymore in both the Ergo and the Moby. No problem with the Moby, but I'm a little hesitant with the Ergo! He meets the 12 pound requirement, but I wonder if he needs the head and neck support the insert offers. He is 11 weeks.