Have the babies been leaving the nest and begun eating on their own?
At 4 weeks of age they should be fledging, relying less and less on feedings from their parents. Millet will likely be their favorite food right now.
If they've been doing this, it's ok to remove them. Just keep a close watch on them, be sure they're thriving and always have an avian vet you can trust for in-person visits.