Would this be rude of me?? Please answer....Any opinions appreciated!?

NO. I think it is perfectly reasonable to ask her to come another time when YOU invite her. You really don't need a 3 year old running around the house with a baby.