Modification of Agreement with regards to child support and legal custody of children

Have been divorced for over two years. At the time of divorce ex-wife lived in martial home with me and 3 kids while she accumulated enough money to move out. We agreed on joint physical/legal custody with no child support (due to the fact she was living in house). Since she moved out last year she has had limited contact with children, usually two weekends a month. I have not limited her time with them, only advised her of school hours. She has never provided any type of set schedule, so I am constantly waiting for indeterminate pick up and drop off times. She has also on several occasions kept the children out of school with no legitamite excuse (ie. doctors visit or illness). I have assumed all the financial burden of the 3 children to include: school costs and recreational activities. Due to the fact that the children are in my "physical custody" for the majority of the time, should I seek a modification of our divorce agreement to include phsical custody and child support.