Learning by Observing and Pitching In