Stellar Flybys Leave a Permanent Mark on Newly Forming Planetary Systems