Man made fabrics such as polyester and even those that are semi-synthetic plant based such as rayon have a significant impact on the environment. Silk is a natural fabric created by the silk worm which feeds on mulberry trees. The mulberry tree does not use much water and pesticides are not used or at a minimum.
The production of polyester and its relatives rayon, viscose and modal, all involve use of harmful chemicals, large amounts of water and energy. Polyester is the true villain because it is made from petroleum, a non renewable fossil fuel, and when we wash clothing containing polyester microfibres are released into the environment and are eventually found in our oceans and waterways. Polyester nylon and cotton including because they are processed, finished, dyed, and coated with a wide range of chemicals are the fibres commonly found in ocean water samples. Silk is the fabric which has least impact on ocean plastics and microfibres.
Silk is biodegradable, so in the unlikely event that you want to throw out your silk garment rather than keep wearing it until it falls apart, it will disintegrate quickly. Also the foliage of the mulberry tree is fodder for cattle and related waste is used as fertilizer.
If you want to read more on this topic here are some useful links: