If you have to cut corners in order for the game to run properly then it's not what you envisioned. Larger texture sizes that have been made, for example, that can't be implemented on one platform but can on another.
Take TLOU REmake for example. The added touches that will be available on the PS5 (and not on the PS4) is an example of what I'm talking about.
Yes, it matters. Do you know how many people participated and argued in a thread when DF compared the XSX and PS5 with a 3rd party game and people continued to say "Another One" for the platform that was a little better than the other one? If it didn't matter, then people wouldn't care about day and date PC/PS releases.