Yes, if all that matters in your life is where you go to school then yes they were held back from achieving that.
For a lot of people higher education is seen as the mantle to climb the socioeconomic ladder and on average the Asian or white kid who was competitive but didn’t get to go to harvard will achieve similar outcomes regardless.
Were they held down though? In California when they eliminated AA at their public universities there wasn’t much change in the economic outcomes for asian and white students. Sure, before maybe they didn’t get to go to the ‘best’ school but on average still had similar outcomes.
This isn’t true for black/Hispanic/native American students whose economic outcome depreciated. These students benefited from being able to break into the social networks provided by elite universities. Something that white/Asian students might already have access to.
So if all that matters is going to elite schools then sure, but there are externalities that are important and not everyone benefits equally from the top echelon of schools. It’s not a perfect system but it’s better than not having it.
A lot of the political discussion I’ve read on here has also been pretty well thought out. I feel like people are taking time to explain their perspective more and even if in general it’s been more left leaning there is definitely more nuance. I was surprised by the quality of some of the discussion around the end of affirmative action.