Veyne wants to discredit two views of discourse that he finds incorrect: discourse as an ideology and discourse as an infrastructure. To Veyne (and apparently Foucault), discourse is not an imposed ideology, nor forces of productions, but a "pair of glasses through which people of all ages observe and react" (p. 49).
Some authors have mentioned that "discourse" is a difficult term to define. Foucault's examples of "parts of a discourse" are adminstrative procedures or legal practices. He is of course famous for his interest in the "ruptures" of discourse -- points in history where our way of seeing and doing things radically changes. So, in a way (my intepretation) discourse could be informally described as Zeitgeist or common sense reasoning behind everyday practices.
However, let's try a completely different angle and assume that Foucault was simply exaggerating. Suppose there is nothing behind the administrative procedures and legal practices of any era -- they are just adaptations of the era's scientific, technical, political and practical knowledge into administration and law. Suppose moreover that there are no real ruptures in what we find in a sequence of archived documents -- there is only gradual change.
Any support for this argument? No, but I do not have to. It's simply an application of Occam's razor ("entities must not be multiplied beyond necessity"), and maybe discourse is an unnecessary entity. And just to illustrate the point by a current development in law: still in the 1980's, in Finnish law, "provoking someone to an act of homosexuality" was punishable by a fine. Soon (and that's about 25 years after this clause was removed from the codex) same-sex marriages will be legal in Finland. Still, why would we call this a rupture in the legal discourse? Most of the laws that were in effect in the 1980's still stand.
Then, how about Foucault's idea of the autonomy of discourse, meaning that for a researcher, there is no need to look into anything behind it (the forces of production, for example)? Foucault's idea is apparently that if there was something like this, it would be already a part of the discourse. I would think that an empiricist would not like this idea -- most people probably think that there are some real phenomena behind statistics, for instance. But maybe even that would be a part of discourse, since Foucault was a strong believer in "no observation without a theory". Consequently? Maybe it is better to see the "autonomous discourse" idea simply as a research programme, not as a profound theory. I can never be proven or disproven, anyway.
Veyne emphasizes, too, that Foucault was a nominalist. And that brings us to another subject .. to be continued..