That is not how a bastard works. To be clear, all of this is archaic, I’m not actually calling anybody a bastard, but the definition of bastard is “a child born to parents who are not married to each other.” He matches that definition because his parents were never married.

Affairs don’t come into it, it’s just some old bullshit cultural and religious ideals about which types of relationships (just the one: heterosexual marriages) children are “supposed” to be born into.