I don’t think he ever actually had a chance at the crown; He was born before his mother married the crown prince. His younger siblings would have a potential claim since they’re the prince’s kids, but he wouldn’t any claim since he’s not the prince’s child. He’s even excluded from some of the royal portraits.