It took the Tapolca seat with a majority of just under 300 votes.
Jobbik looks set to present a major challenge to the governing Fidesz party in parliamentary elections due in three years' time.
Correspondents say the party has softened its rhetoric in recent years and has gained support from mainstream voters.
Lajos Rig's margin of victory was narrow, but his win was symbolically enormous, says the BBC's Nick Thorpe in Budapest.
"The mood in Hungary is for a change of government and with Jobbik Hungary finally has a force to change the government,'' party leader Gabor Vona told supporters in Tapolca.
It re-enforces their reputation as the most successful nationalist party in Europe and is another setback for Hungary's Prime Minister Viktor Orban.
"There are times when the ball hits the post," said Mr Orban, a keen footballer, on his Facebook page as he admitted defeat.
Fidesz easily won the last national and European elections in 2014, when Jobbik took around 20% of the vote.
But since then the governing party has seen its reputation falter, with allegations of corruption, and opposition to measures like a planned internet tax and limits on Sunday trading.
Fidesz lost its two-thirds parliamentary majority in February after another by-election defeat.
The far-right Jobbik party in Hungary has won its first ever individual constituency seat in parliament.