Elections are being held in 27 English county councils and seven unitary authorities, last contested in 2009, as well as in Anglesey in Wales.
Labour are defending the South Shields seat in a contest caused by the resignation of David Miliband in March.
Mayoral contests are also being held in Doncaster and North Tyneside.
Polls for all the contests opened at 07:00 BST, and close at 22:00 BST.
Blue sky and sunshine are covering much of England, as millions of voters choose who will represent them on their local council.
More than 2,300 seats are up for grabs in county councils and unitary authorities across the length and breadth of England, including Derbyshire, Lancashire, Somerset, Essex, Surrey, Leicestershire, Kent, Durham, Bristol and Cornwall.
The Conservatives and Labour are putting up candidates in most seats, with 2,263 and 2,168 candidates respectively.
The Lib Dems have 1,763 candidates. UKIP is fielding 1,745 candidates, three times as many as it did the last time these seats were fought in 2009, and the Greens have 893 candidates.
Other parties standing include the Trade Unionist and Socialist Coalition, which is fighting 120 seats, the BNP, with 99 candidates and the English Democrats, with 38 candidates. There are around 900 independent candidates.
No elections are taking place in London, Scotland or Northern Ireland.
The majority of council results are expected on Friday between 11:00 BST and 18:30 BST, although six councils - Lincolnshire, Dorset, Somerset, Essex, Gloucestershire and Hampshire - will declare their results in the early hours of Friday morning.
The result for the South Shields by-election is also expected early on Friday morning.
Voters across England are going to the polls for council elections and a parliamentary by-election in South Shields.