Ms Yellen said current valuations were "quite high", but added she did not see any bubbles forming.
Chinese shares saw the biggest falls in the region with the Shanghai Composite closing down 2.8% at 4,112.21 - marking a third day of declines.
Hong Kong's Hang Seng was down 1.3% at 27,289.97 - falling for the sixth day.
Several companies on the mainland have announced share offerings through the week, leading investors to pull funds from other stocks.
Investors were also worried about reported new moves by regulators to cool China's market, such as capping the size of margin financing and limiting the type of stocks investors can buy with borrowed money.
In Japan, the benchmark Nikkei 225 closed down 1.23% at 19,291.99 after being closed all week for a holiday.
Australia's benchmark S&P/ASX 200 index closed down 0.82% at 5,645.70.
Shares in Australia had suffered their biggest one-day fall in more than two years on Wednesday following disappointing results from Australia's biggest banks.
Investors are concerned about the country's big four banks facing slower growth.
National Australia Bank (NAB), the last of the big banks to announce their results this week, reported a 5.4% rise in cash profit for the six months to March.
It also announced a 5.5bn Australian dollar ($4.4bn; Â£2.8bn) capital-raising exercise which will involve the selling of new shares.
NAB wants to demerge and float its troubled British bank, Clydesdale, and shore up its balance sheet.
The lender's capital-raising announcement has been reported by some of Australia's media as being the largest in the country's corporate history.
Also weighing on investor confidence were the latest figures on Australia's unemployment rate, which showed a slight rise to 6.2% in April.
In South Korea, the benchmark Kospi index closed down 0.65% at 2,091.00.
Asian shares fell on Thursday, tracking losses seen on US markets after Federal Reserve chair Janet Yellen expressed concerns over share valuations.