Not only are middle class suffering from stagnant incomes, their wealth hasn't grown at all either.

That's led to the widest wealth gap on record between the middle class and the rich.

The median household net worth of middle-income Americans remained at $96,500 between 2010 and 2013, according to a new report from the Pew Research Center, which looked at Federal Reserve Bank data.

Upper-income households, however, saw their wealth grow to $639,400 last year, up from $595,300.

Related: Elizabeth Warren: 'Wealth trickles up'

That means the rich have 6.6 times more wealth than the middle class, a figure that's grown from 4.1 in 1998 and 3.4 in 1983.

It's also a record 69 times the wealth of lower-income Americans, who had accumulated only $9,300 as of last year.

Related: Whites get wealthier, while Blacks and Hispanics lag further behind

Part of the reason for the gap stems from how the rich and the middle class build wealth. The former are more likely to invest in the stock market, which has been on a tear in recent years. The latter have more of their net worth tied up in the housing market, which hasn't recovered as much.

That's also why the Great Recession had a bigger impact on the net worth of the middle class. Back in 2007, before the housing crash, the middle class saw its median wealth soar to $158,400.

The rich also haven't recovered fully from the downturn, but they are a lot closer to their 2007 peak of $718,000.

Looking longer term, the rich have more than doubled the size of their nest eggs over the past three decades, while the middle class have inched up 2.3%.

Pew defines middle income as family of four with a household income between $44,000 and $132,000. Some 46% of American household fall into middle income under its methodology, which adjusts for family size. Upper-income Americans are those who earn more than $132,000 for a family of four.

Share your story: Has your wealth gone up or down in recent years?