Millions of Americans aren't working. Why?

John Makely | NBC News

Call it the million-worker mystery.

A large chunk of American adults are no longer in the labor force. That has left economists divided over how many of them are voluntarily not working-or even looking for work-because they wanted to retire, go to school or take care of family members, versus how many have been forced out because they couldn't find a job.

"Almost everyone who's looked rigorously at the numbers thinks both of those things are going on," said Heidi Shierholz, an economist with the Economic Policy Institute, a left-leaning think tank.

What they can't agree on is what is more prevalent-leaving the workplace on purpose or getting left out even as the economy improves.

The distinction is important because it would help economists understand whether the job market is on its way to a healthy recovery, or whether the current unemployment rate of 6.7 percent vastly underestimates how many Americans actually need a job.

The labor force participation rate-or the share of Americans ages 16 or over who are working or have been actively looked for a job in the past four weeks-has generally been decreasing since about 2000. That's happened as an aging population has headed toward retirement and more young Americans have spent time in college before heading into the workforce.

Read More For love and money: Many parents want to work

But that drop accelerated after the nation went into recession in December 2007. The labor force participation rate fell about 1 percentage point between 2000 and 2007, from about 67 percent to 66 percent, before dropping another 4 percentage points-from about 66 percent in 2007 to about 63 percent today.

That big drop came during a time of incredible economic turmoil, and one problem economists face when parsing through the statistics is that people lead complicated lives that may not easily fit into a box on a chart.

A mother who gave birth in 2008 may have taken time out of the paid workforce after having that baby, for example, but only because she didn't think she could find a job that paid well enough to cover child care costs. Another mom may have gone back to work although she hadn't planned to work, because her husband lost his job and they needed the income.

An older man may have gotten laid off at age 61, and, finding few other options, opted to retire years earlier than planned. Meanwhile, his friend may have planned to retire early but instead decided to stick it out a few more years, until her nest egg fattened up a bit.

Take the case of Jennifer Dias.

Dias, who now lives in Holbrook, N.Y., was working as a medical writer for a large pharmaceutical company in 2007, with plans to retire early from that job and start her own business to keep busy and earn some money.

"I would always want to be doing something," she said.

She was in her 50s then, and living a middle class life that included a nice house, salary and job. Then the stock market tanked, making her early retirement plan uncertain, about the same time as her employer became the target of a drawn-out takeover.

While she had once regularly fielded calls from recruiters, she now faced the possibility of a layoff with few job prospects in sight.

She lost her job in February 2010, and the market losses combined with the job stress left her feeling nervous about taking the risk of starting her own business.

Read More Stupid things employers ask job candidates

Dias eventually ended up moving in with her parents and starting school to reinvent herself as a physician's assistant. Then in 2012 came devastating news: She had been diagnosed with cancer.

Dias, now 60, is on disability and being treated for liver cancer. She said she would probably be considered retired, but she would like to work and dreams of one day having at least a freelance job when she is healthier.

The financial hit of losing a job and not finding a new one has been devastating, as have her serious health problems. But she said the most difficult thing has been losing the prestigious career she worked so hard for, and not being able to find another job.

"That's probably the hardest part of it all," she said, "because we base our identities on our jobs."

Opt out, or left out?

Shierholz estimates that there are currently more than 5 million missing workers who are not included in the unemployment rate because they are neither working nor looking for work, but who might be if the job market were stronger.

She bases her estimate on a comparison of employment projections before the Great Recession to the actual situation now.

Some economists agree with her.

"It's pretty clear in my mind that these people are missing from the labor force," said Dean Baker, co-director of the Center for Economic and Policy Research, a left-leaning think tank.

He said that's because there have been significant losses among so-called "prime age workers." These are workers who are between age 25 and 54, and therefore are likely too old to be in college and too young to retire.

The labor force participation rate for prime-age workers fell from about 83 percent in March 2007 to about 81 percent last month, according to the Bureau of Labor Statistics.

"These people just didn't decide to drop out of the labor market," Baker said.

Other economists say that in more recent years at least, much of the drop-off can be explained by aging baby boomers retiring.

Shigeru Fujita, a senior economist with the Federal Reserve Bank of Philadelphia, recently took a look at employment statistics for 2012 and 2013. That was a time when the unemployment rate was falling so quickly that many economists were speculating that the only possible explanation was that a bunch of people were just giving up on looking for work, and therefore were no longer counted as unemployed because they hadn't recently looked for a job.

Fujita concluded that most of the people who left the labor market in the past two years were part of a retirement wave. He said that means they are unlikely to jump back into the labor market any time soon.

"I think the basic message is that the labor market is actually in a relatively healthy situation," Fujita said.

Read More Stay-at-home dads struggle to shake Mr. Mom image

Joel Naroff, economist with Naroff Economic Advisors, said it's certainly true that some people have become discouraged by the bad job situation, but he thinks the downturn has only accelerated by a little bit a long-term trend toward baby boomers getting out of the job market.

That analysis would be good news for people who have a job or are looking for one-and might find that they are getting higher salary offers, or even raises to stay on the job.

"I think the labor market is clearly tightening," he said.

-By CNBC's Allison Linn.