Rumors that the last major American rare earth elements mining and processing company may seek bankruptcy protection sent its shares plummeting last week, leading to questions over both the future of the company and whether a major rare earths mining business can even be viable outside China.

Analysts and experts who spoke with CNBC said that bankruptcy would not necessarily spell the end of Molycorp but that, apart from reducing its debt, the company still faces major challenges. The novel technologies that it developed to make rare earths processing less environmentally dangerous still need improvements before they can be productive, analysts say, and the company has to find a way to sell product at a time when prices are severely depressed due to competition from an influx of illegal mining operations in China.

Molycorp is one of very few companies outside China that mine and sell rare earth elements—a group of 17 elements in the periodic table that are found in everything from smartphones and plasma TVs to cars and fighter jets. The company's stock was trading around $75 a share in 2011, fueled, in part, by high demand for rare earths and Chinese export policies that tightened the supply of the elements.

On Monday, shares were trading just above 40 cents.

Recent news reports say the company is planning to file for bankruptcy protection, and analysts who follow the company say it is buried under mountains of debt and struggling with historically low prices for rare earths.

"We have been waiting for them to file for bankruptcy, and we have been anticipating this for some time given the dynamics of the global rare earths market and debt structure of Molycorp" said Michael Gambardella, an analyst for JP Morgan.

Kevin Starke, an analyst with CRT Capital, told CNBC that the company will likely file for bankruptcy, but that there is no reason yet to believe it will shut down entirely.

"In reality, bankruptcy would be more like a new beginning for Molycorp," Starke said. Molycorp spokesman Jim Sims said he could not speak about the company's future.

Surviving will not be easy.

First, the company will have to reduce the bulk of a $1.7 billion debt it amassed making expensive acquisitions and dealing with cost overruns at the company's mine and facility, Starke said.

"It paid top-of-market prices for acquisitions and used debt to finance a lot of the deals," said Kevin Starke, a managing director for CRT Capital.

The company had other challenges as well, though, said Dudley Kingsnorth, a professor of mineral and energy economics at Curtin University in Australia.

"They underestimated the complexities of starting up a project employing 'new' technologies to reduce the environmental footprint and could not have foreseen the dramatic fall in prices due to the uncontrolled illegal mining and processing in China," Kingsnorth said.

Rare earth elements need to be separated from the minerals in which they are embedded. Developing the processes to do this are complex and require huge investments. They also can be environmentally harmful, and Molycorp has spent a lot money trying to find methods that have minimal environmental impact.

"They are trying to make a fairly environmentally dangerous business environmentally compliant in an environmentally conscious place like California," Starke said.

The biggest unforeseen hurdle Molycorp and other producers are battling now are the historically low prices for rare earths, which many, including those at Molycorp, say is the result of unauthorized mining operations in the south of China. Many of these companies are small and completely unregulated, and use mining and extraction methods that Kingsnorth said are causing environmental damage there. The Chinese government has been trying to crack down on these producers, Kingsnorth said, but that is easier said than done, especially if demand begins to outstrip supply.

There are some experts in China who say the country's production quotas are not meeting demand, which is in part fueling the market for illegal suppliers.

Government-authorized production quotas in China run about 105,000 metric tonnes of rare earths every year, and illegal operations are estimated to be selling another 40,000 metric tonnes on top of that, Kingsnorth said. That extra supply alone is enough to drive down prices.

But many illegal producers undercut their regulated counterparts' prices by 50 percent. Unregulated companies can afford to, as they are probably not having to spend money meeting China's increasingly stringent environmental regulations, said Molycorp spokesman Sims.

This is hurting even legitimate producers in China, not just non-Chinese companies.

Most experts contacted by CNBC estimate that Chinese firms control between 90 and 95 percent of the market, a position they achieved after pushing out many of the formerly dominant players with lower prices in the 1990s.

The Chinese government curtailed the export of rare earths around 2010 in order to encourage their use among Chinese companies. The World Trade Organization forced the country to remove these restrictions this year, which has boosted exports again.

As a result many consider that the Chinese have a considerable competitive advantage that will continue to make it difficult for non-Chinese businesses to compete.

"I can't see how anyone outside of China really knows what is going on in this industry," said JP Morgan's Gambardella.

Molycorp is not the company that has struggled recently. But Australian rare earths firm Lynas—the only other major non-Chinese rare earths company—is paying off its own debts, and announced its first month of positive free cash flow in March.