Michael Kors stock not as haute as before

Michael Kors (KORS), which markets itself as a brand for the jet set, may soon be in for a rough ride.

For the first time since going public in late 2011, shares of the fashion company appeared at risk of declining on the day of an earnings release. The stock traded down as much as 4 percent Wednesday morning and was up slightly in midday trading. That's a far cry from the average gain of 8 percent over the previous eight reporting days, according to FactSet.

The muted response surprised some investors, given the company's impressive performance. Kors has topped consensus earnings estimates handily in every reporting period since going public. In the quarter through March, Kors reported earnings per share of 78 cents, well above analyst expectations of 68 cents.

Read More Can Instagram and Bethany Mota save this retailer?

What's different this time? One possible explanation is that investors have realized that the company's guidance is overly conservative, say some shareholders privately. Brokers often stick close to company guidance for fear of retribution from companies that want to keep expectations within reach. That can keep a lid on consensus forecasts while investors quietly anticipate a better outcome.

Indeed, a strong performance appeared priced into the stock before the earnings release arrived. Kors shares have risen an impressive 19 percent so far in 2014 as the company's sales continue to soar. That compares with steep share price declines for the likes of Coach (COH) and Ralph Lauren (RL).

But some investors are watching closely for signs that the company's remarkable growth may not be sustainable. On a conference call Wednesday morning, Kors said it expected long-term sales growth would be at a "double-digit" percentage rate. That compared with a previous long-term estimate of 20 to 25 percent revenue growth that Kors reiterated as recently as November. The company didn't respond to multiple requests for comment from CNBC.

One area of vulnerability could be the key North American market, where Kors generated 84 percent of its sales in the year through March.

Read More For once, Amazon won't leave a path of destruction

Kors has about 300 stores in the U.S., compared with roughly 550 for rival Coach, according to Paul Lejuez, an analyst with Wells Fargo. But Kors also sells a large amount of merchandise through department stores like Macy's. Lejuez estimates that Kors generates about $1.3 billion in U.S. wholesale revenue versus $225 million for Coach.

Existing wholesale revenue is potentially an issue because Kors might not be able to add a new retail store at a mall where it already generates big sales. The bar is especially high for Kors because it earns far more sales per retail store than rivals like Coach.

Of course, such concerns haven't yet come to fruition. The company said Wednesday that gross margins in retail haven't changed much from last year, even as Kors moved ahead quickly with store additions.

The real issue is that investors didn't respond as they have in the past to a very impressive earnings report. While Kors' earnings multiple has come down significantly over the last couple of years, it still trades at 24.3 times consensus forward earnings. Given how skittish investors have been recently, the company will need a perfect showing to keep the stock in the stratosphere.

-By CNBC's John Jannarone.