Either limestone/caliche, or as Rasputin suggested, construction debris. Not many other possibilities, and Rasputin deserves a real pat on the back for thinking outside the box on that one.
If you can provide a geographic location, we can probably tell you if natural limestone or caliche is a promising candidate. (I'm familiar with both, as well as their geographical distribution.) If it's construction debris (another situation I'm familiar with), the rocks will likely be accompanied by nails, pieces of wood, and other (s)crap the contractor figured it would be cheaper to just bury on site rather than to pay someone to sort it out and have it hauled away.