I went ahead and did a calculation: how collisionless would "rocky" dark matter be?
That calculation is done for the Earth's "local" dark matter: isotropic 220 km/s orbits through a 0.3 GeV/cm^3 mean density. I gave it 5g/cm^3 density, somewhere between stone and iron.
Look at those numbers. If you built the Milky Way using Volkswagen-sized rocks as the dark matter, they'd last four thousand years between collisions; they'd be dust and plasma. Use 500 m asteroids, they'd last a million years before colliding and pulverizing. (Remember, these are 220 km/s collisions; they make Shoemaker-Levy look wimpy.) A 10^6 m planetoid could last for a gigayear---at least that survives a full Galactic orbit!---but at that point we're into the stuff that the EROS surveys have ruled out. Sub-meter-scale dust, of course, is not collisionless at all which is why it's never been even in the ballpark of viable dark matter candidates.