Wood in the perimeter drains on a foating slab?

My basement is a floating slab and the perimeter drain is clogged with crap (dirt, basement debris, etc.) I started cleaning it out and I noticed that there are 3-inch wide shims of rotting untreated wood under the debris. They appear to be the shims/forms that were used when they poured the floor, but I though that those normally came out. Should I leave this rotting mess or dig it out? If I dig it out, do I need some sort of spacer put back in?