You should have a look at xfs.
I used to use ext4 earlier, traversing through /usr/portage
used to be very slow. When I switched xfs, speed increased
drastically.
This might be kind of unrelated, but makes sense.
I guess traversing through directories may be faster with
XFS, but in my experience ext4 perfoms better than XFS in
regard to operations (cp, rm) on small files.
I read that there are some tuning options for XFS and small
files, but never tried it.
But if somone seconds XFS I will try it too.