I think what you can do is push your darker shadows more, specifically the shadow beneath the hat around the eyes, have that the darkest. Having that and, some other shadows, darker allows you more variance and contrast overall. That way so many shadows aren’t the same value range and, as you said, could distract, and you can now go darker with others and add shadows with the hand.