“… staring down/All his young days into the harbor where/His ball went…”

Do you think the boy has had the ball for a long time? Is it linked to the memories of days when he played with it?

Yes, it seems like the boy has had the ball for a long time. When it bounced into the water, all his memories of the days of childhood flashed in front of him. This led to a realisation that those moments would not come back, just like the ball. He can buy new balls and can similarly create new moments, but those that are gone would not return.