People always presume "the aliens" will have superior technology. It could just as easily be the opposite.
Maybe we'll be the life form that cracks time, enabling us to travel hundreds of light years in a day. If neither does, then neither will meet.

I don't find the idea of not being alone in the universe terrifying at all. There's a near absolute likelihood that I'll never encounter these alien life or ever gain any knowledge of its existence. If the even more unlikelihood of this alien life being advanced enough to achieve communication with this planet in my lifetime or even visit Earth I doubt they would travel billions and billions of miles just to conquer or kill us. That would be like me buying a plane ticket to fly to Australia just to punch a random stranger in the face, then turn around and fly home again. Also, there nothing on Earth aside from the biological life here that is unique or even difficult to find elsewhere in the universe. Water and minerals are abundant elsewhere and aliens would have no reason to try and get them from us.

Unless the aliens are all assholes. Then we might be in trouble.