Much like the idiom “get to the point”, this expression uses “point” to mean “the basic idea of a conversation, what you’re trying to show or persuade”.

So, when our listener doesn’t understand, or instead focuses on something the speaker considers less important, we say that the listener missed the point, and then we try to explain it to them in a different way.

In debate class, I argued that taxes needed to be lowered, but my opponent completely missed my point and argued instead about spending.

Of course, it’s also the source of many great jokes:

“Waiter, what’s the fly doing in my soup?”

“The backstroke.”

In this joke we can see that the waiter completely missed the point and answered a totally different question.