bash – Making sense of the regex in grep command

A repeated capturing group, such as (.)*, puts in the
memory what was captured last. That’s where the confusion is
coming from.

As your group is a wildcard ((.)), it will match anything
and whatever was matched last will remain in \1. Simpler
example (again in the more common (enhanced) regex style):

/(.)*/
aabbcc     # matches everything, but \1 contains `c`

This might be strange at first glance, but makes sense: it
repeats a wildcard of a single character. So, matches
everything. Finally, whatever was last captures is what
remains in capture group memory.

To capture a repeated occurrence, one needs to repeat
explicitly:

/(.)\1*/
aabbcc     # matches `aa` and \1 contains `a`

In the example above, (.) matches the first character and
then it is repeated.

Back to your original expression (here expressed in grep’s
basic notation):

^\(.\)*\1$

Beginning tied to the start (^), it matches anything
(\(.\)), as many times as possible (*), and then asks
for a repeat of the last matched character at the end ($).
In plain English:

Match any line that ends with a repeated character.

Achieving what you originally wanted (lines starting and
ending with the same character) is possible with the regex
you already mentioned in your question:

grep '^\(.\).*\1$' filename