Evaluating two functions according to their performance
Time Complexity

Function A cost \$\mathcal{O}\left(n\right)\$

toLowerCase, split cost \$\mathcal{O}\left(n\right)\$
forEach cost \$\mathcal{O}\left(n\right)\$
Object.entries, filter cost \$\mathcal{O}\left(n\right)\$

counts[x] read / write may cost \$\mathcal{O}\left(1\right)\$ on average




Function B cost cost \$\mathcal{O}\left(n^2\right)\$

toLowerCase, split still cost \$\mathcal{O}\left(n\right)\$
filter cost \$\mathcal{O}\left(n^2\right)\$

indexOf, lastIndexOf cost \$\mathcal{O}\left(n\right)\$
filter repeat above operation \$\mathcal{O}\left(n\right)\$ times, thus \$\mathcal{O}\left(n\cdot n\right)\$





Time consume most occurred in loops. Not only forEach, fliter, but also toLowerCase, split, indexOf are loops. You didn't count the performance of nested loops correctly.
Also, notice that big-O notation is targeted to describe how fast the time consume increase when size of input increase. It not necessarily means an algorithm with better big-O notation will always be faster.
Implementation

Do not use str.split(''), use [...str] if you can. The split one won't handle some unicode points correctly. For example, [...'面'].length is 3 but '面'.split('').length is 5.
Use Map if you can (no need to support old browsers). Object is not an ideal way to use as a dictionary. Only if you want to keep ES5 support, you may use Object.create(null) instead.
You may count how many entries has at least count 2 during iteration by tracking if count[x] == 2 after increase. So there is no need to iterate Object.entries later.