RIT Capital announced it will not make a formal bid, after confirming reports last week that it was in the early stages of informal talks about merger.
Alliance Trust invests nearly Â£3bn worth of shareholder assets.
It will continue to face a fundamental review of what it does during this summer.
That could still lead to a sale, or outsourcing investment to other asset management firms. And following the review, it is still possible that a deal could be done with RIT.
But the announcement on Tuesday takes Alliance Trust out of the stock market's formal takeover and mergers procedures and oversight by the Takeover Panel.
Activist investors last year forced a change of strategy, aimed at reducing the discount of company value to underlying assets, which involved replacement of board members and chief executive Katherine Garrett-Cox.
A statement from RIT Capital said: "Following careful analysis and constructive discussions with representatives of Alliance Trust, RIT has concluded that it would not be in the best interests of its shareholders to make an offer for Alliance Trust and accordingly announces that it does not intend to make an offer to acquire Alliance Trust.
"The board of RIT respects the process which Alliance Trust are going through and wishes them well with their strategic review."
In response, the Alliance Trust statement said: "The board of Alliance Trust is implementing a series of changes to enhance shareholder value. This process is well under way and has already started to make good progress, lowering costs, narrowing the discount and allowing for the creation of a fully independent board of directors.
"This process of development continues and includes a strategic review of the group, encompassing a broad range of potential courses of action."
Alliance Trust was buying shares on the stock exchange on a daily basis, as part of its strategy to increase shareholder value. Its announcement stated that practice is to be stopped until the review is complete.
Talks about a possible takeover of Alliance Trust in Dundee by the asset fund linked to the Rothschild banking dynasty have been called off.