This caused growers of U.S. sugar and tobacco to complain to Roosevelt, who remonstrated with his Secretary of War. Taft expressed unwillingness to change his position, and threatened to resign;[46] Roosevelt hastily dropped the matter.[47] Taft returned to the islands in 1905, leading a delegation of congressmen, and again in 1907, to open the first Philippine Assembly.[48]
On both of his Philippine trips as Secretary of War, Taft went to Japan, and met with officials there.[49] The meeting in July 1905 came a month before the Portsmouth Peace Conference, which would end the Russo-Japanese War with the Treaty of Portsmouth. Taft met with Japanese Prime Minister Katsura Tarō. After that meeting, the two signed a memorandum. It contained nothing new but instead reaffirmed official positions: Japan had no intention to invade the Philippines, and the U.S. that it did not object to Japanese control of Korea.[50] There were U.S. concerns about the number of Japanese laborers coming to the American West Coast, and during Taft's second visit, in September 1907, Tadasu Hayashi, the foreign minister, informally agreed to issue fewer passports to them.