Samuel Fox at Derby and via Stanhope's residence. By 1791, Joyce had joined the Unitarian Society (fuller name Unitarian Society for Promoting Christian Knowledge and the Practice of Virtue, to which was added "by the distribution of books"). In the following years he was involved in publications, and the Society had as associate Joseph Johnson. He joined the Essex Street Chapel congregation, and formed a long-standing relationship with Theophilus Lindsey, its founder. When Lindsey retired from the chapel in 1793, Joyce for a period was minister there with John Disney, who had shared the position with Lindsey from 1783. Joyce took the afternoon services, continuing until 1804 when Thomas Belsham took over the chapel. Ahead of the penal transportation of the Scottish Martyrs to Liberty and Maurice Margarot of the LCS, Joyce on 28 March 1794 proposed an address of support from the SCI. It contained the sentiment "A full and fair Representation of the People of Great Britain we seek, with all the ardour of men and Britons". From 4 April the SCI and LCS worked together, to nominate delegates to a convention of the "Friends of Liberty". Joyce was chosen, with Holcroft, Kyd, William Sharp and Thomas Wardle. Treason charge and acquittal A dozen activists were arrested in May, followed by the passing of the Habeas Corpus Suspension Act 1794, and Joyce was one of those picked up. On 12 May Thomas Hardy of the LCS was taken into custody at home by Bow Street Runners and King's Messengers. Shortly after that an ambiguous letter from Joyce to John Horne Tooke of the SCI, written within hours of Hardy's arrest. was intercepted by the authorities. John Thelwall was arrested on 13 May, at Beaufort Buildings, Strand, London where he was lecturing. Horne Tooke himself was arrested on 16 May. Others detained in the sweep included the silversmith John Baxter, successor to Margarot at the LCS; the businessman John Richter, LCS and an SCI associate; and Bonney and Kyd of the LCI. On 14 May 1794 Joyce was at Stanhope's house Chevening in Kent, and was arrested while talking to Stanhope's sons, by John King and a King's Messenger. The charge was "treasonable practices". The arrest occurred at around 8 am. Joyce was being examined by the Privy Council by about 1 pm that day. He refused to answer any questions without a lawyer, which he was not allowed.