In this paper, a spectrally coded optical code division multiple access (OCDMA) system using a hybrid modulation scheme has been investigated. The idea is to propose an effective approach for simultaneous improvement of the system capacity and security. Data formats, NRZ (non-return to zero), DQPSK (differential quadrature phase shift keying), and PoISk (polarisation shift keying) are used to get the orthogonal modulated signal. It is observed that the proposed hybrid modulation provides efficient utilisation of bandwidth, increases the data capacity and enhances the data confidentiality over existing OCDMA systems. Further, the proposed system performance is compared with the current state-of-the-art OCDMA schemes. (C) 2015 Elsevier Inc. All rights reserved.