Now days there is exponential growth in the volume of online users causes variety and volumes of data. So the rapid development of internet web technology the big data and big user adopting NoSQL. It is a large scale distributed computing and cloud storage services have brought great challenges to the traditional relational database. NoSQL databases are very scalable, accomplishing higher availability easily handle unstructured data. As we know that the security is the main concern which is faced by many big organizations. In document oriented NoSQL database, MongoDB has many security issues and vulnerabilities such as unencrypted data files, weak authorization, injection attack, weak authentication between client and servers which can be reduced by efficient and secure cryptographic techniques. In this research paper we discussed about two public key cryptographic techniques named as ECC and RSA. The research paper proposed a hybrid security architecture in application server of MongoDB. We analyze their encryption/decryption time of document data in MongoDB. We provide better security model in order to provide high level of encryption mechanism, good access control, high authentication system and better throughput of the system we also compare their key sizes. For better security mechanism new security protocol architecture has been designed which fulfills the basic parameter of security areas. This architecture uses public and private key cryptography and provides security primitives such as Confidentiality, Integrity, and Authentication. These security primitives achieved by the help of elliptical curve cryptography, RSA/DSA algorithm, and message digest MD5.