What OP code can I use in the scripting system to make a smart contract that acts similar to a lottery
You might be able to combine a hash lock with a conditional payout.

Here's a somewhat simplified example that shows how to make a payout depend on knowledge of a secret. Bob makes a bet with Alice that she can't guess Bob's secret before a deadline. Alice proves she knows the secret by publishing the preimage to its hash value, which Bob publishes. Alice and Bob each contribute one bitcoin. The winner receives two bitcoin. If Bob doesn't reveal the secret after the deadline, Alice takes all the money.

There are two one-bitcoin inputs, signed respectively by Alice and Bob.

There is one two-bitcoin output. It is protected by a nested conditional challenge script that looks something like this:

IF
  // Alice spends with secret before deadline
  OP_HASH256 &lt;hash&gt; OP_EQUALVERIFY &lt;Alice pubkey&gt; OP_CHECKSIG
ELSE
  // deadline has expired
  &lt;deadline&gt; OP_CHECKLOCKTIMEVERIFY OP_DROP

  IF
    // Bob spends by revealing the secret
    OP_HASH256 &lt;hash&gt; OP_EQUALVERIFY &lt;Bob pubkey&gt; OP_CHECKSIG
  ELSE
    // Alice spends if Bob refuses to publish the secret
    &lt;Alice pubkey&gt; OP_CHECKSIG
  ENDIF
ENDIF


To spend the output before the deadline, Alice uses this response script:

&lt;Alice signature&gt; &lt;secret&gt; &lt;1&gt;


After the deadline, Bob uses this response script:

&lt;0&gt; &lt;Bob signature&gt; &lt;secret&gt; &lt;1&gt; &lt;0&gt;


If Bob fails to spend the output after the deadline (revealing the secret), Alice takes all the money with this response script:

&lt;0&gt; &lt;Alice signature&gt; &lt;0&gt; &lt;0&gt;


There's a problem with this approach, though. If Alice doesn't guess the secret and Bob publishes it, Alice can use the secret to make her own transaction claiming the output. This sets up a race condition in which Alice and Bob each have pending transactions trying to spend the output.

One way to fix this problem would be to give Bob a grace period immediately after the deadline (say, one day). During the grace period, Alice would be unable to spend, but Bob could.