Spending transaction with custom output I've recently entered the bitcoin arena, and bitcoinj was my tool choice. Experimenting on regression test mode, I was able to create the following custom transaction. It has a modified locking script and would need a modified unlocking, input, script. 

 //Sender: sig pubk [dup() hash() pubk equ() chksig() OP_VERIFY() match_me eqal()] 
 //Receiver: matcher sig pubk [dup() hash() pubk equ() chksig()]

 Script locking = new ScriptBuilder()
      .op(OP_DUP)
      .op(OP_HASH160)
      .data(address3.getHash160())
      .op(OP_EQUALVERIFY).op(OP_CHECKSIG)
      .op(OP_VERIFY)/*gets rid of the true left by previous checksig*/
      .data(match_me)/*32bytes to be matched in unlocking script*/
      .op(OP_EQUAL)/*checks the equality of match_me with one provided by receiver*/
      .build();

 Transaction contract = new Transaction(params);  

 contract.addOutput(COIN.multiply(25), locking);                           
 SendRequest req = SendRequest.forTx(contract); 
 wallet2.wallet().completeTx(req);
 wallet2.peerGroup().broadcastTransaction(req.tx);


Here, I'm trying to create this transaction that won't be spent until the receiver provides data that matches 'match_me'. The transaction half successfully spends the coins from the sender's wallet, subtracting 25 BTC from wallet2. However, wallet3 was still empty even after generating a new block. 

 Transaction:
   bd52d8ef024715096b5177d581121122dc1ce9f8925c105a496bb290691ba1a7
   updated: 2016-10-02T23:19:14Z
      in   PUSHDATA(72)     [3045022100f5664de61bd0c9576d7dd2aaf98f2ba4e60c1013e55a36e62bf1d47c38a0fb5c02206872b35821f05069df5d2302dadc55011e6e6d7569e2e46c655deb9265390f3f01] PUSHDATA(33)[035562c04ff3ddb5a8ba566f4cd84c3e47478997ab8730bd923a4260fc9dfe246e] 50.00 BTC
      outpoint:7738e333b27700e835d3c118444d9f2181033c8d796ca116c66d795140ad64df:1 hash160:67d4ea4c50e80497380b7eb2f901664dc341db4b
 out  DUP HASH160 PUSHDATA(20)[d713cf366138ad0d0aac8c148ca724c18a8c4e4d] EQUALVERIFY CHECKSIG VERIFY PUSHDATA(16)[e04fd020ea3a6910a2d808002b30309d] EQUAL 25.00 BTC
 out  DUP HASH160 PUSHDATA(20)[2515ad4a14c15cc363796684149fd23ae1c3146e] EQUALVERIFY CHECKSIG 24.999877 BTC
 fee  0.00050204 BTC/kB, 0.000123 BTC for 245 bytes
 prps USER_PAYMENT


I speculate that it could be the case that wallet3 doesn't recognize the custom transaction, so it doesn't include it in its balance. Do you think so?
If I decided to spend the transaction using the txid, how would I create the inputs to unlock it? how do I create scriptsig with (.data(matcher).data(sig).data(pubkey))?

Thank you
Typically wallets do not support spending coins with any scriptPubKey except the ones they hand out themselves (in the sense that an address is a shorthand for a particulatlr scriptPubKey).

This is for good reasons. It's impossible for wallets to scan for all possible combinations and alterations of scripts with keys they know, so it would inherently be a subset anyway. It being a subset would just create unclear expectations, as senders may wonder what exactly is permitted. Furthermore, it is not the sender's business to decide what script the receiver should accept money at.

Compare this to a real life scenario where you're asked to pay someone by slipping an envelope under their door. Instead, you take the envelope and put it under their doormat. You wouldn't expect this to be a fulfillment of the payment either.