How to display a block only for an article with a specific node id? I have quickly seen view reference and context module. Though they can display a view or block to a specific content type, I'm not sure if they can do it within a specific node.

As an example: I want to display a block only for an article with node id 3.

Can that be done?
I would steer clear of using PHP for block visibility.  If you type something wrong you can break any number of things.

Just go to your block settings, and take a look at the vertical tabs at the bottom of the page.  The first option is Pages.  Just change the radio button to "Only the listed pages" and enter node/3 in the box.

EDIT

If you use the Views Reference module mentioned in your question, then you can quite easily embed a Views block as one of your fields just as any other field.  The field itself can't be exclusive to a node ID, only an entity type.  However, you could just leave it blank for the other entities of the same type and use Field Permissions if you are worried about other people using the field incorrectly/at all.

One unrelated helpful hint with Views Reference - I generally tag my "reference-able" views with the same tag (i.e. Embeddable, Referenceable e.t.c.).  then you can easily select which Views to make available to the Views Reference field.

You MIGHT be able to embed blocks with Entity Reference combined with BEAN or Boxes, but I'm not 100% sure.  Failing this you are probably looking at breaking up the entire $content variable in the template and embedding it yourself (which would mean much more maintenance if fields get added or removed), using Views and turning the whole content types into blocks that can be positioned (MUCH effort).