The imported object does not become the active object in the scene
Active object is the one on which the actions in 3d view are performed. For instance, if you press tab the mode of only the active object changes. There can, however, be multiple selected objects. Left click an object in outliner to make it active, deselect everything and you will still be able to tab the object into edit mode (as it is active but not selected). 

Excuse me if this all is known piece of info but since you said "I tried bpy.ops.object.select_all(action='DESELECT') before importing the new object but it didn't work and the active object is still another object in the scene.", I guess you are thinking that by deselecting an object you would also negate its active state.

So the answer to your question is: 
The imported object is selected but not active by default. To make it active, 


deselect all the objects before you do the ops.import_scene.obj and do bpy.ops.object.select_all(action='DESELECT')
make the selected object active explicitly. Since the imported object will be the only one selected, you can simply do bpy.context.scene.objects.active = [ob for ob in bpy.data.objects if ob.select == True][0]
or better as suggested in the comments do:

bpy.context.scene.objects.active = bpy.context.selected_objects[0]