Reconstructing Tree Limbs that are Occluded by Leaves How would would one approximately reconstruct the branches in a photo of a tree, where portions of the branches are covered by leaves?

I can extract the visible portions of the branches by threshholding then opening with line segments at a few different angles.  But how would I then (approximately) fill in the spaces that were covered by leaves?

Suppose we start with this drawing of a tree:



load the image as grayscale

uu = mmreadgray('tree2.jpg')



threshold the darker colors; convert to binary; remove connected regions under 200px area

vv = mmareaopen(mmbinary(mmthreshad(mmneg(uu), 200)), 20)



open vv with 10px line segments at 90 degrees, 180 degrees, 135 degrees and 45 degrees; add these images together to form a marker image zz

zz = mmaddm(mmaddm(mmaddm(mmopen(vv, mmseline(10, 90)), mmopen(vv, mmseline(10, 180))), mmopen(vv, mmseline(10, 135))), mmopen(vv, mmseline(10, 45)))



reconstruct the visible limb portions in vv from marker image zz

aa = mminfrec(zz, vv)



display result aa on top of original image uu; a reasonable extraction of limbs is achieved

mmshow(uu, aa)


I would appreciate any thoughts on how to now approximately reconstruct the entire limb.  (Or improvements to the steps given above).  Thank you!
The masked area hidden by leaves is very large and while the border and shading seems to imply what is hidden to a human being it is unsuited to "inpainting" which relies on pretty consistent boundary, such as where a foreground object temporarily obscures something continuing behind.

The branch orientation is a start but not enough data on its own to imply how the twigs extend without having some external concept of an oak tree versus a willow, poplar or pine.  That is more about 3d fractal modeling than about image restoration.

The least obviously visible algorithms for in-painting involve texture synthesis called "Healing" in Photoshop.  G'MIC and Resynth are popular open-source tools for that available as plug-ins for GIMP.  While visual continuity is quite good they have no concept of a tree or depth, just 2-d patch statistics so would need input from an artist as well in the example given.  The mask is white background on the outer edges and thick branches on the inside .. so texture interpolation gives a weird unnatural result.