Acoustical Design considerations for Active Noise Cancellation Project for Air vent
I first wasn't sure that this was in the scope of a digital signal processing site, but then decided that it is about limits and environments for doing DSP, so it's quite on-topic.

Your paper (what paper? Please, please, get used to always correctly cite!) hit the spot:


  If the acoustical design of the system is not optimized,the digital controller may not be able to attenuate the undesired noise adequately.


I think you understand quite well what they're saying: You've got to deal with signals in the real world, i.e. with actual, physical, pressure waves. You're doing that by modeling the real world mathematically.

This can of course only be a success if your reality is "easy" enough to fit your model. For example, your ANC probably assumes that there's a position for the noise emitter, and position(s) for the cancellation signal emitter(s). What it probably doesn't involve is that with a bad acoustic design, noise might be leaking out of the back of your emitter, bounce off a distant wall, and come back with a huge, random phase shift.

This is science. In science, you try to reduce the amount of uncontrollable effects as far as possible to make as many things certain as possible.

Your design is a good one, in principle: Cardboard is pretty effective at absorbing sound. However, you weren't really too careful when constructing your system:


gaps. Close them.
Walls. Make them thicker.
backs of speakers, open for absorption and emission of ambient noise. Pad them.
Not too sure about the bottom of your "tube": Is it dampening cardboard or actually the table, which probably is much better at transporting power of a speaker lying flat atop than air is?
Overall system exposed to ambient noise. I'd get a big cardboard, put down a layer of insulation (e.g. styrofoam), put everything inside, fill things up with insulating material.
Is omni-directional really the pattern you're looking for in microphones? This all sounds like you want your noise source mic to be highly directional to only capture the noise, and not the effect your own emissions have at the noise source.


Then, make sure all the components, especially your sensors, work very well:


are your microphones (especially that right in front of the noise emitter) perhaps clipping?
if not 1., then, how clean is the signal your speaker produces? Maybe you want to try and produce a few tones at different volumes, and look at the spectra (coming from an RF background, I'd propose full correlation-based channel sounding, but I don't know how well that works).
Verify your model: unplug your cancellation speakers. Run your device, and record what they would have produced as sound to a file, as well as what your microphones heard. Simulate what the mics at the places where you want your noise to be cancelled should have heard based on these files. Do exactly the same with the  speakers plugged in. Compare simulation with actual microphone recording.