text: We have numerous projects, some of which are very large. Some of these involve not only large jars, but also large
  loose codebase directories in their classpaths.
  With this reality (this is not going to change), NetBeans has in one respect continuously gotten worse despite its
  numerous improvements: the IDE is ever more insistent on scanning all of these directories before letting you do real
  work -- even in 6.1 where this is purported not to be the case. Also in cases where one rebuilding the output jar in a
  project upon which many, many other open projects depend the IDE will decide that their sources needs to be rescanned in
  their entirety in cases. This is *not* a recipe for large code set scalability.
  Perhaps we're alone in this boat -- or perhaps others in it have already jumped ship to Eclipse with its incremental
  compilation logic, but I wanted to raise the issue for community discussion. I did so on nbuser and did not seem to get
  any response, so I'm recording this as an issue. This really should be noted as a defect in many respects due to its
  severity, but I'm calling it an enhancement to avoid any arguments over issue type.
  In my book I think the IDE needs to provide more control over when code scanning is done at a bare minimum -- especially
  for those unfortunate enough to have loose codebase directories. It should also likely provide some control over the
  priority the background code scanning thread is given -- as this thread has been aggressive enough to render the
  remainder of the IDE barely usable. There's also got to be a better way than rescanning all downstream projects in
  their entirety when you rebuild a project upon which many others depend. Finally, I'd like to see the IDE live up to
  the promises in the 6.1 release notes and allow me to do "Go to Type", etc, with existing code completion information
  without waiting for all scanning to complete. When I start the IDE I know I have full (and up-to-date) code completion,
  etc, data in many cases yet the IDE refuses to "Go To Type", etc, until it has rescanned the world.
  On top of all this, I really think "Go to Type", etc, should still be faster, though I certainly recognize progress in
  this regard. That said, as a point of comparison I've used NetBeans' classfile module to analyze all our class files,
  store cross-references in a relational database (with appropriate indexing, of course), and then fronted this with a web
  app. The results of "Go to Type" type lookups (and many others) are *very* fast in all cases with this web app (even
  when running everything involved on the same laptop that I'm running NetBeans on). I guess I should integrate that with
  the IDE, but I'd think the IDE should do this sort of thing better than my somewhat ad hoc web app! [Also I limit the
  fidelity of my Find Usages to which class is using another and don't record/divulge which lines the usages occur upon.]
  Too many things here, if I forgot to answer something please lat me know.
  1) Probably one of the problem is that you use the option which prefers the output.jar to sources, which causes all the project's depending on the output.jar
  being rescanned when you rebuild the output.jar.
  2) >I think the IDE needs to provide more control over when code scanning - can you provide more details? Thread priority is clear, but not sure if it will
  help, most OSes increase dynamic priority to threads doing syscalls (IO) which is case of this thread. But it can be tried.
  3)> There's also got to be a better way than rescanning all downstream projects in their entirety when you rebuild a project upon which many others
  depend. This is caused by preferring output.jar.
  4)>data in many cases yet the IDE refuses to "Go To Type", etc, until it has rescanned the world - sounds as a bug. Adding pflaska to cc, who is working on
  this feature.
  5)> The Goto type in NB 6.1 needs to wait until the source root is indexed, there is quite big delay, which should be fixed in 6.5, again pflaska should
  know more.
  6) The classfiles indexing is very fast. The rt.jar (~18000 classes) is scanned and stored in less than 10s, the problem is class files doesn't contain all
  information, for project you need to prefer sources which is much more time expensive, but I believe it can be faster than now.
  The most important, at least from my point of view, is to prevent non needed indexing.
  I believe pflaska will work on this in NB 6.5.
  Thanks for all the information.
  I don't know that we've used any "prefer jars to sources" option per se. We have used -J-DCacheClassPath.keepJars=true,
  which we absolutely must for now as per issue #134992 and issue #121853. See these issues as to why we're using this.
  We don't want to "prefer jars to source" overall -- we just want a means of dealing with classes resulting from non-Java
  sources (without having to split them into separate projects, which is a non-starter since they're interspersed in the
  same source trees as the Java sources).
  Overall indexing/scanning appears to be pretty fast except when:
  1) Indexing a large loose codebase directory containing classes and other files.
  * There really ought to be a way to set a maximum reindexing frequency
  for such directories (e.g. once a day) and then to force reindexing on demand.
  2) NetBeans decides a large source directory needs to be recompiled/scanned.
  * This can grind NetBeans to a halt and make even other apps hard to use during
  this interval.
  On what is really another front entirely "Go To Type", etc, are still *far* slower than they should be. I don't see any
  reason that the code index/scan can't produce data that's just as efficient to search as the relational database case I
  noted above. This should be blindingly fast.
  Note that the need for -J-DCacheClassPath.keepJars=true is now gone in NetBeans 6.5 (at least for us) due to issue
  #134992 being resolved.
  Not using this option has resolved the issue where building one project causes all downstream projects to do a code
  completion/navigation database rescan/recompile.
  The other issues still remain, however. I often try to use code completion or navigation only to get "scanning in
  progress" and no action. Sometimes when this occurs there is no visible indication that such activity is in progress in
  the status bar. Moreover, scanning should never prevent code completion or navigation. As each source tree is being
  updated its previous code completion/navigation data should be used with a warning that the data may be stale.
  [Actually code completion/navigation should be prevented when there is *no* code completion/navigation data whatsoever,
  but that's rarely the case.]
  I second everything Jesse mentioned with respect to code completion and go-to-type becoming useless too often while
  scanning is taking place. In addition, I want to add that "Find Usages" is also VERY slow (I'm running 08/17 daily
  build of 6.5). On our project (3825 Java source files), I just did a "Find Usages" on a static method and it took 33
  seconds to find 26 usages. Granted, my machine is no speed deamon (single core Dell Latitude D810 laptop with 2GB
  memory), but it's not exactly a low-end box either. I'm running Ubuntu 8.10 w. JDK 1.6b10 (build 25).
  The various forms of "Find Usages" are also nearly instantaneous (unless the result set gets really large) with my
  little relational database plus web app.
  I cannot confirm that "Find usages" is fast. Sometimes it takes up to 30 seconds until the progress window is
  initialized and the progress bar "starts moving". This happens on a single core as well as on a dual core machine as
  well. Most of the time, cancelling the dialog and re-issuing a "Find usage" is then faster.
  In my previous comment I meant that the "find usages" query types are nearly instantaneous against our (huge) data set
  when the searching is done via the static analysis web app backed by a relational database that I mentioned in my
  original description.
  "Find usages" is far from fast. Part of this, getting from which classes used something to which lines in those classes
  used something, is somewhat understandable to me -- as my web app only tracks usages to the class level and then uses
  heuristics to highlight the source lines that appear to be involved. That said, getting to the list of classes that
  use/subclass/whatever something should be far, far faster than it is.
  Still I can live with "Find Usages" being a bit slower as I use it less. Code completion and navigation must be always
  ready (no "scanning in progress" business) and always nearly instantaneous.
  > I often try to use code completion or navigation only to get "scanning in progress" and no action.
  So "Go to Type" or code completion *triggers* the scanning by itself?
  I wonder if this is ever supposed to happen. Bug?
  I don't know that "Go To Type" or code completion triggers the scanning.
  Rather I get a "Scanning in Progress" tooltip instead of a code completion pop-up for instance. All the while there is
  no scanning or compiling note or progress bar in the status bar -- so the scanning could have been going on for some
  time without the IDE telling me or it is possible that no scanning is actually occurring and the IDE is confused.
  Either way it is not very helpful :-)
  I can confirm that "Go To Type", "Go To File" and the auto-completion features in NetBeans usually are downright
  unusable in large projects. I've seen reponse times up to a minute or two, with no visual feedback. Find and grep in a
  unix shell is unfortunately much faster.
  Eclipse and IntelliJ has near-instantaneous response on similar operations on the same project base, I don't see why
  NetBeans have to be that slow.
  I've tested this on NetBeans 6.5 and 7.0-dev.
  I am being so bold as to change this to a defect. I don't know how I was ever talked into letting this be an
  "enhancement". Sure I argued for some things that were an enhancement, but overall this is about having usable behavior
  for code scanning and scan data usage, e.g. Go To Type.
  As per a recent nbuser post, I can sum up the inter-related issues as:
  1. Scanning is fairly slow, whether this can be improved is unclear.
  2. Scanning occurs when you do not want it.
  * You should be able to tell it "honestly your data for X is good enough for now, don't scan X now".
  * This is especially horrific if you have a large directory containing loose class files instead of a jar
  somewhere (an unfortunate situation, but truly necessary in some cases), but also applies to large jars.
  * Addressing this is much less critical if (4) is fully addressed.
  3. Scanning sometimes occurs behind your back without clear indication in the GUI
  * Sometimes you'll try to do an operation and you'll get "scanning in progress, please wait", yet there's no
  indication of scanning in progress in the status bar, much less of what's being scanned.
  4. While scanning is occurring the IDE is unnecessarily crippled
  * One should still be able to do any/all "Go To Type", etc, navigations, debugger, etc, without any ill-effects
  from scanning except: lack of new data being produced by the scan and a warning of this lack. If jar X is being scanned
  but you already had scan data for jar X, then the previous scan data for that jar should be used while jar X is being
  scanned -- with a warning, of course. There should be no "scanning in progress, please wait" or "Go To Type" dialogs
  missing results that were present in both the old and new jar.
  * Scan data for directories in the classpath should be stored as well and treated in the same fashion just
  described for jars -- no difference.
  5. After scanning is actually done searches using the scanned data are often slower than expected.
  * I scan our class files with NetBeans' classfile module and pump loads of static analysis data into a
  relational database running on my laptop. The equivalent of Go To Type, Find Usages, etc, searches are generally
  noticeably faster with my database searches than they are for NetBeans.
  * Overall, the speed of Go To Type and other navigation using the scan database is critical. If this is near
  instantaneous, NetBeans seems miraculous to users and allows them greater productivity on large bodies of code. If this
  is not, then NetBeans is a bloated whale. It's that simple for users.
  This really should be a P1 bug -- it is killing NB usage, and is NB complaint #1 through 5 of those I work with, both
  those who use NB and those who refuse to. I'm setting it to P1 knowing it will get bumped back down, but maybe someone
  will notice it in the interim.
  Not a p1, I think.
  I will try to investigate. Thanks for your help.
  Thanks for giving this some attention. No, it really is probably not a P1, just a high P2 that someone talked me into
  letting be labeled an enhancement.
  Note that this isn't just a matter of slow "find usages", "go to type", and code completion ("go to type", Ctrl-B, and
  code completion being the critical use cases here for speed as they're things you do mid-thought-stream, whereas "find
  usages" is an activity to itself). The slow scanning combined with an inability to continue reliably using the old scan
  data during a scan that is updating said data are just as big of issues.
  There's also the matter of fragile scan data that some developers are seeing around here and on the list. This should
  be a separate bug but I don't have enough information to file a meaningful one as I've not reproduced this personally.
  I have watched it be reproduced in front of me, though. Essentially this involves having a set of projects that build
  fine and scan fine, showing no inappropriate error markers. The developer proceeds to compile certain Java sources (not
  just any Java source will cause the issue) and then the NB editor starts marking all sorts of things with "cannot find
  symbol" sorts of errors. This seems to occur more in 6.5 and has caused a good number of developers to revert to 6.1.
  Overall there are some big problems in this area that are causing NB users to switch away or strongly consider doing so
  -- and preventing Eclipse users from considering NB. Scan reliability and speed, both on scan update and scan data
  usage is right up there with basic editor speed as far as criticality to IDE usability. This area needs attention!
  You wrote that you are working on some very large projects. Is it possible to download some of them, to test performance?
  Can you attach information how big they are (how many source roots, jars, source files, megabytes...).
  What type of NetBeans project you are using?
  Do you have some numbers, like start NetBeans with foo project opened (+info about size of project) + fresh userdir =>
  scanning takes 5 minutes?
  Opening the same project with Eclipse => 1 minute...
  We will try to do some comparison anyway, but I will appreciate any data.
  I open the source for the Liferay portal every now and then, but I make sure it's closed before I close NetBeans.
  Otherwise the next startup wil increase my coffee consumption too much :)
  Is there some sort of module you can give me that will just collect size metric information you're interested in from my
  open projects?
  We're using free-form web projects with a complex Ant framework being invoked for the builds. The source is all quite
  proprietary and thus I'd have to go through various legalities to hand out a copy.
  I can try to generate some numbers for NB -- again a NB module (or log level configuration) that produces such metrics
  would be nice to ensure consistency and allow many developers to easily give you numbers. I don't actually use Eclipse
  myself at all, so I can't give Eclipse numbers. I don't know that Eclipse is faster except insofar as NB users move to
  it for this reason and Eclipse users who might otherwise dislike Eclipse won't consider NB for this reason (amongst others).
  Overall I'm not dead set of scanning being dramatically faster. If NB is still perfectly usable while it is scanning
  and navigation / code completion using the scan data is lightning fast, that would be a big step forward. The fact that
  scanning can sometimes hog too much of the CPU for the IDE to be usable while scanning is a big problem (and a
  separately filed issue). The fact you can't do certain operations at all while scanning but instead get a "please wait"
  is unacceptable [unless scanning is all but instantaneous irrespective of project size/number]. The fact that "go to
  type", etc, forget the existence of types that it new about from previous scans while rescanning a jar is also
  unacceptable [even *if* scanning is all but instantaneous]. Finally, the long delay in getting results from "Go to
  Type" or Ctrl-B in many cases [sometimes it is quite fast] is unacceptable. If all of those items are fixed, then the
  speed of scanning itself is much, much less critical.
  From memory I remember that if you have 4-5 projects open in Netbeans and you have over a total of 100 classes then
  Netbeans will begin crawling. Please note I have no retested this recently but it used to be the case a few months ago.
  I was surprised by how quickly things deteriorated. 100 classes is not large by any stretch of the imagination. I don't
  want to point you guys in too many different directions. I think the best first step would be to measure how our
  performance degrades as the number of classes increases. I suspect (but we need to prove this) that performance degrades
  exponentially.
  Creating some module that can collect performance data is great idea! Thanks. I will ask here if we have something like
  this.
  I see the need for at least 2 different areas of metrics:
  1) Something to report on the number and size of projects loaded
  * For each project list how many source files, how many K of source files,
  how many jars referenced, how many MB of jars referenced, etc.
  * List totals for all open projects
  * Developers would like the ability to generate such a report anyway; this would be a nice feature overall
  2) Something to report on time consumed for various IDE tasks
  * This may just be a set of timing loggers that you can tell users to enable when they complain. Thus this
  may not be so much one new module as enhancements to the logging in many modules.
  * There may be some general points where one could insert timing from which lots of things might benefit
  quickly, e.g. the NetBeans background task processor could time each task it processes.
  Apart from these items, however, I have been pushing for the NetBeans CPU profiler (and VisualVM) to offer a simple
  sampling mode which one could use to determine which areas of a large application to re-test with NetBeans CPU profiling
  instrumentation. Apart from the general utility of such a feature, it would be great if NB offered a simple "profile me
  [start/stop]" action which would capture a sampling profile during some operation. Users could then be directed to do
  this for any operation that isn't performing as well as they'd like (or that seems to hang as long as it doesn't hang
  the whole UI) and attach the result to an issue. This would seem to streamline the process of addressing performance
  issues in NB overall.
  Please, update TM.
  One of our projects has over 7900 source files and has 56 libraries in the classpath.
  Many of these source files are quite large often with numerous inner classes. Many of the classes are quite
  interdependent. Many of the libraries are quite large. A good number of the libraries are outputs of other projects.
  There are then dozens of projects depending on this project (and other projects besides). These projects are also often
  fairly large, but not as large, say 1100 source files in cases. Projects generally have between 1 and 3 source roots.
  [All project source file counts were provided by SQE, so hopefully it is reliable in that regard.]
  The primary output jar from the 7900 source file project is around 27MB (compressed) by the way. That includes a lot of
  .ser files as well, but class files are definitely the bulk of the jar's content.
  As for the "Profile Me" feature, it's already in current dev builds (will be in M2). See issue 153221, and.
  As for the metrics, would be nice, definitely to have some plugin that would collect various useful data. It's not that
  clear, though, what the useful data should be. The size of the project, number of classes, etc, definitely matter, but
  is not the main problem. See gtzabari having problems with only 100 classes. OTOH we (NetBeans developers) work on quite
  large project, with tons of projects, zillions classes, many dependencies, but don't see significant problems - neither
  with scanning nor with code completion or Go to Type navigation. We also do some measurements on a reasonably large
  project (20MB of sources) where we measure startup of the IDE, time to open the project, time to complete the scan, time
  to use code completion, time to build the project, consumed memory, etc. In all these areas NetBeans is comparable or
  even better than Eclipse. So all in all, it's not generally true that NetBeans is slow on big projects. There are other
  (bigger) problems than size.
  The various scanning and related problems seem to be always tied to the particular project, its configuration, custom
  tasks running as part of build, etc. There's usually some problem causing the IDE to scan sources too often, or scan
  parts of classpath that would not need to be scanned. Whether it is misconfigured project, or too sensitive/fragile
  mechanism of detecting and updating changes on classpath, I don't know (I'm not expert on the java infrastructure).
  Anyway, this means that we are able to do very little with general statements like that scanning is slow etc. Ideally we
  need the project itself to test it and profile. So if you can, please send us your projects, with description what to
  try. There are also other ways: One is the new "Profile Me" feature, another one is to use a special startup switch to
  create scanning log:
  -J-Dorg.netbeans.modules.java.source.usages.RepositoryUpdater.level=FINEST
  The log will contain what scanning happened and why. Cane use FINE level for less details. Since the log can be pretty
  huge, it's necessary to find the case where you see scanning which should not happen, and run the IDE with the switch
  only for reproducing this one problem. From the log we might be able to find out what is wrong.
  BTW we also plan to create an experimental module which would turn off some of automatic classpath updates/scanning and
  give the user the option to invoke the refresh/rescan explicitly. See issue 158177. There are concerns this could lead
  to various errors and inconsistencies, but is probably worth trying if it can solve some problems.
  Glad to hear it about the "profile me" progress. I've been begging for almost exactly this for some time. Will this be
  made part of 6.7? Please!!! This is huge to both NB and VisualVM.
  As for scan speed, I can try the logging, but again I'm not really too concerned about scan speed by itself. I realize
  they're going to take *some* time in my case however hard you try. I'm concerned by loss of functionality/usability
  during these scans and speed of navigation features based on the scan data.
  Re "Profile Me" - it's already in, you can download a daily build and try it.
  Re Logging - the log should show why scanning happens when it is not expected. It can also show that bigger amount of
  files is scanned than needed. I.e. it may reveal bugs in the scanning infrastructure, or problems with project setup.
  As for IDE usable even during scanning (i.e. use the previous version of data until scan is finished) - that was
  discussed several times, but always found too difficult to implement. See also issue 133943. For 6.7 we hope for an
  improvement of Go to type navigation during scanning, at least (new indexing API should allow that).
  tpavek is right. We need a *specific* testcase (complete with the source-codes if possible) to work against. Otherwise
  we'll spin our wheels trying to fix a theoretical problem without being able to verify whether our work made a
  difference or not.
  Most of us work for commercial companies and aren't allowed to send source code.
  This might sound silly, but throughout the conversation, NB devs always indicate that they have no problem with a pretty
  big project: Netbeans! Can the NB developers put together a nice self-contained zip file with the entire NB project &
  src and let people who have trouble with their project try it out to see if they also have problems with the NB project.
  If so, we learned something.
  (I realize that the NB src is readily available. If it's already *very* easy and self-contained, then maybe just point
  to it) and we can try it. I just made the above suggestion on the off-chance that building NB takes more than simply
  downloading and opening/building in the IDE).
  Create a library project with perhaps 100 classes in a jar file. Change the build.xml for that library so that
  "-post-jar" copies the jar file into a "public directory" that is the location that the library manager has in its
  configuration for the library.
  Next, create 5-10 projects with approximately 500 classes in each, have each of those projects reference this library.
  Make sure all of these projects are open in the IDE along with the library.
  Finally, open some .java files in one or more of those 5-10 projects.
  Make a change in the library and rebuild it.
  As it's building start editing and trying to do work on those java files. Build them, try and run things and do normal
  development tasks. At some point, the IDE will start recompiling those 5-10 projects because the library timestamp changed.
  During that time, you will not be able to work.
  As for NB sources, can be downloaded as one zip file e.g. from here:
  To test it, you can open whatever project in the IDE - e.g. "form". Note that nobody opens all projects, but having
  just one project opened brings in all the projects it depends on (transitively), which is typically quite big number.
  You can let the IDE to open all the dependent projects explicitly. We usually don't build entire NetBeans in NetBeans,
  possibly just the module being worked on (plus dependencies).
  When opening the project, you'll see quite long initial scan (much shorter next time when IDE is restarted). Then you
  can try to jump to some class to test Go to Type speed (try e.g. FileObject or RepositoryUpdater), try to modify some
  file to see code completion speed, do some refactoring, etc.
  This way you can get some idea of speed at which the IDE can work on a big set of projects...
  I have a particular configuration which I illustrated the details thereof. It creates problems for the implementation
  of scanning and error badging related to classpath detected changes. Many others have lamented on the mailing list the
  same types of problems which they experience, and based on what I can tell, they have similar project numbers and
  layouts and are experiencing the same problem.
  I don't know how else to explain what I see... I know that the NB team says you all can develop the IDE just fine. I
  also know that there are specific optimizations in the IDE's behavior regarding netbeans platform development.
  I am not developing a netbeans platform app and I am not developing the netbeans IDE (a form of the platform). I and
  many others it seems, need help with fixing the problems that occur in our environments.
  > I and many others it seems, need help with fixing the problems that occur in our environments.
  That's exactly why we ask for the details. We need to know how your environments look like. Thanks Gregg for describing
  yours, we should take a look at it. Let me summarize your case, someone from NB java team can comment more on that.
  By copying a built JAR file to a "public directory" which is used by other projects you are changing the classpath of
  these projects. When this happens, the IDE can see the entire JAR file changing, but has no idea what actually has
  changed inside the JAR file. It does not compare it to previous state to determine some "incremental" changes. At this
  point it needs to rescan the projects that depend on the JAR file because there could have been potentially any change
  affecting them (some errors in source code may appear, some existing may be resolved). In reality, there is usually no
  significant change (affecting public API) and such huge scanning is not needed. It's even bigger problem if you need to
  rebuild your library JAR.
  This is a simple view on the problem. Someone from java team please correct me if I misrepresented something.
  Now the question is whether you can alter your project setup (or at least try - to see if it can really help), or if
  NetBeans can behave differently even in the case it has not the sources associated. I only know about issue 158177,
  which could help solving this, but it is only an experiment.
  With regard to "If you rebuild your libraries often, then you should have your dependency
  defined on the project level". When you build using non-NetBeans ant scripts (i.e. freeform project) and a non-NetBeans
  dependency management, for example using Ivy, then you want to avoid project-level dependencies in NetBeans in the
  general case, as the results of depending on local project sources won't be consistent with your build infrastructure.
  It would be acceptable while you work on two dependent projects simultaneously, with dependent changes. but not
  otherwise, because the reason for the build infrastructure is for the builds to not be dependent on the IDE or local
  settings. In this scenario, which for us is the default scenario, what would at least be needed then is the ability to
  quickly switch between project-level dependencies and classpath dependencies in NetBeans, without the world being
  rescanned.
  For some reason all of this doesn't seem to be a problem in Eclipse, and our impression is that this is a killer
  advantage of Eclipse over NetBeans in "enterprisy" environment, no matter how much better NetBeans maybe in other areas.
  "Depending on local project sources" should be only for the IDE, and possibly need not affect the building
  infrastructure. Only the IDE would resolve against the sources, building would work with JAR files. So in other words it
  should be possible to specify the sources for any JAR/library on classpath. I'm not sure right now if that is possible
  in such general way (for free form projects configuration; let's say it should be there). But I don't see how else the
  IDE could determine that it should ignore a changed JAR on classpath. If Eclipse does that without rescanning, does it
  catch and show errors correctly in depending projects if you e.g. delete a public method in the library and rebuild it?.
  As for "only the IDE would resolve against the sources", that's exactly what would cause inconsistencies: Unless these
  local sources are exactly the same version as the ones the jar you build against was built from, the IDE can show
  errors from its internal compilation where the actual build won't produce errors, or vice versa, and code completion
  and refactorings will not use the correct information.
  OK, so to sum up the main use case here:
  There is a "library" project which is built quite often, but separately from other projects that use it. The separation
  is intentional and the dependency should only be on the produced JAR file (its copy in fact). Relating the JAR file to
  the library project sources is not wanted, because as explained, the JAR file is held separately and may not correspond
  to the sources. (Direct dependency between projects would ensure the consistency and eliminate the scanning problem, but
  then the projects would not be separated as requested.)
  Now the problem with this setup is that rebuilding the library and then switching to some of the projects that use it
  cause "entire world" rescanning which makes the IDE unusable. The rescanning is needed, but Eclipse is supposed to
  handle this situation better because it's doing it "on background" while the IDE is still responsive and usable,
  navigation works, nothing is blocked etc. Seems its able to use the index and data even during the scan, updating it
  continuously.
  From this viewpoint, NetBeans does not scan needlessly but should try to fix the scanning itself to be more user
  friendly (as the issue 133943 is about). It is not obvious the problem should be solved by making the IDE scan less (not
  react on some changes, etc, as issue 158177 is basically about). So matthies, greggwon - is this basically your
  standpoint? I must admit I thought the second one was the main problem and also the difference from Eclipse. Still it
  can be both...
  1. dependended on by the currently edited file
  2. In the package of the currently edited file
  3. In the package of any other opened file with depended on being first
  4. In the current project
  5. In the projects of other dependent and opened in the editor files.
  If the keyboard goes idle for more than 30 seconds, the IDE could then start assessing the effects on other projects.
  It could initially just mark dependent projects (for example if a library changed) as "dirty" and use the project
  dependency information to note which files might be dependent on that library and mark those in particular as dirty.
  But it really should not do anything behind the scenes until the user is not actively using the IDE. As the user
  changes the currently edited file, the 1-5 above should be the order of processing to work on updating the "view of the
  world".
  If I break something in a library API, I do it on purpose and I know what to change to fix everything and I don't need
  the IDE telling me I can't work on what I need to fix until it makes sure that another, unrelated project has no problems.
  That's what I currently see happening when I update a library. I can't edit the current project because the IDE is off
  compiling another unrelated.
  -------------
  Are we then just missing a piece of the chain that the IDE doesn't know about? The source files in my library project
  are known. The output file is known, but the place that I copy it to, the library manager designated path, is not part
  of the puzzle. Should it be? Is there a different way that a freeform project can make this part of the chain visible
  to the IDE?
  I agree with greggwon on the point that the scanning could be more demand-driven. For example, if member code
  completion is invoked on a variable whose class is known to have out-of-date scan data, then specifically this class
  could be scanned as needed to update the member list for code completion, rather than waiting that "the entire world"
  has been scanned. The code completion could be dynamically updated, so that the user isn't blocked even during this
  demand-driven scan.
  Similar "targeted" scanning would be helpful for Find Usages and refactorings. For such operations the user could also
  be given the option to have the operation performed immediately based on the current, partially out-of-date scan data,
  because the user may know that the yet-to-be-scanned changes don't affect the operation, or is willing to take the risk.
  Unlike greggwon, I think that "the entire world" should *eventually* be fully scanned, to correctly update the error
  badges etc. But this could be done with lower priority and shouldn't disrupt or prevent user operations.
  I suspect that there's also room for improvement in scanning and indexing speed even intra-project, though this is
  probably a separate issue. Rescanning sometimes appears to be significantly slower than equivalent recompiling. Or for
  example when adding a new method to a widely used interface in a largish project, and then immediately removing the
  method again using Safely Delete, the Safely Delete and the Refreshing Workspace that ensues afterwards takes ages
  (okay, minutes). It also appears that use of generics and circular dependencies (e.g. visitor pattern) affects scanning
  times disproportionately high. Another thing is that CPU usage rarely seems to be the bottleneck for scanning, so maybe
  the performance of I/O and index storage layout could be improved. I'm just speculating here though.
  ."
  That says it all in my book. Scans on really big stuff are never going to be sufficiently fast to avoid causing the
  user lost productivity unless (1) nothing is blocked during scanning and (2) previous scan results are replaced at a
  fine-grained level rather than being forgotten at a macroscopic level thus leaving you with huge gaps in your data
  during rescans.
  The indexing api (part of the parsing api) allows access to the class index data in time of scanning, the java features may use it. The 6.7 scan is much slower
  than it was in the 6.5. One of the reason may be creation of the FileObjects where Files were used before. I will try to return the possibility to use the Files,
  this requires the java indexer not to touch the FileObjects, I will create JavaFileManager based on the Indexables and Dusan should fix the indexer.
  I suspect that most people can only do one thing at a time, with the exception of batch refactoring, so only a small percentage of files change between
  builds and it is wasteful to scan entire "finished" code bases (like libraries) on every timestamp change. Instead, we should store a priority based change-
  map that starts all classes with (priority) 1 change, and every time a change is found increment the priority. With logic "scan only files that have changed
  the most number of times" we could scan 'suspect' classes in real time and leave the low change rate classes for idle/background scanning. If this
  information was stored between sessions, it could also greatly reduce startup time.
  For example, we maintain a common "toolkit" class library that is shared between three products. Most of the classes do not change, but there are a few
  enums that are regularly expanded. Instead of the whole 3mb jar, I would much rather see only these 2 files scanned in real time when the project is built.
  Not only do most files not change, but I would go so far as to say that in most cases, the majority of classes in a library are not used. By joining a map of
  changes with a map of usages, it may be possible to heuristically scan a small number, say %5, of classes initially without compromising utility or the
  "NetBeans magic" that makes it such a good IDE.
  As much of a nightmare as it would be (I surely wouldn't want to do it...) a full out relational database with the ability to hold interfaces, method
  definitions, and javadoc may be the only 'real' solution.
  Doing a relational database containing the initial pieces, e.g. answering what classes are defined and where, what
  methods and fields they define, which classes use them, etc, is not that hard. I can say this as I've done it.
  Now I currently do this in bulk using batch statements for many megabytes of jars in one swoop, but it is easy enough to
  see how one could do a transaction for smaller updates, e.g. of the data related to a given source file, thus allowing
  continuous access to the most accurate information possible.
  With appropriate indexing on the resulting tables the queries for things like "go to type" are nearly instantaneous as well.
  Of course I did this on work time, so it's not open source...
  Created attachment 79469 [details]
  Thread dump when 'Scanning Projects' happens every 10 seconds
  I do not think that relation database will help. Queries to Lucene index is fast enough without any doubt.
  Scanning time is composed from several things and they together takes a log time:
  i) opening indexes,
  ii) computing classpath,
  iii) going through all files and checking timestamps.
  iv) updating changed files.
  When using "Go To Type" feature, again,
  i) classpath computation,
  ii) index search.
  Especially for API support type projects, classpath computation (scope for search) is not fast enough.
  I have 2 big issues with the existing speed.
  1) Indexing/scanning takes time for large sets of code (unsurprising) *and* one can't use large portions of the IDE
  during this indexing. If the latter were solved, then I'd be okay with 6.5's indexing speed. One possibility here
  would be a separate relational database transaction for each source file being scanned, but there are certainly other
  approaches.
  2) Go To Type (and other critical quick navigation actions) can be maddeningly slow at times. I've not seen the
  equivalent take so long with a relational database. Perhaps this is all due to computing classpaths. In any case this
  needs fixing.
  It might be my imagination, but 200904070200 seems much better about the indexing thing. I'm not sure if any code has changed here or not, but it seems
  like it has.
  Now, a single class will disappear without warning from the IDE's 'world'. That is it looked OK just minutes before, but suddenly one will get errors about a
  class not existing even though it compiles and runs fine. It's not always the same class, actually I've never seen it do the same one twice.
  It is also not noticeable until the IDE has been running for a while, which makes me think of issue 162176 and the slow memory leak. Perhaps the asynchronous code is all and well when fresh, but there it gets bogged down somewhere. That's for yall to decide :)
  Attaching a test case. This makes it very easy to see what this issue is talking about.
  Steps to reproduce:
  1) Decompress attached file. It is in .zip format
  2) Open the project in NetBeans 6.7 (It was created using build 200904270201)
  3) Open the "BreakNBManyFiles.java" under the Default Package in the editor
  4) Press <shift> + F6 to run the file. This will create a package under auto.gen.code in "src" and "test"
  Problems you will encounter:
  A) Open up one of the packages, and try to select all files. This takes several seconds on a fast computer.
  B) Deleting all files takes a minimum of 30 seconds, if it does not freeze the IDE
  C) Running the test file more than once will trigger the "Scanning Files" loop-of-death (Each time generates a new set of files)
  D) The JVM maxes out on physical memory, and starts swapping (fresh user directory on Java SE install with 3.0gb virtual after running the test project)
  Created attachment 81114 [details]
  See BreakNBManyFiles.java in the Default Package
  This issue was discussed during last VOC meeting #8 [1] and it was strongly suggested to fix this after 6.7.
  [1]
  So performance and scalability of the most critical core functionality is always "next release"?
  Sadly this sounds a lot like other software products (that shall remain nameless), but that's no excuse.
  Try the "Scan On Demand" module from Development Update Centre. That should reduce the scanning. Bear in mind that usage of module is at your own
  risk! Any issue reported when module is on will be considered as a bug against experimental module.
  The project scan *MUST* be improved in 6.7 otherwise you may as well just install Eclipse for remaining the NB
  developers here, they have almost given up on NB and this will just be the death nail.
  I really don't think you guys understand how badly NB is coming up in comparison to other IDEs in this area.
  Along these lines, while I can't reasonably share/upload my company's projects or source code as a test case, I will be
  at JavaOne and could certainly let some NetBeans folk poke around in these via NetBeans on my laptop.
  I'm guessing that there may well be others who are greatly pained by this issue who are similarly equipped. It might be
  good for the NetBeans guys to organize a "show me your irreproducible NetBeans problems" meet up or some such.
  [No, I won't be attending CommunityOne or at least most of it -- as I've attended it every year thus far and haven't
  found it worth the extra day of lodging and losing part of my weekend.]
  I work for a big sized company and I have used Netbeans since the first version. We are more than 30 developers working
  in the same J2EE projects. We are willing to use Netbeans but this issue and issue #42683 are preventing us to migrate
  from JBuilder.
  Comparing with other IDEs Netbeans lacks the stability and scalability for enterprise sized projects, and in every new
  version this issues are postponed. Aren't this more important than new languages or eye-candy?
  Scanning is hell. If you open the IDEs with ten or more projects it becames very slowly an unstable, freezing sometimes.
  Opening a single java or jsp file may take 5 seconds to appear. If the are many classes in the project badging of
  compiling errors are almost always wrong... If you compile the full project with dependecies takes forever, and after
  that the badges remain wrong.
  IMHO Netbeans is sadly becoming a tool for educational purposes, but not for the real work.
  Discussion here is getting long, so I would like to summarize it.
  THIS BUG IS NOT ABOUT SLOW NETBEANS, BECAUSE NETBEANS ARE FAST!!!
  We have several comparison test on different large projects, and we KNOW that NetBeans are comparable to other IDEs on
  these projects. We are developing NetBeans in NetBeans, so we REALLY KNOW that ... NetBeans are fast!!! (NetBeans source
  base is really big)
  So, this issue is about one concrete usecase:
  User has custom Ant build script (free form project) that generates something to the Sources.
  (Free form project sources are defined in free form project properties - "Java Sources" and "Java Sources Classpath" lists.)
  Your Ant project should not modify files / Jars that are listed there!!!
  If you broke that rule - you have troubles. NetBeans listens on that folders / Jars, and starts scanning.
  The fact that NetBeans are listening on source path changes is design decision (not a bug) and we are not going to
  change it. It is very useful feature.
  So, what can we do with it? Stop listening on source path during build + add some manual rescan? We are investigating,
  but there is no simple solution.
  And what can you do with it? You can try to modify your build script to separate source folders / Jars from Output
  folders / Jars. And you can modify your "Java Sources" and "Java Sources Classpath" lists to not contain any files
  generated / modified by your build script. It should help to fix this issue for you.
  And if you see some other issue why your NetBeans are slow, FILE ANOTHER BUG, PLEASE!
  This issue is about "Source path modification initiated from build script starts scanning." only.
  jjancura: I don't think it is very useful to say that the problem doesn't exist after all the comments and complaints on
  the issue. It's great that the NetBeans team uses NetBeans for development and doesn't see any issues w/ performance.
  However, when someone was designing / building this auto-scanning feature they envisioned a particular usage model for
  NetBeans, and it appears (rightly or wrongly) that how people use the IDE falls outside that envisioned usage model,
  which causes them trouble (resulting in a slow IDE).
  The suggestions that you gave are useful in analyzing the problems and hopefully folks will go back and analyze their
  projects for the issues that you describe. There is a problem with pointing the finger at the users and saying "You're
  wrong": if the users set up their project wrong, there is a chance that the IDE didn't necessarily support them in
  creating a "correct" project setup. The next steps that you identified sound reasonable - it would be nice if there is
  some resolution to this SOON, you can see a lot of loyal / long time NetBeans users upset over this.
  One potential issue in what you describe as the source of the problem is that NetBeans can relaunch the scanning of
  something changes in the "Java Sources Classpath" setup : it seems more or less normal for this path to contain class
  directories and jars. It seems just as normal that an Ant build will modify some of those during the build. In a
  different issue, jglick pointed out that if a classes directory or a jar is associated w/ sources, the IDE wouldn't
  rescan even if the artifact changed. Thus, an additional suggestion to review the project structures is : are all of
  your freeform projects correctly configured to identify the "output" of their source (so that NB can associated classes
  and jar w/ source and not scan when the clasess or jars change during the build).
  jjancura: I don't agree with your appraisement and narrowing of the issue. The scanning/refreshing has a design and/or
  implementation flaw which is not just related to custom Ant tasks. The scanning is often a problem too in projects or
  project groups without custom Ant tasks. Excessive scanning and/or wrong error badges happens in the context of:
  - in-between saving of files with compilation errors
  - larger refactorings
  - CVS updates
  - building of dependee or sub-projects, as common in J2EE project structures
  - concurrent executions/combinations of the above, which NB doesn't (and shouldn't) prevent
  - projects with heavy use of generics and/or recursive dependencies (source model appears to be buggy)
  - incomplete or inconsistent source model data
  - ...others may add more
  The problem is not necessarily the rescanning or "refreshing workspace" itself, but that it
  - is sometimes modal in that it is blocking certain user actions
  - consumes too much CPU
  - does a lot of I/O (much more than e.g. javac for the same set of source files)
  - assumes a bug-free source and dependency model
  - assumes no concurrent actions (or so it seems)
  I believe there's both a performance problem in that updates of the source model appear to consume more resources than
  they should (and more than they appear to in other IDEs), and, as far as I can gather, a conceptual problem in that the
  design seems to be to "stop the world" until the source model is (believed to be) complete and consistent again, as
  opposed to being able to work with a possibly incomplete and inconsistent, incrementally updated model (which appears
  to be how other IDEs work).
  So, source path modifications caused by Ant execution may well be an individual issue in itself, but is not the larger
  issue this Issuezilla issue was about, IMO.
  NB is *NOT* fast!!. The "project scan" issues are an embarrassment. Look at the message boards.
  As most devolopers use other IDEs when working on large existing projects we can't just change the project structure for
  all developers to suite NB when the other IDEs have no trouble with the existing project structure.
  I have never seen Eclispe block the developer while trying to navigate to a class. With NB it seems to block far too often.
  @jjancura When you say "And if you see some other issue why your NetBeans are slow, FILE ANOTHER BUG, PLEASE!"
  The summery of this issue has been changed. According to my records on 30 APR 2009 (as of my test case), the summery was
  [Issue 134990] Java support is slow on big projects
  Then, today (5 MAY 2009), I received this by email:
  User jjancura changed the following:
  What |Old value |New value
  ================================================================================
  Summary|Java support is slow on bi|Source path modification i
  |g projects (code completio|nitiated from build script
  |n, find usages) | starts scanning.
  --------------------------------------------------------------------------------
  Instead of changing issues so that they no longer exist, then denying that they are issues, how about putting effort into fixing them?
  Perhaps on NB source, NB works fine. All of the NBM that I have seen have custom ANT scripts, not the scripts that NB generates when a user creates a
  project. Have you considered that of the 36 registered users who voted for this issue, that the majority of them do not use custom ANT scripts?
  I will file another issue; But coverups should have died with the CCCP.. Grow up.
  Fist, I must say that I do not see any big disagreement here. All we are trying to make NetBeans better, faster. We
  should only clarify how to do it effectively.
  Messages like "NetBeans are slow", "NetBeans are ugly", "NetBeans are buggy" are not fixable issues. See. There is description how to write useful issue (see mainly "One
  problem - one issue", "Provide detailed description", and "Provide steps to reproduce" sections)
  "NetBeans are slow" is great point to discussion. We definitely need such great source of feedback from community. But
  it should be discussed on mailing lists. Visibility of one issue is definitely lower than a number of people watching
  nbusers mailing list. We need more users to discuss it, more feedback! Issues like "Source path modification initiated
  from build script starts scanning." can be one of results of such "NetBeans are slow" discussion. You probably
  understand that useful bug report have to be concrete, reproducible, measurable...
  I have identified that "Source path modification initiated from build script starts scanning." is root cause of many
  problems described here. That is why this issue is now about this usecase only. But it does not mean that we will throw
  rest or reproducible usecases mentioned in this issue out of window. We will check all concrete user reports from this
  issue and file separate issues for them.
  Please,
  If you would like to discuss NetBeans performance, go to nbusers (more users = more feedback). You can tag your message
  with "[PERFORMANCE]" to be more visible.
  If you would like to discuss NetBeans architecture go to nbdev mailing list. Themes like "Manual or automatic scanning",
  "source models", "incrementaly updating scanning" are great points for discussion there. I am not able to attach all
  developers, architects and other people responsible for NetBeans architecture to this single issue, but all of them are
  definitely on nbdev! We can find some solutions there, and file concrete issues / RFEs than.
  And, please,
  if you have some concrete, reproducible usecases that are not covered by this "Source path modification initiated from
  build script starts scanning." usecase - file separate issues for them, please!
  Thanks
  This is ridiculous. By changing the summary, the issue no longer applies to me. But look! The problem still exists.
  You may as well have changed it to, "My car won't start because there is no fuel in it".
  Madness.
  jjancura's initial post ruffled a few feathers, but his second post makes perfect sense. One possible solution that
  might please everyone is to transform this issue into an umbrella issue, with multiple sub-issues (one per problem).
  I've seen the Netbeans team do this before...
  The predominate issue is that scanning, no matter what initiates it, requires in its current design, too much CPU and
  disk I/O bandwidth. Part of that is based on the limited ability of or incorrectly established content of a cached view
  of the source and class file content.
  There really are bugs involved. As was recently reported, one such logic bug was found. I still contend from a point
  of ignorance that there are architectural bugs which need to be dealt with to remedy the problem overall.
  Clearly the dev team are defensive about where they believe the software is at. The users who are having this issue
  impact their productivity, some in quite dramatic ways based on what we read in this issue, are not content with that
  positioning.
  Progress will only happen if we can all get on the same page and agree that there are issues that need to be addressed.
  The main reason I noticed this is because in 6.1 IT DOESN'T HAPPEN. When I "upgraded" to 6.5 is when the problem
  appeared. Maybe it doesn't matter at this point but it may be a clue as to what the problem is.
  Also a possibly important note is that it isn't bad from the beginning. It seems to be manageable until some threshold
  is crossed (time, number of clean rebuilds?) and then the project scanning goes into its never ending mode.
  I'd like to add my two cents to this problem.
  First I'd like to add that many of the developers here have a lot of very positive things to say about NetBeans. It's
  amazing how much the product has improved over the past couple of years. We like the way the GUI is laid out. We love
  the EJB generation wizards. Some of us have used the Swing drag-and-drop editor, and I've heard good things about that.
  We also like the way plugins are handled.
  Having said that, the "scanning..." problem described in this ticket is really killing us. No joke. We consider this
  to be a high priority defect, if there ever was one. Although it's not our primary build mechanism for making installs,
  we do compilations within Netbeans just to make sure everything is Kosher, and we stick with the default ANT script
  without any custom ANT tasks.
  As mentioned previously, we wish we had more control over the scanning process, and we wish that it didn't consume so
  many resources.
  Please fix this ASAP.
  I'd like to second kidvid comments. The reason I care so much about the "scanning" issues is that I love all the other
  parts of the IDE, so much better than Eclipse in so many ways but one. The "scanning" issues are what have forced the
  bulk of the developers here off NB.
  As was already mentioned, please try the Scan on Demand module, especially if you experience frequent or long re-scans:
  Note the module is experimental, it's not meant to be the fix for the scanning problems, but it might give us valuable
  idea how the performance can really improve by reducing scanning and what other problems may arise. And it may
  actually help in many cases. We continue working on understanding and fixing the scanning issues on other fronts too.
  I tried again the scenario described by theshadow27 in:
  in the latest build of NetBeans 6.8. Here are my findings:
  A) Select all files: yes it is slow, IDE autodetected frozen AWT queue. Reported as a separate issue 172191.
  B) Deleting all files: also slow, reported as a separate issue 172199.
  C) Running the test file more than once: there was a brief period of scanning, but nothing unreasonable or unexpected.
  Seemed fine to me.
  D) Also did not experience this (though my machine has 4GB of RAM).
  > B) Deleting all files: also slow, reported as a separate issue 172199.
  That's about performing the delete operation itself. There's also issue 168267 about optimizing the preparatory phase of
  the delete (i.e. not to initialize refactoring when Safe Delete may not be chosen at all, and not to block invocation of
  the action if scanning is running). Planned for 6.8.
  There is one other sub-scenario that leads to undesirable scanning:
  1. Create a Java SE application
  2. Create some valid XML schema in this application
  3. Create JAXB binding against this schema. This will generate the sources and build them.
  4. Use these generated sources in some other classes of the project (e.g. the Main class)
  5. Clean the project (which deletes the generated sources) -> sources are scanned
  6. Build the project (which re-generates sources) -> sources are also scanned
  BTW, let's really, really use this bug only for the "sources generated by build script" use case - for any other use
  cases where the build script does not generate Java sources, separate bug reports need to be filed. Thanks.
  What about the annotation processor case?
  For instance, in Java 6 one can even create an annotation processor to be invoked as part of javac that takes
  @SpecialClass
  public class MyClass
  extends MyBaseClass
  {
  // ...
  }
  and auto generates MyBaseClass.java. Even in less "extreme" use cases, it is common for an annotation processor to
  create (and update?) source files as part of javac invocation.
  Re pjiricka's last comment (desc 74): could you please be more specific about what should happen, in your opinion?
  Specifically, should there be error badges for missing classes after step 5 (the classes are missing in reality, are
  they not)? If not, why (usually, such situations are reported as P2s, why it is not a P2 in this particular case)?
  Should the code completion show the classes that are missing? What should happen after step 6? Should the error badges
  disappear? Should the newly appeared classes be visible in the code completion? Thanks.
  I don't have a strong opinion about what should happen after step 5 - anything we do is somehow "incorrect".
  After step 6, the generated classes should of course be visible in code completion, there should be no error badges etc.
  Regarding the performance aspect, it would be nice if after steps 5 and 6, sources were not rescanned (after step 6,
  everything is the same as was before step 5, so the IDE could reuse what it already knew at that point).
  It's not really the scanning itself that is undesirable, but the fact that it invalidates code completion data until
  the scanning is completed, and prevents all kinds of actions (source navigation, hints, refactorings) that rely on that
  data. Of course the scanning also shouldn't consume CPU and I/O resources to the point of significantly slowing down
  other operations.
  It might be worth looking at how Eclipse handles such use cases, because it doesn't seem to have these scanning issues
  and users appear to be happy with how it does things.
  The fact that scanning disables code completion and source navigation and louses up error badging, etc, is an underlying
  issue behind most current scanning issues. This behavior is simply untenable/unacceptable. Instead, scanning should
  behave like a transaction -- other threads should be able to continue seamlessly using old data until new data is
  provided by the scanning thread. Sure there should be a means for the other threads to check for a scan in progress,
  e.g. so refactoring could *warn* that the results may be inaccurate due to a scan in progress. Sure there's a question
  as to appropriate transaction isolation granularity, e.g. should each source file's scan data be updated it is scanned
  or should this be delayed until a package or source root level. These details can clearly be worked out. The critical
  thing is to have scanning block nothing and never remove old scan data until the new replacement scan data is available
  (which should be an atomic replacement). Anything less than this and NetBeans will forever remain in crippled scan hell.
  Once this is resolved, then avoiding unnecessary scans becomes purely a CPU/disk grinding issue. Assuming one is not
  too strapped for CPU/disk resources and the scan thread is not too aggressive, this then becomes a *much* lower priority
  issue.
  See also some ideas given by Peter Nabbefeld: issue 174511.
  For sources generated by the annotation processor the scan is not triggered in NB 6.9 (the annotation processing has to be correctly configured). For freeform see issue #185457.
  Based on the Comment #78 of the original reporter and also the previous Comment #77 I am marking this as duplicate of #182653. The overall sentiment here is (if I understand correctly) that majority of the users want transactional index and the symptoms described by this bug would not solve the general dissatisfaction with the current state. Only solving #182653 will improve the situation so these reports are in fact the same.
  *** This bug has been marked as a duplicate of bug 182653 ***
  Just my small input to this issue...I've tried netbeans v8, v8.0.1, v7.3.1, and v6.9.1 and all of the versions have a major performance bug with the project scanning. The problem is major in that netbeans in unusable during any scan, which appears to get triggered for no apparent reason.
  Using sysinternal process monitor, I can see that netbeans is doing a full rescan of my entire project every few minutes.
  I'm going to try v6.1 of netbeans and if that still causes this issue, I'm done with netbeans.
  When I first reverted to v6.9.1, the problem didn't happen for several weeks, but now the program is worthless due to the performance.
  I've tried turning off all of the code completion settings, but the scan is still a full scan of the project. I've tried the "scan on demand"...which doesn't work.
  Obviously someone at Oracle isn't getting the picture that this is a show-stopper for many developers, and will likely lead me to jump to another IDE. It can't be that difficult to just only update the changed files, a simple check of the file data or which file was saved would be an easy fix and only scan those files, instead of the ENTIRE PROJECT!
  P.S. I'm a very long time MS VS6 user, due primarily to it's performance. I still use VS6 to open 100+ MiB files as it has no issue with very large ASCII files.
  In fact the scan is done for minimal set of files.
  There are 2 types of update scans in case of the java project:
  1st) The project metadata change like classpath change.
  Such a change requires a full rescan of all sources in affected project as
  the metadata change affects all sources. The sources which are now compilable
  may become invalid and vice versa.
  2nd) Single file(s) change like save of file (external file modification).
  Such a change has 2 phases. In the first phase just the changed file(s) is scanned.
  In the second phase the hash of class API signatures from the changed files are computed and compared to the last know API signatures. If changed (the class API has changed and it can affect other files) the usages of the class is found and the classes which are using the changed class are rescanned.
  The scope of the second phase can be set in the Tools/Options/Editor/Editor/Hints/Java/Dependency Scanning.
  If the full scan happens often it can be cause either by frequent metadata changes (custom build steps), changes in signatures of file with has lots of usages (in this case you can limit the Dependency Scanning) or it can be caused by Annotation Processor.
  If you can share some test project or attach the nps file of such a scan I can help.