1. A cache coherency controller for a distributed, scalable shared memory systems, said system including a scalable plurality of nodes, comprising: shared memory distributed to the node memory at each node for storing a plurality of storage words at addressable memory locations in each of a plurality of cache lines; said node memory subdivided into a first section for changeable data and a second section for unchangeable data; a status bit associated with each of said storage words for defining whether the corresponding memory location contains changeable or constant data; a distributed invalidation directory at each node associated with said first section for listing and tracking which nodes have copies of each cache line in said first section, said invalidation directory being expandable when necessary by using an overflow directory so as not to limit the number of nodes that can access each cache line; a memory controller at each node for determining whether an address in shared memory to which access is being sought by a first thread is located in local memory or remote memory; and if the access is remote, for signaling the node processor that a remote read is required for said first thread, enabling said node processor selectively to respond by switching program threads; generating a read request message for the cache line containing a requested storage word to the remote node having the memory address being accessed; receiving the requested cache line from said remote node; storing the requested cache line to local cache; and signaling the node processor that the requested data is available; and if data is stored to a cache line which resides in said changeable portion of memory, for invalidating copies of said cache line stored at remote nodes.