<story><title>Megatron-Turing NLG 530B, the World’s Largest Generative Language Model</title><url>https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/</url></story><parent_chain><item><author>alecst</author><text>Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole.&lt;p&gt;How much of a simplification is that? And how much does the accuracy of such a model matter, in the grand scheme of things?</text></item><item><author>singularity2001</author><text>Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own.&lt;p&gt;Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.</text></item><item><author>cs702</author><text>So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network.&lt;p&gt;Trillion-parameter models are surely within reach in the near term -- and that&amp;#x27;s only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you&amp;#x27;re talking really big numbers.&lt;p&gt;I know the figures are not comparable apples-to-apples, but still, I find myself &lt;i&gt;in awe&lt;/i&gt; looking at how far we&amp;#x27;ve come in just the last few years, to the point that we&amp;#x27;re realistically contemplating the possibility of seeing dense neural networks with hundreds of trillions of parameters used for real-world applications in our lifetime.&lt;p&gt;We sure live in interesting times.</text></item></parent_chain><comment><author>sydd</author><text>oh finally something that I learned a lot about :)&lt;p&gt;Such research is the area of computational neuroscience - one thing that such people do is try to model parts of the brain (or just a single neuron) with computers.&lt;p&gt;A Neuron (=nerve cell in the brain) is a very complex beast. In rough terms they work like this: They collect signals (electrical impulses) via their small appendages called dendrites. when the sum of the signals reaches a certain threshold a large electrical impulse is generated at the cell body that will travel trough its &amp;quot;output&amp;quot; appendage (called axon) that connected to another neuron&amp;#x27;s cell body or to its dendrite.&lt;p&gt;Neurons display a dazzling variety in all these parameters:&lt;p&gt;- In morphology, e.g. they can look like a pine tree &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.scholarpedia.org&amp;#x2F;article&amp;#x2F;Pyramidal_neuron&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.scholarpedia.org&amp;#x2F;article&amp;#x2F;Pyramidal_neuron&lt;/a&gt; (I really recommend scholarpedia, also this article has a nice animation on how electrical impulses propagate) or like a sea urchin.&lt;p&gt;- it really matters where the cell gets its impulse from: A neuron stimulated near its cell body will be much more sensitive to the input than being stimulated far away.&lt;p&gt;- Their response characteristics are wildly varied too. Some give off one large impulse, some a quick burst of impulses. Some are preventing others from giving out impulses from stimulation (inhibitor neurons)&lt;p&gt;- This whole mess can be modulated with chemical compounds that are released by the body -- some make some neurons more sensitive, some less.&lt;p&gt;- Also we still discover every year some new mechanism that modulates how they function.&lt;p&gt;The issue is that this results in such a complex system that a modern PC cant even simulate 1 detailed neuron model realtime (these tools are open source, try them out! for example &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;neuron.yale.edu&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;neuron.yale.edu&amp;#x2F;&lt;/a&gt; ). Now we know that we&amp;#x27;re simulating things that likely do not matter (e.g. we don&amp;#x27;t need a neuron model that consist of 10.000+ segments), but we do not know which parts we need to remove to have a faithful simulation. Also we might simply simulate some parts wrong because our knowledge of the subject is not enough.&lt;p&gt;But on the upside we&amp;#x27;ve reached some great things already, for example we know how our brain calculates from our head and eye position the orientation of the things we&amp;#x27;re looking at</text></comment>
<story><title>Megatron-Turing NLG 530B, the World’s Largest Generative Language Model</title><url>https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/</url></story><parent_chain><item><author>alecst</author><text>Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole.&lt;p&gt;How much of a simplification is that? And how much does the accuracy of such a model matter, in the grand scheme of things?</text></item><item><author>singularity2001</author><text>Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own.&lt;p&gt;Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.</text></item><item><author>cs702</author><text>So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network.&lt;p&gt;Trillion-parameter models are surely within reach in the near term -- and that&amp;#x27;s only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you&amp;#x27;re talking really big numbers.&lt;p&gt;I know the figures are not comparable apples-to-apples, but still, I find myself &lt;i&gt;in awe&lt;/i&gt; looking at how far we&amp;#x27;ve come in just the last few years, to the point that we&amp;#x27;re realistically contemplating the possibility of seeing dense neural networks with hundreds of trillions of parameters used for real-world applications in our lifetime.&lt;p&gt;We sure live in interesting times.</text></item></parent_chain><comment><author>visarga</author><text>&amp;gt; All we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is that?&lt;p&gt;A lot. Parallel optimization is an art form. These models are trained on static datasets, they can&amp;#x27;t intervene in the environment to infer causal relations, so they need legs and hands.</text></comment>