<story><title>LoRA from scratch: implementation for LLM finetuning</title><url>https://lightning.ai/lightning-ai/studios/code-lora-from-scratch?view=public&amp;section=all</url></story><parent_chain><item><author>rsweeney21</author><text>It&amp;#x27;s still strange to me to work in a field of computer science where we say things like &amp;quot;we&amp;#x27;re not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best.&amp;quot;</text></item></parent_chain><comment><author>TacticalCoder</author><text>&amp;gt; &amp;quot;we&amp;#x27;re not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best.&amp;quot;&lt;p&gt;Isn&amp;#x27;t it the same for anything that uses a Monte Carlo simulation to find a value? At times you&amp;#x27;ll end up on a local maxima (instead of the best&amp;#x2F;correct) answer, but it works.&lt;p&gt;We cannot solve something used a closed formula so we just do a billion (or whatever) random samplings and find what we&amp;#x27;re after.&lt;p&gt;I&amp;#x27;m not saying it&amp;#x27;s the same for LLMs but &amp;quot;trying a bunch of different values and see which one works best&amp;quot; is something we do a lot.</text></comment>
<story><title>LoRA from scratch: implementation for LLM finetuning</title><url>https://lightning.ai/lightning-ai/studios/code-lora-from-scratch?view=public&amp;section=all</url></story><parent_chain><item><author>rsweeney21</author><text>It&amp;#x27;s still strange to me to work in a field of computer science where we say things like &amp;quot;we&amp;#x27;re not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best.&amp;quot;</text></item></parent_chain><comment><author>SkyMarshal</author><text>That bottom-up tinkering is kinda how CS started in the US, as observed by Dijkstra himself:  &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.cs.utexas.edu&amp;#x2F;users&amp;#x2F;EWD&amp;#x2F;transcriptions&amp;#x2F;EWD06xx&amp;#x2F;EWD611.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.cs.utexas.edu&amp;#x2F;users&amp;#x2F;EWD&amp;#x2F;transcriptions&amp;#x2F;EWD06xx&amp;#x2F;E...&lt;/a&gt;&lt;p&gt;Ideally we want theoretical foundations, but sometimes random explorations are necessary to tease out enough data to construct or validate theory.</text></comment>