title: Using linear approximation to perform addition and subtraction | error barrier
body: <p>I'm attempting my first solo project, after taking an introductory course to machine learning, where I'm trying to use linear approximation to predict the outcome of addition/subtraction of two numbers.</p>
  
  <p>I have 3 features: first number, subtraction/addition (0 or 1), and second number.
  So my input looks something like this:</p>
  
  <pre><code>3 0 1
  4 1 2
  3 0 3
  </code></pre>
  
  <p>With corresponding output like this:</p>
  
  <pre><code>2
  6
  0
  </code></pre>
  
  <p>I have (I think) successfully implemented logistic regression algorithm, as the squared error does gradually decrease, but in 100 values, ranging from 0 to 50, the squared error value flattens out at around 685.6 after about 400 iterations.</p>
  
  <p><a href="https://i.stack.imgur.com/LjYCK.png" rel="nofollow noreferrer">Graph: Squared Error vs Iterations</a>
   .</p>
  
  <p>To fix this, I have tried using a larger dataset for training, getting rid of regularization, and normalizing the input values.</p>
  
  <p>I know that one of the steps to fix high bias is to add complexity to the approximation,  but I want to maximize the performance at this particular level. Is it possible to go any further on this level?</p>
  
  <p>My linear approximation code in Octave:</p>
  
  <pre><code>% Iterate
  for i = 1 : iter
      % hypothesis
      h = X * Theta;
  
      % reg theta prep
      regTheta = Theta;
      regTheta(:, 1) = 0;
  
      % cost calc
      J(i, 2) = (1 / (2 * m)) * (sum((h - y) .^ 2) + lambda * sum(sum(regTheta .^ 2,1),2));
  
      % theta calc
      Theta = Theta -  (alpha / m) * ((h - y)' * X)' + lambda * sum(sum(regTheta, 1), 2);
  end
  </code></pre>
  
  <p>Note: I'm using 0 for lambda, as to ignore regularization. </p>