How should feedback be given for "silly mistakes" on an exam
First, notice that students will typically be able to tell which of their mistakes were silly and which more significant.  So for silly mistakes you don't need to write feedback other than to circle where they went wrong, and indicate the number of points taken off (or granted).

For a more serious mistake you can write more detailed feedback.  Sometimes feedback here isn't necessary, either, other than to write, "see solution."

Second, you can figure out what types of errors and what frequency of errors you want to bring the grade down from A to B, from A to C, etc., and decide how many points to remove based on that.

More importantly: what's a well written exam like?  Example: I'm asking them to solve this eigenvalue-eigenvector problem.  If the student has a vague idea what is being sought, I want him to get one point.  If he sets things up well but doesn't know what to do next, he gets two points.  If he follows through well but for whatever reason didn't quite get to the perfectly correct punch line, he'll get 3 points.  Perfect, complete answer: 4 points. (This is just an example of the point distribution.  You might end up with a different scoring structure.) In short, the scoring should be integral to the exam design from the beginning.

Most importantly: as a TA, you should be getting special guidance from the professor for grading a midterm exam.