As an inexperienced/unknown recommendation letter writer, what should I do to make my letter stand out?
My advice is to start your letter by explaining who you are and your background.  For example, you could mention where you went to graduate school and that you think the student could succeed in that graduate program.  This will give the reader a sense that you have some idea of what will be expected of the student in graduate school.  

Beyond that, you should focus on the same things that more experienced letter writers do.  

You need to explain how you know the applicant and give the reader some sense that you've had enough interaction with the student to be able to judge their chances of success.  Simply having had a student in a large class isn't really enough here.  On the other hand, "I was Johnny's instructor in a senior level topology class with 12 students.  Johnny frequently met with me during office hours to ask questions about aspects of the subject that we were not able to cover in class." tells me that you have had many discussions with the student and really know the student well.  

You should comment on both the applicant's intellectual ability and their work habits.  Ideally, we want applicants who are smart and work hard.  An applicant who isn't very smart but smart enough and who works very hard might be a good candidate for admission to a master's program but might be a very poor choice for a PhD program.  An applicant who is brilliant but lazy might start to work harder in graduate school and could turn out to be really successful.  Some faculty are willing to take a chance on brilliant but lazy students.  

You should talk about the applicant's personality and how they get along with other students and faculty members.  No one wants to work with a student who is not fun to be around because they're argumentative or depressed all the time.  

You should talk about the applicant's communication skills (both writing and oral presentations) and in mathematics you should comment specifically about their ability to write mathematical proofs.  

If it's relevant (and it would be for any area of applied mathematics) you should comment on the applicant's computing skills.  What languages and specialized packages have they used in their work with you?  Have they been able to produce programs of any substantial size?  

You should check that what you've written matches up with the student's transcript and their statement of purpose.  For example, don't say that they're one of your top 5% students if their GPA is 3.0.  Don't say that Johnny would be a great master's student if he's applying to a PhD program.  If the "story" told by the applicant's application file isn't consistent, then I'll be much less likely to admit a student.