Tokenizing each document in a large document of documents
Tokenizer method:


You use ref incorrectly, all reference types are always passed by reference, please read the MSDN or any other online resource to understand the ref
Your code doesn't follow naming conventions: local variables, parameters should be camelCased (e.g. StopWordSet)
use string.Split(new char[0], StringSplitOptions.RemoveEmptyEntries) overload to split on all whitespace characters and remove empty strings during tokenizing, rather than trimming result
strings are immutable, so the statement item.Trim() is useless unless you assign it to something
Tokenizer method always returns an empty list since tokens (not tokenz) is only intialized and not used. It looks like tokenz parameter is redundant and tokens variable should be used instead
To improve performance it might be better to change the type of stopwords parameter to ISet&lt;string&gt; and use it for matching, rather than create a new StopWordSet each time
As a result you'll get a cleaner version of Tokenizer: 

private static IEnumerable&lt;string&gt; Tokenizer(StreamReader sr, ISet&lt;string&gt; stopwords)//function for tokenizing
{
    string line;
    List&lt;string&gt; tokens = new List&lt;string&gt;();//list of strings called tokens

    while ((line = sr.ReadLine()) != null)//as long as the streamreader has something
    {
        foreach (string item in line.Split(new char[0], StringSplitOptions.RemoveEmptyEntries))//split amongst strings
        {
            if (item.StartsWith("&lt;") &amp; item.EndsWith("&gt;"))
            {
                if (item == "&lt;/DOC&gt;")
                    tokens.Add(item); //adds the doc tags for later separation use
            }
            else
            {
                string newitem = Regex.Replace(item, @"[^A-Za-z0-9]+", "", RegexOptions.IgnoreCase).ToLower();
                if (!stopwords.Contains(newitem))
                    tokens.Add(newitem);
            }
        }
    }
    return tokens;
}



Now let's look at the AddToDictionaryAndCount method:


naming conventions - do name your variables/parameters properly, it's absolutely not clear what is the difference between doccounter2 parameter, doccounter and secondcounter variables. Rename them to smth. like documentIndex, numberOfTermOccurances etc. 
while (counter &lt;= tokens.LastIndexOf("&lt;/DOC&gt;")) is the killer, you're looking for "&lt;/DOC&gt;" entry on each iteration. Cache the calculated value instead.
since you know the number of iterations and you always increment the index it's better to use for instead of while
not sure why you pass docFreqCounter as parameter, it looks like is being used as a local variable.


As to performance improvements (other than caching LastIndexOf) - currently you scan the through the tokens list for each and every item in myLexicon. Assuming that lexicon is built from tokens it would be much better to scan the tokens list only once, tracking which items you've already counted and where are the doc boundaries. Since you haven't provided the meaning of all parameters participating in this method it's hard to suggest a proper solution, but here is the first approximation:

public static Dictionary&lt;string, Dictionary&lt;int, int&gt;&gt; AddToDictionaryAndCount(List&lt;string&gt; tokens)
{
    var result = new Dictionary&lt;string, Dictionary&lt;int,int&gt;&gt;();
    var documentIndex = 0; //tracks the current document index

    var lastDocumentIndex = tokens.LastIndexOf("&lt;/DOC&gt;");
    for (int i = 0; i &lt;= lastDocumentIndex; i++)
    {
        var token = tokens[i];
        if (token == "&lt;/DOC&gt;")
        {
            //finalize stats for the document. not sure what goes here.
            //add the logic corresponding to "doccounter2 &lt; doccounter"
            documentIndex++;
            continue;
        }

        Dictionary&lt;int, int&gt; documentStats;
        if (!result.TryGetValue(token, out documentStats))
            documentStats = result[token] = new Dictionary&lt;int, int&gt;();

        documentStats[documentIndex]++;
    }
    return result;
}