How to loop sklearn linear regression by values within a column - python
Question: I am relatively new to python and programming in general.

I have a csv with three columns, an identifier, distance, and date. The data
looks something like:

    
    
    id, date, distance
    
    1, 1850, 150
    1, 1950, 200
    1, 1990, 250
    2, 1850, 130
    2, 1950, 180
    2, 1990, 210
    3, 1850, 200
    3, 1950, 220
    3, 1990, 250
    

etc...

I'm trying to use sklearn.linear_model LinearRegression. To run a regression
on each of these series. So I need to basically separate the columns by id. I
want a separate result for each group of id's (1, 2, 3, etc).

I was able to do this easily with R by using:

    
    
    reg1 <- lmList(distance ~ date | id, mydata)
    

However, for several reasons, I need to do this with python.

I've imported the csv using pandas.read_csv. Then I assumed I need to
attribute each of these to a variable:

    
    
    dataset = pandas.read_csv(location, sep = ",", header = 0)
    x = dataset['date']
    y = dataset['distance']
    z = dataset['id']
    

Then apparently to use them in LinearRegression() and model.fit(x,y) I needed
to use

    
    
    x2 = numpy.array(x).T and
    y2 = numpy.array(x).T
    

So now that that is all set up, what should I do to actually loop the
regression through those id values?

when I try:

    
    
    for n in main_reg2.id:
        model1 = LinearRegression()
        model1.fit(xT1,yT)
        print(model1.coef_, model1.intercept_)
        break 
    

It just gives me one result for the whole dataset. And I'm not sure where to
tell it to go through the id values.

Any help would be greatly appreciated. Sorry for the mixing of jargon, I've
had to dabble in 5 different programming and statistical languages over the
past 2 years of grad school and am very confused sometimes.

Perhaps I'm overthinking this, or maybe there is some way towards the
beginning that I can separate the data?

Also, how would I create a table from this that lists the coefficients and
intercepts and whatnot.

Thanks

Updated code so far: with open(r'file_location','rb') as f: reader =
csv.reader(f) headers = reader.next() for fields in reader: your_list =
list(reader)

    
    
    z = [_[0] for _ in your_list]
    x = [_[1] for _ in your_list]
    y = [_[2] for _ in your_list]
    
    id_max = max(z)
    new_list = [[]]
    for _ in range(id_max):
        new_list.append([])
    
    for index in range(len(z)):
        id = z[index]
        new_entry = [x[index],y[index]]
        print "Adding to index", id, new_entry
        new_list[id].append(new_entry)
    print new_list 
    

I used header = reader.next() code to get rid of the headers. When I was doing
the max(z) before, the header was coming up as the max...

Going through line by line: I get...

    
    
    id_max = max(z)
    print(id_max)
    999
    

Which I already know is wrong since the max is 1576.

But moving on.... after the append I get

    
    
    Type Error: range() integer end argument expected, got str
    

Which I don't understand since z is a list of integers from 1 to 1576.

Some examples of the indexes are:

    
    
    z[1] = '10'
    y[1] = '2011'
    x[1] = '515.8938'
    


Answer: I'd simply separate the data by ID at the very start: make this a list of
lists:

    
    
    data_list = [
        [1, 1850, 150],
        [1, 1950, 200],
        [1, 1990, 250],
        [2, 1850, 130],
        [2, 1950, 180],
        [2, 1990, 210],
        [3, 1850, 200],
        [3, 1950, 220],
        [3, 1990, 250]
    ]
    
    z = [_[0] for _ in data_list]
    x = [_[1] for _ in data_list]
    y = [_[2] for _ in data_list]
    
    # Create a list of empty lists, one for each ID
    id_max = max(z)
    new_list = [[]]
    for _ in range(id_max):
        new_list.append([])
    
    # For each row, add the data to the sub-list for the row ID;
    #   ID 0 is not used.
    for index in range(len(z)):
        id = z[index]
        new_entry = [x[index], y[index]]
        print "Adding to index", id, new_entry
        new_list[id].append(new_entry)
    
    print new_list
    

new_list is now a list of lists, each sub-list having rows that share the same
id. There are more "Pythonic" ways to do some of these things, but I kept the
code to where I expect you can follow the logic.

Output, complete with trace:

    
    
    Adding to index 1 [1850, 150]
    [[], [[1850, 150]], [], []]
    Adding to index 1 [1950, 200]
    [[], [[1850, 150], [1950, 200]], [], []]
    Adding to index 1 [1990, 250]
    [[], [[1850, 150], [1950, 200], [1990, 250]], [], []]
    Adding to index 2 [1850, 130]
    [[], [[1850, 150], [1950, 200], [1990, 250]], [[1850, 130]], []]
    Adding to index 2 [1950, 180]
    [[], [[1850, 150], [1950, 200], [1990, 250]], [[1850, 130], [1950, 180]], []]
    Adding to index 2 [1990, 210]
    [[], [[1850, 150], [1950, 200], [1990, 250]], [[1850, 130], [1950, 180], [1990, 210]], []]
    Adding to index 3 [1850, 200]
    [[], [[1850, 150], [1950, 200], [1990, 250]], [[1850, 130], [1950, 180], [1990, 210]], [[1850, 200]]]
    Adding to index 3 [1950, 220]
    [[], [[1850, 150], [1950, 200], [1990, 250]], [[1850, 130], [1950, 180], [1990, 210]], [[1850, 200], [1950, 220]]]
    Adding to index 3 [1990, 250]
    [[], [[1850, 150], [1950, 200], [1990, 250]], [[1850, 130], [1950, 180], [1990, 210]], [[1850, 200], [1950, 220], [1990, 250]]]
    

