convert list of tuple of tuple to list of tuple in pySpark Code:

def find_collinear(rdd):
    op = rdd.map( lambda x: (find_slope(x)[0][1],x) )
    op = op.groupByKey().mapValues(lambda x:[a for a in x])
    op = op.map(lambda x:x[1])
    return op

def find_slope(x):
    p1 = x[0]
    p2 = x[1]
    if p1[0] == p2[0] :
        slope = "inf"
    else:
       slope = (p2[1] - p1[1]) / (p2[0] - p1[0])
    t1 = tuple([x[0], slope])
    t2 = tuple([t1, x[1]])
    return t2

test_rdd = sc.parallelize(
    [((4, 2), (2, 1)), ((4, 2), (-3, 4)), ((4, 2), (6, 3)),
     ((2, 1), (4, 2)), ((2, 1), (-3, 4)), ((2, 1), (6, 3)),
     ((-3, 4), (4, 2)), ((-3, 4), (2, 1)), ((-3, 4), (6, 3)),
     ((6, 3), (4, 2)), ((6, 3), (2, 1)), ((6, 3), (-3, 4))])

temp1 = find_collinear(test_rdd).collect()


Output

[[((4, 2), (2, 1)), ((4, 2), (6, 3)), 
  ((2, 1), (4, 2)), ((2, 1), (6, 3)),  
  ((6, 3), (4, 2)),  ((6, 3), (2, 1))], 
 [((4, 2), (-3, 4)), ((-3, 4), (4, 2))], 
 [((2, 1), (-3, 4)), ((-3, 4), (2, 1))], 
 [((-3, 4), (6, 3)), ((6, 3), (-3, 4))]
]


Expect output:

[((6, 3), (4, 2), (2, 1)), ((4, 2), (-3, 4)), ((-3, 4), (2, 1)), ((6, 3), (-3, 4))]


How can I get the expected output from/instead of the actual.
To get the unique elements you can convert the tuples to a set with a couple of comprehensions like:

Code:

[tuple({t for y in x for t in y}) for x in data]


How:

Inside of a list comprehension, this code creates a set via a set comprehension {}. This will gather up the unique tuples. Two loops are needed inside of the set comprehension:

for y in x for t in y


because the tuples of interest are themselves inside of a tuple.   

Test Code:

data = [
    [
        ((4, 2), (2, 1)),
        ((4, 2), (6, 3)),
        ((2, 1), (4, 2)),
        ((2, 1), (6, 3)),
        ((6, 3), (4, 2)),
        ((6, 3), (2, 1))
    ], [
        ((4, 2), (-3, 4)),
        ((-3, 4), (4, 2))
    ], [
        ((2, 1), (-3, 4)),
        ((-3, 4), (2, 1))
    ], [
        ((-3, 4), (6, 3)),
        ((6, 3), (-3, 4))
    ]
]

expected = [
    ((6, 3), (4, 2), (2, 1)),
    ((4, 2), (-3, 4)),
    ((-3, 4), (2, 1)),
    ((6, 3), (-3, 4))
 ]

assert expected == [tuple({t for y in x for t in y}) for x in data]