Algorithm that receives a dictionary, converts it to a GET string, and is optimized for big data I found this question online as an example from a technical interview and it seems to be a flawed question in many ways. It made me curious how I would answer it.  So, If you were on a technical Python interview and asked to do the following:


  Write an algorithm that receives a dictionary, converts it to a GET string, and is optimized for big data.


Which option would you consider the best answer? Any other code related comments are welcome.

Common:

import requests
base_url = "https://api.github.com"
data = {'per_page': 10}
node = 'users/arctelix/repos'


Option 1:

My first thought was just answer the question in the simplest form and use pagination to control the size of the data returned.

def get_query_str(node, data=None):
    # base query
    query_str = "%s/%s" % (base_url, node)
    # build query params dict
    query_params = "&amp;".join(["%s=%s" % (k,str(v)) 
                             for k, v in data.items()])
    if query_params:
        query_str += "?%s" % query_params
    return query_str

print("\n--Option 1--\n")
url = get_query_str(node, data)
print("url = %s" % url)


Option 2:

Well, that's not really optimized for big data and the requests library will convert a dict to params for me. Secondly, a generator would be a great way to keep memory in check with very large data sets.

def get_resource(node, data=None):
    url = "%s/%s" % (base_url, node)
    print("geting resource : %s %s" % (url, data))
    resp = requests.get(url, params=data)
    json = resp.json()
    yield json

print("\n--Option 2--\n")
results = get_resource(node, data)
for r in results:
    print(r)


Option 3:

Just in case the interviewer was really looking to see if I knew how join() and a list comprehension could be used to convert a dictionary to a string of query parameters. Let's put it all together and use a generator for not only the pages, but the objects as well. get_query_str is totally unnecessary, but again the task was to write something that returned a "GET string"..

class Github:
    base_url = "https://api.github.com"

    def get_query_str(self, node, data=None):
        # base query
        query_str = "%s/%s" % (self.base_url, node)
        # build query params dict
        query_params = "&amp;".join(["%s=%s" % (k,str(v)) 
                                 for k, v in data.items()])
        if query_params:
            query_str += "?%s" % query_params
        return query_str

    def get(self, node, data=None):
        data = data or {}
        data['per_page'] = data.get('per_page', 50)
        page = range(0,data['per_page'])
        p=0
        while len(page) == data['per_page']:
            data['page'] = p
            query = self.get_query_str(node, data)
            page = list(self.req_resource(query))
            p += 1
            yield page

    def req_resource(self, query):
        print("geting resource : %s" % query)
        r = requests.get(query)
        j = r.json()
        yield j


gh = Github()
pages = gh.get(node, data)
print("\n--Option 3--\n")
for page in pages:
    for repo in page:
        print("repo=%s" % repo)
Considered as an interview question, my response would be, "what do you mean by 'optimized for big data'?" Wikipedia says,


  Big data is a term for data sets that are so large or complex that traditional data processing applications are inadequate to deal with them


and it's not clear how this is related to the problem of converting a dictionary to a query string (which would normally be solved by calling urllib.parse.urlencode). So there seems to be a hidden requirement here. Or possibly the interviewer is confused. Either way, you need to go through the requirements capture and analysis process before starting to write code.