python_code/data_scraping.py
def pull_cases_by_court(court_id):
    """
    Given a court id, pull_cases_by_court will iterate through every cluster online, check if the case (cluster &
    opinion) corresponding to that cluster has been downloaded ( via is_downloaded() ) and will build up a list of
    [case_number, parent_directory] tuples that will enable quick iteration and creation of a dictionary of cases
    in other functions via Class_Case.json_to_case.

    :param court_id: court id by which to filter
    :return:
        case_numbers_and_parent_directories:
            list of the case numbers and the parents directories where they are located
    """
    start = time.time()

    # Initial info about the online clusters database
    clusters_dict = helper_functions.url_to_dict(clusters_url)
    num_to_inspect = clusters_dict['count']
    inspected = 0.
    page = starting_page

    # List of case numbers and parent directories that will eventually be returns
    case_numbers_and_parent_directories = []

    # Make sure we already have subdirectories for this court. Create them if not.
    subdirectories = [data_dir + r'/clusters/%s' % court_id, data_dir + r'/opinions/%s' % court_id]
    for subdir in subdirectories:
        if not os.path.isdir(subdir):
            os.makedirs(subdir)

    # As long as there are more pages to iterate through
    while clusters_dict['next'] is not None:

        # Iterate through each cluster on current page
        for individual_cluster in clusters_dict['results']:
            time.sleep(2)  # Make sure we don't overload the server
            inspected += 1.
            pct_inspected = inspected / num_to_inspect

            url_id_number = individual_cluster['resource_uri'].rsplit('/')[-2]
            new_cluster_url = site_dict_url + r'clusters/%s' % url_id_number
            new_cluster_dict = helper_functions.url_to_dict(new_cluster_url)

            new_cluster_docket_url = new_cluster_dict['docket']
            new_cluster_docket_dict = helper_functions.url_to_dict(new_cluster_docket_url)

            try:
                new_cluster_court = new_cluster_docket_dict['court'].rsplit('/')[-2]

                if new_cluster_court == court_id:
                    file_number = new_cluster_dict['citation_id']
                    parent_directory = court_id
                    case_numbers_and_parent_directories.append([file_number, parent_directory])

                    if not is_downloaded(file_number, parent_directory):
                        download_case(new_cluster_dict, file_number, parent_directory)
                    # print '%s DOWNLOADED. (%s)' % (file_number, pct_inspected)

                else:
                    pass
                    # print '%s passed. (%s)' % (new_cluster_dict['citation_id'], pct_inspected)

            except KeyError:  # If there is no court data
                pass

        print 'page %s checked. (%s)' % (page, pct_inspected)
        page += 1
        clusters_dict = helper_functions.url_to_dict(clusters_dict['next'])

    # Iterate through each cluster on last page
    for individual_cluster in clusters_dict['results']:
        time.sleep(2)  # Make sure we don't overload the server

        new_cluster_url = individual_cluster['cluster']
        new_cluster_dict = helper_functions.url_to_dict(new_cluster_url)
        new_cluster_docket_url = new_cluster_dict['dockets']
        new_cluster_docket_dict = helper_functions.url_to_dict(new_cluster_docket_url)
        try:
            new_cluster_court = new_cluster_docket_dict['court']

            if new_cluster_court == court_id:
                file_number = new_cluster_dict['citation_id']
                parent_directory = court_id
                case_numbers_and_parent_directories.append([file_number, parent_directory])

                if not is_downloaded(file_number, parent_directory):
                    download_case(new_cluster_dict, file_number, parent_directory)
        except KeyError:  # If there is no court data
            pass

    finish = time.time()
    elapsed = finish - start
    print "Download took %s minutes" % elapsed / 60.
    return case_numbers_and_parent_directories

    Given a court id, pull_cases_by_court will iterate through every cluster online, check if the case (cluster &
    opinion) corresponding to that cluster has been downloaded ( via is_downloaded() ) and will build up a list of
    [case_number, parent_directory] tuples that will enable quick iteration and creation of a dictionary of cases
    in other functions via Class_Case.json_to_case.

    :param court_id: court id by which to filter
    :return:
        case_numbers_and_parent_directories:
            list of the case numbers and the parents directories where they are located
    
Given a court id, pull_cases_by_court will iterate through every cluster online, check if the case (cluster &
opinion) corresponding to that cluster has been downloaded ( via is_downloaded() ) and will build up a list of
[case_number, parent_directory] tuples that will enable quick iteration and creation of a dictionary of cases
in other functions via Class_Case.json_to_case.
Given a court id pull_cases_by_court will iterate through every cluster online check if the case ( cluster & opinion ) corresponding to that cluster has been downloaded ( via is_downloaded () ) and will build up a list of [ case_number parent_directory ] tuples that will enable quick iteration and creation of a dictionary of cases in other functions via Class_Case . json_to_case .
def pull_cases_by_court(court_id):
    start = time.time()
    clusters_dict = helper_functions.url_to_dict(clusters_url)
    num_to_inspect = clusters_dict['count']
    inspected = 0.
    page = starting_page
    case_numbers_and_parent_directories = []
    subdirectories = [data_dir + r'/clusters/%s' % court_id, data_dir + r'/opinions/%s' % court_id]
    for subdir in subdirectories:
        if not os.path.isdir(subdir):
            os.makedirs(subdir)
    while clusters_dict['next'] is not None:
        for individual_cluster in clusters_dict['results']:
            time.sleep(2)  
            inspected += 1.
            pct_inspected = inspected / num_to_inspect
            url_id_number = individual_cluster['resource_uri'].rsplit('/')[-2]
            new_cluster_url = site_dict_url + r'clusters/%s' % url_id_number
            new_cluster_dict = helper_functions.url_to_dict(new_cluster_url)
            new_cluster_docket_url = new_cluster_dict['docket']
            new_cluster_docket_dict = helper_functions.url_to_dict(new_cluster_docket_url)
            try:
                new_cluster_court = new_cluster_docket_dict['court'].rsplit('/')[-2]
                if new_cluster_court == court_id:
                    file_number = new_cluster_dict['citation_id']
                    parent_directory = court_id
                    case_numbers_and_parent_directories.append([file_number, parent_directory])
                    if not is_downloaded(file_number, parent_directory):
                        download_case(new_cluster_dict, file_number, parent_directory)
                else:
                    pass
            except KeyError:  
                pass
        print 'page %s checked. (%s)' % (page, pct_inspected)
        page += 1
        clusters_dict = helper_functions.url_to_dict(clusters_dict['next'])
    for individual_cluster in clusters_dict['results']:
        time.sleep(2)  
        new_cluster_url = individual_cluster['cluster']
        new_cluster_dict = helper_functions.url_to_dict(new_cluster_url)
        new_cluster_docket_url = new_cluster_dict['dockets']
        new_cluster_docket_dict = helper_functions.url_to_dict(new_cluster_docket_url)
        try:
            new_cluster_court = new_cluster_docket_dict['court']
            if new_cluster_court == court_id:
                file_number = new_cluster_dict['citation_id']
                parent_directory = court_id
                case_numbers_and_parent_directories.append([file_number, parent_directory])
                if not is_downloaded(file_number, parent_directory):
                    download_case(new_cluster_dict, file_number, parent_directory)
        except KeyError:  
            pass
    finish = time.time()
    elapsed = finish - start
    print "Download took %s minutes" % elapsed / 60.
    return case_numbers_and_parent_directories
def pull_cases_by_court ( court_id ) : start = time . time ( ) clusters_dict = helper_functions . url_to_dict ( clusters_url ) num_to_inspect = clusters_dict [ 'count' ] inspected = 0. page = starting_page case_numbers_and_parent_directories = [ ] subdirectories = [ data_dir + r'/clusters/%s' % court_id , data_dir + r'/opinions/%s' % court_id ] for subdir in subdirectories : if not os . path . isdir ( subdir ) : os . makedirs ( subdir ) while clusters_dict [ 'next' ] is not None : for individual_cluster in clusters_dict [ 'results' ] : time . sleep ( 2 ) inspected += 1. pct_inspected = inspected / num_to_inspect url_id_number = individual_cluster [ 'resource_uri' ] . rsplit ( '/' ) [ - 2 ] new_cluster_url = site_dict_url + r'clusters/%s' % url_id_number new_cluster_dict = helper_functions . url_to_dict ( new_cluster_url ) new_cluster_docket_url = new_cluster_dict [ 'docket' ] new_cluster_docket_dict = helper_functions . url_to_dict ( new_cluster_docket_url ) try : new_cluster_court = new_cluster_docket_dict [ 'court' ] . rsplit ( '/' ) [ - 2 ] if new_cluster_court == court_id : file_number = new_cluster_dict [ 'citation_id' ] parent_directory = court_id case_numbers_and_parent_directories . append ( [ file_number , parent_directory ] ) if not is_downloaded ( file_number , parent_directory ) : download_case ( new_cluster_dict , file_number , parent_directory ) else : pass except KeyError : pass print 'page %s checked. (%s)' % ( page , pct_inspected ) page += 1 clusters_dict = helper_functions . url_to_dict ( clusters_dict [ 'next' ] ) for individual_cluster in clusters_dict [ 'results' ] : time . sleep ( 2 ) new_cluster_url = individual_cluster [ 'cluster' ] new_cluster_dict = helper_functions . url_to_dict ( new_cluster_url ) new_cluster_docket_url = new_cluster_dict [ 'dockets' ] new_cluster_docket_dict = helper_functions . url_to_dict ( new_cluster_docket_url ) try : new_cluster_court = new_cluster_docket_dict [ 'court' ] if new_cluster_court == court_id : file_number = new_cluster_dict [ 'citation_id' ] parent_directory = court_id case_numbers_and_parent_directories . append ( [ file_number , parent_directory ] ) if not is_downloaded ( file_number , parent_directory ) : download_case ( new_cluster_dict , file_number , parent_directory ) except KeyError : pass finish = time . time ( ) elapsed = finish - start print "Download took %s minutes" % elapsed / 60. return case_numbers_and_parent_directories
Given a court id, pull_cases_by_court will iterate through every cluster online, check if the case (cluster &
opinion) corresponding to that cluster has been downloaded ( via is_downloaded() ) and will build up a list of
[case_number, parent_directory] tuples that will enable quick iteration and creation of a dictionary of cases
in other functions via Class_Case.json_to_case.
Given a court id pull_cases_by_court will iterate through every cluster online check if the case ( cluster & opinion ) corresponding to that cluster has been downloaded ( via is_downloaded () ) and will build up a list of [ case_number parent_directory ] tuples that will enable quick iteration and creation of a dictionary of cases in other functions via Class_Case . json_to_case .
"""
    Given a court id, pull_cases_by_court will iterate through every cluster online, check if the case (cluster &
    opinion) corresponding to that cluster has been downloaded ( via is_downloaded() ) and will build up a list of
    [case_number, parent_directory] tuples that will enable quick iteration and creation of a dictionary of cases
    in other functions via Class_Case.json_to_case.

    :param court_id: court id by which to filter
    :return:
        case_numbers_and_parent_directories:
            list of the case numbers and the parents directories where they are located
    """ # Initial info about the online clusters database # List of case numbers and parent directories that will eventually be returns # Make sure we already have subdirectories for this court. Create them if not. # As long as there are more pages to iterate through # Iterate through each cluster on current page # Make sure we don't overload the server # print '%s DOWNLOADED. (%s)' % (file_number, pct_inspected) # print '%s passed. (%s)' % (new_cluster_dict['citation_id'], pct_inspected) # If there is no court data # Iterate through each cluster on last page # Make sure we don't overload the server # If there is no court data
list of the case numbers and the parents directories where they are located
list of the case numbers and the parents directories where they are located
court_id
court id by which to filter
court id by which to filter