问:
What is the best practice for creating a View from a Single Lookup Table?
I have a fairly large table mostly consisting of lookup id's referencing a single lookup table. I understand that this is generally not considered best practice, but unfortunately that aspect is outside of my control.

What I wish to do is create a View that converts the lookup id's into their corresponding descriptions (25-30 lookup columns in this particular table). My options, from what I've read, seem to be the following. My question is which of these (or any other method I'm not thinking of) would provide the best combination of of performance, readability and extensibility.

Simply creating a Left JOIN for each column with a separate table alias.

Inline Table-Valued Function using a subquery to return the desired lookup

Inline Scalar-Valued Function using the new 2019 SQL Server optimizations for Scalar UDF's.

1 seems to be the most straightforward, but would be cumbersome to maintain and read, and wouldn't be able to translate to other tables I need this functionality for.

2 seems to be more extensible but not by much and I'm not sure how much performance would suffer compared to #1.

3 is my preference, as it's extremely extensible and makes the view much more readable. However I understand this is a new addition to MSSQL so I'm doubly concerned of potential performance issues vs 1.

Does anyone have any input/advice on this particular approach?

EDIT: Here's an example that highlights the potential burden of using straight joins. This particular view required me to incorporate a combination of short codes and descriptions into one. I think an inline UDF would make this much more readable and flexible, but before I go through all the work of converting it, I just wanted to get a feel of if it would even be a practical option...

```<code>SELECT dbo.Clients.client_id, dbo.Clients.ssn, dbo.ClientEpisode.admission_date, dbo.Clients.rin, ISNULL(admission_type.lookup_code, admit_type.lookup_code)
                     AS admission_type, dbo.ClientEpisode.initial_contact_date, dbo.ClientEpisode.assessment_date, dbo.Clients.first_name, dbo.Clients.last_name,
                     dbo.Clients.middle_initial, dbo.Clients.dob, dbo.Clients.address_1 + ' ' + ISNULL(dbo.Clients.address_2, '') AS address, dbo.Clients.city, dbo.Clients.state,
                     dbo.Clients.zip_code, geocode.lookup_code AS geocode, race.lookup_code + ' - ' + race.lookup_desc AS race,
                     hispanic_origin.lookup_code + ' - ' + hispanic_origin.lookup_desc AS hispanic_origin, dbo.Clients.gender, dbo.Clients.veteran_status,
                     marital_status.lookup_code + ' - ' + marital_status.lookup_desc AS marital_status, dbo.Clients.family_size, dbo.Clients.num_receiving_child_support,
                     dbo.Clients.num_of_children, dbo.Clients.num_removed_from_custody, pregnant.lookup_code + ' - ' + pregnant.lookup_desc AS pregnant,
                     livingarrange.lookup_code + ' - ' + livingarrange.lookup_desc AS livingarrange, paysource.lookup_code + ' - ' + paysource.lookup_desc AS paysource,
                     medicaidmco.lookup_code + ' - ' + medicaidmco.lookup_desc AS medicaidmco, empstatus.lookup_code + ' - ' + empstatus.lookup_desc AS empstatus,
                     unemployment.lookup_code + ' - ' + unemployment.lookup_desc AS unemployment, schooljob.lookup_code + ' - ' + schooljob.lookup_desc AS schooljob,
                     dbo.Clients.highest_grade_completed AS education, dbo.Clients.annual_income, income_source.lookup_code + ' - ' + income_source.lookup_desc AS income_source,
                      hearing_status.lookup_code + ' - ' + hearing_status.lookup_desc AS hearing_status,
                     family_income_override.lookup_code + ' - ' + family_income_override.lookup_desc AS family_income_override,
                     preferred_language.lookup_code + ' - ' + preferred_language.lookup_desc AS preferred_language, dbo.Clients.english_speaking,
                     priorepisodes.lookup_code + ' - ' + priorepisodes.lookup_desc AS prior_treatment_episodes,
                     referral_source.lookup_code + ' - ' + referral_source.lookup_desc AS referral_source, referral_office.lookup_code + ' - ' + referral_office.lookup_desc AS referral_office,
                      dbo.ClientEpisode.dcfs_involvement, dbo.ClientEpisode.misa_involvement, dbo.ClientAdmissionData.numarrests, dbo.ClientAdmissionData.numgroups,
                     attendgroups.lookup_code AS attendgroups, dbo.Clients.support_in_recovery, problemarea.lookup_code + ' - ' + problemarea.lookup_desc AS problemarea,
                     primdrug.lookup_code + ' - ' + primdrug.lookup_desc AS primdrug, primdrugfreq.lookup_code + ' - ' + primdrugfreq.lookup_desc AS primdrugfreq,
                     primdrugrout.lookup_code + ' - ' + primdrugrout.lookup_desc AS primdrugrout, primdrugsevr.lookup_code + ' - ' + primdrugsevr.lookup_desc AS primdrugsevr,
                     dbo.ClientAdmissionData.primdrugage, secdrug.lookup_code + ' - ' + secdrug.lookup_desc AS secdrug,
                     secdrugfreq.lookup_code + ' - ' + secdrugfreq.lookup_desc AS secdrugfreq, secdrugrout.lookup_code + ' - ' + secdrugrout.lookup_desc AS secdrugrout,
                     secdrugsevr.lookup_code + ' - ' + secdrugsevr.lookup_desc AS secdrugsevr, dbo.ClientAdmissionData.secdrugage,
                     tertdrug.lookup_code + ' - ' + tertdrug.lookup_desc AS tertdrug, tertdrugfreq.lookup_code + ' - ' + tertdrugfreq.lookup_desc AS tertdrugfreq,
                     tertdrugrout.lookup_code + ' - ' + tertdrugrout.lookup_desc AS tertdrugrout, tertdrugsevr.lookup_code + ' - ' + tertdrugsevr.lookup_desc AS tertdrugsevr,
                     dbo.ClientAdmissionData.tertdrugage, dbo.ClientAdmissionData.currentlevel, dbo.ClientAdmissionData.leveldate, dbo.ClientAdmissionData.formtype,
                     dbo.ClientAdmissionData.date_entered, dbo.ClientAdmissionData.emp_name FROM dbo.ClientEpisode INNER JOIN
                     dbo.ClientAdmissionData ON dbo.ClientEpisode.episode_id = dbo.ClientAdmissionData.episode_id INNER JOIN
                     dbo.Clients ON dbo.ClientAdmissionData.client_id = dbo.Clients.client_id LEFT OUTER JOIN
                     dbo.LookupDict AS geocode ON dbo.ClientAdmissionData.geocode = geocode.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS admission_type ON dbo.ClientEpisode.admission_type = admission_type.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS admit_type ON dbo.ClientAdmissionData.admittype = admit_type.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS race ON dbo.Clients.race = race.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS hispanic_origin ON dbo.Clients.hispanic_origin = hispanic_origin.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS pregnant ON dbo.ClientAdmissionData.pregnant = pregnant.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS livingarrange ON dbo.ClientAdmissionData.livingarrang = livingarrange.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS medicaidmco ON dbo.ClientEpisode.medicaid_mco_provider = medicaidmco.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS paysource ON dbo.ClientEpisode.pay_source = paysource.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS empstatus ON dbo.ClientAdmissionData.employment = empstatus.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS unemployment ON dbo.ClientAdmissionData.unemployment = unemployment.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS schooljob ON dbo.ClientAdmissionData.schooljob = schooljob.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS family_income_override ON dbo.Clients.family_income_override = family_income_override.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS income_source ON dbo.Clients.income_source = income_source.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS preferred_language ON dbo.Clients.preferred_language = preferred_language.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS hearing_status ON dbo.Clients.hearing_status = hearing_status.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS referral_source ON dbo.ClientEpisode.referral_source = referral_source.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS referral_office ON dbo.ClientEpisode.referral_office = referral_office.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS priorepisodes ON dbo.Clients.prior_treatment_episodes = priorepisodes.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS problemarea ON dbo.ClientAdmissionData.problemarea = problemarea.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS primdrug ON dbo.ClientAdmissionData.primdrug = primdrug.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS primdrugfreq ON dbo.ClientAdmissionData.primdrugfreq = primdrugfreq.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS primdrugrout ON dbo.ClientAdmissionData.primdrugrout = primdrugrout.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS primdrugsevr ON dbo.ClientAdmissionData.primdrugsevr = primdrugsevr.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS secdrug ON dbo.ClientAdmissionData.secdrug = secdrug.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS secdrugfreq ON dbo.ClientAdmissionData.secdrugfreq = secdrugfreq.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS secdrugrout ON dbo.ClientAdmissionData.secdrugrout = secdrugrout.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS secdrugsevr ON dbo.ClientAdmissionData.secdrugsevr = secdrugsevr.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS tertdrug ON dbo.ClientAdmissionData.tertdrug = tertdrug.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS tertdrugfreq ON dbo.ClientAdmissionData.tertdrugfreq = tertdrugfreq.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS tertdrugrout ON dbo.ClientAdmissionData.tertdrugrout = tertdrugrout.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS tertdrugsevr ON dbo.ClientAdmissionData.tertdrugsevr = tertdrugsevr.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS marital_status ON dbo.Clients.marital_status = marital_status.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS attendgroups ON dbo.ClientAdmissionData.attendgroups = attendgroups.lookup_id
</code>```

```<code>
</code>```

答:
I am clearly missing something because I don't see how a scalar UDF would be more "extensible" -- unless you mean that you can change the data model and not change the view.

I don't see what is cumbersome about using left joins for this purpose, so a view seems like an obvious solution.  If you do use scalar UDF, then computed columns using those UDFs seems like a good approach.

That said, there are two more important considerations.  If performance is an issue, then try out the different approaches (on reasonably-sized data) to see which is best.  Second, what are you most comfortable with?  You will need to write, maintain, document, and test the code.

Body:
I have a fairly large table mostly consisting of lookup id's referencing a single lookup table. I understand that this is generally not considered best practice, but unfortunately that aspect is outside of my control.

What I wish to do is create a View that converts the lookup id's into their corresponding descriptions (25-30 lookup columns in this particular table). My options, from what I've read, seem to be the following. My question is which of these (or any other method I'm not thinking of) would provide the best combination of of performance, readability and extensibility.

Simply creating a Left JOIN for each column with a separate table alias.

Inline Table-Valued Function using a subquery to return the desired lookup

Inline Scalar-Valued Function using the new 2019 SQL Server optimizations for Scalar UDF's.

1 seems to be the most straightforward, but would be cumbersome to maintain and read, and wouldn't be able to translate to other tables I need this functionality for.

2 seems to be more extensible but not by much and I'm not sure how much performance would suffer compared to #1.

3 is my preference, as it's extremely extensible and makes the view much more readable. However I understand this is a new addition to MSSQL so I'm doubly concerned of potential performance issues vs 1.

Does anyone have any input/advice on this particular approach?

EDIT: Here's an example that highlights the potential burden of using straight joins. This particular view required me to incorporate a combination of short codes and descriptions into one. I think an inline UDF would make this much more readable and flexible, but before I go through all the work of converting it, I just wanted to get a feel of if it would even be a practical option...

SELECT dbo.Clients.client_id, dbo.Clients.ssn, dbo.ClientEpisode.admission_date, dbo.Clients.rin, ISNULL(admission_type.lookup_code, admit_type.lookup_code)
                     AS admission_type, dbo.ClientEpisode.initial_contact_date, dbo.ClientEpisode.assessment_date, dbo.Clients.first_name, dbo.Clients.last_name,
                     dbo.Clients.middle_initial, dbo.Clients.dob, dbo.Clients.address_1 + ' ' + ISNULL(dbo.Clients.address_2, '') AS address, dbo.Clients.city, dbo.Clients.state,
                     dbo.Clients.zip_code, geocode.lookup_code AS geocode, race.lookup_code + ' - ' + race.lookup_desc AS race,
                     hispanic_origin.lookup_code + ' - ' + hispanic_origin.lookup_desc AS hispanic_origin, dbo.Clients.gender, dbo.Clients.veteran_status,
                     marital_status.lookup_code + ' - ' + marital_status.lookup_desc AS marital_status, dbo.Clients.family_size, dbo.Clients.num_receiving_child_support,
                     dbo.Clients.num_of_children, dbo.Clients.num_removed_from_custody, pregnant.lookup_code + ' - ' + pregnant.lookup_desc AS pregnant,
                     livingarrange.lookup_code + ' - ' + livingarrange.lookup_desc AS livingarrange, paysource.lookup_code + ' - ' + paysource.lookup_desc AS paysource,
                     medicaidmco.lookup_code + ' - ' + medicaidmco.lookup_desc AS medicaidmco, empstatus.lookup_code + ' - ' + empstatus.lookup_desc AS empstatus,
                     unemployment.lookup_code + ' - ' + unemployment.lookup_desc AS unemployment, schooljob.lookup_code + ' - ' + schooljob.lookup_desc AS schooljob,
                     dbo.Clients.highest_grade_completed AS education, dbo.Clients.annual_income, income_source.lookup_code + ' - ' + income_source.lookup_desc AS income_source,
                      hearing_status.lookup_code + ' - ' + hearing_status.lookup_desc AS hearing_status,
                     family_income_override.lookup_code + ' - ' + family_income_override.lookup_desc AS family_income_override,
                     preferred_language.lookup_code + ' - ' + preferred_language.lookup_desc AS preferred_language, dbo.Clients.english_speaking,
                     priorepisodes.lookup_code + ' - ' + priorepisodes.lookup_desc AS prior_treatment_episodes,
                     referral_source.lookup_code + ' - ' + referral_source.lookup_desc AS referral_source, referral_office.lookup_code + ' - ' + referral_office.lookup_desc AS referral_office,
                      dbo.ClientEpisode.dcfs_involvement, dbo.ClientEpisode.misa_involvement, dbo.ClientAdmissionData.numarrests, dbo.ClientAdmissionData.numgroups,
                     attendgroups.lookup_code AS attendgroups, dbo.Clients.support_in_recovery, problemarea.lookup_code + ' - ' + problemarea.lookup_desc AS problemarea,
                     primdrug.lookup_code + ' - ' + primdrug.lookup_desc AS primdrug, primdrugfreq.lookup_code + ' - ' + primdrugfreq.lookup_desc AS primdrugfreq,
                     primdrugrout.lookup_code + ' - ' + primdrugrout.lookup_desc AS primdrugrout, primdrugsevr.lookup_code + ' - ' + primdrugsevr.lookup_desc AS primdrugsevr,
                     dbo.ClientAdmissionData.primdrugage, secdrug.lookup_code + ' - ' + secdrug.lookup_desc AS secdrug,
                     secdrugfreq.lookup_code + ' - ' + secdrugfreq.lookup_desc AS secdrugfreq, secdrugrout.lookup_code + ' - ' + secdrugrout.lookup_desc AS secdrugrout,
                     secdrugsevr.lookup_code + ' - ' + secdrugsevr.lookup_desc AS secdrugsevr, dbo.ClientAdmissionData.secdrugage,
                     tertdrug.lookup_code + ' - ' + tertdrug.lookup_desc AS tertdrug, tertdrugfreq.lookup_code + ' - ' + tertdrugfreq.lookup_desc AS tertdrugfreq,
                     tertdrugrout.lookup_code + ' - ' + tertdrugrout.lookup_desc AS tertdrugrout, tertdrugsevr.lookup_code + ' - ' + tertdrugsevr.lookup_desc AS tertdrugsevr,
                     dbo.ClientAdmissionData.tertdrugage, dbo.ClientAdmissionData.currentlevel, dbo.ClientAdmissionData.leveldate, dbo.ClientAdmissionData.formtype,
                     dbo.ClientAdmissionData.date_entered, dbo.ClientAdmissionData.emp_name FROM dbo.ClientEpisode INNER JOIN
                     dbo.ClientAdmissionData ON dbo.ClientEpisode.episode_id = dbo.ClientAdmissionData.episode_id INNER JOIN
                     dbo.Clients ON dbo.ClientAdmissionData.client_id = dbo.Clients.client_id LEFT OUTER JOIN
                     dbo.LookupDict AS geocode ON dbo.ClientAdmissionData.geocode = geocode.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS admission_type ON dbo.ClientEpisode.admission_type = admission_type.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS admit_type ON dbo.ClientAdmissionData.admittype = admit_type.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS race ON dbo.Clients.race = race.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS hispanic_origin ON dbo.Clients.hispanic_origin = hispanic_origin.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS pregnant ON dbo.ClientAdmissionData.pregnant = pregnant.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS livingarrange ON dbo.ClientAdmissionData.livingarrang = livingarrange.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS medicaidmco ON dbo.ClientEpisode.medicaid_mco_provider = medicaidmco.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS paysource ON dbo.ClientEpisode.pay_source = paysource.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS empstatus ON dbo.ClientAdmissionData.employment = empstatus.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS unemployment ON dbo.ClientAdmissionData.unemployment = unemployment.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS schooljob ON dbo.ClientAdmissionData.schooljob = schooljob.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS family_income_override ON dbo.Clients.family_income_override = family_income_override.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS income_source ON dbo.Clients.income_source = income_source.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS preferred_language ON dbo.Clients.preferred_language = preferred_language.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS hearing_status ON dbo.Clients.hearing_status = hearing_status.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS referral_source ON dbo.ClientEpisode.referral_source = referral_source.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS referral_office ON dbo.ClientEpisode.referral_office = referral_office.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS priorepisodes ON dbo.Clients.prior_treatment_episodes = priorepisodes.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS problemarea ON dbo.ClientAdmissionData.problemarea = problemarea.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS primdrug ON dbo.ClientAdmissionData.primdrug = primdrug.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS primdrugfreq ON dbo.ClientAdmissionData.primdrugfreq = primdrugfreq.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS primdrugrout ON dbo.ClientAdmissionData.primdrugrout = primdrugrout.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS primdrugsevr ON dbo.ClientAdmissionData.primdrugsevr = primdrugsevr.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS secdrug ON dbo.ClientAdmissionData.secdrug = secdrug.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS secdrugfreq ON dbo.ClientAdmissionData.secdrugfreq = secdrugfreq.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS secdrugrout ON dbo.ClientAdmissionData.secdrugrout = secdrugrout.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS secdrugsevr ON dbo.ClientAdmissionData.secdrugsevr = secdrugsevr.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS tertdrug ON dbo.ClientAdmissionData.tertdrug = tertdrug.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS tertdrugfreq ON dbo.ClientAdmissionData.tertdrugfreq = tertdrugfreq.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS tertdrugrout ON dbo.ClientAdmissionData.tertdrugrout = tertdrugrout.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS tertdrugsevr ON dbo.ClientAdmissionData.tertdrugsevr = tertdrugsevr.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS marital_status ON dbo.Clients.marital_status = marital_status.lookup_id LEFT OUTER JOIN
                     dbo.LookupDict AS attendgroups ON dbo.ClientAdmissionData.attendgroups = attendgroups.lookup_id


来源: stackexchange

问:
ReportViewer - how to filter on Multi Value Parameter
I have a report that uses a Multi Value parameter. I have added this to the report and can specify the parameter values in code.

Now I want to filter the data on the report by the multiple value parameter. Eg, I pass in a list of account no's and I want to filter the data on the report so only records with the passed in account no's are shown.

In the tablix properties there is a filters option which looks like the place I need to set up the filters.

I have added a new one, selected the Account Number column on the report. The operator that seems relevant to me is the 'in' operator. So show records that a 'in' this list. However, If i select the 'in' filter the expression text box is disabled. Is this what im meant to use?

答:
Found it:

You should filter on The name of the parameter rather than an expression.

eg:

```<code>[@CostCentres]
</code>```

rather than:

```<code>=Parameters!CostCentres.Value
</code>```

ContentLicense: CC BY-SA 3.0

Body:
Found it:

You should filter on The name of the parameter rather than an expression.

eg:

[@CostCentres]

rather than:

=Parameters!CostCentres.Value


问:
CoreData performance using NSFetchedResultsController. UITableView freezes during data import
I have the app similar to (almost the same code):
https://developer.apple.com/documentation/coredata/loading_and_displaying_a_large_data_feed

The app loads and creates/updates 100000+ records from JSON to CoreData at launch.

App only consists of the screen with single tableview (with data from import).
NSFetchedResultsController is used to show data in table view.

When app imports data in database the UI freezes, I try to scroll the tableview it also freezes even at small scrolls.

I see 100% load of the main thread during the import, so UI will freeze.

PrivateQueueConcurrencyType contexts(used for bg batch saves) have mainQueueConcurrencyType as parent.
mainQueueConcurrencyType have privateQueueConcurrencyType as parent with persistentStoreCoordinator connected to it. (as described https://medium.com/soundwave-stories/core-data-cffe22efe716)

Is it possible somehow to make import process in background(on the other thread) and not to block main thread in this case?

How not to affect main thread almost at all during large import?

Code:

Core data stack

```<code>private lazy var managedObjectModel: NSManagedObjectModel = {
    guard let modelURL = Bundle.main.url(forResource: "Earthquakes", withExtension: "momd") else {
        fatalError("Unable to Find Data Model")
    }
    guard let managedObjectModel = NSManagedObjectModel(contentsOf: modelURL) else {
        fatalError("Unable to Load Data Model")
    }

    return managedObjectModel
}()

private lazy var persistentStoreCoordinator: NSPersistentStoreCoordinator = {
    let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: managedObjectModel)
    let fileManager = FileManager.default
    let storeName = "test.sqlite"
    let documentsDirectoryURL = fileManager.urls(for: .documentDirectory, in: .userDomainMask)[0]
    let persistentStoreURL = documentsDirectoryURL.appendingPathComponent(storeName)

    do {
        try persistentStoreCoordinator.addPersistentStore(ofType: NSSQLiteStoreType,
                                                          configurationName: nil,
                                                          at: persistentStoreURL,
                                                          options: nil)
    } catch {
        fatalError("Unable to Load Persistent Store")
    }

    return persistentStoreCoordinator
}()

 lazy var parentContext: NSManagedObjectContext = {
    let moc = NSManagedObjectContext(concurrencyType: .privateQueueConcurrencyType)
    moc.persistentStoreCoordinator = persistentStoreCoordinator
    return moc
}()

 lazy var context: NSManagedObjectContext = {
    let moc = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
    moc.parent = parentContext
    return moc
}()

 /**
 Fetches the earthquake feed from the remote server, and imports it into Core Data.

 Because this server does not offer a secure communication channel, this example
 uses an http URL and adds "earthquake.usgs.gov" to the "NSExceptionDomains" value
 in the apps's info.plist. When you commmunicate with your own servers, or when
 the services you use offer a secure communication option, you should always
 prefer to use https.
*/
func fetchQuakes(completionHandler: @escaping (Error?) -> Void) {

    // Create a URL to load, and a URLSession to load it.
    guard let jsonURL = URL(string: earthquakesFeed) else {
        completionHandler(QuakeError.urlError)
        return
    }
    let session = URLSession(configuration: .default)

    // Create a URLSession dataTask to fetch the feed.
    let task = session.dataTask(with: jsonURL) { data, _, error in

        // Alert the user if no data comes back.
        guard let data = data else {
            completionHandler(QuakeError.networkUnavailable)
            return
        }

        // Decode the JSON and import it into Core Data.
        do {
            // Decode the JSON into codable type GeoJSON.
            let decoder = JSONDecoder()
            var geoJSON = try decoder.decode(GeoJSON.self, from: data)

            geoJSON.quakePropertiesArray.append(contentsOf: geoJSON.quakePropertiesArray)
            geoJSON.quakePropertiesArray.append(contentsOf: geoJSON.quakePropertiesArray)
            geoJSON.quakePropertiesArray.append(contentsOf: geoJSON.quakePropertiesArray)
            geoJSON.quakePropertiesArray.append(contentsOf: geoJSON.quakePropertiesArray)
            print(geoJSON.quakePropertiesArray.count)

            // Import the GeoJSON into Core Data.
            try self.importQuakes(from: geoJSON)

        } catch {
            // Alert the user if data cannot be digested.
            completionHandler(QuakeError.wrongDataFormat)
            return
        }
        completionHandler(nil)
    }
    // Start the task.
    task.resume()
}

/**
 Imports a JSON dictionary into the Core Data store on a private queue,
 processing the record in batches to avoid a high memory footprint.
*/
private func importQuakes(from geoJSON: GeoJSON) throws {

    guard !geoJSON.quakePropertiesArray.isEmpty else { return }

    // Create a private queue context.
    //let taskContext = persistentContainer.newBackgroundContext()
    let taskContext = NSManagedObjectContext(concurrencyType: .privateQueueConcurrencyType)
    taskContext.parent = context

    taskContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
    // Set unused undoManager to nil for macOS (it is nil by default on iOS)
    // to reduce resource requirements.
    taskContext.undoManager = nil

    // Process records in batches to avoid a high memory footprint.
    let batchSize = 256
    let count = geoJSON.quakePropertiesArray.count

    // Determine the total number of batches.
    var numBatches = count / batchSize
    numBatches += count % batchSize > 0 ? 1 : 0

    for batchNumber in 0 ..< numBatches {

        // Determine the range for this batch.
        let batchStart = batchNumber * batchSize
        let batchEnd = batchStart + min(batchSize, count - batchNumber * batchSize)
        let range = batchStart..<batchEnd

        // Create a batch for this range from the decoded JSON.
        let quakesBatch = Array(geoJSON.quakePropertiesArray[range])

        // Stop the entire import if any batch is unsuccessful.
        if !importOneBatch(quakesBatch, taskContext: taskContext) {
            return
        }

    }
}

/**
 Imports one batch of quakes, creating managed objects from the new data,
 and saving them to the persistent store, on a private queue. After saving,
 resets the context to clean up the cache and lower the memory footprint.

 NSManagedObjectContext.performAndWait doesn't rethrow so this function
 catches throws within the closure and uses a return value to indicate
 whether the import is successful.
*/
private func importOneBatch(_ quakesBatch: [QuakeProperties], taskContext: NSManagedObjectContext) -> Bool {

    var success = false

    // taskContext.performAndWait runs on the URLSession's delegate queue
    // so it won’t block the main thread.
    taskContext.performAndWait {
        // Create a new record for each quake in the batch.
        for quakeData in quakesBatch {

            // Create a Quake managed object on the private queue context.
            guard let quake = NSEntityDescription.insertNewObject(forEntityName: "Quake", into: taskContext) as? Quake else {
                print(QuakeError.creationError.localizedDescription)
                return
            }
            // Populate the Quake's properties using the raw data.
            do {
                try quake.update(with: quakeData)
            } catch QuakeError.missingData {
                // Delete invalid Quake from the private queue context.
                print(QuakeError.missingData.localizedDescription)
                taskContext.delete(quake)
            } catch {
                print(error.localizedDescription)
            }
        }

        // Save all insertions and deletions from the context to the store.
        if taskContext.hasChanges {
            do {
                try taskContext.save()
                context.performAndWait {
                    try? context.save()
                }

            } catch {
                print("Error: \(error)\nCould not save Core Data context.")
                return
            }
            // Reset the taskContext to free the cache and lower the memory footprint.
            taskContext.reset()

        }

        success = true
    }
    return success
}

    // MARK: - NSFetchedResultsController

/**
 A fetched results controller delegate to give consumers a chance to update
 the user interface when content changes.
 */
weak var fetchedResultsControllerDelegate: NSFetchedResultsControllerDelegate?

/**
 A fetched results controller to fetch Quake records sorted by time.
 */
lazy var fetchedResultsController: NSFetchedResultsController<Quake> = {

    // Create a fetch request for the Quake entity sorted by time.
    let fetchRequest = NSFetchRequest<Quake>(entityName: "Quake")
    fetchRequest.sortDescriptors = [NSSortDescriptor(key: "time", ascending: false)]

    // Create a fetched results controller and set its fetch request, context, and delegate.
    let controller = NSFetchedResultsController(fetchRequest: fetchRequest,
                                                managedObjectContext: context,
                                                sectionNameKeyPath: nil, cacheName: nil)

    controller.delegate = fetchedResultsControllerDelegate

    // Perform the fetch.
    do {
        try controller.performFetch()
    } catch {
        fatalError("Unresolved error \(error)")
    }

    return controller
}()
</code>```

ViewController code:

```<code>// MARK: - UITableViewDataSource

extension QuakesViewController {

    override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {

        guard let cell = tableView.dequeueReusableCell(withIdentifier: "QuakeCell", for: indexPath) as? QuakeCell else {
            print("Error: tableView.dequeueReusableCell doesn'return a QuakeCell!")
            return QuakeCell()
        }
        guard let quake = dataProvider.fetchedResultsController.fetchedObjects?[indexPath.row] else { return cell }

        cell.configure(with: quake)
        return cell
    }

    override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        return dataProvider.fetchedResultsController.fetchedObjects?.count ?? 0
    }
}

// MARK: - NSFetchedResultsControllerDelegate

extension QuakesViewController: NSFetchedResultsControllerDelegate {

    /**
     Reloads the table view when the fetched result controller's content changes.
     */
    func controllerDidChangeContent(_ controller: NSFetchedResultsController<NSFetchRequestResult>) {
        tableView.reloadData()
    }
}
</code>```

答:
All the UI are performed at main thread, so you can try to use a global queue to perform this fetch.

I think that a global queue with .utility QoS will be enough to solve this problem.

Here a sample:

```<code>DispatchQueue.global(qos: .utility).async {
            //do the fetch here
}
</code>```

The global queues are concurrent, the utility QoS (quality of service) tells the GCD that this task should not perform in main thread and the .async make the execution doesn't block this global queue.

If it's not totally clear i can post a most complete sample code for you :)

Good luck!

ContentLicense: CC BY-SA 4.0

Body:
I have the app similar to (almost the same code):
https://developer.apple.com/documentation/coredata/loading_and_displaying_a_large_data_feed

The app loads and creates/updates 100000+ records from JSON to CoreData at launch.

App only consists of the screen with single tableview (with data from import).
NSFetchedResultsController is used to show data in table view.

When app imports data in database the UI freezes, I try to scroll the tableview it also freezes even at small scrolls.

I see 100% load of the main thread during the import, so UI will freeze.

PrivateQueueConcurrencyType contexts(used for bg batch saves) have mainQueueConcurrencyType as parent.
mainQueueConcurrencyType have privateQueueConcurrencyType as parent with persistentStoreCoordinator connected to it. (as described https://medium.com/soundwave-stories/core-data-cffe22efe716)

Is it possible somehow to make import process in background(on the other thread) and not to block main thread in this case?

How not to affect main thread almost at all during large import?

Code:

Core data stack

private lazy var managedObjectModel: NSManagedObjectModel = {
    guard let modelURL = Bundle.main.url(forResource: "Earthquakes", withExtension: "momd") else {
        fatalError("Unable to Find Data Model")
    }
    guard let managedObjectModel = NSManagedObjectModel(contentsOf: modelURL) else {
        fatalError("Unable to Load Data Model")
    }

    return managedObjectModel
}()

private lazy var persistentStoreCoordinator: NSPersistentStoreCoordinator = {
    let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: managedObjectModel)
    let fileManager = FileManager.default
    let storeName = "test.sqlite"
    let documentsDirectoryURL = fileManager.urls(for: .documentDirectory, in: .userDomainMask)[0]
    let persistentStoreURL = documentsDirectoryURL.appendingPathComponent(storeName)

    do {
        try persistentStoreCoordinator.addPersistentStore(ofType: NSSQLiteStoreType,
                                                          configurationName: nil,
                                                          at: persistentStoreURL,
                                                          options: nil)
    } catch {
        fatalError("Unable to Load Persistent Store")
    }

    return persistentStoreCoordinator
}()

 lazy var parentContext: NSManagedObjectContext = {
    let moc = NSManagedObjectContext(concurrencyType: .privateQueueConcurrencyType)
    moc.persistentStoreCoordinator = persistentStoreCoordinator
    return moc
}()

 lazy var context: NSManagedObjectContext = {
    let moc = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
    moc.parent = parentContext
    return moc
}()

 /**
 Fetches the earthquake feed from the remote server, and imports it into Core Data.

 Because this server does not offer a secure communication channel, this example
 uses an http URL and adds "earthquake.usgs.gov" to the "NSExceptionDomains" value
 in the apps's info.plist. When you commmunicate with your own servers, or when
 the services you use offer a secure communication option, you should always
 prefer to use https.
*/
func fetchQuakes(completionHandler: @escaping (Error?) -> Void) {

    // Create a URL to load, and a URLSession to load it.
    guard let jsonURL = URL(string: earthquakesFeed) else {
        completionHandler(QuakeError.urlError)
        return
    }
    let session = URLSession(configuration: .default)

    // Create a URLSession dataTask to fetch the feed.
    let task = session.dataTask(with: jsonURL) { data, _, error in

        // Alert the user if no data comes back.
        guard let data = data else {
            completionHandler(QuakeError.networkUnavailable)
            return
        }

        // Decode the JSON and import it into Core Data.
        do {
            // Decode the JSON into codable type GeoJSON.
            let decoder = JSONDecoder()
            var geoJSON = try decoder.decode(GeoJSON.self, from: data)

            geoJSON.quakePropertiesArray.append(contentsOf: geoJSON.quakePropertiesArray)
            geoJSON.quakePropertiesArray.append(contentsOf: geoJSON.quakePropertiesArray)
            geoJSON.quakePropertiesArray.append(contentsOf: geoJSON.quakePropertiesArray)
            geoJSON.quakePropertiesArray.append(contentsOf: geoJSON.quakePropertiesArray)
            print(geoJSON.quakePropertiesArray.count)

            // Import the GeoJSON into Core Data.
            try self.importQuakes(from: geoJSON)

        } catch {
            // Alert the user if data cannot be digested.
            completionHandler(QuakeError.wrongDataFormat)
            return
        }
        completionHandler(nil)
    }
    // Start the task.
    task.resume()
}

/**
 Imports a JSON dictionary into the Core Data store on a private queue,
 processing the record in batches to avoid a high memory footprint.
*/
private func importQuakes(from geoJSON: GeoJSON) throws {

    guard !geoJSON.quakePropertiesArray.isEmpty else { return }

    // Create a private queue context.
    //let taskContext = persistentContainer.newBackgroundContext()
    let taskContext = NSManagedObjectContext(concurrencyType: .privateQueueConcurrencyType)
    taskContext.parent = context

    taskContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
    // Set unused undoManager to nil for macOS (it is nil by default on iOS)
    // to reduce resource requirements.
    taskContext.undoManager = nil

    // Process records in batches to avoid a high memory footprint.
    let batchSize = 256
    let count = geoJSON.quakePropertiesArray.count

    // Determine the total number of batches.
    var numBatches = count / batchSize
    numBatches += count % batchSize > 0 ? 1 : 0

    for batchNumber in 0 ..< numBatches {

        // Determine the range for this batch.
        let batchStart = batchNumber * batchSize
        let batchEnd = batchStart + min(batchSize, count - batchNumber * batchSize)
        let range = batchStart..<batchEnd

        // Create a batch for this range from the decoded JSON.
        let quakesBatch = Array(geoJSON.quakePropertiesArray[range])

        // Stop the entire import if any batch is unsuccessful.
        if !importOneBatch(quakesBatch, taskContext: taskContext) {
            return
        }

    }
}

/**
 Imports one batch of quakes, creating managed objects from the new data,
 and saving them to the persistent store, on a private queue. After saving,
 resets the context to clean up the cache and lower the memory footprint.

 NSManagedObjectContext.performAndWait doesn't rethrow so this function
 catches throws within the closure and uses a return value to indicate
 whether the import is successful.
*/
private func importOneBatch(_ quakesBatch: [QuakeProperties], taskContext: NSManagedObjectContext) -> Bool {

    var success = false

    // taskContext.performAndWait runs on the URLSession's delegate queue
    // so it won’t block the main thread.
    taskContext.performAndWait {
        // Create a new record for each quake in the batch.
        for quakeData in quakesBatch {

            // Create a Quake managed object on the private queue context.
            guard let quake = NSEntityDescription.insertNewObject(forEntityName: "Quake", into: taskContext) as? Quake else {
                print(QuakeError.creationError.localizedDescription)
                return
            }
            // Populate the Quake's properties using the raw data.
            do {
                try quake.update(with: quakeData)
            } catch QuakeError.missingData {
                // Delete invalid Quake from the private queue context.
                print(QuakeError.missingData.localizedDescription)
                taskContext.delete(quake)
            } catch {
                print(error.localizedDescription)
            }
        }

        // Save all insertions and deletions from the context to the store.
        if taskContext.hasChanges {
            do {
                try taskContext.save()
                context.performAndWait {
                    try? context.save()
                }

            } catch {
                print("Error: \(error)\nCould not save Core Data context.")
                return
            }
            // Reset the taskContext to free the cache and lower the memory footprint.
            taskContext.reset()

        }

        success = true
    }
    return success
}

    // MARK: - NSFetchedResultsController

/**
 A fetched results controller delegate to give consumers a chance to update
 the user interface when content changes.
 */
weak var fetchedResultsControllerDelegate: NSFetchedResultsControllerDelegate?

/**
 A fetched results controller to fetch Quake records sorted by time.
 */
lazy var fetchedResultsController: NSFetchedResultsController<Quake> = {

    // Create a fetch request for the Quake entity sorted by time.
    let fetchRequest = NSFetchRequest<Quake>(entityName: "Quake")
    fetchRequest.sortDescriptors = [NSSortDescriptor(key: "time", ascending: false)]

    // Create a fetched results controller and set its fetch request, context, and delegate.
    let controller = NSFetchedResultsController(fetchRequest: fetchRequest,
                                                managedObjectContext: context,
                                                sectionNameKeyPath: nil, cacheName: nil)

    controller.delegate = fetchedResultsControllerDelegate

    // Perform the fetch.
    do {
        try controller.performFetch()
    } catch {
        fatalError("Unresolved error \(error)")
    }

    return controller
}()

ViewController code:

// MARK: - UITableViewDataSource

extension QuakesViewController {

    override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {

        guard let cell = tableView.dequeueReusableCell(withIdentifier: "QuakeCell", for: indexPath) as? QuakeCell else {
            print("Error: tableView.dequeueReusableCell doesn'return a QuakeCell!")
            return QuakeCell()
        }
        guard let quake = dataProvider.fetchedResultsController.fetchedObjects?[indexPath.row] else { return cell }

        cell.configure(with: quake)
        return cell
    }

    override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        return dataProvider.fetchedResultsController.fetchedObjects?.count ?? 0
    }
}

// MARK: - NSFetchedResultsControllerDelegate

extension QuakesViewController: NSFetchedResultsControllerDelegate {

    /**
     Reloads the table view when the fetched result controller's content changes.
     */
    func controllerDidChangeContent(_ controller: NSFetchedResultsController<NSFetchRequestResult>) {
        tableView.reloadData()
    }
}

Body:
All the UI are performed at main thread, so you can try to use a global queue to perform this fetch.

I think that a global queue with .utility QoS will be enough to solve this problem.

Here a sample:

DispatchQueue.global(qos: .utility).async {
            //do the fetch here
}

The global queues are concurrent, the utility QoS (quality of service) tells the GCD that this task should not perform in main thread and the .async make the execution doesn't block this global queue.

If it's not totally clear i can post a most complete sample code for you :)

Good luck!


问:
How can I route my Angular API Calls through the hosting server?
I have an application with an angular 7 front end and a dotnet core 2.2 API. The API exists on a protected corporate server, while the angular front end exists on a hosting server. The front end hosting server's ip is whitelisted on the API server's firewall, but because the API calls are coming from the user's IP instead of the hosting server's IP, all of the requests are being blocked.

For example, if User 1 has IP A and makes a request from the frontend application, it hits the API server with IP A, instead of IP B (the frontend server's IP). The request is then blocked because only IP B is whitelisted.

Is there a way around this? Can I force the request to come from the front end server's IP?

答:
That's to be expected, it's the client making the request and not the server.

On the server where your Angular frontend is hosted create a simple proxy that receives API requests from the clients and then forwards them on to your internal server.  You can write it in nearly any language that your server supports.

Then update your Angular app to point to this new URL and all requests to your internal API server will come from your whitelisted IP.


问:
? time delay, when using === or <=
When we use

```<code>input reg [7:0] ast, f_out;

ast === f_out ;

ast <= ast + 8'b00000001;
</code>```

for those operations " === and <= ", Have any time delay been occurred ?

EDIT: I think something small like 1 unit time, am I wrong ?

used language : verilog

答:
Time delays are most commonly specified using #.  Since I see no # in your code, there is probably no delay.

Another way to add delays is to use a specify block, and you don't show that either.  There are plenty of examples of specify in the Verilog IEEE Std.

You can check for delays yourself by running a simulation and printing time values:

```<code>$display($time);
</code>```

Title: ? time delay, when using === or <=

Body:
When we use

input reg [7:0] ast, f_out;

ast === f_out ;

ast <= ast + 8'b00000001;

for those operations " === and <= ", Have any time delay been occurred ?

EDIT: I think something small like 1 unit time, am I wrong ?

used language : verilog

Body:
Time delays are most commonly specified using #.  Since I see no # in your code, there is probably no delay.

Another way to add delays is to use a specify block, and you don't show that either.  There are plenty of examples of specify in the Verilog IEEE Std.

You can check for delays yourself by running a simulation and printing time values:

$display($time);


问:
Where to invoke VIPS_INIT in a C shared library
I am working on a C shared library that would contain one function that receives a filename, and returns a pointer to the C-array representation of the image.

I read VIPS_INIT needs to be set in the beginning of the program, with argv[0] as its argument. However, since this is a shared library, I'm not sure where exactly do I need to place this.

答:
libvips uses argv[0] to help discover message catalogues if its been relocated. If you don't need a relocatable package, you can just pass "" and it'll be fine.

Other notes: be careful not to init from more than one thread at the same time, and make sure init finishes before you try any processing.


问:
How do I support Internet Explorer in an Angular 8 application?
When I generate a project with Angular CLI (8.0.0), I run ng serve, open the application up in Internet Explorer and I am presented with a blank screen.

I had a look at the polyfills.ts files and I uncommented the following lines:

```<code>    import 'classlist.js';
    import 'web-animations-js';
</code>```

I've also removed all core.js imports as Angular 8 supports core.js 3.0 directly.

I've also ran npm i.

package.json:

```<code>"dependencies": {
    "@angular/animations": "~8.0.0",
    "@angular/common": "~8.0.0",
    "@angular/compiler": "~8.0.0",
    "@angular/core": "~8.0.0",
    "@angular/forms": "~8.0.0",
    "@angular/platform-browser": "~8.0.0",
    "@angular/platform-browser-dynamic": "~8.0.0",
    "@angular/router": "~8.0.0",
    "classlist.js": "^1.1.20150312",
    "rxjs": "~6.4.0",
    "tslib": "^1.9.0",
    "web-animations-js": "^2.3.1",
    "zone.js": "~0.9.1"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~0.800.0",
    "@angular/cli": "~8.0.0",
    "@angular/compiler-cli": "~8.0.0",
    "@angular/language-service": "~8.0.0",
    "@types/node": "~8.9.4",
    "@types/jasmine": "~3.3.8",
    "@types/jasminewd2": "~2.0.3",
    "codelyzer": "^5.0.0",
    "jasmine-core": "~3.4.0",
    "jasmine-spec-reporter": "~4.2.1",
    "karma": "~4.1.0",
    "karma-chrome-launcher": "~2.2.0",
    "karma-coverage-istanbul-reporter": "~2.0.1",
    "karma-jasmine": "~2.0.1",
    "karma-jasmine-html-reporter": "^1.4.0",
    "protractor": "~5.4.0",
    "ts-node": "~7.0.0",
    "tslint": "~5.15.0",
    "typescript": "~3.4.3"
  }
</code>```

tsconfig.json:

```<code>{
  "compileOnSave": false,
  "compilerOptions": {
    "baseUrl": "./",
    "outDir": "./dist/out-tsc",
    "sourceMap": true,
    "declaration": false,
    "module": "esnext",
    "moduleResolution": "node",
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "importHelpers": true,
    "target": "es2015",
    "typeRoots": [
      "node_modules/@types"
    ],
    "lib": [
      "es2018",
      "dom"
    ]
  }
}
</code>```

EDIT:

browserlists:

```<code># This file is used by the build system to adjust CSS and JS output to support the specified browsers below.
# For additional information regarding the format and rule options, please see:
# https://github.com/browserslist/browserslist#queries

# You can see what browsers were selected by your queries by running:
#   npx browserslist

> 0.5%
last 2 versions
Firefox ESR
not dead
IE 9-11 # For IE 9-11 support, remove 'not'.
</code>```

EDIT 2:

The console in Internet Explorer (11) shows the following errors:

polyfills.js:
Syntax error (3168, 5)
(line 3168 beginning) -> class Zone {

vendor.js:
Syntax error (156, 1)
(line 156 beginning) -> class PlatformLocation {

main.ts:
Syntax error (95, 20)
(points to the AppComponent)

What else do I need to do?

答:
Note: my original reply was from Reddit (https://www.reddit.com/r/Angular2/comments/buup6a/)

Check your tsconfig.json
While upgrading to Angular 8, the target changed to es2015 for me, so with ng serve I encountered many many problems. While compiling, the dist folder had both es5 and es2015 versions.

Basically, compiling for production will create both versions for new and older browsers like IE11

To test IE11 on development environment, I created a dev environment in angular.json where I specified a copy of tsconfig which I called tsconfig.dev.json where the target is set to es5. Run ng s -c=dev and voilá!

Body:
When I generate a project with Angular CLI (8.0.0), I run ng serve, open the application up in Internet Explorer and I am presented with a blank screen.

I had a look at the polyfills.ts files and I uncommented the following lines:

    import 'classlist.js';
    import 'web-animations-js';

I've also removed all core.js imports as Angular 8 supports core.js 3.0 directly.

I've also ran npm i.

package.json:

"dependencies": {
    "@angular/animations": "~8.0.0",
    "@angular/common": "~8.0.0",
    "@angular/compiler": "~8.0.0",
    "@angular/core": "~8.0.0",
    "@angular/forms": "~8.0.0",
    "@angular/platform-browser": "~8.0.0",
    "@angular/platform-browser-dynamic": "~8.0.0",
    "@angular/router": "~8.0.0",
    "classlist.js": "^1.1.20150312",
    "rxjs": "~6.4.0",
    "tslib": "^1.9.0",
    "web-animations-js": "^2.3.1",
    "zone.js": "~0.9.1"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~0.800.0",
    "@angular/cli": "~8.0.0",
    "@angular/compiler-cli": "~8.0.0",
    "@angular/language-service": "~8.0.0",
    "@types/node": "~8.9.4",
    "@types/jasmine": "~3.3.8",
    "@types/jasminewd2": "~2.0.3",
    "codelyzer": "^5.0.0",
    "jasmine-core": "~3.4.0",
    "jasmine-spec-reporter": "~4.2.1",
    "karma": "~4.1.0",
    "karma-chrome-launcher": "~2.2.0",
    "karma-coverage-istanbul-reporter": "~2.0.1",
    "karma-jasmine": "~2.0.1",
    "karma-jasmine-html-reporter": "^1.4.0",
    "protractor": "~5.4.0",
    "ts-node": "~7.0.0",
    "tslint": "~5.15.0",
    "typescript": "~3.4.3"
  }

tsconfig.json:

{
  "compileOnSave": false,
  "compilerOptions": {
    "baseUrl": "./",
    "outDir": "./dist/out-tsc",
    "sourceMap": true,
    "declaration": false,
    "module": "esnext",
    "moduleResolution": "node",
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "importHelpers": true,
    "target": "es2015",
    "typeRoots": [
      "node_modules/@types"
    ],
    "lib": [
      "es2018",
      "dom"
    ]
  }
}

EDIT:

browserlists:

# This file is used by the build system to adjust CSS and JS output to support the specified browsers below.
# For additional information regarding the format and rule options, please see:
# https://github.com/browserslist/browserslist#queries

# You can see what browsers were selected by your queries by running:
#   npx browserslist

> 0.5%
last 2 versions
Firefox ESR
not dead
IE 9-11 # For IE 9-11 support, remove 'not'.

EDIT 2:

The console in Internet Explorer (11) shows the following errors:

polyfills.js:
Syntax error (3168, 5)
(line 3168 beginning) -> class Zone {

vendor.js:
Syntax error (156, 1)
(line 156 beginning) -> class PlatformLocation {

main.ts:
Syntax error (95, 20)
(points to the AppComponent)

What else do I need to do?


问:
Is there a standard docstring format to show what signature is expected for an argument that takes a function?
My __init__ method accepts another function as an argument called func_convert:

```<code>class Adc:
    """Reads data from ADC

    """
    def __init__(self, func_convert):
        """Setup the ADC

        Parameters
        ----------
        func_convert : [???]
            user-supplied conversion function
        """
        self._func_convert = func_convert

    def read(self):
        data = 0 #some fake data
        return self._func_convert(data)
</code>```

The argument func_convert allows a custom scaling function to be supplied once at instantiation that gets called to convert the data each time it's read.  The function must accept a single int argument and return a single float.  One possible example would be:

```<code>def adc_to_volts(value):
    return value * 3.0 / 2**16 - 1.5

adc = Adc(adc_to_volts)
volts = adc.read()

</code>```

Is there a standard way to document what the expected signature of func_convert is in the parameters section of the __init__ docstring?  If it makes a difference, I'm using numpy docstring style (I think).

答:
If you want to follow numpy docstring style, there are some examples from numpy which show how function parameters are described:

1)

```<code>apply_along_axis(func1d, axis, arr, *args, **kwargs)
    ...

    Parameters
    ----------
    func1d : function (M,) -> (Nj...)
        This function should accept 1-D arrays. It is applied to 1-D
        slices of `arr` along the specified axis.
</code>```

2)

```<code>apply_over_axes(func, a, axes)
    ...

    Parameters
    ----------
    func : function
        This function must take two arguments, `func(a, axis)`.
</code>```

3)

```<code>set_string_function(f, repr=True)
    ...

    Parameters
    ----------
    f : function or None
        Function to be used to pretty print arrays. The function should expect
        a single array argument and return a string of the representation of
        the array. If None, the function is reset to the default NumPy function
        to print arrays.
</code>```

TLDR: they are described manually, without any special syntax or guidelines. If your goal is to create docstrings similar to numpy's, you can describe them in any way you want. But I strongly suggest following @jfaccioni answer and using type hints.

Body:
My __init__ method accepts another function as an argument called func_convert:

class Adc:
    """Reads data from ADC

    """
    def __init__(self, func_convert):
        """Setup the ADC

        Parameters
        ----------
        func_convert : [???]
            user-supplied conversion function
        """
        self._func_convert = func_convert

    def read(self):
        data = 0 #some fake data
        return self._func_convert(data)

The argument func_convert allows a custom scaling function to be supplied once at instantiation that gets called to convert the data each time it's read.  The function must accept a single int argument and return a single float.  One possible example would be:

def adc_to_volts(value):
    return value * 3.0 / 2**16 - 1.5

adc = Adc(adc_to_volts)
volts = adc.read()

Is there a standard way to document what the expected signature of func_convert is in the parameters section of the __init__ docstring?  If it makes a difference, I'm using numpy docstring style (I think).

Body:
If you want to follow numpy docstring style, there are some examples from numpy which show how function parameters are described:

1)

apply_along_axis(func1d, axis, arr, *args, **kwargs)
    ...

    Parameters
    ----------
    func1d : function (M,) -> (Nj...)
        This function should accept 1-D arrays. It is applied to 1-D
        slices of `arr` along the specified axis.

2)

apply_over_axes(func, a, axes)
    ...

    Parameters
    ----------
    func : function
        This function must take two arguments, `func(a, axis)`.

3)

set_string_function(f, repr=True)
    ...

    Parameters
    ----------
    f : function or None
        Function to be used to pretty print arrays. The function should expect
        a single array argument and return a string of the representation of
        the array. If None, the function is reset to the default NumPy function
        to print arrays.

TLDR: they are described manually, without any special syntax or guidelines. If your goal is to create docstrings similar to numpy's, you can describe them in any way you want. But I strongly suggest following @jfaccioni answer and using type hints.


问:
what is the optimal way to show differences between two data sets
I am trying to show differences  of two different datasets in and efficient way. I need to support Hadoop.  I have tried the following methods I now but none have been optimal for Hadoop</p>

```<code>def main(args: Array[String]) {
    System.setProperty("hadoop", "C:\\Users\\7777777\\App\\SparkHadoop\\);
    val sc = new SparkContext("local[*]","name")
    //val sc = new SparkContext()
    val spark = SparkSession.builder().config("warehouse.dir","C:\\Users\\777777\\workspace\\OracleSparkImport\\warehouse").appName("name").getOrCreate();
   import spark.implicits._
    val baseConfig = ConfigFactory.load()
    val config = ConfigFactory.parseFile(new File("C:/Users/7777/workspace/OracleSparkReport/resources_new/application.properties")).withFallback(baseConfig)
    val dbConfigs = config.getConfig("db");
    val connectionStr = dbConfigs.getString("connectionstr");

       print(dbConfigs);
       var queryStrKey2 = "q2" ;
       var queryStr2 = dbConfigs.getString(queryStrKey2);
       var queryStrKey3 = "q3";
       var queryStr3 = dbConfigs.getString(queryStrKey3);

       var query2 = "(" + queryStr2 + ") rep";
       var query3 = "(" + queryStr3 + ") rep3";

    var df2 = spark.read.format("jdbc")
            .option("url", connectionStr)
            .option("dbtable",query2)
            .option("driver", "oracle.jdbc.driver.OracleDriver")
            .load();

    var df1 = spark.read.format("jdbc")
            .option("url", connectionStr)
            .option("dbtable",query3)
            .option("driver", "oracle.jdbc.driver.OracleDriver")
            .load();
</code>```

答:
IMHO, I dont understand what do you mean by "optimal" one way is like below.

Option 1 :

If you want to find schema differences between 2 dataframes

```<code>/**
    * getAllSchemaDiff
    *
    * @param schema1 Map[String, (DataType, Boolean)]
    * @param schema2 Map[String, (DataType, Boolean)]
    * @return Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])]
    */
  def getAllSchemaDiff(schema1: Map[String, (DataType, Boolean)],
                       schema2: Map[String, (DataType, Boolean)]
                      ): Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])] = {

    val common = ((schema1.keySet).intersect(schema2.keySet)).map(_.toLowerCase).toList.distinct
    LOGGER.info(" common columns are   " + common.mkString("\n"))

    val distinctkeys = ((schema1.keys) ++ (schema2.keys)).map(_.toLowerCase).toList.distinct
    LOGGER.info("distinctkeys - > " + distinctkeys)
    distinctkeys.flatMap { columnName: String =>
      val schema1FieldOpt: Option[(DataType, Boolean)] = schema1.get(columnName)
      val schema2FieldOpt: Option[(DataType, Boolean)] = schema2.get(columnName)

      if (schema1FieldOpt == schema2FieldOpt) None
      else Some(columnName -> (schema1FieldOpt, schema2FieldOpt))
    }.toMap
  }
</code>```

where getMapFromSchema is...

```<code> /**
        * getMapFromSchema -Extract relevant information: name (key), type & nullability (values) of columns
        *
        * @param df
        * @return
        */
      def getMapFromSchema(df: DataFrame): Map[String, (DataType, Boolean)] = {
        val a: Seq[(String, (DataType, Boolean))] = df.schema.map { structField: StructField =>
          structField.name.toLowerCase -> (structField.dataType, structField.nullable)
        }.toList

        a.toMap
      }
</code>```

if you want find column wise differences :

```<code>/**
    * columnWiseDifferences - finds columnWise differences between 2 dataframes one is source and another is target
    *
    * @param sourceDataFrame
    * @param targetDataFrame
    * @return
    */

  def columnWiseDifferences(sourceDataFrame: DataFrame, targetDataFrame: DataFrame) = {
    val columns = sourceDataFrame.schema.fields.map(_.name)
    //    LOGGER.info("source schema")
    //    sourceDataFrame.printSchema
    //    LOGGER.info("target schema")
    //    targetDataFrame.printSchema
    LOGGER.info("Source except target")
    var selectiveDifferences = columns.map(col => sourceDataFrame.select(col).except(targetDataFrame.select(col)))

    // columns contains different values.
    selectiveDifferences.map(diff => {
      if (diff.count > 0) diff.show
    })

    selectiveDifferences = columns.map(col => targetDataFrame.select(col).except(sourceDataFrame.select(col)))
    LOGGER.info("target except source")

    // columns contains different values.
    selectiveDifferences.map(diff => {
      if (diff.count > 0) diff.show
    })
  }
</code>```

Full example :

```

pacakage com.examples

import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.types.{DataType, StructField}
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.slf4j.{Logger, LoggerFactory}

/**
  * @author Ram Ghadiyaram
  */
object ComparisionUtil {

  def main(args: Array[String]): Unit = {
     val logger = org.apache.log4j.Logger.getLogger("org")
    logger.setLevel(Level.WARN)
    val spark: SparkSession = SparkSession.builder().appName(this.getClass.getName).master("local[*]").getOrCreate()
    import spark.implicits._
    val df1 = Seq(
      (1, "Ram", ""),
      (2, "william peck", ""),
      (3, "peck", "")
    ).toDF("id", "name", "grade")
    val df2 = Seq(
      (1, "Ram", 3),
      (2, "william peck", 3),
      (3, "peck", 3)
    ).toDF("id", "name", "grade")
    val diffMap = getAllSchemaDiff(getMapFromSchema(df1), getMapFromSchema(df2))
        LOGGER.info("schema differences are  " + diffMap)

    val srcNTgtDataFrames = getDataFramesWithCommonColumns(df1, df2)
    columnWiseDifferences(srcNTgtDataFrames._1, srcNTgtDataFrames._2)
  }

  private[this] val LOGGER: org.slf4j.Logger = LoggerFactory.getLogger(this.getClass)

  /**
    * getAllSchemaDiff
    *
    * @param schema1 Map[String, (DataType, Boolean)]
    * @param schema2 Map[String, (DataType, Boolean)]
    * @return Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])]
    */
  def getAllSchemaDiff(schema1: Map[String, (DataType, Boolean)],
                       schema2: Map[String, (DataType, Boolean)]
                      ): Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])] = {

    val common = ((schema1.keySet).intersect(schema2.keySet)).map(_.toLowerCase).toList.distinct
    LOGGER.info(" common columns are   " + common.mkString("\n"))

    val distinctkeys = ((schema1.keys) ++ (schema2.keys)).map(_.toLowerCase).toList.distinct
    LOGGER.info("distinctkeys - > " + distinctkeys)
    distinctkeys.flatMap { columnName: String =>
      val schema1FieldOpt: Option[(DataType, Boolean)] = schema1.get(columnName)
      val schema2FieldOpt: Option[(DataType, Boolean)] = schema2.get(columnName)

      if (schema1FieldOpt == schema2FieldOpt) None
      else Some(columnName -> (schema1FieldOpt, schema2FieldOpt))
    }.toMap
  }

  /**
    * getDataFramesWithCommonColumns
    */
  def getDataFramesWithCommonColumns(srcData: DataFrame, targetData: DataFrame) = {
    val schema1 = getMapFromSchema(srcData)
    val schema2 = getMapFromSchema(targetData)
    val common = ((schema1.keySet).intersect(schema2.keySet)).map(_.toLowerCase).toList.distinct
    LOGGER.info(" srcData " + srcData.schema.treeString)
    LOGGER.info(" targetData " + targetData.schema.treeString)
    LOGGER.info(" **** \n\n\n\ncommon columns in source and target are  \n " + common.mkString("\n"))

    val df1 = srcData.selectExpr(common: _*)
    val df2 = targetData.selectExpr(common: _*)
    (df1, df2)
  }

  /**
    * getMapFromSchema -Extract relevant information: name (key), type & nullability (values) of columns
    *
    * @param df
    * @return
    */
  def getMapFromSchema(df: DataFrame): Map[String, (DataType, Boolean)] = {
    val a: Seq[(String, (DataType, Boolean))] = df.schema.map { structField: StructField =>
      structField.name.toLowerCase -> (structField.dataType, structField.nullable)
    }.toList

    a.toMap
  }

  /**
    * columnWiseDifferences - finds columnWise differences between 2 dataframes one is source and another is target
    *
    * @param sourceDataFrame
    * @param targetDataFrame
    * @return
    */

  def columnWiseDifferences(sourceDataFrame: DataFrame, targetDataFrame: DataFrame) = {
    val columns = sourceDataFrame.schema.fields.map(_.name)
        LOGGER.info("source schema")
        sourceDataFrame.printSchema
        LOGGER.info("target schema")
        targetDataFrame.printSchema
    LOGGER.info("Source except target")
    var selectiveDifferences = columns.map(col => sourceDataFrame.select(col).except(targetDataFrame.select(col)))

    // columns contains different values.
    selectiveDifferences.map(diff => {
      if (diff.count > 0) diff.show
    })

    selectiveDifferences = columns.map(col => targetDataFrame.select(col).except(sourceDataFrame.select(col)))
    LOGGER.info("target except source")

    // columns contains different values.
    selectiveDifferences.map(diff => {
      if (diff.count > 0) diff.show
    })
  }

  /**
    * getListFromSchemaWithKeysOnly -Extract relevant information: name (key) of columns
    *
    * @param df
    * @return
    */
  def getListFromSchemaWithKeysOnly(df: DataFrame): List[String] = {
    val a = df.schema.map { structField: StructField =>
      structField.name.toLowerCase
    }.toList
    a
  }
}

```

Result :

```
2019-05-30 12:44:49 INFO  ComparisionUtil$:48 -  common columns are   id
name
grade
2019-05-30 12:44:49 INFO  ComparisionUtil$:51 - distinctkeys - > List(id, name, grade)
2019-05-30 12:44:49 INFO  ComparisionUtil$:29 - schema differences are  Map(grade -> (Some((StringType,true)),Some((IntegerType,false))))
2019-05-30 12:44:49 INFO  ComparisionUtil$:68 -  srcData root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- grade: string (nullable = true)

2019-05-30 12:44:49 INFO  ComparisionUtil$:69 -  targetData root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- grade: integer (nullable = false)

2019-05-30 12:44:49 INFO  ComparisionUtil$:70 -  ****

common columns in source and target are
 id
name
grade
2019-05-30 12:44:49 INFO  ComparisionUtil$:101 - source schema
root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- grade: string (nullable = true)

2019-05-30 12:44:49 INFO  ComparisionUtil$:103 - target schema
root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- grade: integer (nullable = false)

2019-05-30 12:44:49 INFO  ComparisionUtil$:105 - Source except target
+-----+
|grade|
+-----+
|     |
+-----+

2019-05-30 12:44:52 INFO  ComparisionUtil$:114 - target except source
+-----+
|grade|
+-----+
|    3|
+-----+

```

Note : if you want to use column wise differences in below code snippet, you can ensure that if rowcounts of your oracle tables are
  not matching then go for it. otherwise columnwisedifferences will take
  some time if you are using this with large datasets.

if you dont want column wise differences and schema differences you can adapt below approach using except on dataframe... for the above dataframes created.

Option 2 :

Another option using except which might be simple is
left hand side except right hand side

```<code>println("Another option using except which might be simple is ")
    val difference1 = (df1.except(df2))
    println("left hand side except right hand side ")
    difference1.show()
    val difference2 = (df2.except(df1))
    println("right hand side except left hand side ")
    difference2.show()
</code>```

Result :

```<code>Another option using except which might be simple is
left hand side except right hand side
+---+------------+-----+
| id|        name|grade|
+---+------------+-----+
|  2|william peck|     |
|  3|        peck|     |
|  1|         Ram|     |
+---+------------+-----+

right hand side except left hand side
+---+------------+-----+
| id|        name|grade|
+---+------------+-----+
|  3|        peck|    3|
|  1|         Ram|    3|
|  2|william peck|    3|
+---+------------+-----+
</code>```

Body:
I am trying to show differences  of two different datasets in and efficient way. I need to support Hadoop.  I have tried the following methods I now but none have been optimal for Hadoop

def main(args: Array[String]) {
    System.setProperty("hadoop", "C:\\Users\\7777777\\App\\SparkHadoop\\);
    val sc = new SparkContext("local[*]","name")
    //val sc = new SparkContext()
    val spark = SparkSession.builder().config("warehouse.dir","C:\\Users\\777777\\workspace\\OracleSparkImport\\warehouse").appName("name").getOrCreate();
   import spark.implicits._
    val baseConfig = ConfigFactory.load()
    val config = ConfigFactory.parseFile(new File("C:/Users/7777/workspace/OracleSparkReport/resources_new/application.properties")).withFallback(baseConfig)
    val dbConfigs = config.getConfig("db");
    val connectionStr = dbConfigs.getString("connectionstr");

       print(dbConfigs);
       var queryStrKey2 = "q2" ;
       var queryStr2 = dbConfigs.getString(queryStrKey2);
       var queryStrKey3 = "q3";
       var queryStr3 = dbConfigs.getString(queryStrKey3);

       var query2 = "(" + queryStr2 + ") rep";
       var query3 = "(" + queryStr3 + ") rep3";

    var df2 = spark.read.format("jdbc")
            .option("url", connectionStr)
            .option("dbtable",query2)
            .option("driver", "oracle.jdbc.driver.OracleDriver")
            .load();

    var df1 = spark.read.format("jdbc")
            .option("url", connectionStr)
            .option("dbtable",query3)
            .option("driver", "oracle.jdbc.driver.OracleDriver")
            .load();

Body:
IMHO, I dont understand what do you mean by "optimal" one way is like below.

Option 1 :

If you want to find schema differences between 2 dataframes

/**
    * getAllSchemaDiff
    *
    * @param schema1 Map[String, (DataType, Boolean)]
    * @param schema2 Map[String, (DataType, Boolean)]
    * @return Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])]
    */
  def getAllSchemaDiff(schema1: Map[String, (DataType, Boolean)],
                       schema2: Map[String, (DataType, Boolean)]
                      ): Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])] = {

    val common = ((schema1.keySet).intersect(schema2.keySet)).map(_.toLowerCase).toList.distinct
    LOGGER.info(" common columns are   " + common.mkString("\n"))

    val distinctkeys = ((schema1.keys) ++ (schema2.keys)).map(_.toLowerCase).toList.distinct
    LOGGER.info("distinctkeys - > " + distinctkeys)
    distinctkeys.flatMap { columnName: String =>
      val schema1FieldOpt: Option[(DataType, Boolean)] = schema1.get(columnName)
      val schema2FieldOpt: Option[(DataType, Boolean)] = schema2.get(columnName)

      if (schema1FieldOpt == schema2FieldOpt) None
      else Some(columnName -> (schema1FieldOpt, schema2FieldOpt))
    }.toMap
  }

where getMapFromSchema is...

 /**
        * getMapFromSchema -Extract relevant information: name (key), type & nullability (values) of columns
        *
        * @param df
        * @return
        */
      def getMapFromSchema(df: DataFrame): Map[String, (DataType, Boolean)] = {
        val a: Seq[(String, (DataType, Boolean))] = df.schema.map { structField: StructField =>
          structField.name.toLowerCase -> (structField.dataType, structField.nullable)
        }.toList

        a.toMap
      }

if you want find column wise differences :

/**
    * columnWiseDifferences - finds columnWise differences between 2 dataframes one is source and another is target
    *
    * @param sourceDataFrame
    * @param targetDataFrame
    * @return
    */

  def columnWiseDifferences(sourceDataFrame: DataFrame, targetDataFrame: DataFrame) = {
    val columns = sourceDataFrame.schema.fields.map(_.name)
    //    LOGGER.info("source schema")
    //    sourceDataFrame.printSchema
    //    LOGGER.info("target schema")
    //    targetDataFrame.printSchema
    LOGGER.info("Source except target")
    var selectiveDifferences = columns.map(col => sourceDataFrame.select(col).except(targetDataFrame.select(col)))

    // columns contains different values.
    selectiveDifferences.map(diff => {
      if (diff.count > 0) diff.show
    })

    selectiveDifferences = columns.map(col => targetDataFrame.select(col).except(sourceDataFrame.select(col)))
    LOGGER.info("target except source")

    // columns contains different values.
    selectiveDifferences.map(diff => {
      if (diff.count > 0) diff.show
    })
  }

Full example :

pacakage com.examples

import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.types.{DataType, StructField}
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.slf4j.{Logger, LoggerFactory}

/**
  * @author Ram Ghadiyaram
  */
object ComparisionUtil {

  def main(args: Array[String]): Unit = {
     val logger = org.apache.log4j.Logger.getLogger("org")
    logger.setLevel(Level.WARN)
    val spark: SparkSession = SparkSession.builder().appName(this.getClass.getName).master("local[*]").getOrCreate()
    import spark.implicits._
    val df1 = Seq(
      (1, "Ram", ""),
      (2, "william peck", ""),
      (3, "peck", "")
    ).toDF("id", "name", "grade")
    val df2 = Seq(
      (1, "Ram", 3),
      (2, "william peck", 3),
      (3, "peck", 3)
    ).toDF("id", "name", "grade")
    val diffMap = getAllSchemaDiff(getMapFromSchema(df1), getMapFromSchema(df2))
        LOGGER.info("schema differences are  " + diffMap)

    val srcNTgtDataFrames = getDataFramesWithCommonColumns(df1, df2)
    columnWiseDifferences(srcNTgtDataFrames._1, srcNTgtDataFrames._2)
  }

  private[this] val LOGGER: org.slf4j.Logger = LoggerFactory.getLogger(this.getClass)

  /**
    * getAllSchemaDiff
    *
    * @param schema1 Map[String, (DataType, Boolean)]
    * @param schema2 Map[String, (DataType, Boolean)]
    * @return Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])]
    */
  def getAllSchemaDiff(schema1: Map[String, (DataType, Boolean)],
                       schema2: Map[String, (DataType, Boolean)]
                      ): Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])] = {

    val common = ((schema1.keySet).intersect(schema2.keySet)).map(_.toLowerCase).toList.distinct
    LOGGER.info(" common columns are   " + common.mkString("\n"))

    val distinctkeys = ((schema1.keys) ++ (schema2.keys)).map(_.toLowerCase).toList.distinct
    LOGGER.info("distinctkeys - > " + distinctkeys)
    distinctkeys.flatMap { columnName: String =>
      val schema1FieldOpt: Option[(DataType, Boolean)] = schema1.get(columnName)
      val schema2FieldOpt: Option[(DataType, Boolean)] = schema2.get(columnName)

      if (schema1FieldOpt == schema2FieldOpt) None
      else Some(columnName -> (schema1FieldOpt, schema2FieldOpt))
    }.toMap
  }

  /**
    * getDataFramesWithCommonColumns
    */
  def getDataFramesWithCommonColumns(srcData: DataFrame, targetData: DataFrame) = {
    val schema1 = getMapFromSchema(srcData)
    val schema2 = getMapFromSchema(targetData)
    val common = ((schema1.keySet).intersect(schema2.keySet)).map(_.toLowerCase).toList.distinct
    LOGGER.info(" srcData " + srcData.schema.treeString)
    LOGGER.info(" targetData " + targetData.schema.treeString)
    LOGGER.info(" **** \n\n\n\ncommon columns in source and target are  \n " + common.mkString("\n"))

    val df1 = srcData.selectExpr(common: _*)
    val df2 = targetData.selectExpr(common: _*)
    (df1, df2)
  }

  /**
    * getMapFromSchema -Extract relevant information: name (key), type & nullability (values) of columns
    *
    * @param df
    * @return
    */
  def getMapFromSchema(df: DataFrame): Map[String, (DataType, Boolean)] = {
    val a: Seq[(String, (DataType, Boolean))] = df.schema.map { structField: StructField =>
      structField.name.toLowerCase -> (structField.dataType, structField.nullable)
    }.toList

    a.toMap
  }

  /**
    * columnWiseDifferences - finds columnWise differences between 2 dataframes one is source and another is target
    *
    * @param sourceDataFrame
    * @param targetDataFrame
    * @return
    */

  def columnWiseDifferences(sourceDataFrame: DataFrame, targetDataFrame: DataFrame) = {
    val columns = sourceDataFrame.schema.fields.map(_.name)
        LOGGER.info("source schema")
        sourceDataFrame.printSchema
        LOGGER.info("target schema")
        targetDataFrame.printSchema
    LOGGER.info("Source except target")
    var selectiveDifferences = columns.map(col => sourceDataFrame.select(col).except(targetDataFrame.select(col)))

    // columns contains different values.
    selectiveDifferences.map(diff => {
      if (diff.count > 0) diff.show
    })

    selectiveDifferences = columns.map(col => targetDataFrame.select(col).except(sourceDataFrame.select(col)))
    LOGGER.info("target except source")

    // columns contains different values.
    selectiveDifferences.map(diff => {
      if (diff.count > 0) diff.show
    })
  }

  /**
    * getListFromSchemaWithKeysOnly -Extract relevant information: name (key) of columns
    *
    * @param df
    * @return
    */
  def getListFromSchemaWithKeysOnly(df: DataFrame): List[String] = {
    val a = df.schema.map { structField: StructField =>
      structField.name.toLowerCase
    }.toList
    a
  }
}

Result :

2019-05-30 12:44:49 INFO  ComparisionUtil$:48 -  common columns are   id
name
grade
2019-05-30 12:44:49 INFO  ComparisionUtil$:51 - distinctkeys - > List(id, name, grade)
2019-05-30 12:44:49 INFO  ComparisionUtil$:29 - schema differences are  Map(grade -> (Some((StringType,true)),Some((IntegerType,false))))
2019-05-30 12:44:49 INFO  ComparisionUtil$:68 -  srcData root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- grade: string (nullable = true)

2019-05-30 12:44:49 INFO  ComparisionUtil$:69 -  targetData root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- grade: integer (nullable = false)

2019-05-30 12:44:49 INFO  ComparisionUtil$:70 -  ****

common columns in source and target are
 id
name
grade
2019-05-30 12:44:49 INFO  ComparisionUtil$:101 - source schema
root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- grade: string (nullable = true)

2019-05-30 12:44:49 INFO  ComparisionUtil$:103 - target schema
root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- grade: integer (nullable = false)

2019-05-30 12:44:49 INFO  ComparisionUtil$:105 - Source except target
+-----+
|grade|
+-----+
|     |
+-----+

2019-05-30 12:44:52 INFO  ComparisionUtil$:114 - target except source
+-----+
|grade|
+-----+
|    3|
+-----+

Note : if you want to use column wise differences in below code snippet, you can ensure that if rowcounts of your oracle tables are
  not matching then go for it. otherwise columnwisedifferences will take
  some time if you are using this with large datasets.

if you dont want column wise differences and schema differences you can adapt below approach using except on dataframe... for the above dataframes created.

Option 2 :

Another option using except which might be simple is
left hand side except right hand side

println("Another option using except which might be simple is ")
    val difference1 = (df1.except(df2))
    println("left hand side except right hand side ")
    difference1.show()
    val difference2 = (df2.except(df1))
    println("right hand side except left hand side ")
    difference2.show()

Result :

Another option using except which might be simple is
left hand side except right hand side
+---+------------+-----+
| id|        name|grade|
+---+------------+-----+
|  2|william peck|     |
|  3|        peck|     |
|  1|         Ram|     |
+---+------------+-----+

right hand side except left hand side
+---+------------+-----+
| id|        name|grade|
+---+------------+-----+
|  3|        peck|    3|
|  1|         Ram|    3|
|  2|william peck|    3|
+---+------------+-----+


问:
Is it possible to construct an adjacency matrix for a graph without storing indexes of nodes
I have managed to implement the solution with storing an index of each node, but is it possible to it without the indexes? My assignment is to implement a bloxrolz game, sort of. In the example I'm about to show you, I am out of ideas how to write code for checking if two nodes are adjacent to one another.

This is just a random field for the game. When reading from a file, I store every charachter here in a linked list, with a x,y coordinate and ID(if the char is not '-'). Only those that are not "-" are to be stored in a adjacency matrix since they represent the playground and "-" are nodes that could be changed to "-".

I have successfully implemented a solution in a way that I'm not sure if its legitimate or not. Can you help me figure out how to do this without the indexes.

答:
An adjacency matrix is by definition a square matrix where each node is represented by a row and a column. If the entry at row i and column j is 1 (or some value you pick), then those two nodes are connected in the graph. Therefore, you need some way to map rows and columns to nodes and vice versa. Putting the nodes in a list and using each node's index in the list is one way to do that, but it's not the only way. You could give each node an ID number and, as long as each node's ID is unique and within the bounds of the matrix, you can use that. You could store a pointer to the corresponding node at the beginning or end of each row and column. In short, any method that lets you find the entries in the matrix for a given pair of nodes is sufficient.

That said, because you're talking about ordered lists of rows and columns, any method you pick will be similar to putting the nodes in a list and using the index of a node in the list. Think hard about what problem you're actually trying to solve.


问:
How can I condense all these similar views into one?
All of these views are very similar, and I would like to condense them into one view.

```<code>class SummerIntents(TemplateView):

    template_name = 'admin/hr/intent_forms/summer_intents.html'

    @user_is_staff
    def dispatch(self, request, *args, **kwargs):
        return super(SummerIntents, self).dispatch(request, *args, **kwargs)

    def get_context_data(self, **kwargs):
        context = super(SummerIntents, self).get_context_data(**kwargs)

        # These functions are found in util.py
        update_date = get_update_date()
        active_users = get_active_users(self)

        context['summer_info'] = SummerInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        return context

    def post(self, request, *args, **kwargs):
        context = super(SummerIntents, self).get_context_data(**kwargs)
        file_name = "summer_intent_forms"

        update_date = get_update_date()
        active_users = get_active_users(self)
        info = SummerInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        student_intents = get_student_intents(active_users, update_date, 'summer', info)

        return intents_to_csv(student_intents, file_name)

class WinterIntents(TemplateView):

    template_name = 'admin/hr/intent_forms/winter_intents.html'

    @user_is_staff
    def dispatch(self, request, *args, **kwargs):
        return super(WinterIntents, self).dispatch(request, *args, **kwargs)

    def get_context_data(self, **kwargs):
        context = super(WinterIntents, self).get_context_data(**kwargs)

        # These functions are found in util.py
        update_date = get_update_date()
        active_users = get_active_users(self)

        context['winter_info'] = WinterInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        return context

    def post(self, request, *args, **kwargs):
        context = super(WinterIntents, self).get_context_data(**kwargs)
        file_name = "winter_intent_forms"

        update_date = get_update_date()
        active_users = get_active_users(self)
        info = WinterInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        student_intents = get_student_intents(active_users, update_date, 'winter', info)

        return intents_to_csv(student_intents, file_name)

class FallIntents(TemplateView):

    template_name = 'admin/hr/intent_forms/fall_intents.html'

    @user_is_staff
    def dispatch(self, request, *args, **kwargs):
        return super(FallIntents, self).dispatch(request, *args, **kwargs)

    def get_context_data(self, **kwargs):
        context = super(FallIntents, self).get_context_data(**kwargs)

        # These functions are found in util.py
        update_date = get_update_date()
        active_users = get_active_users(self)

        context['fall_info'] = FallInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        return context

    def post(self, request, *args, **kwargs):
        context = super(FallIntents, self).get_context_data(**kwargs)
        file_name = "fall_intent_forms"

        update_date = get_update_date()
        active_users = get_active_users(self)
        info = FallInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        student_intents = get_student_intents(active_users, update_date, 'fall')

        return intents_to_csv(student_intents, file_name, info)
</code>```

I am open to one main view they all inherit from, but would that mean I have to pass some variables through context? Such as update_date and active_users? I would prefer not to add another method on top of these ones, especially since they are already so short and a main view to inherit from wouldn't do too much good. Secondly, they each have a post method that saves the templates as CSV file when a button is clicked, and I don't know how to condense that so that all the buttons still function properly. Finally, they all have different templates, would I have to condense them into one file as well? That's what I'm hoping to do eventually.

答:
I'm not sure what you mean by having to "pass some variables through context", but inheritance is definitely the way to go here.

Note even within each view you have some duplication, such as getting the Info objects which can be abstracted out into a separate method. Note also that you can avoid defining dispatch just to decorate it if instead you add the django.contrib.auth.mixins.UserPassesTestMixin and define test_func.

```<code>class Intents(UserPassesTestMixin, TemplateView):

    def test_func(self):
        return self.request.user.is_staff

    def get_template_names(self):
         return 'admin/hr/intent_forms/{}_intents.html'.format(self.season_name)

    def get_info_items(self):
        update_date = get_update_date()
        active_users = get_active_users(self)
        return self.info_model.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')

    def get_context_data(self, **kwargs):
        kwargs['{}_info'.format(self.season_name)] = self.get_info_items()
        return super().get_context_data(**kwargs)

    def post(self, request, *args, **kwargs):
        file_name = "{}_intent_forms".format(self.season_name)
        info = self.get_info_items()
        student_intents = get_student_intents(active_users, update_date, self.season_name, info)

        return intents_to_csv(student_intents, file_name)

class SummerIntents(Intents):
    season_name = 'summer'
    info_model = SummerInfo

class WinterIntents(Intents):
    season_name = 'winter'
    info_model = WinterInfo

class FallIntents(Intents):
    season_name = 'fall'
    info_model = FallInfo
</code>```

(Also, it's a bit odd to have get_active_users as a utility method that accepts the view class as a parameter. Are you sure that shouldn't be a method on the view?)

Body:
All of these views are very similar, and I would like to condense them into one view.

class SummerIntents(TemplateView):

    template_name = 'admin/hr/intent_forms/summer_intents.html'

    @user_is_staff
    def dispatch(self, request, *args, **kwargs):
        return super(SummerIntents, self).dispatch(request, *args, **kwargs)

    def get_context_data(self, **kwargs):
        context = super(SummerIntents, self).get_context_data(**kwargs)

        # These functions are found in util.py
        update_date = get_update_date()
        active_users = get_active_users(self)

        context['summer_info'] = SummerInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        return context

    def post(self, request, *args, **kwargs):
        context = super(SummerIntents, self).get_context_data(**kwargs)
        file_name = "summer_intent_forms"

        update_date = get_update_date()
        active_users = get_active_users(self)
        info = SummerInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        student_intents = get_student_intents(active_users, update_date, 'summer', info)

        return intents_to_csv(student_intents, file_name)

class WinterIntents(TemplateView):

    template_name = 'admin/hr/intent_forms/winter_intents.html'

    @user_is_staff
    def dispatch(self, request, *args, **kwargs):
        return super(WinterIntents, self).dispatch(request, *args, **kwargs)

    def get_context_data(self, **kwargs):
        context = super(WinterIntents, self).get_context_data(**kwargs)

        # These functions are found in util.py
        update_date = get_update_date()
        active_users = get_active_users(self)

        context['winter_info'] = WinterInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        return context

    def post(self, request, *args, **kwargs):
        context = super(WinterIntents, self).get_context_data(**kwargs)
        file_name = "winter_intent_forms"

        update_date = get_update_date()
        active_users = get_active_users(self)
        info = WinterInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        student_intents = get_student_intents(active_users, update_date, 'winter', info)

        return intents_to_csv(student_intents, file_name)

class FallIntents(TemplateView):

    template_name = 'admin/hr/intent_forms/fall_intents.html'

    @user_is_staff
    def dispatch(self, request, *args, **kwargs):
        return super(FallIntents, self).dispatch(request, *args, **kwargs)

    def get_context_data(self, **kwargs):
        context = super(FallIntents, self).get_context_data(**kwargs)

        # These functions are found in util.py
        update_date = get_update_date()
        active_users = get_active_users(self)

        context['fall_info'] = FallInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        return context

    def post(self, request, *args, **kwargs):
        context = super(FallIntents, self).get_context_data(**kwargs)
        file_name = "fall_intent_forms"

        update_date = get_update_date()
        active_users = get_active_users(self)
        info = FallInfo.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')
        student_intents = get_student_intents(active_users, update_date, 'fall')

        return intents_to_csv(student_intents, file_name, info)

I am open to one main view they all inherit from, but would that mean I have to pass some variables through context? Such as update_date and active_users? I would prefer not to add another method on top of these ones, especially since they are already so short and a main view to inherit from wouldn't do too much good. Secondly, they each have a post method that saves the templates as CSV file when a button is clicked, and I don't know how to condense that so that all the buttons still function properly. Finally, they all have different templates, would I have to condense them into one file as well? That's what I'm hoping to do eventually.

Body:
I'm not sure what you mean by having to "pass some variables through context", but inheritance is definitely the way to go here.

Note even within each view you have some duplication, such as getting the Info objects which can be abstracted out into a separate method. Note also that you can avoid defining dispatch just to decorate it if instead you add the django.contrib.auth.mixins.UserPassesTestMixin and define test_func.

class Intents(UserPassesTestMixin, TemplateView):

    def test_func(self):
        return self.request.user.is_staff

    def get_template_names(self):
         return 'admin/hr/intent_forms/{}_intents.html'.format(self.season_name)

    def get_info_items(self):
        update_date = get_update_date()
        active_users = get_active_users(self)
        return self.info_model.objects.filter(employee__in=active_users, date_modified__gte=update_date.date, active=True).order_by('position','employee__last_name')

    def get_context_data(self, **kwargs):
        kwargs['{}_info'.format(self.season_name)] = self.get_info_items()
        return super().get_context_data(**kwargs)

    def post(self, request, *args, **kwargs):
        file_name = "{}_intent_forms".format(self.season_name)
        info = self.get_info_items()
        student_intents = get_student_intents(active_users, update_date, self.season_name, info)

        return intents_to_csv(student_intents, file_name)

class SummerIntents(Intents):
    season_name = 'summer'
    info_model = SummerInfo

class WinterIntents(Intents):
    season_name = 'winter'
    info_model = WinterInfo

class FallIntents(Intents):
    season_name = 'fall'
    info_model = FallInfo

(Also, it's a bit odd to have get_active_users as a utility method that accepts the view class as a parameter. Are you sure that shouldn't be a method on the view?)