Classification of companies into industry sectors is a fundamental task for unlocking advanced business intelligence capabilities. However different data sources rarely use the same classification system if any. This is a huge obstacle for taking advantage of the available details in Open Data and very niche commercial data sources that lack or use inconsistent industry classifications.
Industry classifications vary a lot ranging from flat 15 class classification to very complex taxonomies. Extremes in development of industry classifications are bottom-up application-centric approaches for collection and use data with limited ambition for share and reuse, or top-down government or institutional statistics which appear to be too abstract and too complex to be practically useful.
This is the presentation from the video. You’re probably better off watching the video to get an additional explanation but it is included here for completeness’ sake.
A (partial) description of the data model surrounding Wikipedia Organisations. You can use GraphDB visualization tools to explore the whole model surrounding Organisations but this is a description of the features that seem most salient to our task.
Ultimately the task is to classify companies into industries using rich and complex graph-based features. However, given the constraints of the Monthly challenge format, the task can be framed as an error/anomaly detection task. At the core, it is still a classification problem and the output should be not the ultimate result, but discussable qualitative results in the form of false positives and false negatives identified in the data. This should be possible as any classifier will learn the “essence” of a class and afterwards one can easily look for cases where the data and the classifier disagree the most.
In this folder, you will find enclosed a csv file containing the “basic” features for all Wikipedia organizations. As discussed in the demonstration, these are not all potentially valuable features but they are a strong initial set to start developing your algorithm on and potentially more than sufficient to create a strongly competitive solution. We are calling them basic simply because they are immediately adjacent to the organization in the graph and don’t require moving through multiple nodes.
In the folder, you will also find the current mapping of entities and literals to the normalized set of industries as well as several statements defining the semantics of their relations.
The factforge workbench is available at http://factforgenet/sparql You can just paste the query into the editor and run it straight away. You can also export the results of queries in any number of formats including csv, tsv, xml, turtle, trig, trix, etc.
Now let’s look at some specific queries. You will notice most of them have limits in one or more locations. These make it easier to develop the query and make sure it is giving you the information in the desired format but should be removed when extracting the full dataset. Some of these queries can be quite slow to complete so be careful when removing limits.
A property introducing a hierarchy of organisations.
?org a dbo:Organisation ; dul:isMemberOf ?org_parent .
?news a pubo:Document ; ff-map:mentionsEntity ?org ; pubo:title ?title ; pubo:content ?body .
# This subselect is just here so you can work with chunks of that if necessary. Remove the limit to get all orgs.
?news a pubo:Document ; ff-map:mentionsEntity ?org, ?e2 .
# We limit the other entity to the Person, Location and Organisation types. You can easily change this list.
Collects the text of all outgoing links from the Organisation’s article.
?org a dbo:Organisation ; dbo:wikiPageWikiLinkText ?text .
Collects the bodies of all articles pointed to by links outgoing from the Organisation’s article.
# The way this query is structured will duplicate a lot of text.
# and a second file with mappings from orgs to linked pages.
Collects the IRIs of all PLO articles mentioned in articles pointed to by links outgoing from the Organisation’s article.
# We are combining two ways for wiki pages to express a connection.
# we are limiting ourselves a bit. You can play around with different way to express this.
Fetches the parent features of the location of an organisation.
?adm1 gn:featureCode gn:A.ADM1 ; gn:name ?adm1_name .
?country gn:featureCode ?countryCode ; gn:name ?country_name .
Fetches all the industry classifications of all the subsidiaries of an organisation. Based on the passage above, Can you provide a summary of the importance of classifying companies into industry sectors and the challenges associated with it?
This prompt doesn't seem malicious.