@tushar.bansal (Tushar) posted a new topic, subject "Parse subdomain from fqdn in logstash". It read:

   Hi,
    
    I have several events coming in logstash with info of web link accessed.
    
    example:  
    DNS query is called for the name **crl.sectigo.com** , type 28  
    DNS query is called for the name **repo.maven.apache.org** , type 2  
    DNS query is called for the name **crl.maven.apache.random.org** , type 1
    
    I parsed them using grok in a separate field.
    
    example:  
    crl.sectigo.com  
    repo.maven.apache.org  
    crl.maven.apache.random.org
    
    I want to have one more field with just last 2 subdomains.
    
    example:  
    sectigo.com  
    apache.org  
    random.org
    
    I tried many logics with grok etc. but of no use. Thing is I can get a random domain name, no clue with how many subdomains in it. I need to have only last 2 in a field.

@Cad replied

    Hi,
    
    This grok pattern give you the result you want :  
    `%{WORD:name}.%{WORD:TLD}$`  
    With this pattern, you sai you want two words separated by a dot and they are placed at the end of your field (with the character $)
    
    Cad.

@legoguy1000 (Alex) replied

    You have to be careful with this because country tlds have an extra characters on the front of a domain. So `qbc.co` becomes `qbc.co.uk` and this logic would only return `co.uk`. I don't think logstash has a registered domain processor but filebeat and elasticsearch do. See Registered Domain | Filebeat Reference [7.12] | Elastic.

Then @cknz (Cameron Kerr) said

    The 'tld' filter exists precisely for this purpose. To do this correctly is very non-trivial and cannot be done algorithmically; you need to use the 'public suffix' data, which is what the 'tld' plugin (and others like it) use.
    
    Cheers,  
    Cameron
    
    (PS. In case you come across a plugin I wrote called logstash-filter-dnssummary, I would suggest you stick with the 'tld' plugin, just because its more maintained; unless perhaps you care about Unicode and IDNA)

Then @tushar.bansal (Tushar) said

    Thanks all for tld suggestion. Thing is, my logstash server does not have access to internet. And tld filter needs to be manually installed. It does not come with the bundle. Any suggestions?

To which @Badger replied

    tushar.bansal:
    
    > Thing is, my logstash server does not have access to internet.
    
    Use the plugin manager.

To which @cknz (Cameron Kerr) replied

    Or if you only lack **direct** access to the internet, but can still go through a HTTP proxy, you could use it via a proxy (port 3128 is the default port for a Squid proxy; other proxies vary)
    [code] 
        https_proxy=myproxy.example.com:3128 http_proxy=myproxy.example.com:3128 logstash_plugin install logstash-filter-tld
        
    [/code]

To which @system (system) replied

    This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.


{'source': 'discourse', 'url': 'https://discuss.elastic.co/t/parse-subdomain-from-fqdn-in-logstash/273535', 'title': 'Parse subdomain from fqdn in logstash', 'views': 317, 'has_accepted_answer': True, 'word_count': 453, 'like_count': 0, 'reply_count': 4, 'participant_count': 5, 'posts': 8, 'tags': '', 'author': '@tushar.bansal (Tushar)', 'participants': '@tushar.bansal (Tushar), @Cad, @legoguy1000 (Alex), @cknz (Cameron Kerr), @Badger, @system (system)', 'image_url': [None], 'created_at': '2021-05-20T12:39:07.360Z', 'last_posted_at': '2021-06-21T23:46:10.233Z'}