<p dir="auto">From <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/water4u99/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/water4u99">@water4u99</a> on 2016-11-18T22:28:43Z</p>
<h5 dir="auto">ISSUE TYPE</h5>
<ul dir="auto">
<li>Bug Report</li>
</ul>
<h5 dir="auto">COMPONENT NAME</h5>
<p dir="auto">template</p>
<h5 dir="auto">ANSIBLE VERSION</h5>
<p dir="auto">2.0 and higher</p>
<h5 dir="auto">CONFIGURATION</h5>
<p dir="auto">[ssh_connection]<br>
control_path = %(directory)s/%%C</p>
<h5 dir="auto">OS / ENVIRONMENT</h5>
<p dir="auto">Mac OS X 10.11.6<br>
Centos 6.x, 7.x</p>
<h5 dir="auto">SUMMARY</h5>
<p dir="auto">In the input .j2 file, we substitute a variable with an environment variable that has a line/string that contains a grok expression containing "(?m)\n" .  The output generated by the template module in versions 2.0 and later, treats the \n as actual line break.  Where as versions up to 1.9.6 retains the literal "(?m)\n" without replacing the \n with an actual line break.  We see the line break after we upgraded the Ansible version to 2.x.</p>
<p dir="auto">Any way we can work around this issue?  Thank you for your help.</p>
<h5 dir="auto">STEPS TO REPRODUCE</h5>

<p dir="auto">Our execution flow is probably not the nicest - we want to reengineer it soon.  Basic steps:</p>
<ol dir="auto">
<li>Run a shell script with ansible-playbook command that pass in an env variable with (?m)\n literal.</li>
<li>Playbook calls a main yaml file and assigns shell environment var to a included task yaml file.</li>
<li>The task yaml file invokes the template module.</li>
</ol>
<p dir="auto">In the snippet below I stripped out other lines/vars for clarity.</p>

<p dir="auto">main shell</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="set GROK_PATTERN_GENERAL_ERROR_PG=&quot;%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})&quot;

ansible-playbook -i ../common/host.inventory \
                -${VERBOSE} \
                t.yml \
                ${CHECK_ONLY} \
                --extra-vars &quot;hosts='${HOST}'
                              xlogstash_grok_general_error='${GROK_PATTERN_GENERAL_ERROR_PG}'
                             &quot;"><pre class="notranslate"><code class="notranslate">set GROK_PATTERN_GENERAL_ERROR_PG="%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})"

ansible-playbook -i ../common/host.inventory \
                -${VERBOSE} \
                t.yml \
                ${CHECK_ONLY} \
                --extra-vars "hosts='${HOST}'
                              xlogstash_grok_general_error='${GROK_PATTERN_GENERAL_ERROR_PG}'
                             "
</code></pre></div>
<p dir="auto">t.yml</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---
- hosts: 127.0.0.1
  connection: local

  tasks:
    - include_vars: ../common/defaults/main.yml
    - name: generate logstash kafka logscan filter config file
      include: tasks/t.yml
      vars:
          logstash_grok_general_error: &quot;{{xlogstash_grok_general_error}}&quot;
"><pre class="notranslate"><code class="notranslate">---
- hosts: 127.0.0.1
  connection: local

  tasks:
    - include_vars: ../common/defaults/main.yml
    - name: generate logstash kafka logscan filter config file
      include: tasks/t.yml
      vars:
          logstash_grok_general_error: "{{xlogstash_grok_general_error}}"

</code></pre></div>
<p dir="auto">tasks/t.yml</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---
    - name: generate logstash kafka logscan filter config file
      template: src=../common/templates/my.conf.j2
                dest=&quot;./500-filter.conf&quot;
"><pre class="notranslate"><code class="notranslate">---
    - name: generate logstash kafka logscan filter config file
      template: src=../common/templates/my.conf.j2
                dest="./500-filter.conf"

</code></pre></div>
<p dir="auto">my.conf.j2</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        grok {
                break_on_match =&gt; &quot;true&quot;
                match =&gt; [
                        &quot;message&quot;, &quot;{{logstash_grok_general_error}}&quot;
                ]
        }"><pre class="notranslate"><code class="notranslate">        grok {
                break_on_match =&gt; "true"
                match =&gt; [
                        "message", "{{logstash_grok_general_error}}"
                ]
        }
</code></pre></div>

<p dir="auto">Note the (?m)\n are still on the same line.</p>
<h5 dir="auto">EXPECTED RESULTS</h5>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        grok {
                break_on_match =&gt; &quot;true&quot;
                match =&gt; [
                        &quot;message&quot;, &quot;%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})&quot;
                ]
        }"><pre class="notranslate"><code class="notranslate">        grok {
                break_on_match =&gt; "true"
                match =&gt; [
                        "message", "%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})"
                ]
        }
</code></pre></div>
<h5 dir="auto">ACTUAL RESULTS</h5>

<p dir="auto">Note (?m)\n now has the \n as actual line break.</p>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        grok {
                break_on_match =&gt; &quot;true&quot;
                match =&gt; [
                        &quot;message&quot;, &quot;%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)
%{USER:logerror}%{GREEDYDATA})&quot;
                ]
        }"><pre class="notranslate"><code class="notranslate">        grok {
                break_on_match =&gt; "true"
                match =&gt; [
                        "message", "%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)
%{USER:logerror}%{GREEDYDATA})"
                ]
        }
</code></pre></div>
<p dir="auto">Copied from original issue: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="190437223" data-permission-text="Title is private" data-url="https://github.com/ansible/ansible-modules-core/issues/5674" data-hovercard-type="issue" data-hovercard-url="/ansible/ansible-modules-core/issues/5674/hovercard" href="https://github.com/ansible/ansible-modules-core/issues/5674">ansible/ansible-modules-core#5674</a></p>
<h5 dir="auto">ISSUE TYPE</h5>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Bug Report"><pre class="notranslate"><code class="notranslate">Bug Report
</code></pre></div>
<h5 dir="auto">COMPONENT NAME</h5>
<p dir="auto">template</p>
<h5 dir="auto">ANSIBLE VERSION</h5>
<p dir="auto">2.0 and higher<br>
CONFIGURATION</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[ssh_connection]
control_path = %(directory)s/%%C"><pre class="notranslate"><code class="notranslate">[ssh_connection]
control_path = %(directory)s/%%C
</code></pre></div>
<h5 dir="auto">OS / ENVIRONMENT</h5>
<p dir="auto">Mac OS X 10.11.6<br>
Centos 6.x, 7.x<br>
SUMMARY</p>
<p dir="auto">In the input .j2 file, we substitute a variable with an environment variable that has a line/string that contains a grok expression containing <code class="notranslate">(?m)\n</code> . The output generated by the template module in versions 2.0 and later, treats the \n as actual line break. Where as versions up to 1.9.6 retains the literal <code class="notranslate">(?m)\n</code> without replacing the \n with an actual line break. We see the line break after we upgraded the Ansible version to 2.x.</p>
<p dir="auto">Any way we can work around this issue? Thank you for your help.</p>
<h5 dir="auto">STEPS TO REPRODUCE</h5>
<p dir="auto">Our execution flow is probably not the nicest - we want to reengineer it soon. Basic steps:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Run a shell script with ansible-playbook command that pass in an env variable with `(?m)\n` literal.
Playbook calls a main yaml file and assigns shell environment var to a included task yaml file.
The task yaml file invokes the template module."><pre class="notranslate"><code class="notranslate">Run a shell script with ansible-playbook command that pass in an env variable with `(?m)\n` literal.
Playbook calls a main yaml file and assigns shell environment var to a included task yaml file.
The task yaml file invokes the template module.
</code></pre></div>
<p dir="auto">In the snippet below I stripped out other lines/vars for clarity.</p>
<p dir="auto">main shell</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="set GROK_PATTERN_GENERAL_ERROR_PG=&quot;%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})&quot;"><pre class="notranslate"><code class="notranslate">set GROK_PATTERN_GENERAL_ERROR_PG="%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})"
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ansible-playbook -i ../common/host.inventory \
                -${VERBOSE} \
                t.yml \
                ${CHECK_ONLY} \
                --extra-vars &quot;hosts='${HOST}'
                              xlogstash_grok_general_error='${GROK_PATTERN_GENERAL_ERROR_PG}'
                             &quot;"><pre class="notranslate"><code class="notranslate">ansible-playbook -i ../common/host.inventory \
                -${VERBOSE} \
                t.yml \
                ${CHECK_ONLY} \
                --extra-vars "hosts='${HOST}'
                              xlogstash_grok_general_error='${GROK_PATTERN_GENERAL_ERROR_PG}'
                             "
</code></pre></div>
<p dir="auto">t.yml</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---
- hosts: 127.0.0.1
  connection: local

  tasks:
    - include_vars: ../common/defaults/main.yml
    - name: generate logstash kafka logscan filter config file
      include: tasks/t.yml
      vars:
          logstash_grok_general_error: &quot;{{xlogstash_grok_general_error}}&quot;"><pre class="notranslate"><code class="notranslate">---
- hosts: 127.0.0.1
  connection: local

  tasks:
    - include_vars: ../common/defaults/main.yml
    - name: generate logstash kafka logscan filter config file
      include: tasks/t.yml
      vars:
          logstash_grok_general_error: "{{xlogstash_grok_general_error}}"
</code></pre></div>
<p dir="auto">tasks/t.yml</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---
    - name: generate logstash kafka logscan filter config file
      template: src=../common/templates/my.conf.j2
                dest=&quot;./500-filter.conf&quot;"><pre class="notranslate"><code class="notranslate">---
    - name: generate logstash kafka logscan filter config file
      template: src=../common/templates/my.conf.j2
                dest="./500-filter.conf"
</code></pre></div>
<p dir="auto">my.conf.j2</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        grok {
                break_on_match =&gt; &quot;true&quot;
                match =&gt; [
                        &quot;message&quot;, &quot;{{logstash_grok_general_error}}&quot;
                ]
        }"><pre class="notranslate"><code class="notranslate">        grok {
                break_on_match =&gt; "true"
                match =&gt; [
                        "message", "{{logstash_grok_general_error}}"
                ]
        }
</code></pre></div>
<p dir="auto">Note the <code class="notranslate">(?m)\n</code> are still on the same line.</p>
<h5 dir="auto">EXPECTED RESULTS</h5>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        grok {
                break_on_match =&gt; &quot;true&quot;
                match =&gt; [
                        &quot;message&quot;, &quot;%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})&quot;
                ]
        }"><pre class="notranslate"><code class="notranslate">        grok {
                break_on_match =&gt; "true"
                match =&gt; [
                        "message", "%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})"
                ]
        }
</code></pre></div>
<h5 dir="auto">ACTUAL RESULTS</h5>
<p dir="auto">Note <code class="notranslate">(?m)\n</code> now has the <code class="notranslate">\n</code> as actual line break.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        grok {
                break_on_match =&gt; &quot;true&quot;
                match =&gt; [
                        &quot;message&quot;, &quot;%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)
%{USER:logerror}%{GREEDYDATA})&quot;
                ]
        }"><pre class="notranslate"><code class="notranslate">        grok {
                break_on_match =&gt; "true"
                match =&gt; [
                        "message", "%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)
%{USER:logerror}%{GREEDYDATA})"
                ]
        }
</code></pre></div>