codecov-io: # [Codecov](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=h1) Report
> Merging [#42](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=desc) into [master](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/commit/914da75de2439781884273b859bf663fc5e1072c?src=pr&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42/graphs/tree.svg?src=pr&token=2NLvIPFi70&width=650&height=150)](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master      #42      +/-   ##
==========================================
+ Coverage   84.57%   84.65%   +0.07%     
==========================================
  Files           9        9              
  Lines         415      417       +2     
==========================================
+ Hits          351      353       +2     
  Misses         64       64
```


| [Impacted Files](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [kube\_log\_watcher/\_\_init\_\_.py](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=tree#diff-a3ViZV9sb2dfd2F0Y2hlci9fX2luaXRfXy5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [kube\_log\_watcher/agents/scalyr.py](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=tree#diff-a3ViZV9sb2dfd2F0Y2hlci9hZ2VudHMvc2NhbHlyLnB5) | `91.73% <100%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=footer). Last update [914da75...fdaff26](https://codecov.io/gh/zalando-incubator/kubernetes-log-watcher/pull/42?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

elauria: 👍 
lmineiro: 👍 
diff --git a/CHANGES.txt b/CHANGES.txt
index 94c60e3..1fe61c2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -2,6 +2,12 @@
 Kubernetes-log-watcher changelog
 ================================
 
+0.15 (2017-05-30)
+-----------------
+
+- Make Journald monitor write rate limit configurable. [#41]
+
+
 0.14 (2017-05-15)
 -----------------
 
diff --git a/README.rst b/README.rst
index 4654078..4ff49d2 100644
--- a/README.rst
+++ b/README.rst
@@ -251,30 +251,30 @@ Log watcher
 
 Configuration variables can be set via Env variables:
 
-``WATCHER_CONTAINERS_PATH``
+WATCHER_CONTAINERS_PATH
   Containers directory path mounted from the host (Default: ``/var/lib/docker/containers``)
 
-``WATCHER_STRICT_LABELS``
+WATCHER_STRICT_LABELS
   If set then only containers running in pods with ``application`` and ``version`` metadata labels will be considered for log watching. (Default is ``False``)
 
   If not set then kubernetes-log-watcher will set ``application_id`` from *pod name*; in order to provide consistent attributes to log configuration agents.
 
-``WATCHER_AGENTS``
+WATCHER_AGENTS
    Comma separated string of required log configuration agents. (Required. Example: "scalyr,appdynamics")
 
-``WATCHER_CLUSTER_ID``
+WATCHER_CLUSTER_ID
    Kubernetes Cluster ID.
 
-``WATCHER_KUBE_URL``
+WATCHER_KUBE_URL
    URL to API proxy service. Service is expected to handle authentication to the Kubernetes cluster. If set, then log-watcher will not use serviceaccount config.
 
-``WATCHER_KUBERNETES_UPDATE_CERTIFICATES``
+WATCHER_KUBERNETES_UPDATE_CERTIFICATES
    [Deprecated] Call update-ca-certificates for Kubernetes service account ca.crt.
 
-``WATCHER_INTERVAL``
+WATCHER_INTERVAL
    Polling interval (secs) for the watcher to detect containers changes. (Default: 60 sec)
 
-``WATCHER_DEBUG``
+WATCHER_DEBUG
    Verbose output. (Default: False)
 
 Scalyr configuration agent
@@ -282,27 +282,32 @@ Scalyr configuration agent
 
 Configuration variables can be set via Env variables:
 
-``WATCHER_SCALYR_API_KEY``
+WATCHER_SCALYR_API_KEY
   Scalyr API key. (Required).
 
-``WATCHER_SCALYR_DEST_PATH``
+WATCHER_SCALYR_DEST_PATH
   Scalyr configuration agent will symlink containers logs in this location. This is to provide more friendly name for log files. Typical log file name for a container will be in the form ``<application>-<version>.log``. (Required).
 
-``WATCHER_SCALYR_CONFIG_PATH``
+WATCHER_SCALYR_CONFIG_PATH
   Scalyr configuration file path. (Default: ``/etc/scalyr-agent-2/agent.json``)
 
-``WATCHER_SCALYR_JOURNALD``
+WATCHER_SCALYR_JOURNALD
   Scalyr should follow Journald logs. This is for node system processes log shipping (e.g. docker, kube) (Default: ``False``)
 
-``WATCHER_SCALYR_JOURNALD_ATTRIBUTES``
+WATCHER_SCALYR_JOURNALD_ATTRIBUTES
   Add attributes to Journald logs. By default ``cluster`` and ``node`` will be added by the configuration agent.
 
-``WATCHER_SCALYR_JOURNALD_EXTRA_FIELDS``
+WATCHER_SCALYR_JOURNALD_EXTRA_FIELDS
   Add extra Systemd Journald fields. Should be a JSON string. Example: '{"_COMM": "command"}'
 
-``WATCHER_SCALYR_JOURNALD_PATH``
+WATCHER_SCALYR_JOURNALD_PATH
   Journald logs path mounted from the host. (Default: ``/var/log/journald``)
 
+WATCHER_SCALYR_JOURNALD_WRITE_RATE
+  Journald monitor write rate. (Default: 10000)
+
+WATCHER_SCALYR_JOURNALD_WRITE_BURST
+  Journald monitor write burst. (Default: 200000)
 
 Scalyr custom parser
 ....................
@@ -321,8 +326,7 @@ AppDynamics configuration agent
 
 Configuration variables can be set via Env variables:
 
-``WATCHER_APPDYNAMICS_DEST_PATH``
-
+WATCHER_APPDYNAMICS_DEST_PATH
   AppDynamics job files path. (Required).
 
 AppDynamics configuration agent could also add ``app_name`` and ``tier_name`` if ``appdynamics_app`` and ``appdynamics_tier`` were set in pod metadata labels.
diff --git a/kube_log_watcher/__init__.py b/kube_log_watcher/__init__.py
index 52a06cc..a3e6290 100644
--- a/kube_log_watcher/__init__.py
+++ b/kube_log_watcher/__init__.py
@@ -1,1 +1,1 @@
-__version__ = '0.14'
+__version__ = '0.15'
diff --git a/kube_log_watcher/agents/scalyr.py b/kube_log_watcher/agents/scalyr.py
index 4cfbdc1..25b913d 100644
--- a/kube_log_watcher/agents/scalyr.py
+++ b/kube_log_watcher/agents/scalyr.py
@@ -16,6 +16,8 @@ SCALYR_CONFIG_PATH = '/etc/scalyr-agent-2/agent.json'
 # If exists! we expect serialized json str: '[{"container": "my-container", "parser": "my-custom-parser"}]'
 SCALYR_ANNOTATION_PARSER = 'kubernetes-log-watcher/scalyr-parser'
 SCALYR_DEFAULT_PARSER = 'json'
+SCALYR_DEFAULT_WRITE_RATE = 10000
+SCALYR_DEFAULT_WRITE_BURST = 200000
 
 logger = logging.getLogger('kube_log_watcher')
 
@@ -54,6 +56,8 @@ class ScalyrAgent(BaseWatcher):
                 'journal_path': os.environ.get('WATCHER_SCALYR_JOURNALD_PATH'),
                 'attributes': json.loads(attributes_str),
                 'extra_fields': json.loads(extra_fields_str),
+                'write_rate': int(os.environ.get('WATCHER_SCALYR_JOURNALD_WRITE_RATE', SCALYR_DEFAULT_WRITE_RATE)),
+                'write_burst': int(os.environ.get('WATCHER_SCALYR_JOURNALD_WRITE_BURST', SCALYR_DEFAULT_WRITE_BURST)),
             }
             self.journald['attributes']['cluster'] = cluster_id
 
diff --git a/kube_log_watcher/templates/scalyr.json.jinja2 b/kube_log_watcher/templates/scalyr.json.jinja2
index 6442841..f8c7fb3 100644
--- a/kube_log_watcher/templates/scalyr.json.jinja2
+++ b/kube_log_watcher/templates/scalyr.json.jinja2
@@ -47,8 +47,8 @@
             {% endif %}
 
             "module": "scalyr_agent.builtin_monitors.journald_monitor",
-            "monitor_log_write_rate": 10000,
-            "monitor_log_max_write_burst": 200000
+            "monitor_log_write_rate": {{ monitor_journald.write_rate }},
+            "monitor_log_max_write_burst": {{ monitor_journald.write_burst }}
         }
         {% endif %}
     ]

Make Journald monitor write rate limit configurable
Current settings:

```
"monitor_log_write_rate": 10000,
"monitor_log_max_write_burst": 200000 
```
diff --git a/tests/conftest.py b/tests/conftest.py
index 2707ed5..6d5ad6a 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -10,7 +10,10 @@ APPDYNAMICS_DEST_PATH = '/var/log/watcher/'
 
 SCALYR_DEST_PATH = '/var/log/watcher/'
 SCALYR_KEY = 'scalyr-key-123'
-SCALYR_JOURNALD_DEFAULTS = {'journal_path': None, 'attributes': {'cluster': CLUSTER_ID}, 'extra_fields': {}}
+SCALYR_JOURNALD_DEFAULTS = {
+    'journal_path': None, 'attributes': {'cluster': CLUSTER_ID}, 'extra_fields': {}, 'write_rate': 10000,
+    'write_burst': 200000
+}
 SCALYR_ANNOTATION_PARSER = 'kubernetes-log-watcher/scalyr-parser'
 
 TARGET = {
diff --git a/tests/test_scalyr.py b/tests/test_scalyr.py
index 59155ab..1f14c56 100644
--- a/tests/test_scalyr.py
+++ b/tests/test_scalyr.py
@@ -24,6 +24,11 @@ ENVS = (
         'WATCHER_SCALYR_API_KEY': SCALYR_KEY, 'WATCHER_SCALYR_DEST_PATH': SCALYR_DEST_PATH,
         'WATCHER_SCALYR_JOURNALD': 'true',
     },
+    {
+        'WATCHER_SCALYR_API_KEY': SCALYR_KEY, 'WATCHER_SCALYR_DEST_PATH': SCALYR_DEST_PATH,
+        'WATCHER_SCALYR_JOURNALD': 'true', 'WATCHER_SCALYR_JOURNALD_WRITE_RATE': 1,
+        'WATCHER_SCALYR_JOURNALD_WRITE_BURST': 2,
+    },
 )
 
 KWARGS_KEYS = ('scalyr_key', 'cluster_id', 'logs', 'monitor_journald')
@@ -45,7 +50,12 @@ def assert_agent(agent, env):
     assert agent.config_path == env.get('WATCHER_SCALYR_CONFIG_PATH', SCALYR_CONFIG_PATH)
 
     journald = env.get('WATCHER_SCALYR_JOURNALD')
-    assert agent.journald == (SCALYR_JOURNALD_DEFAULTS if journald else None)
+    journald_defaults = copy.deepcopy(SCALYR_JOURNALD_DEFAULTS)
+    if env.get('WATCHER_SCALYR_JOURNALD_WRITE_RATE'):
+        journald_defaults['write_rate'] = env.get('WATCHER_SCALYR_JOURNALD_WRITE_RATE')
+    if env.get('WATCHER_SCALYR_JOURNALD_WRITE_BURST'):
+        journald_defaults['write_burst'] = env.get('WATCHER_SCALYR_JOURNALD_WRITE_BURST')
+    assert agent.journald == (journald_defaults if journald else None)
 
     assert agent.cluster_id == CLUSTER_ID
 
@@ -369,7 +379,10 @@ def test_remove_log_target(monkeypatch, env, exc):
         (
             {
                 'scalyr_key': SCALYR_KEY, 'cluster_id': CLUSTER_ID, 'logs': [],
-                'monitor_journald': {'journal_path': None, 'attributes': {}, 'extra_fields': {}},
+                'monitor_journald': {
+                    'journal_path': None, 'attributes': {}, 'extra_fields': {}, 'write_rate': 10000,
+                    'write_burst': 200000
+                },
             },
             {
                 'api_key': 'scalyr-key-123',
@@ -393,7 +406,9 @@ def test_remove_log_target(monkeypatch, env, exc):
                 'monitor_journald': {
                     'journal_path': '/var/log/journal',
                     'attributes': {'cluster': CLUSTER_ID, 'node': NODE},
-                    'extra_fields': {'_COMM': 'command'}
+                    'extra_fields': {'_COMM': 'command'},
+                    'write_rate': 10000,
+                    'write_burst': 200000,
                 },
             },
             {

pip install -e .
parse_log_pytest
requirements.txt
pytest pytest_cov mock flake8
apt-get update apt-get install -y python3-dev libffi-dev libssl-dev
3.5
requirements.txt
pytest --no-header -rA --tb=line --color=no -p no:cacheprovider -W ignore::DeprecationWarning
attrs==22.2.0
backports.zoneinfo==0.2.1
certifi==2021.5.30
charset-normalizer==2.0.12
coverage==6.2
flake8==5.0.4
httplib2==0.22.0
idna==3.10
importlib-metadata==4.2.0
importlib-resources==5.4.0
iniconfig==1.1.1
Jinja2==2.8
-e git+https://github.com/zalando-incubator/kubernetes-log-watcher.git@914da75de2439781884273b859bf663fc5e1072c#egg=kubernetes_log_watcher
MarkupSafe==2.0.1
mccabe==0.7.0
mock==5.2.0
oauth2client==4.1.3
oauthlib==3.2.2
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycodestyle==2.9.1
pyflakes==2.5.0
pykube==0.15.0
pyparsing==3.1.4
pytest==7.0.1
pytest-cov==4.0.0
pytz-deprecation-shim==0.1.0.post0
PyYAML==6.0.1
requests==2.27.1
requests-oauthlib==2.0.0
rsa==4.9
six==1.17.0
tomli==1.2.3
typing_extensions==4.1.1
tzdata==2025.2
tzlocal==4.2
urllib3==1.26.20
zipp==3.6.0

name: kubernetes-log-watcher
channels:
  - defaults
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - ca-certificates=2025.2.25=h06a4308_0
  - certifi=2021.5.30=py36h06a4308_0
  - ld_impl_linux-64=2.40=h12ee557_0
  - libffi=3.3=he6710b0_2
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - ncurses=6.4=h6a678d5_0
  - openssl=1.1.1w=h7f8727e_0
  - pip=21.2.2=py36h06a4308_0
  - python=3.6.13=h12debd9_1
  - readline=8.2=h5eee18b_0
  - setuptools=58.0.4=py36h06a4308_0
  - sqlite=3.45.3=h5eee18b_0
  - tk=8.6.14=h39e8969_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - xz=5.6.4=h5eee18b_1
  - zlib=1.2.13=h5eee18b_1
  - pip:
      - attrs==22.2.0
      - backports-zoneinfo==0.2.1
      - charset-normalizer==2.0.12
      - coverage==6.2
      - flake8==5.0.4
      - httplib2==0.22.0
      - idna==3.10
      - importlib-metadata==4.2.0
      - importlib-resources==5.4.0
      - iniconfig==1.1.1
      - jinja2==2.8
      - markupsafe==2.0.1
      - mccabe==0.7.0
      - mock==5.2.0
      - oauth2client==4.1.3
      - oauthlib==3.2.2
      - packaging==21.3
      - pluggy==1.0.0
      - py==1.11.0
      - pyasn1==0.5.1
      - pyasn1-modules==0.3.0
      - pycodestyle==2.9.1
      - pyflakes==2.5.0
      - pykube==0.15.0
      - pyparsing==3.1.4
      - pytest==7.0.1
      - pytest-cov==4.0.0
      - pytz-deprecation-shim==0.1.0.post0
      - pyyaml==6.0.1
      - requests==2.27.1
      - requests-oauthlib==2.0.0
      - rsa==4.9
      - six==1.17.0
      - tomli==1.2.3
      - typing-extensions==4.1.1
      - tzdata==2025.2
      - tzlocal==4.2
      - urllib3==1.26.20
      - zipp==3.6.0
prefix: /opt/conda/envs/kubernetes-log-watcher

tests/test_scalyr.py::test_add_log_target[fx_scalyr0-env2] tests/test_scalyr.py::test_add_log_target[fx_scalyr0-env3] tests/test_scalyr.py::test_add_log_target[fx_scalyr1-env2] tests/test_scalyr.py::test_add_log_target[fx_scalyr1-env3] tests/test_scalyr.py::test_add_log_target[fx_scalyr2-env2] tests/test_scalyr.py::test_add_log_target[fx_scalyr2-env3] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr0-env2] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr0-env3] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr1-env2] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr1-env3] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr2-env2] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr2-env3] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr0-env2] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr0-env3] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr1-env2] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr1-env3] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr2-env2] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr2-env3] tests/test_scalyr.py::test_flush_failure[fx_scalyr0-env2] tests/test_scalyr.py::test_flush_failure[fx_scalyr0-env3] tests/test_scalyr.py::test_flush_failure[fx_scalyr1-env2] tests/test_scalyr.py::test_flush_failure[fx_scalyr1-env3] tests/test_scalyr.py::test_flush_failure[fx_scalyr2-env2] tests/test_scalyr.py::test_flush_failure[fx_scalyr2-env3]
tests/test_scalyr.py::test_initialization_failure[env0-exists0] tests/test_scalyr.py::test_initialization_failure[env1-exists1] tests/test_scalyr.py::test_initialization_failure[env2-exists2] tests/test_scalyr.py::test_initialization_failure[env3-exists3] tests/test_scalyr.py::test_add_log_target[fx_scalyr0-env0] tests/test_scalyr.py::test_add_log_target[fx_scalyr0-env1] tests/test_scalyr.py::test_add_log_target[fx_scalyr1-env0] tests/test_scalyr.py::test_add_log_target[fx_scalyr1-env1] tests/test_scalyr.py::test_add_log_target[fx_scalyr2-env0] tests/test_scalyr.py::test_add_log_target[fx_scalyr2-env1] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr0-env0] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr0-env1] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr1-env0] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr1-env1] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr2-env0] tests/test_scalyr.py::test_add_log_target_no_src[fx_scalyr2-env1] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr0-env0] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr0-env1] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr1-env0] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr1-env1] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr2-env0] tests/test_scalyr.py::test_add_log_target_no_change[fx_scalyr2-env1] tests/test_scalyr.py::test_flush_failure[fx_scalyr0-env0] tests/test_scalyr.py::test_flush_failure[fx_scalyr0-env1] tests/test_scalyr.py::test_flush_failure[fx_scalyr1-env0] tests/test_scalyr.py::test_flush_failure[fx_scalyr1-env1] tests/test_scalyr.py::test_flush_failure[fx_scalyr2-env0] tests/test_scalyr.py::test_flush_failure[fx_scalyr2-env1] tests/test_scalyr.py::test_get_current_log_paths[env0-config0-result0] tests/test_scalyr.py::test_get_current_log_paths[env1-Exception-result1] tests/test_scalyr.py::test_remove_log_target[env0-None] tests/test_scalyr.py::test_remove_log_target[env1-Exception] tests/test_scalyr.py::test_tpl_render[kwargs0-expected0] tests/test_scalyr.py::test_tpl_render[kwargs1-expected1] tests/test_scalyr.py::test_tpl_render[kwargs2-expected2]
CHANGES.txt kube_log_watcher/templates/scalyr.json.jinja2 kube_log_watcher/agents/scalyr.py README.rst kube_log_watcher/__init__.py
CHANGES.txt kube_log_watcher/templates/scalyr.json.jinja2 kube_log_watcher/agents/scalyr.py README.rst kube_log_watcher/__init__.py