prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  AirflowMacroPluginRemovedRule fails on non-python files
  **Apache Airflow version**: 1.10.14
  
  
  **Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
  
  **Environment**:
  
  - **Cloud provider or hardware configuration**: X
  - **OS** (e.g. from /etc/os-release): X
  - **Kernel** (e.g. `uname -a`): X
  - **Install tools**: X
  - **Others**: X
  
  **What happened**:
  
  The `AirflowMacroPluginRemovedRule` seems unable to process non-standard python files (e.g. `.xlsx`) and chokes out with an unhelpful error message.:
  
  ```python
  ========================================================================================================================================================== STATUS ==========================================================================================================================================================
  
  Check for latest versions of apache-airflow and checker...........................................................................................................................................................................................................................................................SUCCESS
  Traceback (most recent call last):
    File "/Users/madison/programs/anaconda3/envs/memphis-airflow/bin/airflow", line 37, in <module>
      args.func(args)
    File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 88, in run
      all_problems = check_upgrade(formatter, rules)
    File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 37, in check_upgrade
      rule_status = RuleStatus.from_rule(rule)
    File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/problem.py", line 44, in from_rule
      result = rule.check()
    File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/airflow_macro_plugin_removed.py", line 52, in check
      problems.extend(self._check_file(file_path))
    File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/airflow_macro_plugin_removed.py", line 42, in _check_file
      for line_number, line in enumerate(file_pointer, 1):
    File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/codecs.py", line 322, in decode
      (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 16: invalid start byte
  ```
  
  **What you expected to happen**:
  
  I expected the macro to skip over files it could not process/understand
  
  **How to reproduce it**:
  
  Add an `.xlsx` or other binary document to the DAGs folder and run the upgrade check.
  
  
  **Suggested resolution**:
  
  I think it's fine to fail out on these files (it led us to add certain items to the `.airflowignore` which should have been there anyway) but I had to modify the upgrade rule directly to tell me _which_ files were the problem. A more helpful error message here, and possibly a message prompting users to add said files to their `.airflowignore` would be ideal.
  
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `airflow/upgrade/rules/airflow_macro_plugin_removed.py`
  Content:
  ```
  1 # Licensed to the Apache Software Foundation (ASF) under one
  2 # or more contributor license agreements.  See the NOTICE file
  3 # distributed with this work for additional information
  4 # regarding copyright ownership.  The ASF licenses this file
  5 # to you under the Apache License, Version 2.0 (the
  6 # "License"); you may not use this file except in compliance
  7 # with the License.  You may obtain a copy of the License at
  8 #
  9 #   http://www.apache.org/licenses/LICENSE-2.0
  10 #
  11 # Unless required by applicable law or agreed to in writing,
  12 # software distributed under the License is distributed on an
  13 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  14 # KIND, either express or implied.  See the License for the
  15 # specific language governing permissions and limitations
  16 # under the License.
  17 
  18 from __future__ import absolute_import
  19 
  20 from airflow import conf
  21 from airflow.upgrade.rules.base_rule import BaseRule
  22 from airflow.utils.dag_processing import list_py_file_paths
  23 
  24 
  25 class AirflowMacroPluginRemovedRule(BaseRule):
  26 
  27     title = "Remove airflow.AirflowMacroPlugin class"
  28 
  29     description = "The airflow.AirflowMacroPlugin class has been removed."
  30 
  31     MACRO_PLUGIN_CLASS = "airflow.AirflowMacroPlugin"
  32 
  33     def _change_info(self, file_path, line_number):
  34         return "{} will be removed. Affected file: {} (line {})".format(
  35             self.MACRO_PLUGIN_CLASS, file_path, line_number
  36         )
  37 
  38     def _check_file(self, file_path):
  39         problems = []
  40         class_name_to_check = self.MACRO_PLUGIN_CLASS.split(".")[-1]
  41         with open(file_path, "r") as file_pointer:
  42             for line_number, line in enumerate(file_pointer, 1):
  43                 if class_name_to_check in line:
  44                     problems.append(self._change_info(file_path, line_number))
  45         return problems
  46 
  47     def check(self):
  48         dag_folder = conf.get("core", "dags_folder")
  49         file_paths = list_py_file_paths(directory=dag_folder, include_examples=False)
  50         problems = []
  51         for file_path in file_paths:
  52             problems.extend(self._check_file(file_path))
  53         return problems
  54
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/airflow/upgrade/rules/airflow_macro_plugin_removed.py b/airflow/upgrade/rules/airflow_macro_plugin_removed.py
  --- a/airflow/upgrade/rules/airflow_macro_plugin_removed.py
  +++ b/airflow/upgrade/rules/airflow_macro_plugin_removed.py
  @@ -39,9 +39,12 @@
           problems = []
           class_name_to_check = self.MACRO_PLUGIN_CLASS.split(".")[-1]
           with open(file_path, "r") as file_pointer:
  -            for line_number, line in enumerate(file_pointer, 1):
  -                if class_name_to_check in line:
  -                    problems.append(self._change_info(file_path, line_number))
  +            try:
  +                for line_number, line in enumerate(file_pointer, 1):
  +                    if class_name_to_check in line:
  +                        problems.append(self._change_info(file_path, line_number))
  +            except UnicodeDecodeError:
  +                problems.append("Unable to read python file {}".format(file_path))
           return problems
   
       def check(self):
  @@ -49,5 +52,7 @@
           file_paths = list_py_file_paths(directory=dag_folder, include_examples=False)
           problems = []
           for file_path in file_paths:
  +            if not file_path.endswith(".py"):
  +                continue
               problems.extend(self._check_file(file_path))
           return problems
  
verification_info: {"golden_diff": "diff --git a/airflow/upgrade/rules/airflow_macro_plugin_removed.py b/airflow/upgrade/rules/airflow_macro_plugin_removed.py\n--- a/airflow/upgrade/rules/airflow_macro_plugin_removed.py\n+++ b/airflow/upgrade/rules/airflow_macro_plugin_removed.py\n@@ -39,9 +39,12 @@\n         problems = []\n         class_name_to_check = self.MACRO_PLUGIN_CLASS.split(\".\")[-1]\n         with open(file_path, \"r\") as file_pointer:\n-            for line_number, line in enumerate(file_pointer, 1):\n-                if class_name_to_check in line:\n-                    problems.append(self._change_info(file_path, line_number))\n+            try:\n+                for line_number, line in enumerate(file_pointer, 1):\n+                    if class_name_to_check in line:\n+                        problems.append(self._change_info(file_path, line_number))\n+            except UnicodeDecodeError:\n+                problems.append(\"Unable to read python file {}\".format(file_path))\n         return problems\n \n     def check(self):\n@@ -49,5 +52,7 @@\n         file_paths = list_py_file_paths(directory=dag_folder, include_examples=False)\n         problems = []\n         for file_path in file_paths:\n+            if not file_path.endswith(\".py\"):\n+                continue\n             problems.extend(self._check_file(file_path))\n         return problems\n", "issue": "AirflowMacroPluginRemovedRule fails on non-python files\n**Apache Airflow version**: 1.10.14\r\n\r\n\r\n**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):\r\n\r\n**Environment**:\r\n\r\n- **Cloud provider or hardware configuration**: X\r\n- **OS** (e.g. from /etc/os-release): X\r\n- **Kernel** (e.g. `uname -a`): X\r\n- **Install tools**: X\r\n- **Others**: X\r\n\r\n**What happened**:\r\n\r\nThe `AirflowMacroPluginRemovedRule` seems unable to process non-standard python files (e.g. `.xlsx`) and chokes out with an unhelpful error message.:\r\n\r\n```python\r\n========================================================================================================================================================== STATUS ==========================================================================================================================================================\r\n\r\nCheck for latest versions of apache-airflow and checker...........................................................................................................................................................................................................................................................SUCCESS\r\nTraceback (most recent call last):\r\n  File \"/Users/madison/programs/anaconda3/envs/memphis-airflow/bin/airflow\", line 37, in <module>\r\n    args.func(args)\r\n  File \"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py\", line 88, in run\r\n    all_problems = check_upgrade(formatter, rules)\r\n  File \"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py\", line 37, in check_upgrade\r\n    rule_status = RuleStatus.from_rule(rule)\r\n  File \"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/problem.py\", line 44, in from_rule\r\n    result = rule.check()\r\n  File \"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/airflow_macro_plugin_removed.py\", line 52, in check\r\n    problems.extend(self._check_file(file_path))\r\n  File \"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/airflow_macro_plugin_removed.py\", line 42, in _check_file\r\n    for line_number, line in enumerate(file_pointer, 1):\r\n  File \"/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/codecs.py\", line 322, in decode\r\n    (result, consumed) = self._buffer_decode(data, self.errors, final)\r\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 16: invalid start byte\r\n```\r\n\r\n**What you expected to happen**:\r\n\r\nI expected the macro to skip over files it could not process/understand\r\n\r\n**How to reproduce it**:\r\n\r\nAdd an `.xlsx` or other binary document to the DAGs folder and run the upgrade check.\r\n\r\n\r\n**Suggested resolution**:\r\n\r\nI think it's fine to fail out on these files (it led us to add certain items to the `.airflowignore` which should have been there anyway) but I had to modify the upgrade rule directly to tell me _which_ files were the problem. A more helpful error message here, and possibly a message prompting users to add said files to their `.airflowignore` would be ideal.\r\n\n", "before_files": [{"content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nfrom __future__ import absolute_import\n\nfrom airflow import conf\nfrom airflow.upgrade.rules.base_rule import BaseRule\nfrom airflow.utils.dag_processing import list_py_file_paths\n\n\nclass AirflowMacroPluginRemovedRule(BaseRule):\n\n    title = \"Remove airflow.AirflowMacroPlugin class\"\n\n    description = \"The airflow.AirflowMacroPlugin class has been removed.\"\n\n    MACRO_PLUGIN_CLASS = \"airflow.AirflowMacroPlugin\"\n\n    def _change_info(self, file_path, line_number):\n        return \"{} will be removed. Affected file: {} (line {})\".format(\n            self.MACRO_PLUGIN_CLASS, file_path, line_number\n        )\n\n    def _check_file(self, file_path):\n        problems = []\n        class_name_to_check = self.MACRO_PLUGIN_CLASS.split(\".\")[-1]\n        with open(file_path, \"r\") as file_pointer:\n            for line_number, line in enumerate(file_pointer, 1):\n                if class_name_to_check in line:\n                    problems.append(self._change_info(file_path, line_number))\n        return problems\n\n    def check(self):\n        dag_folder = conf.get(\"core\", \"dags_folder\")\n        file_paths = list_py_file_paths(directory=dag_folder, include_examples=False)\n        problems = []\n        for file_path in file_paths:\n            problems.extend(self._check_file(file_path))\n        return problems\n", "path": "airflow/upgrade/rules/airflow_macro_plugin_removed.py"}], "after_files": [{"content": "# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#   http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing,\n# software distributed under the License is distributed on an\n# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n# KIND, either express or implied.  See the License for the\n# specific language governing permissions and limitations\n# under the License.\n\nfrom __future__ import absolute_import\n\nfrom airflow import conf\nfrom airflow.upgrade.rules.base_rule import BaseRule\nfrom airflow.utils.dag_processing import list_py_file_paths\n\n\nclass AirflowMacroPluginRemovedRule(BaseRule):\n\n    title = \"Remove airflow.AirflowMacroPlugin class\"\n\n    description = \"The airflow.AirflowMacroPlugin class has been removed.\"\n\n    MACRO_PLUGIN_CLASS = \"airflow.AirflowMacroPlugin\"\n\n    def _change_info(self, file_path, line_number):\n        return \"{} will be removed. Affected file: {} (line {})\".format(\n            self.MACRO_PLUGIN_CLASS, file_path, line_number\n        )\n\n    def _check_file(self, file_path):\n        problems = []\n        class_name_to_check = self.MACRO_PLUGIN_CLASS.split(\".\")[-1]\n        with open(file_path, \"r\") as file_pointer:\n            try:\n                for line_number, line in enumerate(file_pointer, 1):\n                    if class_name_to_check in line:\n                        problems.append(self._change_info(file_path, line_number))\n            except UnicodeDecodeError:\n                problems.append(\"Unable to read python file {}\".format(file_path))\n        return problems\n\n    def check(self):\n        dag_folder = conf.get(\"core\", \"dags_folder\")\n        file_paths = list_py_file_paths(directory=dag_folder, include_examples=False)\n        problems = []\n        for file_path in file_paths:\n            if not file_path.endswith(\".py\"):\n                continue\n            problems.extend(self._check_file(file_path))\n        return problems\n", "path": "airflow/upgrade/rules/airflow_macro_plugin_removed.py"}]}