coveralls: ## Pull Request Test Coverage Report for [Build 266](https://coveralls.io/builds/22384830)

* **24** of **25**   **(96.0%)**  changed or added relevant lines in **2** files are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage increased (+**0.2%**) to **79.565%**

---

|  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [dataflows/processors/select_fields.py](https://coveralls.io/builds/22384830/source?filename=dataflows%2Fprocessors%2Fselect_fields.py#L41) | 23 | 24 | 95.83%
<!-- | **Total:** | **24** | **25** | **96.0%** | -->


|  Totals | [![Coverage Status](https://coveralls.io/builds/22384830/badge)](https://coveralls.io/builds/22384830) |
| :-- | --: |
| Change from base [Build 265](https://coveralls.io/builds/22085756): |  0.2% |
| Covered Lines: | 1355 |
| Relevant Lines: | 1703 |

---
##### 💛  - [Coveralls](https://coveralls.io)

coveralls: ## Pull Request Test Coverage Report for [Build 266](https://coveralls.io/builds/22384830)

* **24** of **25**   **(96.0%)**  changed or added relevant lines in **2** files are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage increased (+**0.2%**) to **79.565%**

---

|  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [dataflows/processors/select_fields.py](https://coveralls.io/builds/22384830/source?filename=dataflows%2Fprocessors%2Fselect_fields.py#L41) | 23 | 24 | 95.83%
<!-- | **Total:** | **24** | **25** | **96.0%** | -->


|  Totals | [![Coverage Status](https://coveralls.io/builds/22384830/badge)](https://coveralls.io/builds/22384830) |
| :-- | --: |
| Change from base [Build 265](https://coveralls.io/builds/22085756): |  0.2% |
| Covered Lines: | 1355 |
| Relevant Lines: | 1703 |

---
##### 💛  - [Coveralls](https://coveralls.io)

diff --git a/PROCESSORS.md b/PROCESSORS.md
index aa07f38..3fe33ec 100644
--- a/PROCESSORS.md
+++ b/PROCESSORS.md
@@ -14,6 +14,7 @@ DataFlows comes with a few built-in processors which do most of the heavy liftin
 
 ### Manipulate row-by-row
 - **add_field** - Adds a column to the data
+- **select_fields** - Selects and keeps some columns in the data
 - **delete_fields** - Removes some columns from the data
 - **add_computed_field** - Adds new fields whose values are based on existing columns
 - **find_replace** - Look for specific patterns in specific fields and replace them with new data
@@ -246,6 +247,26 @@ def add_field(name, type, default=None, resources=None, **options):
   - `None` indicates operation should be done on all resources
   - The index of the resource in the package
 
+#### select_fields
+Select fields (columns) in streamed resources and remove all other fields. Can also be used to reorder schema fields.
+
+`select_fields` accepts a list of resources and list of fields to remove
+
+_Note: if multiple resources provided, all of them should contain all fields to select_
+
+```python
+def select_fields(fields, resources=None):
+    pass
+```
+
+- `fields` - List of field (column) names to keep
+- `resources`
+  - A name of a resource to operate on
+  - A regular expression matching resource names
+  - A list of resource names
+  - `None` indicates operation should be done on all resources
+  - The index of the resource in the package
+
 #### delete_fields
 Delete fields (columns) from streamed resources
 
diff --git a/dataflows/processors/__init__.py b/dataflows/processors/__init__.py
index 62ec261..2b28afb 100644
--- a/dataflows/processors/__init__.py
+++ b/dataflows/processors/__init__.py
@@ -13,6 +13,7 @@ from .duplicate import duplicate
 from .filter_rows import filter_rows
 from .find_replace import find_replace
 from .join import join, join_self
+from .select_fields import select_fields
 from .set_primary_key import set_primary_key
 from .sort_rows import sort_rows
 from .stream import stream
diff --git a/dataflows/processors/select_fields.py b/dataflows/processors/select_fields.py
new file mode 100644
index 0000000..324b029
--- /dev/null
+++ b/dataflows/processors/select_fields.py
@@ -0,0 +1,45 @@
+from ..helpers.resource_matcher import ResourceMatcher
+
+
+def process_resource(rows, fields):
+    fields = set(fields)
+    for row in rows:
+        row = dict(
+            (k, v)
+            for k, v in row.items()
+            if k in fields
+        )
+        yield row
+
+
+def select_fields(fields, resources=None):
+
+    def func(package):
+        matcher = ResourceMatcher(resources, package.pkg)
+        dp_resources = package.pkg.descriptor.get('resources', [])
+        for resource in dp_resources:
+            if matcher.match(resource['name']):
+                dp_fields = resource['schema'].get('fields', [])
+                dp_fields = dict(
+                    (f['name'], f)
+                    for f in dp_fields
+                )
+                non_existing = [f for f in fields if f not in dp_fields]
+
+                assert len(non_existing) == 0, \
+                    "Can't find following field(s): %s" % '; '.join(non_existing)
+
+                new_fields = [
+                    dp_fields.get(f)
+                    for f in fields
+                ]
+                resource['schema']['fields'] = new_fields
+        yield package.pkg
+
+        for resource in package:
+            if not matcher.match(resource.res.name):
+                yield resource
+            else:
+                yield process_resource(resource, fields)
+
+    return func

Add 'select_fields' processor
As using 'concatenate' for that is not intuitive.
diff --git a/tests/test_lib.py b/tests/test_lib.py
index 33708a0..2d32b36 100644
--- a/tests/test_lib.py
+++ b/tests/test_lib.py
@@ -69,6 +69,19 @@ def test_delete_field():
         [dict(name='y', type='string', format='default')]
 
 
+def test_select_field():
+    from dataflows import select_fields
+    f = Flow(
+        data,
+        select_fields(['y'])
+    )
+    results, dp, _ = f.results()
+    for i in results[0]:
+        assert list(i.keys()) == ['y']
+    assert dp.descriptor['resources'][0]['schema']['fields'] == \
+        [dict(name='y', type='string', format='default')]
+
+
 def test_find_replace():
     from dataflows import find_replace
     f = Flow(

pip install -e .[develop]
parse_log_pytest
requirements.txt
pytest
3.7
requirements/base.txt
pytest --no-header -rA --tb=line --color=no -p no:cacheprovider -W ignore::DeprecationWarning
attrs==24.2.0
awesome-slugify==1.6.5
bitarray==3.3.0
blessed==1.20.0
boto3==1.33.13
botocore==1.33.13
cached-property==1.5.2
cachetools==5.5.2
certifi @ file:///croot/certifi_1671487769961/work/certifi
cffi==1.15.1
chardet==5.2.0
charset-normalizer==3.4.1
click==8.1.8
colorama==0.4.6
cryptography==44.0.2
-e git+https://github.com/datahq/dataflows.git@120bb50005c2585b7f99671506f53c0368091ae5#egg=dataflows
dataflows-tabulator==1.54.3
datapackage==1.15.4
distlib==0.3.9
et-xmlfile==1.1.0
exceptiongroup==1.2.2
filelock==3.12.2
greenlet==3.1.1
idna==3.10
ijson==3.3.0
importlib-metadata==6.7.0
importlib-resources==5.12.0
iniconfig==2.0.0
inquirer==2.10.1
isodate==0.7.2
Jinja2==3.1.6
jmespath==1.0.1
jsonlines==3.1.0
jsonpointer==3.0.0
jsonschema==4.17.3
kvfile==1.1.2
linear-tsv==1.1.0
MarkupSafe==2.1.5
mccabe==0.7.0
openpyxl==3.1.3
packaging==24.0
pkgutil_resolve_name==1.3.10
platformdirs==4.0.0
pluggy==1.2.0
pybloom-live==4.0.0
pycodestyle==2.10.0
pycparser==2.21
pydocstyle==6.1.1
pyflakes==3.0.1
pylama==8.4.1
pyproject-api==1.5.3
pyrsistent==0.19.3
pytest==7.4.4
python-dateutil==2.9.0.post0
python-editor==1.0.4
readchar==4.0.5
regex==2024.4.16
requests==2.31.0
rfc3986==2.0.0
s3transfer==0.8.2
six==1.17.0
snowballstemmer==2.2.0
SQLAlchemy==2.0.40
tableschema==1.21.0
tableschema-sql==2.0.1
tabulate==0.9.0
tabulator==1.53.5
tomli==2.0.1
tox==4.8.0
typing_extensions==4.7.1
unicodecsv==0.14.1
Unidecode==0.4.21
urllib3==1.26.20
virtualenv==20.26.6
wcwidth==0.2.13
xlrd==2.0.1
xmljson==0.2.1
xxhash==3.5.0
zipp==3.15.0

name: dataflows
channels:
  - defaults
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - ca-certificates=2025.2.25=h06a4308_0
  - certifi=2022.12.7=py37h06a4308_0
  - ld_impl_linux-64=2.40=h12ee557_0
  - libffi=3.4.4=h6a678d5_1
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - ncurses=6.4=h6a678d5_0
  - openssl=1.1.1w=h7f8727e_0
  - pip=22.3.1=py37h06a4308_0
  - python=3.7.16=h7a1cb2a_0
  - readline=8.2=h5eee18b_0
  - setuptools=65.6.3=py37h06a4308_0
  - sqlite=3.45.3=h5eee18b_0
  - tk=8.6.14=h39e8969_0
  - wheel=0.38.4=py37h06a4308_0
  - xz=5.6.4=h5eee18b_1
  - zlib=1.2.13=h5eee18b_1
  - pip:
      - attrs==24.2.0
      - awesome-slugify==1.6.5
      - bitarray==3.3.0
      - blessed==1.20.0
      - boto3==1.33.13
      - botocore==1.33.13
      - cached-property==1.5.2
      - cachetools==5.5.2
      - cffi==1.15.1
      - chardet==5.2.0
      - charset-normalizer==3.4.1
      - click==8.1.8
      - colorama==0.4.6
      - cryptography==44.0.2
      - dataflows-tabulator==1.54.3
      - datapackage==1.15.4
      - distlib==0.3.9
      - et-xmlfile==1.1.0
      - exceptiongroup==1.2.2
      - filelock==3.12.2
      - greenlet==3.1.1
      - idna==3.10
      - ijson==3.3.0
      - importlib-metadata==6.7.0
      - importlib-resources==5.12.0
      - iniconfig==2.0.0
      - inquirer==2.10.1
      - isodate==0.7.2
      - jinja2==3.1.6
      - jmespath==1.0.1
      - jsonlines==3.1.0
      - jsonpointer==3.0.0
      - jsonschema==4.17.3
      - kvfile==1.1.2
      - linear-tsv==1.1.0
      - markupsafe==2.1.5
      - mccabe==0.7.0
      - openpyxl==3.1.3
      - packaging==24.0
      - pkgutil-resolve-name==1.3.10
      - platformdirs==4.0.0
      - pluggy==1.2.0
      - pybloom-live==4.0.0
      - pycodestyle==2.10.0
      - pycparser==2.21
      - pydocstyle==6.1.1
      - pyflakes==3.0.1
      - pylama==8.4.1
      - pyproject-api==1.5.3
      - pyrsistent==0.19.3
      - pytest==7.4.4
      - python-dateutil==2.9.0.post0
      - python-editor==1.0.4
      - readchar==4.0.5
      - regex==2024.4.16
      - requests==2.31.0
      - rfc3986==2.0.0
      - s3transfer==0.8.2
      - six==1.17.0
      - snowballstemmer==2.2.0
      - sqlalchemy==2.0.40
      - tableschema==1.21.0
      - tableschema-sql==2.0.1
      - tabulate==0.9.0
      - tabulator==1.53.5
      - tomli==2.0.1
      - tox==4.8.0
      - typing-extensions==4.7.1
      - unicodecsv==0.14.1
      - unidecode==0.04.21
      - urllib3==1.26.20
      - virtualenv==20.26.6
      - wcwidth==0.2.13
      - xlrd==2.0.1
      - xmljson==0.2.1
      - xxhash==3.5.0
      - zipp==3.15.0
prefix: /opt/conda/envs/dataflows

tests/test_lib.py::test_select_field
tests/test_lib.py::test_dump_to_sql tests/test_lib.py::test_stream_simple tests/test_lib.py::test_set_primary_key
tests/test_lib.py::test_add_computed_field tests/test_lib.py::test_add_metadata tests/test_lib.py::test_delete_field tests/test_lib.py::test_find_replace tests/test_lib.py::test_unpivot tests/test_lib.py::test_concatenate tests/test_lib.py::test_filter_rows tests/test_lib.py::test_sort_rows tests/test_lib.py::test_sort_reverse_many_rows tests/test_lib.py::test_duplicate tests/test_lib.py::test_duplicate_many_rows tests/test_lib.py::test_flow_as_step tests/test_lib.py::test_load_from_package tests/test_lib.py::test_load_from_env_var tests/test_lib.py::test_load_from_package_resource_matching tests/test_lib.py::test_load_from_package_resources tests/test_lib.py::test_checkpoint tests/test_lib.py::test_load_from_checkpoint tests/test_lib.py::test_update_resource tests/test_lib.py::test_set_type_resources tests/test_lib.py::test_dump_to_path_use_titles tests/test_lib.py::test_load_dates tests/test_lib.py::test_load_dates_timezones tests/test_lib.py::test_add_field tests/test_lib.py::test_load_empty_headers tests/test_lib.py::test_load_xml tests/test_lib.py::test_save_load_dates
dataflows/processors/select_fields.py dataflows/processors/__init__.py PROCESSORS.md
dataflows/processors/select_fields.py dataflows/processors/__init__.py PROCESSORS.md