EmilStenstrom: This looks great! The coverage and flake8 tests seem to fail on Travis. Do you get the same errors locally? 

Summary:
* There's a test missing for "models.py line 30->exit"
* There's a coverage error for "conllu/models.py:24:16: E126 continuation line over-indented for hanging indent"

Mind fixing those two? Thanks again!
bryant1410: That's weird, tests were passing locally on Python 2.7 and 3.6 and coverage
was 100%. I will check it out later.

On Thu, Sep 13, 2018, 3:46 PM Emil Stenström <notifications@github.com>
wrote:

> This looks great! The coverage and flake8 tests seem to fail on Travis. Do
> you get the same errors locally?
>
> Summary:
>
>    - There's a test missing for "models.py line 30->exit"
>    - There's a coverage error for "conllu/models.py:24:16: E126
>    continuation line over-indented for hanging indent"
>
> Mind fixing those two? Thanks again!
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/EmilStenstrom/conllu/pull/19#issuecomment-421129685>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADuX3cIVLa4OnW6z7LYsyAqk75RvA4sPks5uarXxgaJpZM4WoFFX>
> .
>

EmilStenstrom: This looks fantastic. I'll merge and release a new version to PyPI!
diff --git a/conllu/models.py b/conllu/models.py
index d57a49a..cd98ae4 100644
--- a/conllu/models.py
+++ b/conllu/models.py
@@ -5,25 +5,53 @@ from conllu.parser import ParseException, head_to_token, serialize
 
 DEFAULT_EXCLUDE_FIELDS = ('id', 'deprel', 'xpostag', 'feats', 'head', 'deps', 'misc')
 
-class TokenList(object):
-    tokens = None
+
+class TokenList(list):
     metadata = None
 
     def __init__(self, tokens, metadata=None):
+        super(TokenList, self).__init__(tokens)
         if not isinstance(tokens, list):
             raise ParseException("Can't create TokenList, tokens is not a list.")
 
-        self.tokens = tokens
         self.metadata = metadata
 
-    def __getitem__(self, key):
-        return self.tokens[key]
-
     def __repr__(self):
-        return 'TokenList<' + ', '.join([token['form'] for token in self.tokens]) + '>'
+        return 'TokenList<' + ', '.join(token['form'] for token in self) + '>'
 
     def __eq__(self, other):
-        return self.tokens == other.tokens and self.metadata == other.metadata
+        return super(TokenList, self).__eq__(other) \
+            and (not hasattr(other, 'metadata') or self.metadata == other.metadata)
+
+    def __ne__(self, other):
+        return not self == other
+
+    def clear(self):
+        self[:] = []  # Supported in Python 2 and 3, unlike clear()
+        self.metadata = None
+
+    def copy(self):
+        tokens_copy = self[:]  # Supported in Python 2 and 3, unlike copy()
+        return TokenList(tokens_copy, self.metadata)
+
+    def extend(self, iterable):
+        super(TokenList, self).extend(iterable)
+        if hasattr(iterable, 'metadata'):
+            if hasattr(self.metadata, '__add__') and hasattr(iterable.metadata, '__add__'):
+                self.metadata += iterable.metadata
+            elif type(self.metadata) is dict and type(iterable.metadata) is dict:
+                # noinspection PyUnresolvedReferences
+                self.metadata.update(iterable.metadata)
+            else:
+                self.metadata = [self.metadata, iterable.metadata]
+
+    @property
+    def tokens(self):
+        return self
+
+    @tokens.setter
+    def tokens(self, value):
+        self[:] = value  # Supported in Python 2 and 3, unlike clear()
 
     def serialize(self):
         return serialize(self)
@@ -39,6 +67,7 @@ class TokenList(object):
         root.set_metadata(self.metadata)
         return root
 
+
 class TokenTree(object):
     token = None
     children = None

TokenList could inherit "list"
Hey. I have found this library quite useful. There is a suggestion that I ran into: `TokenList` could maybe inherit `list`, and so be treated like that.

I tried manipulating the sentences (TokenLists) as if they were lists, such as using `len` but then I saw they weren't. I thought about adding `__len__`, but I think it's more appropriate to make them lists, so they natively support other things such as adding them, comparing them, and so on.

What do you think guys?
diff --git a/tests/test_models.py b/tests/test_models.py
index 02d377a..0e5a182 100644
--- a/tests/test_models.py
+++ b/tests/test_models.py
@@ -26,6 +26,55 @@ class TestTokenList(unittest.TestCase):
         tokenlist2.metadata = metadata
         self.assertEqual(tokenlist1, tokenlist2)
 
+    def test_len(self):
+        tokenlist = TokenList([{"id": 1}, {"id": 2}, {"id": 3}])
+        self.assertEqual(3, len(tokenlist))
+
+    def test_clear(self):
+        tokenlist = TokenList([{"id": 1}, {"id": 2}, {"id": 3}], {"meta": "data"})
+        tokenlist.clear()
+        self.assertEqual(len(tokenlist.tokens), 0)
+        self.assertEqual(tokenlist.metadata, None)
+
+    def test_copy(self):
+        tokenlist1 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}], {"meta": "data"})
+        tokenlist2 = tokenlist1.copy()
+        self.assertIsNot(tokenlist1, tokenlist2)
+        self.assertEqual(tokenlist1, tokenlist2)
+
+    def test_extend(self):
+        tokenlist1 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}])
+        tokenlist2 = [{"id": 4}, {"id": 5}, {"id": 6}]
+        tokenlist1.extend(tokenlist2)
+        tokenlist3 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}, {"id": 5}, {"id": 6}])
+        self.assertEqual(tokenlist1, tokenlist3)
+
+        tokenlist4 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}], {"meta1": "data1"})
+        tokenlist5 = TokenList([{"id": 4}, {"id": 5}, {"id": 6}], {"meta2": "data2"})
+        tokenlist4.extend(tokenlist5)
+        tokenlist6 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}, {"id": 5}, {"id": 6}],
+                               {"meta1": "data1", "meta2": "data2"})
+        self.assertEqual(tokenlist4, tokenlist6)
+
+        tokenlist7 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}], "abc")
+        tokenlist8 = TokenList([{"id": 4}, {"id": 5}, {"id": 6}], "de")
+        tokenlist7.extend(tokenlist8)
+        tokenlist9 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}, {"id": 5}, {"id": 6}], "abcde")
+        self.assertEqual(tokenlist7, tokenlist9)
+
+        tokenlist7 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}], "abc")
+        tokenlist8 = TokenList([{"id": 4}, {"id": 5}, {"id": 6}], {"meta2": "data2"})
+        tokenlist7.extend(tokenlist8)
+        tokenlist9 = TokenList([{"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}, {"id": 5}, {"id": 6}],
+                               ["abc", {"meta2": "data2"}])
+        self.assertEqual(tokenlist7, tokenlist9)
+
+    def test_tokens(self):
+        tokenlist = TokenList([{"id": 1}, {"id": 2}, {"id": 3}])
+        self.assertEqual(tokenlist.tokens, [{"id": 1}, {"id": 2}, {"id": 3}])
+        tokenlist.tokens = [{"id": 4}, {"id": 5}]
+        self.assertEqual(tokenlist.tokens, [{"id": 4}, {"id": 5}])
+
     def test_to_tree(self):
         tokenlist = TokenList([
             OrderedDict([("id", 2), ("form", "dog"), ("head", 0)]),
@@ -40,11 +89,13 @@ class TestTokenList(unittest.TestCase):
         )
         self.assertEqual(tokenlist.to_tree(), tree)
 
+
 class TestSerialize(unittest.TestCase):
     def test_serialize_on_tokenlist(self):
         tokenlist = TokenList([{"id": 1}])
         self.assertEqual(tokenlist.serialize(), serialize(tokenlist))
 
+
 class TestTokenTree(unittest.TestCase):
     def test_eq(self):
         metadata = {"meta": "data"}
@@ -67,6 +118,7 @@ class TestTokenTree(unittest.TestCase):
         tree.set_metadata(metadata)
         self.assertEqual(tree.metadata, metadata)
 
+
 class TestSerializeTree(unittest.TestCase):
     def test_missing_id(self):
         tree = TokenTree(token={"form": "hej"}, children=[])
@@ -105,6 +157,7 @@ class TestSerializeTree(unittest.TestCase):
             """)
         )
 
+
 class TestPrintTree(unittest.TestCase):
     def test_print_empty_list(self):
         tree = TokenTree(None, [])

pip install -e .
parse_log_pytest
pytest
coverage flake8 isort pytest
3.6
pytest --no-header -rA --tb=line --color=no -p no:cacheprovider -W ignore::DeprecationWarning
attrs @ file:///opt/conda/conda-bld/attrs_1642510447205/work
certifi==2021.5.30
-e git+https://github.com/EmilStenstrom/conllu.git@1604bada91968725102b626da9537f361fc3178d#egg=conllu
coverage==6.2
flake8==5.0.4
importlib-metadata==4.2.0
iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
isort==5.10.1
mccabe==0.7.0
more-itertools @ file:///tmp/build/80754af9/more-itertools_1637733554872/work
packaging @ file:///tmp/build/80754af9/packaging_1637314298585/work
pluggy @ file:///tmp/build/80754af9/pluggy_1615976315926/work
py @ file:///opt/conda/conda-bld/py_1644396412707/work
pycodestyle==2.9.1
pyflakes==2.5.0
pyparsing @ file:///tmp/build/80754af9/pyparsing_1635766073266/work
pytest==6.2.4
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
typing_extensions @ file:///opt/conda/conda-bld/typing_extensions_1647553014482/work
zipp @ file:///tmp/build/80754af9/zipp_1633618647012/work

name: conllu
channels:
  - defaults
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - attrs=21.4.0=pyhd3eb1b0_0
  - ca-certificates=2025.2.25=h06a4308_0
  - certifi=2021.5.30=py36h06a4308_0
  - importlib_metadata=4.8.1=hd3eb1b0_0
  - iniconfig=1.1.1=pyhd3eb1b0_0
  - ld_impl_linux-64=2.40=h12ee557_0
  - libffi=3.3=he6710b0_2
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - more-itertools=8.12.0=pyhd3eb1b0_0
  - ncurses=6.4=h6a678d5_0
  - openssl=1.1.1w=h7f8727e_0
  - packaging=21.3=pyhd3eb1b0_0
  - pip=21.2.2=py36h06a4308_0
  - pluggy=0.13.1=py36h06a4308_0
  - py=1.11.0=pyhd3eb1b0_0
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pytest=6.2.4=py36h06a4308_2
  - python=3.6.13=h12debd9_1
  - readline=8.2=h5eee18b_0
  - setuptools=58.0.4=py36h06a4308_0
  - sqlite=3.45.3=h5eee18b_0
  - tk=8.6.14=h39e8969_0
  - toml=0.10.2=pyhd3eb1b0_0
  - typing_extensions=4.1.1=pyh06a4308_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - xz=5.6.4=h5eee18b_1
  - zipp=3.6.0=pyhd3eb1b0_0
  - zlib=1.2.13=h5eee18b_1
  - pip:
      - coverage==6.2
      - flake8==5.0.4
      - importlib-metadata==4.2.0
      - isort==5.10.1
      - mccabe==0.7.0
      - pycodestyle==2.9.1
      - pyflakes==2.5.0
prefix: /opt/conda/envs/conllu

tests/test_models.py::TestTokenList::test_clear tests/test_models.py::TestTokenList::test_copy tests/test_models.py::TestTokenList::test_extend tests/test_models.py::TestTokenList::test_len
tests/test_models.py::TestTokenList::test_constructor tests/test_models.py::TestTokenList::test_eq tests/test_models.py::TestTokenList::test_to_tree tests/test_models.py::TestTokenList::test_tokens tests/test_models.py::TestSerialize::test_serialize_on_tokenlist tests/test_models.py::TestTokenTree::test_eq tests/test_models.py::TestTokenTree::test_metadata tests/test_models.py::TestSerializeTree::test_flatten tests/test_models.py::TestSerializeTree::test_missing_id tests/test_models.py::TestPrintTree::test_print_empty_list tests/test_models.py::TestPrintTree::test_print_simple tests/test_models.py::TestPrintTree::test_print_with_children tests/test_models.py::TestPrintTree::test_tree_without_deprel tests/test_models.py::TestPrintTree::test_tree_without_id
conllu/models.py
conllu/models.py