We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
'NoneType' object is not iterable when using summarize
When using `gensim.summarization.summarize`, `TypeError: 'NoneType' object is not iterable` is occasionally thrown when summarizing random complaints in [this dataset of consumer complaints](https://catalog.data.gov/dataset/consumer-complaint-database). I'm not sure what it is about the strings that causes this error to be thrown but I feel the error should not be thrown regardless. Here is a self-contained example:

```
from gensim.summarization import summarize
text = "To whom it may concern, Only after weeks of using my new Wal-Mart master " \
    "card was I given the option to opt out of personal information sharing including :" \
    " Social Security #, income, account balance, payment history, credit history and credit" \
    " scores. According to the supervising operator at the wal-mart credit card call center my" \
    " personal information had already been shared. Furthermore, the letter informing me of my " \
    " rights was apart from my billing statement. Seemingly hidden between some flyers in the same" \
    " envelope. I almost threw it in the recycle bin without seeing it. If this is not illegal it " \
    "certainly appears to be an attempt to deceive me and disseminate my private information without " \
    "my permission. Thank you for your time and effort. Sincerely, XXXX XXXX"
summary = summarize(text)
```

I pasted the stack trace below but I'm sure you'll see it when running the above example.
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-15020c19bb47> in <module>()
      1 from gensim.summarization import summarize
      2 text = "To whom it may concern, Only after weeks of using my new Wal-Mart master "     "card was I given the option to opt out of personal information sharing including :"     " Social Security #, income, account balance, payment history, credit history and credit"     " scores. According to the supervising operator at the wal-mart credit card call center my"     " personal information had already been shared. Furthermore, the letter informing me of my "     " rights was apart from my billing statement. Seemingly hidden between some flyers in the same"     " envelope. I almost threw it in the recycle bin without seeing it. If this is not illegal it "     "certainly appears to be an attempt to deceive me and disseminate my private information without "    "my permission. Thank you for your time and effort. Sincerely, XXXX XXXX"
----> 3 summary = summarize(text)

/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in summarize(text, ratio, word_count, split)
    213 
    214     # Extracts the most important sentences with the selected criterion.
--> 215     extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)
    216 
    217     # Sorts the extracted sentences by apparition order in the original text.

/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in _extract_important_sentences(sentences, corpus, important_docs, word_count)
    112 
    113 def _extract_important_sentences(sentences, corpus, important_docs, word_count):
--> 114     important_sentences = _get_important_sentences(sentences, corpus, important_docs)
    115 
    116     # If no "word_count" option is provided, the number of sentences is

/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in _get_important_sentences(sentences, corpus, important_docs)
     87     hashable_corpus = _build_hasheable_corpus(corpus)
     88     sentences_by_corpus = dict(zip(hashable_corpus, sentences))
---> 89     return [sentences_by_corpus[tuple(important_doc)] for important_doc in important_docs]
     90 
     91 

TypeError: 'NoneType' object is not iterable
```

I'm assuming this is a bug but there's no example in the gensim ~~documentation~~ api page so I'm not entirely sure if I'm using the function correctly.

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `gensim/summarization/summarizer.py`
Content:
```
1 #!/usr/bin/env python
2 # -*- coding: utf-8 -*-
3 #
4 # Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
5 
6 import logging
7 from gensim.summarization.pagerank_weighted import pagerank_weighted as _pagerank
8 from gensim.summarization.textcleaner import clean_text_by_sentences as _clean_text_by_sentences
9 from gensim.summarization.commons import build_graph as _build_graph
10 from gensim.summarization.commons import remove_unreachable_nodes as _remove_unreachable_nodes
11 from gensim.summarization.bm25 import get_bm25_weights as _bm25_weights
12 from gensim.corpora import Dictionary
13 from math import log10 as _log10
14 from six.moves import xrange
15 
16 
17 INPUT_MIN_LENGTH = 10
18 
19 WEIGHT_THRESHOLD = 1.e-3
20 
21 logger = logging.getLogger(__name__)
22 
23 
24 def _set_graph_edge_weights(graph):
25     documents = graph.nodes()
26     weights = _bm25_weights(documents)
27 
28     for i in xrange(len(documents)):
29         for j in xrange(len(documents)):
30             if i == j or weights[i][j] < WEIGHT_THRESHOLD:
31                 continue
32 
33             sentence_1 = documents[i]
34             sentence_2 = documents[j]
35 
36             edge_1 = (sentence_1, sentence_2)
37             edge_2 = (sentence_2, sentence_1)
38 
39             if not graph.has_edge(edge_1):
40                 graph.add_edge(edge_1, weights[i][j])
41             if not graph.has_edge(edge_2):
42                 graph.add_edge(edge_2, weights[j][i])
43 
44     # Handles the case in which all similarities are zero.
45     # The resultant summary will consist of random sentences.
46     if all(graph.edge_weight(edge) == 0 for edge in graph.edges()):
47         _create_valid_graph(graph)
48 
49 
50 def _create_valid_graph(graph):
51     nodes = graph.nodes()
52 
53     for i in xrange(len(nodes)):
54         for j in xrange(len(nodes)):
55             if i == j:
56                 continue
57 
58             edge = (nodes[i], nodes[j])
59 
60             if graph.has_edge(edge):
61                 graph.del_edge(edge)
62 
63             graph.add_edge(edge, 1)
64 
65 
66 def _get_doc_length(doc):
67     return sum([item[1] for item in doc])
68 
69 
70 def _get_similarity(doc1, doc2, vec1, vec2):
71     numerator = vec1.dot(vec2.transpose()).toarray()[0][0]
72     length_1 = _get_doc_length(doc1)
73     length_2 = _get_doc_length(doc2)
74 
75     denominator = _log10(length_1) + _log10(length_2) if length_1 > 0 and length_2 > 0 else 0
76 
77     return numerator / denominator if denominator != 0 else 0
78 
79 
80 def _build_corpus(sentences):
81     split_tokens = [sentence.token.split() for sentence in sentences]
82     dictionary = Dictionary(split_tokens)
83     return [dictionary.doc2bow(token) for token in split_tokens]
84 
85 
86 def _get_important_sentences(sentences, corpus, important_docs):
87     hashable_corpus = _build_hasheable_corpus(corpus)
88     sentences_by_corpus = dict(zip(hashable_corpus, sentences))
89     return [sentences_by_corpus[tuple(important_doc)] for important_doc in important_docs]
90 
91 
92 def _get_sentences_with_word_count(sentences, word_count):
93     """ Given a list of sentences, returns a list of sentences with a
94     total word count similar to the word count provided."""
95     length = 0
96     selected_sentences = []
97 
98     # Loops until the word count is reached.
99     for sentence in sentences:
100         words_in_sentence = len(sentence.text.split())
101 
102         # Checks if the inclusion of the sentence gives a better approximation
103         # to the word parameter.
104         if abs(word_count - length - words_in_sentence) > abs(word_count - length):
105             return selected_sentences
106 
107         selected_sentences.append(sentence)
108         length += words_in_sentence
109 
110     return selected_sentences
111 
112 
113 def _extract_important_sentences(sentences, corpus, important_docs, word_count):
114     important_sentences = _get_important_sentences(sentences, corpus, important_docs)
115 
116     # If no "word_count" option is provided, the number of sentences is
117     # reduced by the provided ratio. Else, the ratio is ignored.
118     return important_sentences if word_count is None else _get_sentences_with_word_count(important_sentences, word_count)
119 
120 
121 def _format_results(extracted_sentences, split):
122     if split:
123         return [sentence.text for sentence in extracted_sentences]
124     return "\n".join([sentence.text for sentence in extracted_sentences])
125 
126 
127 def _build_hasheable_corpus(corpus):
128     return [tuple(doc) for doc in corpus]
129 
130 
131 def summarize_corpus(corpus, ratio=0.2):
132     """
133     Returns a list of the most important documents of a corpus using a
134     variation of the TextRank algorithm.
135     The input must have at least INPUT_MIN_LENGTH (%d) documents for the
136     summary to make sense.
137 
138     The length of the output can be specified using the ratio parameter,
139     which determines how many documents will be chosen for the summary
140     (defaults at 20%% of the number of documents of the corpus).
141 
142     The most important documents are returned as a list sorted by the
143     document score, highest first.
144     """ % INPUT_MIN_LENGTH
145     hashable_corpus = _build_hasheable_corpus(corpus)
146 
147     # If the corpus is empty, the function ends.
148     if len(corpus) == 0:
149         logger.warning("Input corpus is empty.")
150         return
151 
152     # Warns the user if there are too few documents.
153     if len(corpus) < INPUT_MIN_LENGTH:
154         logger.warning("Input corpus is expected to have at least " + str(INPUT_MIN_LENGTH) + " documents.")
155 
156     graph = _build_graph(hashable_corpus)
157     _set_graph_edge_weights(graph)
158     _remove_unreachable_nodes(graph)
159 
160     # Cannot calculate eigenvectors if number of unique words in text < 3. Warns user to add more text. The function ends.
161     if len(graph.nodes()) < 3:
162         logger.warning("Please add more sentences to the text. The number of reachable nodes is below 3")
163         return
164 
165     pagerank_scores = _pagerank(graph)
166 
167     hashable_corpus.sort(key=lambda doc: pagerank_scores.get(doc, 0), reverse=True)
168 
169     return [list(doc) for doc in hashable_corpus[:int(len(corpus) * ratio)]]
170 
171 
172 def summarize(text, ratio=0.2, word_count=None, split=False):
173     """
174     Returns a summarized version of the given text using a variation of
175     the TextRank algorithm.
176     The input must be longer than INPUT_MIN_LENGTH sentences for the
177     summary to make sense and must be given as a string.
178 
179     The output summary will consist of the most representative sentences
180     and will also be returned as a string, divided by newlines. If the
181     split parameter is set to True, a list of sentences will be
182     returned.
183 
184     The length of the output can be specified using the ratio and
185     word_count parameters:
186 
187         ratio should be a number between 0 and 1 that determines the
188         percentage of the number of sentences of the original text to be
189         chosen for the summary (defaults at 0.2).
190         word_count determines how many words will the output contain.
191         If both parameters are provided, the ratio will be ignored.
192 
193     """
194     # Gets a list of processed sentences.
195     sentences = _clean_text_by_sentences(text)
196 
197     # If no sentence could be identified, the function ends.
198     if len(sentences) == 0:
199         logger.warning("Input text is empty.")
200         return
201 
202     # If only one sentence is present, the function raises an error (Avoids ZeroDivisionError).
203     if len(sentences) == 1:
204         raise ValueError("input must have more than one sentence")
205 
206     # Warns if the text is too short.
207     if len(sentences) < INPUT_MIN_LENGTH:
208         logger.warning("Input text is expected to have at least " + str(INPUT_MIN_LENGTH) + " sentences.")
209 
210     corpus = _build_corpus(sentences)
211 
212     most_important_docs = summarize_corpus(corpus, ratio=ratio if word_count is None else 1)
213 
214     # Extracts the most important sentences with the selected criterion.
215     extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)
216 
217     # Sorts the extracted sentences by apparition order in the original text.
218     extracted_sentences.sort(key=lambda s: s.index)
219 
220     return _format_results(extracted_sentences, split)
221
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/gensim/summarization/summarizer.py b/gensim/summarization/summarizer.py
--- a/gensim/summarization/summarizer.py
+++ b/gensim/summarization/summarizer.py
@@ -147,7 +147,7 @@
     # If the corpus is empty, the function ends.
     if len(corpus) == 0:
         logger.warning("Input corpus is empty.")
-        return
+        return []
 
     # Warns the user if there are too few documents.
     if len(corpus) < INPUT_MIN_LENGTH:
@@ -157,10 +157,11 @@
     _set_graph_edge_weights(graph)
     _remove_unreachable_nodes(graph)
 
-    # Cannot calculate eigenvectors if number of unique words in text < 3. Warns user to add more text. The function ends.
+    # Cannot calculate eigenvectors if number of unique documents in corpus < 3.
+    # Warns user to add more text. The function ends.
     if len(graph.nodes()) < 3:
         logger.warning("Please add more sentences to the text. The number of reachable nodes is below 3")
-        return
+        return []
 
     pagerank_scores = _pagerank(graph)
 
@@ -197,7 +198,7 @@
     # If no sentence could be identified, the function ends.
     if len(sentences) == 0:
         logger.warning("Input text is empty.")
-        return
+        return [] if split else u""
 
     # If only one sentence is present, the function raises an error (Avoids ZeroDivisionError).
     if len(sentences) == 1:
@@ -211,6 +212,11 @@
 
     most_important_docs = summarize_corpus(corpus, ratio=ratio if word_count is None else 1)
 
+    # If couldn't get important docs, the algorithm ends.
+    if not most_important_docs:
+        logger.warning("Couldn't get relevant sentences.")
+        return [] if split else u""
+
     # Extracts the most important sentences with the selected criterion.
     extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)

{"golden_diff": "diff --git a/gensim/summarization/summarizer.py b/gensim/summarization/summarizer.py\n--- a/gensim/summarization/summarizer.py\n+++ b/gensim/summarization/summarizer.py\n@@ -147,7 +147,7 @@\n     # If the corpus is empty, the function ends.\n     if len(corpus) == 0:\n         logger.warning(\"Input corpus is empty.\")\n-        return\n+        return []\n \n     # Warns the user if there are too few documents.\n     if len(corpus) < INPUT_MIN_LENGTH:\n@@ -157,10 +157,11 @@\n     _set_graph_edge_weights(graph)\n     _remove_unreachable_nodes(graph)\n \n-    # Cannot calculate eigenvectors if number of unique words in text < 3. Warns user to add more text. The function ends.\n+    # Cannot calculate eigenvectors if number of unique documents in corpus < 3.\n+    # Warns user to add more text. The function ends.\n     if len(graph.nodes()) < 3:\n         logger.warning(\"Please add more sentences to the text. The number of reachable nodes is below 3\")\n-        return\n+        return []\n \n     pagerank_scores = _pagerank(graph)\n \n@@ -197,7 +198,7 @@\n     # If no sentence could be identified, the function ends.\n     if len(sentences) == 0:\n         logger.warning(\"Input text is empty.\")\n-        return\n+        return [] if split else u\"\"\n \n     # If only one sentence is present, the function raises an error (Avoids ZeroDivisionError).\n     if len(sentences) == 1:\n@@ -211,6 +212,11 @@\n \n     most_important_docs = summarize_corpus(corpus, ratio=ratio if word_count is None else 1)\n \n+    # If couldn't get important docs, the algorithm ends.\n+    if not most_important_docs:\n+        logger.warning(\"Couldn't get relevant sentences.\")\n+        return [] if split else u\"\"\n+\n     # Extracts the most important sentences with the selected criterion.\n     extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)\n", "issue": "'NoneType' object is not iterable when using summarize\nWhen using `gensim.summarization.summarize`, `TypeError: 'NoneType' object is not iterable` is occasionally thrown when summarizing random complaints in [this dataset of consumer complaints](https://catalog.data.gov/dataset/consumer-complaint-database). I'm not sure what it is about the strings that causes this error to be thrown but I feel the error should not be thrown regardless. Here is a self-contained example:\r\n\r\n```\r\nfrom gensim.summarization import summarize\r\ntext = \"To whom it may concern, Only after weeks of using my new Wal-Mart master \" \\\r\n    \"card was I given the option to opt out of personal information sharing including :\" \\\r\n    \" Social Security #, income, account balance, payment history, credit history and credit\" \\\r\n    \" scores. According to the supervising operator at the wal-mart credit card call center my\" \\\r\n    \" personal information had already been shared. Furthermore, the letter informing me of my \" \\\r\n    \" rights was apart from my billing statement. Seemingly hidden between some flyers in the same\" \\\r\n    \" envelope. I almost threw it in the recycle bin without seeing it. If this is not illegal it \" \\\r\n    \"certainly appears to be an attempt to deceive me and disseminate my private information without \" \\\r\n    \"my permission. Thank you for your time and effort. Sincerely, XXXX XXXX\"\r\nsummary = summarize(text)\r\n```\r\n\r\nI pasted the stack trace below but I'm sure you'll see it when running the above example.\r\n```\r\n---------------------------------------------------------------------------\r\nTypeError                                 Traceback (most recent call last)\r\n<ipython-input-1-15020c19bb47> in <module>()\r\n      1 from gensim.summarization import summarize\r\n      2 text = \"To whom it may concern, Only after weeks of using my new Wal-Mart master \"     \"card was I given the option to opt out of personal information sharing including :\"     \" Social Security #, income, account balance, payment history, credit history and credit\"     \" scores. According to the supervising operator at the wal-mart credit card call center my\"     \" personal information had already been shared. Furthermore, the letter informing me of my \"     \" rights was apart from my billing statement. Seemingly hidden between some flyers in the same\"     \" envelope. I almost threw it in the recycle bin without seeing it. If this is not illegal it \"     \"certainly appears to be an attempt to deceive me and disseminate my private information without \"    \"my permission. Thank you for your time and effort. Sincerely, XXXX XXXX\"\r\n----> 3 summary = summarize(text)\r\n\r\n/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in summarize(text, ratio, word_count, split)\r\n    213 \r\n    214     # Extracts the most important sentences with the selected criterion.\r\n--> 215     extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)\r\n    216 \r\n    217     # Sorts the extracted sentences by apparition order in the original text.\r\n\r\n/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in _extract_important_sentences(sentences, corpus, important_docs, word_count)\r\n    112 \r\n    113 def _extract_important_sentences(sentences, corpus, important_docs, word_count):\r\n--> 114     important_sentences = _get_important_sentences(sentences, corpus, important_docs)\r\n    115 \r\n    116     # If no \"word_count\" option is provided, the number of sentences is\r\n\r\n/usr/local/lib/python3.6/site-packages/gensim/summarization/summarizer.py in _get_important_sentences(sentences, corpus, important_docs)\r\n     87     hashable_corpus = _build_hasheable_corpus(corpus)\r\n     88     sentences_by_corpus = dict(zip(hashable_corpus, sentences))\r\n---> 89     return [sentences_by_corpus[tuple(important_doc)] for important_doc in important_docs]\r\n     90 \r\n     91 \r\n\r\nTypeError: 'NoneType' object is not iterable\r\n```\r\n\r\nI'm assuming this is a bug but there's no example in the gensim ~~documentation~~ api page so I'm not entirely sure if I'm using the function correctly.\n", "before_files": [{"content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n#\n# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html\n\nimport logging\nfrom gensim.summarization.pagerank_weighted import pagerank_weighted as _pagerank\nfrom gensim.summarization.textcleaner import clean_text_by_sentences as _clean_text_by_sentences\nfrom gensim.summarization.commons import build_graph as _build_graph\nfrom gensim.summarization.commons import remove_unreachable_nodes as _remove_unreachable_nodes\nfrom gensim.summarization.bm25 import get_bm25_weights as _bm25_weights\nfrom gensim.corpora import Dictionary\nfrom math import log10 as _log10\nfrom six.moves import xrange\n\n\nINPUT_MIN_LENGTH = 10\n\nWEIGHT_THRESHOLD = 1.e-3\n\nlogger = logging.getLogger(__name__)\n\n\ndef _set_graph_edge_weights(graph):\n    documents = graph.nodes()\n    weights = _bm25_weights(documents)\n\n    for i in xrange(len(documents)):\n        for j in xrange(len(documents)):\n            if i == j or weights[i][j] < WEIGHT_THRESHOLD:\n                continue\n\n            sentence_1 = documents[i]\n            sentence_2 = documents[j]\n\n            edge_1 = (sentence_1, sentence_2)\n            edge_2 = (sentence_2, sentence_1)\n\n            if not graph.has_edge(edge_1):\n                graph.add_edge(edge_1, weights[i][j])\n            if not graph.has_edge(edge_2):\n                graph.add_edge(edge_2, weights[j][i])\n\n    # Handles the case in which all similarities are zero.\n    # The resultant summary will consist of random sentences.\n    if all(graph.edge_weight(edge) == 0 for edge in graph.edges()):\n        _create_valid_graph(graph)\n\n\ndef _create_valid_graph(graph):\n    nodes = graph.nodes()\n\n    for i in xrange(len(nodes)):\n        for j in xrange(len(nodes)):\n            if i == j:\n                continue\n\n            edge = (nodes[i], nodes[j])\n\n            if graph.has_edge(edge):\n                graph.del_edge(edge)\n\n            graph.add_edge(edge, 1)\n\n\ndef _get_doc_length(doc):\n    return sum([item[1] for item in doc])\n\n\ndef _get_similarity(doc1, doc2, vec1, vec2):\n    numerator = vec1.dot(vec2.transpose()).toarray()[0][0]\n    length_1 = _get_doc_length(doc1)\n    length_2 = _get_doc_length(doc2)\n\n    denominator = _log10(length_1) + _log10(length_2) if length_1 > 0 and length_2 > 0 else 0\n\n    return numerator / denominator if denominator != 0 else 0\n\n\ndef _build_corpus(sentences):\n    split_tokens = [sentence.token.split() for sentence in sentences]\n    dictionary = Dictionary(split_tokens)\n    return [dictionary.doc2bow(token) for token in split_tokens]\n\n\ndef _get_important_sentences(sentences, corpus, important_docs):\n    hashable_corpus = _build_hasheable_corpus(corpus)\n    sentences_by_corpus = dict(zip(hashable_corpus, sentences))\n    return [sentences_by_corpus[tuple(important_doc)] for important_doc in important_docs]\n\n\ndef _get_sentences_with_word_count(sentences, word_count):\n    \"\"\" Given a list of sentences, returns a list of sentences with a\n    total word count similar to the word count provided.\"\"\"\n    length = 0\n    selected_sentences = []\n\n    # Loops until the word count is reached.\n    for sentence in sentences:\n        words_in_sentence = len(sentence.text.split())\n\n        # Checks if the inclusion of the sentence gives a better approximation\n        # to the word parameter.\n        if abs(word_count - length - words_in_sentence) > abs(word_count - length):\n            return selected_sentences\n\n        selected_sentences.append(sentence)\n        length += words_in_sentence\n\n    return selected_sentences\n\n\ndef _extract_important_sentences(sentences, corpus, important_docs, word_count):\n    important_sentences = _get_important_sentences(sentences, corpus, important_docs)\n\n    # If no \"word_count\" option is provided, the number of sentences is\n    # reduced by the provided ratio. Else, the ratio is ignored.\n    return important_sentences if word_count is None else _get_sentences_with_word_count(important_sentences, word_count)\n\n\ndef _format_results(extracted_sentences, split):\n    if split:\n        return [sentence.text for sentence in extracted_sentences]\n    return \"\\n\".join([sentence.text for sentence in extracted_sentences])\n\n\ndef _build_hasheable_corpus(corpus):\n    return [tuple(doc) for doc in corpus]\n\n\ndef summarize_corpus(corpus, ratio=0.2):\n    \"\"\"\n    Returns a list of the most important documents of a corpus using a\n    variation of the TextRank algorithm.\n    The input must have at least INPUT_MIN_LENGTH (%d) documents for the\n    summary to make sense.\n\n    The length of the output can be specified using the ratio parameter,\n    which determines how many documents will be chosen for the summary\n    (defaults at 20%% of the number of documents of the corpus).\n\n    The most important documents are returned as a list sorted by the\n    document score, highest first.\n    \"\"\" % INPUT_MIN_LENGTH\n    hashable_corpus = _build_hasheable_corpus(corpus)\n\n    # If the corpus is empty, the function ends.\n    if len(corpus) == 0:\n        logger.warning(\"Input corpus is empty.\")\n        return\n\n    # Warns the user if there are too few documents.\n    if len(corpus) < INPUT_MIN_LENGTH:\n        logger.warning(\"Input corpus is expected to have at least \" + str(INPUT_MIN_LENGTH) + \" documents.\")\n\n    graph = _build_graph(hashable_corpus)\n    _set_graph_edge_weights(graph)\n    _remove_unreachable_nodes(graph)\n\n    # Cannot calculate eigenvectors if number of unique words in text < 3. Warns user to add more text. The function ends.\n    if len(graph.nodes()) < 3:\n        logger.warning(\"Please add more sentences to the text. The number of reachable nodes is below 3\")\n        return\n\n    pagerank_scores = _pagerank(graph)\n\n    hashable_corpus.sort(key=lambda doc: pagerank_scores.get(doc, 0), reverse=True)\n\n    return [list(doc) for doc in hashable_corpus[:int(len(corpus) * ratio)]]\n\n\ndef summarize(text, ratio=0.2, word_count=None, split=False):\n    \"\"\"\n    Returns a summarized version of the given text using a variation of\n    the TextRank algorithm.\n    The input must be longer than INPUT_MIN_LENGTH sentences for the\n    summary to make sense and must be given as a string.\n\n    The output summary will consist of the most representative sentences\n    and will also be returned as a string, divided by newlines. If the\n    split parameter is set to True, a list of sentences will be\n    returned.\n\n    The length of the output can be specified using the ratio and\n    word_count parameters:\n\n        ratio should be a number between 0 and 1 that determines the\n        percentage of the number of sentences of the original text to be\n        chosen for the summary (defaults at 0.2).\n        word_count determines how many words will the output contain.\n        If both parameters are provided, the ratio will be ignored.\n\n    \"\"\"\n    # Gets a list of processed sentences.\n    sentences = _clean_text_by_sentences(text)\n\n    # If no sentence could be identified, the function ends.\n    if len(sentences) == 0:\n        logger.warning(\"Input text is empty.\")\n        return\n\n    # If only one sentence is present, the function raises an error (Avoids ZeroDivisionError).\n    if len(sentences) == 1:\n        raise ValueError(\"input must have more than one sentence\")\n\n    # Warns if the text is too short.\n    if len(sentences) < INPUT_MIN_LENGTH:\n        logger.warning(\"Input text is expected to have at least \" + str(INPUT_MIN_LENGTH) + \" sentences.\")\n\n    corpus = _build_corpus(sentences)\n\n    most_important_docs = summarize_corpus(corpus, ratio=ratio if word_count is None else 1)\n\n    # Extracts the most important sentences with the selected criterion.\n    extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)\n\n    # Sorts the extracted sentences by apparition order in the original text.\n    extracted_sentences.sort(key=lambda s: s.index)\n\n    return _format_results(extracted_sentences, split)\n", "path": "gensim/summarization/summarizer.py"}], "after_files": [{"content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n#\n# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html\n\nimport logging\nfrom gensim.summarization.pagerank_weighted import pagerank_weighted as _pagerank\nfrom gensim.summarization.textcleaner import clean_text_by_sentences as _clean_text_by_sentences\nfrom gensim.summarization.commons import build_graph as _build_graph\nfrom gensim.summarization.commons import remove_unreachable_nodes as _remove_unreachable_nodes\nfrom gensim.summarization.bm25 import get_bm25_weights as _bm25_weights\nfrom gensim.corpora import Dictionary\nfrom math import log10 as _log10\nfrom six.moves import xrange\n\n\nINPUT_MIN_LENGTH = 10\n\nWEIGHT_THRESHOLD = 1.e-3\n\nlogger = logging.getLogger(__name__)\n\n\ndef _set_graph_edge_weights(graph):\n    documents = graph.nodes()\n    weights = _bm25_weights(documents)\n\n    for i in xrange(len(documents)):\n        for j in xrange(len(documents)):\n            if i == j or weights[i][j] < WEIGHT_THRESHOLD:\n                continue\n\n            sentence_1 = documents[i]\n            sentence_2 = documents[j]\n\n            edge_1 = (sentence_1, sentence_2)\n            edge_2 = (sentence_2, sentence_1)\n\n            if not graph.has_edge(edge_1):\n                graph.add_edge(edge_1, weights[i][j])\n            if not graph.has_edge(edge_2):\n                graph.add_edge(edge_2, weights[j][i])\n\n    # Handles the case in which all similarities are zero.\n    # The resultant summary will consist of random sentences.\n    if all(graph.edge_weight(edge) == 0 for edge in graph.edges()):\n        _create_valid_graph(graph)\n\n\ndef _create_valid_graph(graph):\n    nodes = graph.nodes()\n\n    for i in xrange(len(nodes)):\n        for j in xrange(len(nodes)):\n            if i == j:\n                continue\n\n            edge = (nodes[i], nodes[j])\n\n            if graph.has_edge(edge):\n                graph.del_edge(edge)\n\n            graph.add_edge(edge, 1)\n\n\ndef _get_doc_length(doc):\n    return sum([item[1] for item in doc])\n\n\ndef _get_similarity(doc1, doc2, vec1, vec2):\n    numerator = vec1.dot(vec2.transpose()).toarray()[0][0]\n    length_1 = _get_doc_length(doc1)\n    length_2 = _get_doc_length(doc2)\n\n    denominator = _log10(length_1) + _log10(length_2) if length_1 > 0 and length_2 > 0 else 0\n\n    return numerator / denominator if denominator != 0 else 0\n\n\ndef _build_corpus(sentences):\n    split_tokens = [sentence.token.split() for sentence in sentences]\n    dictionary = Dictionary(split_tokens)\n    return [dictionary.doc2bow(token) for token in split_tokens]\n\n\ndef _get_important_sentences(sentences, corpus, important_docs):\n    hashable_corpus = _build_hasheable_corpus(corpus)\n    sentences_by_corpus = dict(zip(hashable_corpus, sentences))\n    return [sentences_by_corpus[tuple(important_doc)] for important_doc in important_docs]\n\n\ndef _get_sentences_with_word_count(sentences, word_count):\n    \"\"\" Given a list of sentences, returns a list of sentences with a\n    total word count similar to the word count provided.\"\"\"\n    length = 0\n    selected_sentences = []\n\n    # Loops until the word count is reached.\n    for sentence in sentences:\n        words_in_sentence = len(sentence.text.split())\n\n        # Checks if the inclusion of the sentence gives a better approximation\n        # to the word parameter.\n        if abs(word_count - length - words_in_sentence) > abs(word_count - length):\n            return selected_sentences\n\n        selected_sentences.append(sentence)\n        length += words_in_sentence\n\n    return selected_sentences\n\n\ndef _extract_important_sentences(sentences, corpus, important_docs, word_count):\n    important_sentences = _get_important_sentences(sentences, corpus, important_docs)\n\n    # If no \"word_count\" option is provided, the number of sentences is\n    # reduced by the provided ratio. Else, the ratio is ignored.\n    return important_sentences if word_count is None else _get_sentences_with_word_count(important_sentences, word_count)\n\n\ndef _format_results(extracted_sentences, split):\n    if split:\n        return [sentence.text for sentence in extracted_sentences]\n    return \"\\n\".join([sentence.text for sentence in extracted_sentences])\n\n\ndef _build_hasheable_corpus(corpus):\n    return [tuple(doc) for doc in corpus]\n\n\ndef summarize_corpus(corpus, ratio=0.2):\n    \"\"\"\n    Returns a list of the most important documents of a corpus using a\n    variation of the TextRank algorithm.\n    The input must have at least INPUT_MIN_LENGTH (%d) documents for the\n    summary to make sense.\n\n    The length of the output can be specified using the ratio parameter,\n    which determines how many documents will be chosen for the summary\n    (defaults at 20%% of the number of documents of the corpus).\n\n    The most important documents are returned as a list sorted by the\n    document score, highest first.\n    \"\"\" % INPUT_MIN_LENGTH\n    hashable_corpus = _build_hasheable_corpus(corpus)\n\n    # If the corpus is empty, the function ends.\n    if len(corpus) == 0:\n        logger.warning(\"Input corpus is empty.\")\n        return []\n\n    # Warns the user if there are too few documents.\n    if len(corpus) < INPUT_MIN_LENGTH:\n        logger.warning(\"Input corpus is expected to have at least \" + str(INPUT_MIN_LENGTH) + \" documents.\")\n\n    graph = _build_graph(hashable_corpus)\n    _set_graph_edge_weights(graph)\n    _remove_unreachable_nodes(graph)\n\n    # Cannot calculate eigenvectors if number of unique documents in corpus < 3.\n    # Warns user to add more text. The function ends.\n    if len(graph.nodes()) < 3:\n        logger.warning(\"Please add more sentences to the text. The number of reachable nodes is below 3\")\n        return []\n\n    pagerank_scores = _pagerank(graph)\n\n    hashable_corpus.sort(key=lambda doc: pagerank_scores.get(doc, 0), reverse=True)\n\n    return [list(doc) for doc in hashable_corpus[:int(len(corpus) * ratio)]]\n\n\ndef summarize(text, ratio=0.2, word_count=None, split=False):\n    \"\"\"\n    Returns a summarized version of the given text using a variation of\n    the TextRank algorithm.\n    The input must be longer than INPUT_MIN_LENGTH sentences for the\n    summary to make sense and must be given as a string.\n\n    The output summary will consist of the most representative sentences\n    and will also be returned as a string, divided by newlines. If the\n    split parameter is set to True, a list of sentences will be\n    returned.\n\n    The length of the output can be specified using the ratio and\n    word_count parameters:\n\n        ratio should be a number between 0 and 1 that determines the\n        percentage of the number of sentences of the original text to be\n        chosen for the summary (defaults at 0.2).\n        word_count determines how many words will the output contain.\n        If both parameters are provided, the ratio will be ignored.\n\n    \"\"\"\n    # Gets a list of processed sentences.\n    sentences = _clean_text_by_sentences(text)\n\n    # If no sentence could be identified, the function ends.\n    if len(sentences) == 0:\n        logger.warning(\"Input text is empty.\")\n        return [] if split else u\"\"\n\n    # If only one sentence is present, the function raises an error (Avoids ZeroDivisionError).\n    if len(sentences) == 1:\n        raise ValueError(\"input must have more than one sentence\")\n\n    # Warns if the text is too short.\n    if len(sentences) < INPUT_MIN_LENGTH:\n        logger.warning(\"Input text is expected to have at least \" + str(INPUT_MIN_LENGTH) + \" sentences.\")\n\n    corpus = _build_corpus(sentences)\n\n    most_important_docs = summarize_corpus(corpus, ratio=ratio if word_count is None else 1)\n\n    # If couldn't get important docs, the algorithm ends.\n    if not most_important_docs:\n        logger.warning(\"Couldn't get relevant sentences.\")\n        return [] if split else u\"\"\n\n    # Extracts the most important sentences with the selected criterion.\n    extracted_sentences = _extract_important_sentences(sentences, corpus, most_important_docs, word_count)\n\n    # Sorts the extracted sentences by apparition order in the original text.\n    extracted_sentences.sort(key=lambda s: s.index)\n\n    return _format_results(extracted_sentences, split)\n", "path": "gensim/summarization/summarizer.py"}]}