title: How to create a generator/iterator with the Python C API?
body: <p>How do I replicate the following Python code with the Python C API?</p>
  <pre><code>class Sequence():
      def __init__(self, max):
          self.max = max
      def data(self):
          i = 0
          while i &lt; self.max:
              yield i
              i += 1
  </code></pre>
  <p>So far, I have this:</p>
  <pre><code>#include &lt;Python/Python.h&gt;
  #include &lt;Python/structmember.h&gt;
  
  /* Define a new object class, Sequence. */
  typedef struct {
      PyObject_HEAD
      size_t max;
  } SequenceObject;
  
  /* Instance variables */
  static PyMemberDef Sequence_members[] = {
      {&quot;max&quot;, T_UINT, offsetof(SequenceObject, max), 0, NULL},
      {NULL} /* Sentinel */
  };
  
  static int Sequence_Init(SequenceObject *self, PyObject *args, PyObject *kwds)
  {
      if (!PyArg_ParseTuple(args, &quot;k&quot;, &amp;(self-&gt;max))) {
          return -1;
      }
      return 0;
  }
  
  static PyObject *Sequence_data(SequenceObject *self, PyObject *args);
  
  /* Methods */
  static PyMethodDef Sequence_methods[] = {
      {&quot;data&quot;, (PyCFunction)Sequence_data, METH_NOARGS,
       &quot;sequence.data() -&gt; iterator object\n&quot;
       &quot;Returns iterator of range [0, sequence.max).&quot;},
      {NULL} /* Sentinel */
  };
  
  /* Define new object type */
  PyTypeObject Sequence_Type = {
     PyObject_HEAD_INIT(NULL)
     0,                         /* ob_size */
     &quot;Sequence&quot;,                /* tp_name */
     sizeof(SequenceObject),    /* tp_basicsize */
     0,                         /* tp_itemsize */
     0,                         /* tp_dealloc */
     0,                         /* tp_print */
     0,                         /* tp_getattr */
     0,                         /* tp_setattr */
     0,                         /* tp_compare */
     0,                         /* tp_repr */
     0,                         /* tp_as_number */
     0,                         /* tp_as_sequence */
     0,                         /* tp_as_mapping */
     0,                         /* tp_hash */
     0,                         /* tp_call */
     0,                         /* tp_str */
     0,                         /* tp_getattro */
     0,                         /* tp_setattro */
     0,                         /* tp_as_buffer */
     Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags*/
     &quot;Test generator object&quot;,   /* tp_doc */
     0,                         /* tp_traverse */
     0,                         /* tp_clear */
     0,                         /* tp_richcompare */
     0,                         /* tp_weaklistoffset */
     0,                         /* tp_iter */
     0,                         /* tp_iternext */
     0,                         /* tp_methods */
     Sequence_members,          /* tp_members */
     0,                         /* tp_getset */
     0,                         /* tp_base */
     0,                         /* tp_dict */
     0,                         /* tp_descr_get */
     0,                         /* tp_descr_set */
     0,                         /* tp_dictoffset */
     (initproc)Sequence_init,   /* tp_init */
     0,                         /* tp_alloc */
     PyType_GenericNew,         /* tp_new */
  };
  
  static PyObject *Sequence_data(SequenceObject *self, PyObject *args)
  {
      /* Now what? */
  }
  </code></pre>
  <p>But I'm not sure where to go next. Could anyone offer some suggestions?</p>
  <h1>Edit</h1>
  <p>I suppose the main problem I'm having with this is simulating the <code>yield</code> statement. As I understand it, it is a pretty simple-looking, but in reality complex, statement — it creates a generator with its own <code>__iter__()</code> and <code>next()</code> methods which are called automatically. Searching through the docs, it seems to be associated with the <a href="http://docs.python.org/c-api/gen.html" rel="noreferrer">PyGenObject</a>; however, how to create a new instance of this object is unclear. <code>PyGen_New()</code> takes as its argument a <code>PyFrameObject</code>, the only reference to which I can find is <a href="http://docs.python.org/c-api/reflection.html" rel="noreferrer"><code>PyEval_GetFrame()</code></a>, which doesn't seem to be what I want (or am I mistaken?). Does anyone have any experience with this they can share?</p>
  <h1>Further Edit</h1>
  <p>I found this to be clearer when I (essentially) expanded what Python was doing behind the scenes:</p>
  <pre><code>class IterObject():
      def __init__(self, max):
          self.max = max
      def __iter__(self):
          self.i = 0
          return self
      def next(self):
          if self.i &gt;= self.max:
              raise StopIteration
          self.i += 1
          return self.i
  
  class Sequence():
      def __init__(self, max):
          self.max = max
      def data(self):
          return IterObject(self.max)
  </code></pre>
  <p>Technically the sequence is off by one but you get the idea.</p>
  <p>The only problem with this is it's very annoying to create a new object every time one needs a generator — even more so in Python than C because of the required monstrosity that comes with defining a new type. And there can be no <code>yield</code> statement in C because C has no closures. What I did instead (since I couldn't find it in the Python API — <em>please</em> point me to a standard object if it already exists!) was create a simple, generic generator object class that called back a C function for every <code>next()</code> method call. Here it is (note that I have not yet tried compiling this because it is not complete — see below):</p>
  <pre><code>#include &lt;Python/Python.h&gt;
  #include &lt;Python/structmember.h&gt;
  #include &lt;stdlib.h&gt;
  
  /* A convenient, generic generator object. */
  
  typedef PyObject *(*callback)(PyObject *callee, void *info) PyGeneratorCallback;
  
  typedef struct {
      PyObject HEAD
      PyGeneratorCallback callback;
      PyObject *callee;
      void *callbackInfo; /* info to be passed along to callback function. */
      bool freeInfo; /* true if |callbackInfo| should be free'()d when object
                      * dealloc's, false if not. */
  } GeneratorObject;
  
  static PyObject *Generator_iter(PyObject *self, PyObject *args)
  {
      Py_INCREF(self);
      return self;
  }
  
  static PyObject *Generator_next(PyObject *self, PyObject *args)
  {
      return self-&gt;callback(self-&gt;callee, self-&gt;callbackInfo);
  }
  
  static PyMethodDef Generator_methods[] = {
      {&quot;__iter__&quot;, (PyCFunction)Generator_iter, METH_NOARGS, NULL},
      {&quot;next&quot;, (PyCFunction)Generator_next, METH_NOARGS, NULL},
      {NULL} /* Sentinel */
  };
  
  static void Generator_dealloc(GenericEventObject *self)
  {
      if (self-&gt;freeInfo &amp;&amp; self-&gt;callbackInfo != NULL) {
          free(self-&gt;callbackInfo);
      }
      self-&gt;ob_type-&gt;tp_free((PyObject *)self);
  }
  
  PyTypeObject Generator_Type = {
     PyObject_HEAD_INIT(NULL)
     0,                         /* ob_size */
     &quot;Generator&quot;,               /* tp_name */
     sizeof(GeneratorObject),   /* tp_basicsize */
     0,                         /* tp_itemsize */
     Generator_dealloc,         /* tp_dealloc */
     0,                         /* tp_print */
     0,                         /* tp_getattr */
     0,                         /* tp_setattr */
     0,                         /* tp_compare */
     0,                         /* tp_repr */
     0,                         /* tp_as_number */
     0,                         /* tp_as_sequence */
     0,                         /* tp_as_mapping */
     0,                         /* tp_hash */
     0,                         /* tp_call */
     0,                         /* tp_str */
     0,                         /* tp_getattro */
     0,                         /* tp_setattro */
     0,                         /* tp_as_buffer */
     Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags*/
     0,                         /* tp_doc */
     0,                         /* tp_traverse */
     0,                         /* tp_clear */
     0,                         /* tp_richcompare */
     0,                         /* tp_weaklistoffset */
     0,                         /* tp_iter */
     0,                         /* tp_iternext */
     0,                         /* tp_methods */
     0,                         /* tp_members */
     0,                         /* tp_getset */
     0,                         /* tp_base */
     0,                         /* tp_dict */
     0,                         /* tp_descr_get */
     0,                         /* tp_descr_set */
     0,                         /* tp_dictoffset */
     0,                         /* tp_init */
     0,                         /* tp_alloc */
     PyType_GenericNew,         /* tp_new */
  };
  
  /* Returns a new generator object with the given callback function
   * and arguments. */
  PyObject *Generator_New(PyObject *callee, void *info,
                          bool freeInfo, PyGeneratorCallback callback)
  {
      GeneratorObject *generator = (GeneratorObject *)_PyObject_New(&amp;Generator_Type);
      if (generator == NULL) return NULL;
  
      generator-&gt;callee = callee;
      generator-&gt;info = info;
      generator-&gt;callback = callback;
      self-&gt;freeInfo = freeInfo;
  
      return (PyObject *)generator;
  }
  
  /* End of Generator definition. */
  
  /* Define a new object class, Sequence. */
  typedef struct {
      PyObject_HEAD
      size_t max;
  } SequenceObject;
  
  /* Instance variables */
  static PyMemberDef Sequence_members[] = {
      {&quot;max&quot;, T_UINT, offsetof(SequenceObject, max), 0, NULL},
      {NULL} /* Sentinel */
  }
  
  static int Sequence_Init(SequenceObject *self, PyObject *args, PyObject *kwds)
  {
      if (!PyArg_ParseTuple(args, &quot;k&quot;, &amp;self-&gt;max)) {
          return -1;
      }
      return 0;
  }
  
  static PyObject *Sequence_data(SequenceObject *self, PyObject *args);
  
  /* Methods */
  static PyMethodDef Sequence_methods[] = {
      {&quot;data&quot;, (PyCFunction)Sequence_data, METH_NOARGS,
       &quot;sequence.data() -&gt; iterator object\n&quot;
       &quot;Returns generator of range [0, sequence.max).&quot;},
      {NULL} /* Sentinel */
  };
  
  /* Define new object type */
  PyTypeObject Sequence_Type = {
     PyObject_HEAD_INIT(NULL)
     0,                         /* ob_size */
     &quot;Sequence&quot;,                /* tp_name */
     sizeof(SequenceObject),    /* tp_basicsize */
     0,                         /* tp_itemsize */
     0,                         /* tp_dealloc */
     0,                         /* tp_print */
     0,                         /* tp_getattr */
     0,                         /* tp_setattr */
     0,                         /* tp_compare */
     0,                         /* tp_repr */
     0,                         /* tp_as_number */
     0,                         /* tp_as_sequence */
     0,                         /* tp_as_mapping */
     0,                         /* tp_hash */
     0,                         /* tp_call */
     0,                         /* tp_str */
     0,                         /* tp_getattro */
     0,                         /* tp_setattro */
     0,                         /* tp_as_buffer */
     Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags*/
     &quot;Test generator object&quot;,   /* tp_doc */
     0,                         /* tp_traverse */
     0,                         /* tp_clear */
     0,                         /* tp_richcompare */
     0,                         /* tp_weaklistoffset */
     0,                         /* tp_iter */
     0,                         /* tp_iternext */
     0,                         /* tp_methods */
     Sequence_members,          /* tp_members */
     0,                         /* tp_getset */
     0,                         /* tp_base */
     0,                         /* tp_dict */
     0,                         /* tp_descr_get */
     0,                         /* tp_descr_set */
     0,                         /* tp_dictoffset */
     (initproc)Sequence_init,   /* tp_init */
     0,                         /* tp_alloc */
     PyType_GenericNew,         /* tp_new */
  };
  
  static PyObject *Sequence_data(SequenceObject *self, PyObject *args)
  {
      size_t *info = malloc(sizeof(size_t));
      if (info == NULL) return NULL;
      *info = 0;
  
      /* |info| will be free'()d by the returned generator object. */
      GeneratorObject *ret = Generator_New(self, info, true,
                                           &amp;Sequence_data_next_callback);
      if (ret == NULL) {
          free(info); /* Watch out for memory leaks! */
      }
      return ret;
  }
  
  PyObject *Sequence_data_next_callback(PyObject *self, void *info)
  {
      size_t i = info;
      if (i &gt; self-&gt;max) {
          return NULL; /* TODO: How do I raise StopIteration here? I can't seem to find
                        *       a standard exception. */
      } else {
          return Py_BuildValue(&quot;k&quot;, i++);
      }
  }
  </code></pre>
  <p>However, unfortunately, I'm still not finished. The only question I have left is: How do I raise a <code>StopIteration</code> exception with the C API? I can't seem to find it listed in the <a href="http://docs.python.org/c-api/exceptions.html#standard-exceptions" rel="noreferrer">Standard Exceptions</a>. Also, perhaps more importantly, is this the correct way to approach this problem?</p>
  <p>Thanks to anyone that's still following this.</p>