title: storing the python-dictionary with 10000 lines in mysql - instead of printing
body: <p>see below 
  <strong>update 2</strong> i did as you adviced me: </p>
  
  <p><strong>update:</strong> see the important updates: </p>
  
  <p>want to store the data to the mysql-db</p>
  
  <pre><code>{'url': 'http://dom1', 'name': 'miller', 'name2': 'phil man', 'email-adress': 'waddehadde@hotmail.com'}
  {'url': 'http://dom2', 'name': 'jonboy', 'name2': 'Josef dude', 'email-adress': 'waddehadde@hotmail.com'}
  </code></pre>
  
  <p>I have a very simple dataset  but it s quite very big: approx 10 k records. 
  i-connect-to-a-mysql-database-in-python And i found out that i probably can use peewee</p>
  
  <pre><code>import peewee
  from peewee import *
  
  db = MySQLDatabase('jonhydb', user='john',passwd='megajonhy')
  
  class Book(peewee.Model):
      author = peewee.CharField()
      title = peewee.TextField()
  
      class Meta:
          database = db
  
  Book.create_table()
  book = Book(author="me", title='Peewee is cool')
  book.save()
  for book in Book.filter(author="me"):
      print book.title
  
  Peewee is cool
  </code></pre>
  
  <p>I know of two ways to do this, one being storing the number in a file on the server
  the otehr is to serve it in a database, using MySQL to store this. i can use PHP - to store it in the mysql-db. simple, right? But yesterday ive  found out that Python is even better.  I have installed python on my linux-distri</p>
  
  <p>but it seem overcomplicated to me.</p>
  
  <p>So this takes me to a question, is there a way to store a this dataset with using MySQL and python peewee or some simliar leightweight orm!?</p>
  
  <p>first of all - many many thanks for the quick answer</p>
  
  <p>i am on opensuse verision 13.1 
  the mysql db is allready up and running.</p>
  
  <p>well first of all  i had to install peewee and json 
  while i had some isseus installing peewee - see  below 
  i afterwards tried to install simplejson - as a replacement for json 
  then i guess that peewee was installabe. - see below: </p>
  
  <p>had issues with installing peewee</p>
  
  <pre><code>martin@linux-70ce:~&gt;
  martin@linux-70ce:~&gt; git clone https://github.com/coleifer/peewee.git
  Klone nach 'peewee'...
  remote: Reusing existing pack: 5673, done.
  remote: Counting objects: 13, done.
  remote: Compressing objects: 100% (13/13), done.
  remote: Total 5686 (delta 2), reused 0 (delta 0)
  Empfange Objekte: 100% (5686/5686), 3.54 MiB | 102.00 KiB/s, done.
  Löse Unterschiede auf: 100% (3468/3468), done.
  Prüfe Konnektivität... Fertig
  martin@linux-70ce:~&gt; cd peewee
  martin@linux-70ce:~/peewee&gt; python setup.py install
  running install
  error: can't create or remove files in install directory
  
  The following error occurred while trying to add or remove files in the
  installation directory:
  
      [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/test-easy-install-5717.write-test'
  
  The installation directory you specified (via --install-dir, --prefix, or
  the distutils default setting) was:
  
      /usr/lib/python2.7/site-packages/
  
  Perhaps your account does not have write access to this directory?  If the
  installation directory is a system-owned directory, you may need to sign in
  as the administrator or "root" account.  If you do not have administrative
  access to this machine, you may wish to choose a different installation
  directory, preferably one that is listed in your PYTHONPATH environment
  variable.
  
  For information on other options, you may wish to consult the
  documentation at:
  Please make the appropriate changes for your system and try again.
  martin@linux-70ce:~/peewee&gt;
  </code></pre>
  
  <p>afer that - as written above - i installed simplejson and then i tried to install peewee again :: now i guess i had more luck....</p>
  
  <pre><code>martin@linux-70ce:~/peewee&gt; git clone https://github.com/coleifer/peewee.git
  Klone nach 'peewee'...                                                                                                                                                             
  remote: Reusing existing pack: 5673, done.
  remote: Counting objects: 13, done.
  remote: Compressing objects: 100% (13/13), done.
  remote: Total 5686 (delta 2), reused 0 (delta 0)
  Empfange Objekte: 100% (5686/5686), 3.54 MiB | 309.00 KiB/s, done.                                                                                                                 
  fix the conflicts: 100% (3468/3468), done.                                                                                                        
  </code></pre>
  
  <p>Prüfe Konnektivität... Fertig<br>
  martin@linux-70ce:~/peewee>                           </p>
  
  <p>SEE THE FULL CODE - THAT I AM RUNNING </p>
  
  <pre><code>import urllib
  import urlparse
  import re
  from peewee import *
  import json
  
  db = MySQLDatabase('cpan', user='root',passwd='rimbaud')
  
  class User(Model):
      name = TextField()
      cname = TextField()
      email = TextField()
      url = TextField()
  
      class Meta:
          database = db # this model uses the cpan database
  
  
  User.create_table() #ensure table is created
  
  
  url = "http://search.cpan.org/author/?W"
  html = urllib.urlopen(url).read()
  for lk, capname, name in re.findall('&lt;a href="(/~.*?/)"&gt;&lt;b&gt;(.*?)&lt;/b&gt;&lt;/a&gt;&lt;br/&gt;&lt;small&gt;(.*?)&lt;/small&gt;', html):
  
      alk = urlparse.urljoin(url, lk)
  
      data = { 'url':alk, 'name':name, 'cname':capname }
  
      phtml = urllib.urlopen(alk).read()
      memail = re.search('&lt;a href="mailto:(.*?)"&gt;', phtml)
      if memail:
          data['email'] = memail.group(1)
  
  
  data = json.load() #your json data file here
  
  for entry in data: #assuming your data is an array of JSON objects
      user = User.create(name=entry["name"], cname=entry["cname"],
          email=entry["email"], url=entry["url"])
      user.save()
  </code></pre>
  
  <p>and the subsequently results: </p>
  
  <pre><code>       martin@linux-70ce:~/perl&gt; python cpan6.py
  python: can't open file 'cpan6.py': [Errno 2] No such file or directory
  martin@linux-70ce:~/perl&gt; python cpan5.py
  Traceback (most recent call last):
    File "cpan5.py", line 7, in &lt;module&gt;
      from peewee import *
  ImportError: No module named peewee
  martin@linux-70ce:~/perl&gt; 
  </code></pre>
  
  <p>well at the moment i am a bit clueless - love to hear from you
  many thanks in advance!!       </p>
  
  <p><strong>update 2</strong> i did as you adviced me: </p>
  
  <pre><code>martin@linux-70ce:~/peewee&gt; sudo python setup.py install 
  
  t
  linux-70ce:/home/martin/perl # cd ..
  linux-70ce:/home/martin # cd peewee/
  linux-70ce:/home/martin/peewee #  sudo python setup.py install 
  running install
  running bdist_egg
  running egg_info
  creating peewee.egg-info
  writing peewee.egg-info/PKG-INFO
  writing top-level names to peewee.egg-info/top_level.txt
  writing dependency_links to peewee.egg-info/dependency_links.txt
  writing manifest file 'peewee.egg-info/SOURCES.txt'
  reading manifest file 'peewee.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'peewee.egg-info/SOURCES.txt'
  installing library code to build/bdist.linux-i686/egg
  running install_lib
  running build_py
  creating build
  creating build/lib
  copying peewee.py -&gt; build/lib
  copying pwiz.py -&gt; build/lib
  creating build/lib/playhouse
  copying playhouse/tests_shortcuts.py -&gt; build/lib/playhouse
  copying playhouse/read_slave.py -&gt; build/lib/playhouse
  copying playhouse/tests_kv.py -&gt; build/lib/playhouse
  copying playhouse/tests_pwiz.py -&gt; build/lib/playhouse
  copying playhouse/gfk.py -&gt; build/lib/playhouse
  copying playhouse/test_utils.py -&gt; build/lib/playhouse
  copying playhouse/apsw_ext.py -&gt; build/lib/playhouse
  copying playhouse/__init__.py -&gt; build/lib/playhouse
  copying playhouse/tests_pool.py -&gt; build/lib/playhouse
  copying playhouse/tests_apsw.py -&gt; build/lib/playhouse
  copying playhouse/kv.py -&gt; build/lib/playhouse
  copying playhouse/postgres_ext.py -&gt; build/lib/playhouse
  copying playhouse/signals.py -&gt; build/lib/playhouse
  copying playhouse/sqlcipher_ext.py -&gt; build/lib/playhouse
  copying playhouse/tests_gfk.py -&gt; build/lib/playhouse
  copying playhouse/tests_read_slave.py -&gt; build/lib/playhouse
  copying playhouse/pool.py -&gt; build/lib/playhouse
  copying playhouse/tests_csv_loader.py -&gt; build/lib/playhouse
  copying playhouse/tests_berkeleydb.py -&gt; build/lib/playhouse
  copying playhouse/djpeewee.py -&gt; build/lib/playhouse
  copying playhouse/tests_test_utils.py -&gt; build/lib/playhouse
  copying playhouse/migrate.py -&gt; build/lib/playhouse
  copying playhouse/csv_loader.py -&gt; build/lib/playhouse
  copying playhouse/tests_postgres.py -&gt; build/lib/playhouse
  copying playhouse/tests_migrate.py -&gt; build/lib/playhouse
  copying playhouse/tests_djpeewee.py -&gt; build/lib/playhouse
  copying playhouse/tests_sqlite_ext.py -&gt; build/lib/playhouse
  copying playhouse/berkeleydb.py -&gt; build/lib/playhouse
  copying playhouse/proxy.py -&gt; build/lib/playhouse
  copying playhouse/tests_signals.py -&gt; build/lib/playhouse
  copying playhouse/sqlite_ext.py -&gt; build/lib/playhouse
  copying playhouse/tests_sqlcipher_ext.py -&gt; build/lib/playhouse
  copying playhouse/shortcuts.py -&gt; build/lib/playhouse
  creating build/bdist.linux-i686
  creating build/bdist.linux-i686/egg
  copying build/lib/peewee.py -&gt; build/bdist.linux-i686/egg
  creating build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_shortcuts.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/read_slave.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_kv.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_pwiz.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/gfk.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/test_utils.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/apsw_ext.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/__init__.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_pool.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_apsw.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/kv.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/postgres_ext.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/signals.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/sqlcipher_ext.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_gfk.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_read_slave.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/pool.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_csv_loader.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_berkeleydb.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/djpeewee.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_test_utils.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/migrate.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/csv_loader.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_postgres.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_migrate.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_djpeewee.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_sqlite_ext.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/berkeleydb.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/proxy.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_signals.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/sqlite_ext.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/tests_sqlcipher_ext.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/playhouse/shortcuts.py -&gt; build/bdist.linux-i686/egg/playhouse
  copying build/lib/pwiz.py -&gt; build/bdist.linux-i686/egg
  byte-compiling build/bdist.linux-i686/egg/peewee.py to peewee.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_shortcuts.py to tests_shortcuts.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/read_slave.py to read_slave.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_kv.py to tests_kv.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_pwiz.py to tests_pwiz.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/gfk.py to gfk.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/test_utils.py to test_utils.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/apsw_ext.py to apsw_ext.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/__init__.py to __init__.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_pool.py to tests_pool.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_apsw.py to tests_apsw.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/kv.py to kv.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/postgres_ext.py to postgres_ext.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/signals.py to signals.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/sqlcipher_ext.py to sqlcipher_ext.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_gfk.py to tests_gfk.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_read_slave.py to tests_read_slave.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/pool.py to pool.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_csv_loader.py to tests_csv_loader.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_berkeleydb.py to tests_berkeleydb.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/djpeewee.py to djpeewee.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_test_utils.py to tests_test_utils.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/migrate.py to migrate.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/csv_loader.py to csv_loader.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_postgres.py to tests_postgres.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_migrate.py to tests_migrate.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_djpeewee.py to tests_djpeewee.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_sqlite_ext.py to tests_sqlite_ext.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/berkeleydb.py to berkeleydb.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/proxy.py to proxy.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_signals.py to tests_signals.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/sqlite_ext.py to sqlite_ext.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/tests_sqlcipher_ext.py to tests_sqlcipher_ext.pyc
  byte-compiling build/bdist.linux-i686/egg/playhouse/shortcuts.py to shortcuts.pyc
  byte-compiling build/bdist.linux-i686/egg/pwiz.py to pwiz.pyc
  creating build/bdist.linux-i686/egg/EGG-INFO
  installing scripts to build/bdist.linux-i686/egg/EGG-INFO/scripts
  running install_scripts
  running build_scripts
  creating build/scripts-2.7
  copying and adjusting pwiz.py -&gt; build/scripts-2.7
  changing mode of build/scripts-2.7/pwiz.py from 644 to 755
  creating build/bdist.linux-i686/egg/EGG-INFO/scripts
  copying build/scripts-2.7/pwiz.py -&gt; build/bdist.linux-i686/egg/EGG-INFO/scripts
  changing mode of build/bdist.linux-i686/egg/EGG-INFO/scripts/pwiz.py to 755
  copying peewee.egg-info/PKG-INFO -&gt; build/bdist.linux-i686/egg/EGG-INFO
  copying peewee.egg-info/SOURCES.txt -&gt; build/bdist.linux-i686/egg/EGG-INFO
  copying peewee.egg-info/dependency_links.txt -&gt; build/bdist.linux-i686/egg/EGG-INFO
  copying peewee.egg-info/top_level.txt -&gt; build/bdist.linux-i686/egg/EGG-INFO
  zip_safe flag not set; analyzing archive contents...
  creating dist
  creating 'dist/peewee-2.2.5-py2.7.egg' and adding 'build/bdist.linux-i686/egg' to it
  removing 'build/bdist.linux-i686/egg' (and everything under it)
  Processing peewee-2.2.5-py2.7.egg
  Copying peewee-2.2.5-py2.7.egg to /usr/lib/python2.7/site-packages
  Adding peewee 2.2.5 to easy-install.pth file
  Installing pwiz.py script to /usr/bin
  
  Installed /usr/lib/python2.7/site-packages/peewee-2.2.5-py2.7.egg
  Processing dependencies for peewee==2.2.5
  Finished processing dependencies for peewee==2.2.5
  linux-70ce:/home/martin/peewee # 
  </code></pre>
  
  <p>see the latest update - after insalling peewee correctly i runned the script and now see what happened.</p>
  
  <pre><code>import urllib
  import urlparse
  import re
  import peewee
  import json
  
  db = MySQLDatabase('cpan', user='root',passwd='rimbaud')
  
  class User(Model):
      name = TextField()
      cname = TextField()
      email = TextField()
      url = TextField()
  
      class Meta:
          database = db # this model uses the cpan database
  
  
  User.create_table() #ensure table is created
  
  
  url = "http://search.cpan.org/author/?W"
  html = urllib.urlopen(url).read()
  for lk, capname, name in re.findall('&lt;a href="(/~.*?/)"&gt;&lt;b&gt;(.*?)&lt;/b&gt;&lt;/a&gt;&lt;br/&gt;&lt;small&gt;(.*?)&lt;/small&gt;', html):
      alk = urlparse.urljoin(url, lk)
  
      data = { 'url':alk, 'name':name, 'cname':capname }
  
      phtml = urllib.urlopen(alk).read()
      memail = re.search('&lt;a href="mailto:(.*?)"&gt;', phtml)
      if memail:
          data['email'] = memail.group(1)
  
  
  data = json.load() #your json data file here
  
  for entry in data: #assuming your data is an array of JSON objects
      user = User.create(name=entry["name"], cname=entry["cname"],
          email=entry["email"], url=entry["url"])
      user.save()
  </code></pre>
  
  <p>i got back this error. </p>
  
  <pre><code>Traceback (most recent call last):
    File "cpan5.py", line 10, in &lt;module&gt;
      db = MySQLDatabase('cpan', user='root',passwd='rimbaud')
  NameError: name 'MySQLDatabase' is not defined
  linux-70ce:/home/martin/perl # 
  </code></pre>
  
  <p><strong>the newest update:</strong>  from july 14th </p>
  
  <p>assuming this is all right now - i have set up this... so 
  well - but it fails at a certain point.</p>
  
  <pre><code>import urllib
  import urlparse
  import re
  # import peewee
  import json
  from peewee import *
  
  
  
  #from peewee import MySQLDatabase ('cpan', user='root',passwd='rimbaud') 
  
  
  db = MySQLDatabase('cpan', user='root',passwd='rimbaud') 
  
  class User(Model):
      name = TextField()
      cname = TextField()
      email = TextField()
      url = TextField()
  
      class Meta:
          database = db # this model uses the cpan database
  
  
  User.create_table() #ensure table is created
  
  
  url = "http://search.cpan.org/author/?W"
  html = urllib.urlopen(url).read()
  for lk, capname, name in re.findall('&lt;a href="(/~.*?/)"&gt;&lt;b&gt;(.*?)&lt;/b&gt;&lt;/a&gt;&lt;br/&gt;&lt;small&gt;(.*?)&lt;/small&gt;', html):
      alk = urlparse.urljoin(url, lk)
  
      data = { 'url':alk, 'name':name, 'cname':capname }
  
      phtml = urllib.urlopen(alk).read()
      memail = re.search('&lt;a href="mailto:(.*?)"&gt;', phtml)
      if memail:
          data['email'] = memail.group(1)
  
  
  data = json.load('email') #your json data file here
  
  for entry in data: #assuming your data is an array of JSON objects
      user = User.create(name=entry["name"], cname=entry["cname"],
          email=entry["email"], url=entry["url"])
      user.save()
  </code></pre>
  
  <p>guess that there a data-file must exist: one that have been created by the script during the parsing... is this right?</p>
  
  <pre><code>)
  martin@linux-70ce:~/perl&gt; python cpan_100.py
  Traceback (most recent call last):
    File "cpan_100.py", line 47, in &lt;module&gt;
      data = json.load('email') #your json data file here
    File "/usr/lib/python2.7/json/__init__.py", line 286, in load
      return loads(fp.read(),
  AttributeError: 'str' object has no attribute 'read'
  martin@linux-70ce:~/perl&gt; 
  </code></pre>
  
  <p>love to hear from you</p>