text: Build a mail handling service
  My goal for this short coding session is to have a mail handling service that will allow me to list and maninpulate mails through a JSON API REST back end. And have that service pick up when I write a mail to the database and send it automatically. You can see the result of this project at.
  Gain a head-start with mu-project
  For this project I started with cloning the mu-project repository:
  git clone
  This will give me the CRUD endpoint I need to manipulate my mail related resources. After cloning I rename the repository to MailBox and set the remote origin to a new one. For now I will leave the README.md file as it is.
  For the first block we will modify the /config/resources/domain.lisp, /config/resourecs/repository.lisp and the /config/dispatcher/dispatcher.ex files.
  To add the necessary resource definitions add them to the domain.lisp file as follows:
  (define-resource mail () :class (s-prefix "example:Mail") :properties `((:sender :string ,(s-prefix "example:sender")) (:subject :string ,(s-prefix "example:subject")) (:content :string ,(s-prefix "example:content")) (:ready :string ,(s-prefix "example:ready"))) :resource-base (s-url "") :on-path "mails")
  This will create a resource description that we can manipulate on route /mails with the properties sender, title, body and ready.
  Then add the prefix to the repository.lisp file:
  (add-prefix "example" "")
  We are almost there for a first test! The only thing left to do is to add the /mails route to the dispatcher (for more info check the documentation on). To do this add the following block of code to the dispatcher.ex file:
  match "/mails/*path" do Proxy.forward conn, path, "" end
  Now fire this up and lets see what we have by typing:
  docker-compose up
  in the a console in the project root directory. We don’t have a front end yet but with a tool like postman we can make GET, PATCH and POST calls to test the backend functionality.
  A GET call to produces:
  { "data": [], "links": { "last": "/mails/", "first": "/mails/" } }
  Alright! Ok no data yet but we get back resource information.
  Lets do a post to make a new mail resource:
  URL: Headers: {"Content-Type":"application/vnd.api+json"} Body: { "data":{ "attributes":{ "sender":"flowofcontrol@gmail.com", "subject":"Mu Semtech Mail Server", "content":"This is a test for the Mu Semtech Mail Server.", "ready":"no" }, "type":"mails" } }
  This gives us the following reponse:
  { "data": { "attributes": { "sender": "flowofcontrol@gmail.com", "subject": "Mu Semtech Mail Server", "content": "This is a test for the Mu Semtech Mail Server.", "ready": "no" }, "id": "58978C2A6460170009000001", "type": "mails", "relationships": {} } }
  That worked, in about 30 minutes we have a fully functional REST API endpoint for managing mail resources!
  To verify to original get again, this now produces:
  { "data": { "attributes": { "sender": "flowofcontrol@gmail.com", "subject": "Mu Semtech Mail Server", "content": "This is a test for the Mu Semtech Mail Server.", "ready": "no" }, "id": "58978C3A6460170009000002", "type": "mails", "relationships": {} } }
  Enabling the reactive database
  Before we can start writing our reactive mail managing micro-service we will need to add a monitoring service to monitor the DB. This will be a lot easier than it sounds with mu.semte.ch. To start open the docker-compose.yml file and add the following lines at the bottom of the page:
  delta: image: semtech/mu-delta-service:beta-0.7 links: - db:db volumes: - ./config/delta-service:/config environment: CONFIGFILE: "/config/config.properties" SUBSCRIBERSFILE: "/config/subscribers.json"
  This will add the monitoring service to our installation. The last thing to do for now is to change the link on the resource microservice by replacing
  links: - db:database
  with
  links: - delta:database
  The final steps are to create the configuration and subscribers files. Create a file called config.properties at the location config/delta-service/config.properties and write the following lines in that file:
  # made by Langens Jonathan queryURL= updateURL= sendUpdateInBody=true calculateEffectives=true
  and then create config/delta-service/subscribers.json and put this JSON inside:
  { "potentials":[ ], "effectives":[ ] }
  If we do docker-compose rm and then docker-compose up again then the delta service will be booting and already monitoring the changes that happen in the database! Of course we are not doing anything with them yet. So we will create a new micro-service just for this purpose.
  The mail-fetching microservice
  The next step is to build our mail handling microservice. To do this we create a new directory called mail-service in our platform base directory. Then we create a file in that directory called ‘Dockerfile’. We will start from a mu.semte.ch template to make developing this microservice that much quicker. Mu.semte.ch has templates for a bunch of languages ruby, javascript, python, … For this microservice we will go for python 2.7. To do this we simply need to create a web.py file which will serve as the location for our code. Next add the following to the Dockerfile:
  FROM semtech/mu-python-template MAINTAINER Langens Jonathan <flowofcontrol@gmail.com>
  I know it doesn’t say much but it doesn’t need to. The python template will handle the rest.
  Then we need to add some mail manipulating functionality. Since this is not really the objective of this post I create a mail_helpers.py file and paste the following code in there:
  import sys import imaplib import getpass import email import datetime import uuid import helpers def save_mail(sender, date, subject, content): str_uuid = str(uuid.uuid4()) a <>;\n" insert_query += "<> \"" + sender + "\";\n" insert_query += "<> \"" + date + "\";\n" insert_query += "<> \"" + content + "\";\n" insert_query += "<> \"" + subject + "\";\n" insert_query += "<> \"" + str_uuid + "\".\n" insert_query += "}\n}" print "query:\n", insert_query helpers.update(insert_query) def process_mailbox(mailbox): rv, data = mailbox.search(None, "ALL") if rv != 'OK': print "No messages found!" return for num in data[0].split(): rv, data = mailbox.fetch(num, '(RFC822)') if rv != 'OK': print "ERROR getting message", num return msg = email.message_from_string(data[0][1]) content = str(msg.get_payload()) content = content.replace('\n','') save_mail(msg['From'], msg['Date'], msg['Subject'], content)
  As you can see the mail_helpers contain 2 functions, 1 to iterate over all emails in a mailbox and the other to save a single email to the triple store. Easy peasy!
  Next we create a web.py file for more information on how the python template can be used you can visit:. I created the following method to process all mails:
  @app.route("/fetchMails") def fetchMailMethod(): EMAIL_ADDRESS = "address" EMAIL_PWD = "pwd" MAIL_SERVER = imaplib.IMAP4_SSL('imap.gmail.com') try: MAIL_SERVER.login(EMAIL_ADDRESS, EMAIL_PWD) except imaplib.IMAP4.error: print "Logging into mailbox failed! " rv, data = MAIL_SERVER.select("INBOX") if rv == 'OK': mail_helpers.process_mailbox(MAIL_SERVER) MAIL_SERVER.close() MAIL_SERVER.logout() return "ok"
  This method is rather straightforward it just opens a connection to a email address and opens the inbox mailbox. Then selects it for processing thus inserting all mails into the triple store.
  Next week
  This week we have defined a JSONAPI through which we can access our emails, using the standard mu.semte.ch stack. We have also built a custom service which fetches the emails from our mail account and inserts them into the triplestore using the right model. In the next post we will use these services in combination with the delta service, to discover which emails were inserted into the database, and to perform reactive computations on it.
  Stay tuned!