Ningle Tutorial 10: Email

Contents

• Part 1 (Hello World)
• Part 2 (Basic Templates)
• Part 3 (Introduction to middleware and Static File management)
• Part 4 (Forms)
• Part 5 (Environmental Variables)
• Part 6 (Database Connections)
• Part 7 (Envy Configuation Switching)
• Part 8 (Mounting Middleware)
• Part 9 (Authentication System)
• Part 10 (Email)

Introduction

Welcome back to this tutorial series, in this chapter we are going to write a small app for sending email and connect it up to the authentication system we wrote last time, as part of that we will need to expand the settings we have in our main project. We will also look at different ways in which you can send email, from outputting the console (as a dummy test), smtp (simple mail transfer protocol), and the sendgrid service.

Main Package (Part 1)

There isn’t too much to change in this package, the most we will be doing is creating a series of settings objects to test the different email options. Of course, we will be introducing new settings and relying on the envy-ningle package to load them for us. We will also create some templates in our auth application, but we will override them in our application using the templates override mechanism we developed previously.

There will be a number of required settings and some settings that will only be used in certain circumstances, we have seen before in Part 7 (Envy Configuation Switching) how to use these settings objects.

Let’s start with the common, shared settings, we have our :common settings object:

(defconfig :common
  `(:application-root ,(asdf:component-pathname (asdf:find-system :ningle-tutorial-project))
    :installed-apps (:ningle-auth)
    :auth-mount-path ,*auth-mount-path*
    :login-redirect "/"
    :project-name "NTP"                  ; 1: add this
    :token-expiration 3600               ; 2: add this
    :email-admins ("nmunro@duck.com")))  ; 3: add this

Our first setting is simply creating a name for our project that we can use in email titles etc.

The second setting is to add is related to the tokens we created, we might want to lower this during testing, and restore it when we go into production, it makes sense to centralise it! I should have considered this last time, but the chapter was so big, I had to make cuts somewhere!

The third setting is related to mailing project admins, in the event there’s an error we can mail someone (or in this case a list of people), it’s something we will explore, but not necessarily use, because this is, after all, a tutorial project and not a full blown commercial application.

We have four settings objects we need to create to test everything we need, we will continue using sqlite for our config, but we will explore the following email setups:

• dummy (console)
• smtp with ethereal (a demo smtp service)
• smtp with gmail
• sendgrid

Since we are going to write lots of new settings, each of which is going to duplicate the middleware, we are going to explore how to modularize settings (at least a little)!

We will start by defining a new settings block, but it will only contain our middleware settings, we will, for good measure also look into extracting out the database settings, if, for whatever reason, we need to change them in future.

(defconfig |database-settings|
  `((:mito (:sqlite3 :database-name ,(uiop:getenv "SQLITE_DB_NAME")))))

(defconfig |middleware|
  `(:middleware ((:session)
                 ningle-tutorial-project/middleware:refresh-roles
                 ,@|database-settings|
                 (:mount ,*auth-mount-path* ,ningle-auth:*app*)
                 (:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))

We can see here that a small, database specific settings block exists that defines our database connection settings, when then include it inside a middleware settings block which we will now use in our dummy email settings block:

Prior to writing our settings, we will follow good security practices and NOT store details in our repository, so we will need to edit our .env file.

.env

I obviously didn’t include the actual values here, but simply wanted to include the settings names, for clarity.

EMAIL_DEFAULT_FROM=xxx

SMTP_GMAIL_HOST=xxx
SMTP_GMAIL_ACCOUNT_NAME=xxx
SMTP_GMAIL_PASSWORD=xxx

SMTP_ETHEREAL_HOST=xxx
SMTP_ETHEREAL_ACCOUNT_NAME=xxx
SMTP_ETHEREAL_PASSWORD=xxx

SENDGRID_API_KEY=xxx

Some settings only apply to certain configurations, and some settings require some setup, for example if you want to use ethereal, you will need to set up an account and grab the user, account, and password, if you want to use sendgrid, you will need to get an api key etc.

These tasks I leave up to you, but I will mention them as each settings require them, just remember to come back and add in the settings you need.

Dummy Email Backend Settings

(defconfig |dummy-email|
`(:debug T
    ,@|sqlite-middleware|
    :email-backend :dummy                                      ; 1
    :email-default-from ,(uiop:getenv "EMAIL_DEFAULT_FROM")))  ; 2

This helps us really focus on what we are adding in, it’s worth noting that these settings don’t configure anything yet, but they will when we write the email package, but for now we are:

1. Defining an email dummy back end (this will be used to print email to the terminal)
2. Setup a default “from” address

Since this requires the EMAIL_DEFAULT_FROM setting, please ensure you have an actual value stored.

Our next three configs follow a similar pattern.

Ethereal SMTP Email Backend Settings

Ethereal is a free fake smtp service, it’s a great way to check smtp settings are correct prior to potentially spamming an email account with testing emails. We will use this as a test, while I have an example for smtp settings for gmail, this is not a comprehensive guide to every email provider, so etheral should help you test things, if I have not covered your specific email provider, or… Like me, your account was too locked down to use as an email.

Ethereal has a help page where you can find the host settings etc. The SMTP_ETHERAL_ACCOUNT_NAME gets used for the :email-default-from and :email-reply-to as well as part of the :email-auth settings, there will also be an account password when you set an account up, which will be stored as SMTP_ETHEREAL_PASSWORD and used in the :email-auth too.

(defconfig |ethereal-smtp|
  `(:debug T
    ,@|middleware|
    :email-backend :smtp
    :email-smtp-host ,(uiop:getenv "SMTP_ETHEREAL_HOST")
    :email-default-from ,(uiop:getenv "SMTP_ETHEREAL_ACCOUNT_NAME")
    :email-reply-to ,(uiop:getenv "SMTP_ETHEREAL_ACCOUNT_NAME")
    :email-port 587
    :email-auth (,(uiop:getenv "SMTP_ETHEREAL_ACCOUNT_NAME") ,(uiop:getenv "SMTP_ETHEREAL_PASSWORD"))
    :email-ssl :starttls))

Remember: Add the following to your .env file!

• SMTP_ETHEREAL_ACCOUNT_NAME
• SMTP_ETHEREAL_PASSWORD

When we come to test this, we can use their web interface to check if email would have been sent.

GMail SMTP Email Backend Settings

Setting up GMail for smtp can be a little tricky, certain security settings have to be enabled (and certain ones NOT), at a minimum you must have mfa set up on the account, and Google no longer allows username and passwords as authentication, you must set up an “app password” for your application and use that for the authentication.

No big deal really, but it’s some gotchas that you’ll want to be aware of if you are using GMail as your email provider, again this isn’t a tutorial on how to configure GMail for SMTP, this is how to make Common Lisp use it once it is configured.

(defconfig |gmail-smtp|
  `(:debug T
    ,@|middleware|
    :email-backend :smtp
    :email-smtp-host ,(uiop:getenv "SMTP_GMAIL_HOST")
    :email-default-from ,(uiop:getenv "SMTP_GMAIL_ACCOUNT_NAME")
    :email-reply-to ,(uiop:getenv "SMTP_GMAIL_ACCOUNT_NAME")
    :email-port 587
    :email-auth (,(uiop:getenv "SMTP_GMAIL_ACCOUNT_NAME") ,(uiop:getenv "SMTP_GMAIL_PASSWORD"))
    :email-ssl :starttls))

Remember: Add in the following values in your .env file!

• SMTP_GMAIL_HOST
• SMTP_GMAIL_ACCOUNT_NAME
• SMTP_GMAIL_PASSWORD

SendGrid Email Backend Settings

Sendgrid is a popular way to send mass emails, to get set up you will need an account with an api-key. Once you have those, the settings are as follows.

(defconfig |sendgrid|
  `(:debug T
    ,@|middleware|
    :email-backend :sendgrid
    :email-reply-to ,(uiop:getenv "EMAIL_DEFAULT_FROM")
    :sendgrid-api-key ,(uiop:getenv "SENDGRID_API_KEY")))

Remember: Add the following to your .env file!

• EMAIL_DEFAULT_FROM
• SENDGRID_API_KEY

Email Package

Now that we have your config in place, we can look at building an email package, don’t worry though it’s less than 50 lines, so nothing too crazy, we just create a package because Ningle is a micro framework and so we create small packages to work with it. Perhaps in a later version of this series we build a tighter coupled framework, but not right now.

Using my project builder create a new project like so:

(nmunro-project:make-project #p"~/quicklisp/local-projects/ningle-email")

In the project asd file we need to depend on three packages:

1. envy-ningle
2. cl-smtp
3. cl-sendgrid

And with that, we can edit ningle-email/src/main.lisp and write two simple mail functions send-mail and mail-admins.

(defpackage ningle-email
  (:use :cl)
  (:export #:mail-admins
           #:send-mail))

(in-package ningle-email)

(defun mail-admins (subject message)
  "Sends an email to the admins"
  (let ((project-name (envy-ningle:get-config :project-name))
        (admins (envy-ningle:get-config :email-admins)))
    (send-mail (format nil "[~A]: ~A" project-name subject) message admins)))

(defun send-mail (subject content to &key (from (envy-ningle:get-config :email-default-from)))
  "Sends arbitrary email"
  (let ((email-backend (envy-ningle:get-config :email-backend)))
    (case email-backend
        (:dummy
         (progn
            (format t "from: ~A~%" from)
            (format t "to: ~A~%" to)
            (format t "subject: ~A~%" subject)
            (format t "content: ~A~%" content)))

        (:smtp
            (cl-smtp:send-email
                (envy-ningle:get-config :email-smtp-host)
                from
                to
                subject
                message
                :port (or (envy-ningle:get-config :email-port) 587)
                :ssl (or (envy-ningle:get-config :email-ssl) :starttls)
                :authentication (envy-ningle:get-config :email-auth)))

        (:sendgrid
            (sendgrid:send-email
                :to to
                :from from
                :subject subject
                :content message
                :api-key (envy-ningle:get-config :sendgrid-api-key)))

        (otherwise
            (error "Unknown email backend: ~A" email-backend)))))

It may seem a little unusual to define the mail-admins before we have defined our send-mail and while in common lisp it’s possible to compile a function that calls a function that doesn’t yet exist, because it will be compiled immediately after.

Our new mail-admins function will be a simple wrapper around the send-mail function, so we will look at that first.

(defun mail-admins (subject message)
  "Sends an email to the admins"
  (let ((project-name (envy-ningle:get-config :project-name))
        (admins (envy-ningle:get-config :email-admins)))
    (send-mail (format nil "[~A]: ~A" project-name subject) message admins)))

We don’t yet know the shape of our send-mail function, we only know that we will use it, and in fact, thinking about how we will get and pass information into it, will help us see how its interface might be. When we mail our admins, we already know who we are emailing (our admins) and we also know who the email will be from (our application) so in reality we need a subject and message as parameters.

Although we know who is being mailed by who, we might want to make clear what they are being emailed by, our admins probably get a lot of mail, so I have made a choice that the email title will be [NTP]: <project name> in this way it’s clear that the service has mailed them.

We create a let block that grabs the project name from the settings. We also get the list of project admins from the settings in this block too and we simply call send-mail with a subject (our format expression), a message and a list of recipients (our admins), and with that done, we now know our send-mail function has parameters for a subject, a message, and a list of recipients, we might want to change the default sender, so we can add a &key parameter for this, but we will default it to putting the email from the settings.

(defun send-mail (subject content to &key (from (envy-ningle:get-config :email-default-from)))
  "Sends arbitrary email"
  (let ((email-backend (envy-ningle:get-config :email-backend)))
    (case email-backend
        (:dummy
            (progn
                (format t "from: ~A~%" from)
                (format t "to: ~A~%" to)
                (format t "subject: ~A~%" subject)
                (format t "content: ~A~%" content)))

        (:smtp
            (cl-smtp:send-email
                (envy-ningle:get-config :email-smtp-host)
                from
                to
                subject
                message
                :port (or (envy-ningle:get-config :email-port) 587)
                :ssl (or (envy-ningle:get-config :email-ssl) :starttls)
                :authentication (envy-ningle:get-config :email-auth)))

        (:sendgrid
            (sendgrid:send-email
                :to to
                :from from
                :subject subject
                :content message
                :api-key (envy-ningle:get-config :sendgrid-api-key)))

        (otherwise
            (error "Unknown email backend: ~A" email-backend)))))

As you can see, our parameters are quite simply what our mail-admins specified, the only tricky thing is the from parameter, which simply pulls a default value of :email-default-from from our settings, so in most cases the send-mail function will do exactly the right thing, but it’s possible to override the from, if needed.

The rest of this function is really quite simple, it’s just a case that checks the :email-backend setting we defined in our settings and dispatches to another package for the actual logic. The :dummy backend simply prints the email information to the terminal, the :smtp backend delegates to the cl-smtp package, the :sendgrid backend delegates to the :cl-sendgrid package and, finally, if the email backend wasn’t recognised and error is signalled.

That really is all we need to write for our email package, with it complete we can look at integrating it into our project as a whole and into the auth package we built last time.

Auth Package

Since we now created a package we will be relying on, the first thing we need to do is to ensure we include it in the dependencies of this project.

project.asd

:depends-on (:cl-dotenv
             :clack
             :djula
             :cl-forms
             :cl-forms.djula
             :cl-forms.ningle
             :envy-ningle
             :mito
             :ningle
             :local-time
             :cu-sith
             :ningle-email) ; add this

models.lisp

We will make a slight change to the models, but this is only to support the expiration time that we defined in our settings. In our ningle-auth/src/models.lisp file we will make two changes.

(defmethod initialize-instance :after ((token token) &rest initargs &key &allow-other-keys)
  (unless (slot-boundp token 'salt)
    (setf (token-salt token) (ironclad:make-random-salt 16)))

  (unless (slot-boundp token 'expires-at)
    ; change the below line from 3600 to the :token-expiration setting
    (setf (token-expires-at token) (+ (get-universal-time) (envy-ningle:get-config :token-expiration))))) 

And here.

; Again change the token from 3600 to the value stored in the setting
(defmethod generate-token ((user user) purpose &key (expires-in (envy-ningle:get-config :token-expiration)))
    (unless (member purpose +token-purposes+ :test #'string=)
      (error "Invalid token purpose: ~A. Allowed: ~A" purpose +token-purposes+))

    (let* ((salt (ironclad:make-random-salt 16))
           (expires-at (truncate (+ (get-universal-time) expires-in)))
           (base-string (format nil "~A~A~A" (username user) expires-at salt))
           (hash (ironclad:byte-array-to-hex-string (ironclad:digest-sequence :sha256 (babel:string-to-octets base-string)))))
        (create-dao 'token :user user :purpose purpose :token hash :salt salt :expires-at expires-at)))

main.lisp

Now, since we deal a lot with token generations that are actually urls in our application, I decided we should simplify this a little by creating some utlity functions that generate these, as we do call them over and over again under different circumstances.

So, the first thing to add into our main.lisp are these utility functions:

(defun build-url-root (&key (path ""))
  (format nil "~A://~A:~A~A"
    (lack/request:request-uri-scheme ningle:*request*)
    (lack/request:request-server-name ningle:*request*)
    (lack/request:request-server-port ningle:*request*)
    path))

(defun build-activation-link (user token)
  (let ((host (build-url-root :path (envy-ningle:get-config :auth-mount-path))))
    (format nil "~A/verify?user=~A&token=~A~%" host (ningle-auth/models:username user) (ningle-auth/models:token-value token))))

(defun build-reset-link (user token)
  (let ((host (build-url-root :path (envy-ningle:get-config :auth-mount-path))))
    (format nil "~A/reset/process?user=~A&token=~A~%" host (ningle-auth/models:username user) (ningle-auth/models:token-value token))))

In other frameworks there would ideally be a way to build an absolute url from the request object, but ningle is pretty lightweight, so we will make do with these.

We start with the build-url-root, which will build a url from the request object, using the scheme, server name, port, and any path parts. At the moment I don’t do any checking for the port to be 80 or 443, maybe that’s something for later! The intention is this will build up the basic part of our url, and the two functions build-activation-link and build-reset-link will use it to, well, build the links.

Each function will return a string that represents the link it is concerned with building, it doesn’t do anything we weren’t doing before, but instead of building the link in each place it is used, we have one place where the links are built, so that if we need to change it, we easily can. Each function only needs to take a user, and a token, it it then looks up the username and token-value of the objects and we’re pretty much done!

We don’t have too much we need to change here, only three areas or so, let’s start in our /register controller.

We previously had just a username and token and we used format to display this in the terminal, however if we want to do things right and send emails, we need to make some adjustments.

(let* ((user (mito:create-dao 'ningle-auth/models:user :email email :username username :password password))
       (token (ningle-auth/models:generate-token user ningle-auth/models:+email-verification+))
       (link (build-activation-link user token))
       (subject (format nil "Ningle Tutorial Project registration for ~A" user))
       (template "ningle-auth/email/register.txt")
       (content (djula:render-template* template nil :user user :link link)))
(ningle-email:send-mail subject content email)

In addition to the user and token, we need to generate the link we will send using the build-activation-link function we just wrote above, also, since we know our email needs a subject, we create that now in our let* block. Next we will have our template, although we haven’t yet created these, we will next, and our email content will use djula and this template location to render the content and store it ready for us to user in our send-mail invocation. Since this is happening in our /register controller, we already have an email address to send to, so we don’t need to create a new variable for that, it is already in scope.

The next place to make a change is in our /reset controller, there are two areas here where we would change things, thankfully the changes are exactly the same.

((and user token)
    (mito:delete-dao token)
    (let* ((token (ningle-auth/models:generate-token user ningle-auth/models:+password-reset+))
           (link (build-reset-link user token))
           (subject (format nil "Ningle Tutorial Project password reset for ~A" user))
           (template "ningle-auth/email/reset.txt")
           (content (djula:render-template* template nil :user user :link link)))
        (ningle-email:send-mail subject content email)
        (ingle:redirect "/")))

Here, in the case where we have a user and a token object, we perform basically the same set of steps we did before, getting the token, link, subject, template, and content and passing that on into the send-mail function. It’s worth noting that the template we are loading is different (although, again, we haven’t yet written the templates).

(user
    (let* ((token (ningle-auth/models:generate-token user ningle-auth/models:+password-reset+))
           (link (build-reset-link user token))
           (subject (format nil "Ningle Tutorial Project password reset for ~A" user))
           (template "ningle-auth/email/reset.txt")
           (content (djula:render-template* template nil :user user :link link)))
        (ningle-email:send-mail subject content email)
        (ingle:redirect "/")))

This code is identical as the above, we can probably consolidate these down in a refactor later, but we will keep focused on getting our email working first.

The final place to change things is in the /verify controller.

((and token (ningle-auth/models:is-expired-p token))
  (mito:delete-dao token)
  (let* ((new-token (ningle-auth/models:generate-token user ningle-auth/models:+email-verification+))
          (link (build-activation-link user new-token))
          (subject (format nil "Ningle Tutorial Project registration for ~A" user))
          (template "ningle-auth/email/register.txt")
          (content (djula:render-template* template nil :user user :link link)))
      (ningle-email:send-mail subject content (ningle-auth/models:email user))
  (djula:render-template* "ningle-auth/verify.html" nil :title "Verify" :token-reissued t)))

In this case however a new token is being issued, as it has expired at this point in the application lifecycle and needs to be reissued. There’s nothing really new here we haven’t seen before in our previous examples.

The only other thing I have changed is to remove the format line from inside the (not token) and t branches of the cond here, as they’re no longer needed.

And with those changes, we can move onto our templates!

Templates

Since we will be sending email, and our controllers specify that we will be rendering templates we need to set these up, as discussed in Part 9 (Authentication System) we looked into how templates override each other, so we need to ensure our email templates are in the correct place to that our main application can override them, if needed.

Remember: These template must be placed in ningle-auth/src/templates/ningle-auth/email as it’s this directory structure that allows us to override in broader projects!

base.html

Our base.html is going to be really very simple, it provides a content block that other templates can inject content into, but it also serves another purpose, a file we can override in another project and add headers/footers etc without having to override every template.

This is why its content is so small, we’d almost never directly use this, but because it’s a base template that others extend, we can use it!

{% block content %}{% endblock %}

register.html

Our register template will extend the base and provide the information a user will need to continue setting up their account. The template is simple enough (why complicate it?), but you must pay attention to the safe filter that is being used to correctly encode the url.

{% extends "ningle-auth/email/base.txt" %}
{% block content %}
Hello, {{ user.username }}!

Thanks for registering, for security reasons you must verify your account by clicking on this link:
{{ link|safe }}
This link will expire in 1 hour.

If this was not you, you can ignore this email, as an account will not be activated without clicking on the above link.
{% endblock %}

reset.html

The reset template is very similar to the register template, just with some slightly different wording, but just mind and use the safe template filter as before!

{% extends "ningle-auth/email/base.txt" %}
{% block content %}
Hello, {{ user.username }}!

We have received a password change request for your account, to do so, click this url:
{{ link|safe }}
This link will expire in 1 hour.

If this was not you, you can ignore this email, as your password will not be changed without clicking on the above link.
{% endblock %}

Now that we have our controllers wired up to send emails that are rendered from templates, we are ready to finally connect everything up!

Main Package (Part 2)

As we mentioned in the previous section, our ningle-auth email base template can be overridde, and in fact that’s exactly what we are going to do. We need to create the following file in our ningle-tutorial-project project: src/templates/ningle-auth/email/base.txt and we are going to add a footer!

{% block content %}{% endblock %}

Ningle Tutorial Project

It’s not a lot of code, and to be fair, that was the point, we can quickly and easily override the ningle-auth base template and add in a footer (or a header, or both, if you like), into the email base template and everything just works as we need it to.

Conclusion

Mercifully this tutorial is a lot shorter than the last time, and good news! This means we now have everything we need to begin working on a microblog! Authentication and email are very important, but they highlight a trade off in micro frameworks and macro frameworks, in micro frameworks we have to do a lot of the work either connecting up third party packages, or writing our own, but we are done now, and we can focus on what we set out to do.

We will begin next time by looking at users, and how to display information about their followers etc.

Thank you for following this tutorial series, I hope you are finding it as interesting/helpful to read as I am finding it interesting/helpful to write.

Learning Outcomes

Level	Learning Outcome
Remember	Identify the configuration options required for setting up different email backends (dummy, smtp, sendgrid) in a Ningle application. Recall the purpose of the .env file and its role in storing sensitive credentials.
Understand	Explain the difference between dummy, SMTP, and SendGrid email backends and when each might be used. Describe how template overrides in ningle-auth allow flexibility for customizing email content.
Apply	Configure a Ningle project to use different email backends by modifying defconfig settings. Use Djula templates to generate dynamic email content (e.g., activation and reset links).
Analyze	Compare the advantages and trade-offs of using a microframework (Ningle) versus a macro framework for handling email workflows. Examine how token expiration settings affect authentication workflows and security.
Evaluate	Assess the security implications of storing and handling email credentials in .env files. Justify the choice of email backend for different project stages (development, testing, production).
Create	Design and implement a custom email notification (e.g., welcome email, alert system) using the ningle-email package. Extend the project by building reusable utility functions to streamline email workflows beyond registration and password resets.

Github

• The link for this tutorials code is available here.
• The link for the auth app code is available here.
• The link for the email app code is available here.

Resources

• cl-smtp
• cl-sendgrid

Common Lisp HyperSpec

• and
• case
• cond
• defmethod
• defpackage
• defun
• format
• get-universal-time
• in-package
• initialize-instance
• let / let*
• member
• not
• or
• progn
• setf
• slot-boundp
• string=
• truncate
• unless

Reader Macros

• backquote (`)
• comma (,) and ,@ in backquote
• pathname literal (#p)
• quote (‘)