nmunro.github.io

Common Lisp and other programming related things from NMunro

View on GitHub
31 May 2025

Ningle Tutorial 7: Envy Configuration Switching

by NMunro

Contents

Introduction

Welcome back, in this tutorial we will look at how to simplify the complexities introduced last time. We had three different versions of our application, depending on which SQL database we wanted to use, this is hardly an ideal situation, we might want to run SQLite on one environment and PostgreSQL on another, it does not make sense to have to edit code to change such things, we should have code that is generalised and some configuration (like environmental variables) can provide the system with the connection information.

We want to separate our application configuration from our application logic, in software development we might build an application and have different environments in which is can be deployed, and different cloud providers/environments might have different capabilities, for example some providers provide PostgreSQL and others MySQL. As application designers we do not want to concern ourselves with having to patch our application based whatever an environment has provided, it would be better if we had a means by which we could read in how we connect to our databases and defer to that.

This type of separation is very common, in fact it is this separation that ningle itself if for! Just as now we are creating a means to connect to a number of different databases based on config, ningle allows us to run on a number of different webservers, without ningle we would have to write code directly in the way a web server expects, ningle allows us to write more generalised code.

Enter envy, a package that allows us to define different application configurations. Envy will allow us to set up different configurations and switch them based on an environmental variable, just like we wanted. Using this allows us to remove all of our database specific connection code and read it from a configuration, the configuration of which can be changed, the application can be restarted and everything should just work.

We have a slight complication in that we have our migration code, so we will need a way to also extract the active settings, but I wrote a package to assist in this envy-ningle, we will use both these packages to clean up our code.

Installing Packages

To begin with we will need to ensure we have installed and added the packages we need to our project asd file, there are two that we will be installing:

Note: My package (envy-ningle) is NOT in quicklisp, so you will need to clone it using git into your local-packages directory.

Once you have acquired the packages, as normal you will need to add them in the :depends-on section.

:depends-on (:clack
             :cl-dotenv
             :djula
             :cl-forms
             :cl-forms.djula
             :cl-forms.ningle
             :envy         ;; Add this
             :envy-ningle  ;; Add this
             :ingle
             :mito
             :mito-auth
             :ningle)

Writing Application Configs

config.lisp

We must write our application configs somewhere, so we will do that in src/config.lisp, as always when adding a new file to our application we must ensure it is added to the asd file, in the :components section. This will ensure the file will be loaded and compiled when the system is loaded.

:components ((:module "src"
              :components
              ((:file "config")  ;; Add this
               (:file "models")
               (:file "migrations")
               (:file "forms")
               (:file "main"))))

So we should write this file now!

As normal we set up a package, declare what packages we will use (:use :cl :envy) and set the active package to this one. There’s some conventions we must follow using this that may seem unimportant at first, but actually are, specifically the |sqlite|, |mysql|, and |postgresql| they must include the | surrounding the name, (although the name doesn’t have to be sqlite, mysql, or postgresql, those are just what I used based on the last tutorial).

(defpackage ningle-tutorial-project/config
  (:use :cl :envy))
(in-package ningle-tutorial-project/config)

We will start by loading the .env file using the dotenv package, we will remove it from our main.lisp file a little later, but we need to include it here, next we will inform envy of what the name of the environmental variable is that will be used to switch config, in this case APP_ENV.

(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))
(setf (config-env-var) "APP_ENV")

This means that in your .env file you should add the following:

Note: I am using the sqlite config here, but you can use any of the configs below.

APP_ENV=sqlite

We can define a “common” set of configs using the :common label, this differs from the other labels that use the | to surround them. The :common config isn’t one that will actually be used, it just provides a place to store the, well, common, configuration. While we don’t yet necessarily have any shared config at this point, it is important to understand how to achieve it. In this example we set an application-root that all configs will share.

In envy we use the defconfig macro to define a config. Configs take a name, and a list of items. There is a shared configuration which is called :common, that any number of other custom configs that inherit from, their names are arbitary, but must be surrounded by |, for example |staging|, or |production|.

This is the :common we will use in this tutorial:

(defconfig :common
  `(:application-root ,(asdf:component-pathname (asdf:find-system :ningle-tutorial-project))))

We can now define our actual configs, our “development” config will be sqlite, which will define our database connection, however, because mito defines database connections as middleware, we can define the middleware section in our config. Each config will have a different middleware section. Unfortunately there will be some repetition with the (:session) and (:static ...) middleware sections.

(defconfig |sqlite|
  `(:debug T
    :middleware ((:session)
                 (:mito (:sqlite3 :database-name ,(uiop:getenv "SQLITE_DB_NAME")))
                 (:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))

For our MySQL config we have this:

(defconfig |mysql|
  `(:middleware ((:session)
                 (:mito (:mysql
                         :database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "MYSQL_DB_NAME")))
                         :username ,(uiop:getenv "MYSQL_USER")
                         :password ,(uiop:getenv "MYSQL_PASSWORD")
                         :host ,(uiop:getenv "MYSQL_ADDRESS")
                         :port ,(parse-integer (uiop:getenv "MYSQL_PORT"))))
                 (:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))

And finally our PostgreSQL:

(defconfig |postgresql|
  `(:middleware ((:session)
                 (:mito (:postgres
                         :database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "POSTGRES_DB_NAME")))
                         :username ,(uiop:getenv "POSTGRES_USER")
                         :password ,(uiop:getenv "POSTGRES_PASSWORD")
                         :host ,(uiop:getenv "POSTGRES_ADDRESS")
                         :port ,(parse-integer (uiop:getenv "POSTGRES_PORT"))))
                 (:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))

None of this should be especially new, this middleware section should be familiar from last time, simply wrapped up in the envy:defconfig macro.

Here is the file in its entirety:

(defpackage ningle-tutorial-project/config
  (:use :cl :envy))
(in-package ningle-tutorial-project/config)

(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))
(setf (config-env-var) "APP_ENV")

(defconfig :common
  `(:application-root ,(asdf:component-pathname (asdf:find-system :ningle-tutorial-project))))

(defconfig |sqlite|
  `(:debug T
    :middleware ((:session)
                 (:mito (:sqlite3 :database-name ,(uiop:getenv "SQLITE_DB_NAME")))
                 (:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))

(defconfig |mysql|
  `(:middleware ((:session)
                 (:mito (:mysql
                         :database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "MYSQL_DB_NAME")))
                         :username ,(uiop:getenv "MYSQL_USER")
                         :password ,(uiop:getenv "MYSQL_PASSWORD")
                         :host ,(uiop:getenv "MYSQL_ADDRESS")
                         :port ,(parse-integer (uiop:getenv "MYSQL_PORT"))))
                 (:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))

(defconfig |postgresql|
  `(:middleware ((:session)
                 (:mito (:postgres
                         :database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "POSTGRES_DB_NAME")))
                         :username ,(uiop:getenv "POSTGRES_USER")
                         :password ,(uiop:getenv "POSTGRES_PASSWORD")
                         :host ,(uiop:getenv "POSTGRES_ADDRESS")
                         :port ,(parse-integer (uiop:getenv "POSTGRES_PORT"))))
                 (:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))

main.lisp

As mentioned, we need to do some cleanup in our main.lisp, the first is to remove the dotenv code that has been moved into the config.lisp file, but we will also need to take advantage of the envy-ningle package to load the active configuration into the lack builder code.

To remove the dotenv code:

(defvar *app* (make-instance 'ningle:app))

;; remove the line below
(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))

(setf (ningle:route *app* "/")

Now to edit the start function, it should look like the following:

(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
    (djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
    (djula:set-static-url "/public/")
    (clack:clackup
     (lack.builder:builder (envy-ningle:build-middleware :ningle-tutorial-project/config *app*))
     :server server
     :address address
     :port port))

As you can see, all of the previous middleware code that had to be changed if you wanted to switch databases, is now a single line, because envy loads the config based on the environmental variable, the envy-ningle:build-middleware function will then read that config and insert the middleware into the application. I hope you will agree that it is much simpler and makes your application much easier to manage.

If you are not yet convinced and you think you’re fine to keep things as they were, consider that we have duplicated our database connection logic in migrations.lisp and if we decide we do need to change our connection we have to do it in two places, possibly more if we have many models and want to break the code up.

migrations.lisp

We will use the same structure for how we loaded configuration in our main.lisp file, the way we use envy-ningle is different, previously we called the build-middleware function, which is designed to place the config middleware into the lack builder, here we want to get only the database connection information and thus we will use the extract-mito-config (admittedly not the best name), to get the database connection information and use it in mito:connect-toplevel.

(defun migrate ()
  "Explicitly apply migrations when called."
  (format t "Applying migrations...~%")
  (multiple-value-bind (backend args) (envy-ningle:extract-mito-config :ningle-tutorial-project/config)
    (unless backend
      (error "No :mito middleware config found in ENVY config."))
    (apply #'mito:connect-toplevel backend args)
    (mito:ensure-table-exists 'ningle-tutorial-project/models:user)
    (mito:migrate-table 'ningle-tutorial-project/models:user)
    (mito:disconnect-toplevel)
    (format t "Migrations complete.~%")))

As you can see here, we use multiple-value-bind to get the “backend” (which will be one of the three supported SQL databases), and then the arguments that backend expects. If there isn’t a backend, an error is thrown, if there is, we call apply on the mito:connect-toplevel with the “backend” and “args” values.

Testing The Config Switching

Now that all the code has been written, we will want to test it all works. The simplest way to do this is while the value of “APP_ENV” in your .env file is “sqlite”, run the migrations.

(ningle-tutorial-project/migrations:migrate)

You should see the sqlite specific output, if that works, we can then change the value of “APP_ENV” to be “mysql” or “postgresql”, whichever you have available to you, and we can run the migrations again.

(ningle-tutorial-project/migrations:migrate)

This time we would expect to see different sql output, and if you do, you can confirm that the configurating switching is working as expected.

Conclusion

I hope you found that helpful, and that you agree that it’s better to separate our configuration from our actual application code.

To recap, after working your way though this tutorial you should be able to:

Github

Resources

tags: CommonLisp - Lisp - tutorial - YouTube - web - dev