Ningle Tutorial 7: Envy Configuration Switching
by NMunro
Contents
- Part 1 (Hello World)
- Part 2 (Basic Templates)
- Part 3 (Introduction to middleware and Static File management)
- Part 4 (Forms)
- Part 5 (Environmental Variables)
- Part 6 (Database Connections)
- Part 7 (Envy Configuation Switching)
Introduction
Welcome back, in this tutorial we will look at how to simplify the complexities introduced last time. We had three different versions of our application, depending on which SQL database we wanted to use, this is hardly an ideal situation, we might want to run SQLite on one environment and PostgreSQL on another, it does not make sense to have to edit code to change such things, we should have code that is generalised and some configuration (like environmental variables) can provide the system with the connection information.
We want to separate our application configuration from our application logic, in software development we might build an application and have different environments in which is can be deployed, and different cloud providers/environments might have different capabilities, for example some providers provide PostgreSQL and others MySQL. As application designers we do not want to concern ourselves with having to patch our application based whatever an environment has provided, it would be better if we had a means by which we could read in how we connect to our databases and defer to that.
This type of separation is very common, in fact it is this separation that ningle
itself if for! Just as now we are creating a means to connect to a number of different databases based on config, ningle
allows us to run on a number of different webservers, without ningle
we would have to write code directly in the way a web server expects, ningle
allows us to write more generalised code.
Enter envy, a package that allows us to define different application configurations. Envy will allow us to set up different configurations and switch them based on an environmental variable, just like we wanted. Using this allows us to remove all of our database specific connection code and read it from a configuration, the configuration of which can be changed, the application can be restarted and everything should just work.
We have a slight complication in that we have our migration code, so we will need a way to also extract the active settings, but I wrote a package to assist in this envy-ningle, we will use both these packages to clean up our code.
Installing Packages
To begin with we will need to ensure we have installed and added the packages we need to our project asd file, there are two that we will be installing:
Note: My package (envy-ningle) is NOT in quicklisp, so you will need to clone it using git
into your local-packages
directory.
Once you have acquired the packages, as normal you will need to add them in the :depends-on
section.
:depends-on (:clack
:cl-dotenv
:djula
:cl-forms
:cl-forms.djula
:cl-forms.ningle
:envy ;; Add this
:envy-ningle ;; Add this
:ingle
:mito
:mito-auth
:ningle)
Writing Application Configs
config.lisp
We must write our application configs somewhere, so we will do that in src/config.lisp
, as always when adding a new file to our application we must ensure it is added to the asd
file, in the :components section
. This will ensure the file will be loaded and compiled when the system is loaded.
:components ((:module "src"
:components
((:file "config") ;; Add this
(:file "models")
(:file "migrations")
(:file "forms")
(:file "main"))))
So we should write this file now!
As normal we set up a package, declare what packages we will use (:use :cl :envy)
and set the active package to this one. There’s some conventions we must follow using this that may seem unimportant at first, but actually are, specifically the |sqlite|
, |mysql|
, and |postgresql|
they must include the |
surrounding the name, (although the name doesn’t have to be sqlite
, mysql
, or postgresql
, those are just what I used based on the last tutorial).
(defpackage ningle-tutorial-project/config
(:use :cl :envy))
(in-package ningle-tutorial-project/config)
We will start by loading the .env
file using the dotenv
package, we will remove it from our main.lisp
file a little later, but we need to include it here, next we will inform envy of what the name of the environmental variable is that will be used to switch config, in this case APP_ENV
.
(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))
(setf (config-env-var) "APP_ENV")
This means that in your .env
file you should add the following:
Note: I am using the sqlite config here, but you can use any of the configs below.
APP_ENV=sqlite
We can define a “common” set of configs using the :common
label, this differs from the other labels that use the |
to surround them. The :common
config isn’t one that will actually be used, it just provides a place to store the, well, common, configuration. While we don’t yet necessarily have any shared config at this point, it is important to understand how to achieve it. In this example we set an application-root
that all configs will share.
In envy
we use the defconfig
macro
to define a config. Configs take a name, and a list
of items. There is a shared configuration which is called :common
, that any number of other custom configs that inherit from, their names are arbitary, but must be surrounded by |
, for example |staging|
, or |production|
.
This is the :common
we will use in this tutorial:
(defconfig :common
`(:application-root ,(asdf:component-pathname (asdf:find-system :ningle-tutorial-project))))
We can now define our actual configs, our “development” config will be sqlite
, which will define our database connection, however, because mito
defines database connections as middleware, we can define the middleware section in our config. Each config will have a different middleware section. Unfortunately there will be some repetition with the (:session)
and (:static ...)
middleware sections.
(defconfig |sqlite|
`(:debug T
:middleware ((:session)
(:mito (:sqlite3 :database-name ,(uiop:getenv "SQLITE_DB_NAME")))
(:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))
For our MySQL
config we have this:
(defconfig |mysql|
`(:middleware ((:session)
(:mito (:mysql
:database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "MYSQL_DB_NAME")))
:username ,(uiop:getenv "MYSQL_USER")
:password ,(uiop:getenv "MYSQL_PASSWORD")
:host ,(uiop:getenv "MYSQL_ADDRESS")
:port ,(parse-integer (uiop:getenv "MYSQL_PORT"))))
(:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))
And finally our PostgreSQL
:
(defconfig |postgresql|
`(:middleware ((:session)
(:mito (:postgres
:database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "POSTGRES_DB_NAME")))
:username ,(uiop:getenv "POSTGRES_USER")
:password ,(uiop:getenv "POSTGRES_PASSWORD")
:host ,(uiop:getenv "POSTGRES_ADDRESS")
:port ,(parse-integer (uiop:getenv "POSTGRES_PORT"))))
(:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))
None of this should be especially new, this middleware section should be familiar from last time, simply wrapped up in the envy:defconfig
macro
.
Here is the file in its entirety:
(defpackage ningle-tutorial-project/config
(:use :cl :envy))
(in-package ningle-tutorial-project/config)
(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))
(setf (config-env-var) "APP_ENV")
(defconfig :common
`(:application-root ,(asdf:component-pathname (asdf:find-system :ningle-tutorial-project))))
(defconfig |sqlite|
`(:debug T
:middleware ((:session)
(:mito (:sqlite3 :database-name ,(uiop:getenv "SQLITE_DB_NAME")))
(:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))
(defconfig |mysql|
`(:middleware ((:session)
(:mito (:mysql
:database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "MYSQL_DB_NAME")))
:username ,(uiop:getenv "MYSQL_USER")
:password ,(uiop:getenv "MYSQL_PASSWORD")
:host ,(uiop:getenv "MYSQL_ADDRESS")
:port ,(parse-integer (uiop:getenv "MYSQL_PORT"))))
(:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))
(defconfig |postgresql|
`(:middleware ((:session)
(:mito (:postgres
:database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "POSTGRES_DB_NAME")))
:username ,(uiop:getenv "POSTGRES_USER")
:password ,(uiop:getenv "POSTGRES_PASSWORD")
:host ,(uiop:getenv "POSTGRES_ADDRESS")
:port ,(parse-integer (uiop:getenv "POSTGRES_PORT"))))
(:static :root ,(asdf:system-relative-pathname :ningle-tutorial-project "src/static/") :path "/public/"))))
main.lisp
As mentioned, we need to do some cleanup in our main.lisp
, the first is to remove the dotenv
code that has been moved into the config.lisp
file, but we will also need to take advantage of the envy-ningle
package to load the active configuration into the lack
builder code.
To remove the dotenv
code:
(defvar *app* (make-instance 'ningle:app))
;; remove the line below
(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))
(setf (ningle:route *app* "/")
Now to edit the start
function, it should look like the following:
(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
(djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
(djula:set-static-url "/public/")
(clack:clackup
(lack.builder:builder (envy-ningle:build-middleware :ningle-tutorial-project/config *app*))
:server server
:address address
:port port))
As you can see, all of the previous middleware code that had to be changed if you wanted to switch databases, is now a single line, because envy
loads the config based on the environmental variable, the envy-ningle:build-middleware
function will then read that config and insert the middleware into the application. I hope you will agree that it is much simpler and makes your application much easier to manage.
If you are not yet convinced and you think you’re fine to keep things as they were, consider that we have duplicated our database connection logic in migrations.lisp
and if we decide we do need to change our connection we have to do it in two places, possibly more if we have many models and want to break the code up.
migrations.lisp
We will use the same structure for how we loaded configuration in our main.lisp
file, the way we use envy-ningle
is different, previously we called the build-middleware
function, which is designed to place the config middleware into the lack
builder, here we want to get only the database connection information and thus we will use the extract-mito-config
(admittedly not the best name), to get the database connection information and use it in mito:connect-toplevel
.
(defun migrate ()
"Explicitly apply migrations when called."
(format t "Applying migrations...~%")
(multiple-value-bind (backend args) (envy-ningle:extract-mito-config :ningle-tutorial-project/config)
(unless backend
(error "No :mito middleware config found in ENVY config."))
(apply #'mito:connect-toplevel backend args)
(mito:ensure-table-exists 'ningle-tutorial-project/models:user)
(mito:migrate-table 'ningle-tutorial-project/models:user)
(mito:disconnect-toplevel)
(format t "Migrations complete.~%")))
As you can see here, we use multiple-value-bind
to get the “backend” (which will be one of the three supported SQL databases), and then the arguments that backend expects. If there isn’t a backend, an error is thrown, if there is, we call apply
on the mito:connect-toplevel
with the “backend” and “args” values.
Testing The Config Switching
Now that all the code has been written, we will want to test it all works. The simplest way to do this is while the value of “APP_ENV” in your .env
file is “sqlite”, run the migrations.
(ningle-tutorial-project/migrations:migrate)
You should see the sqlite specific output, if that works, we can then change the value of “APP_ENV” to be “mysql” or “postgresql”, whichever you have available to you, and we can run the migrations again.
(ningle-tutorial-project/migrations:migrate)
This time we would expect to see different sql output, and if you do, you can confirm that the configurating switching is working as expected.
Conclusion
I hope you found that helpful, and that you agree that it’s better to separate our configuration from our actual application code.
To recap, after working your way though this tutorial you should be able to:
- Explain what configuration switching is
- Explain why configuration is important
- Discuss the reasons for separating configuration from application code
- Implement your own configurations for applications you write
Github
- The link for this tutorials code is available here.