Migrations help create your database and track how it evolves overtime. Here, we use GoMigrate to achieve this. Some added complexity is added to enable easy extendability and generate better logs throughout your development process.

Migrations go under pkg/db/migrations/<myNewMigration>.go. Its implemention uses Go's init() function, which means they're added to the list in alphabetical order. They migrate in that order (top to bottom) and rollback in the reverse order (bottom up). For this, it is best to maintain the naming convention of YYYYMMDD[00-99]_migration_description.

Here's a sample migration to get you started:

func init() {
	m := &gormigrate.Migration{}

	m.ID = "2022081801_create_cats_table"

	m.Migrate = func(db *gorm.DB) error {
		type Cat struct {
			models.ModelBase
			Name string `gorm:"size:255"`
			Type string `gorm:"size:255"`
		}

		return AutoMigrateAndLog(db, &Cat{}, m.ID)
	}

	m.Rollback = func(db *gorm.DB) error {
		if err := db.Migrator().DropTable("cats"); err != nil {
			logFail(m.ID, err, true)
		}
		logSuccess(m.ID, true)
		return nil
	}

	AddMigration(m)
}

The variable m holds the migration details and is added to the list of migrations at the end. m.ID is the identifier used by gomigrate to keep track of the migrations that already ran. So, make sure to change that for every migration.

Every migration has 2 methods to be implemented, the Migrate() and Rollback() method as described above. Make sure you use the logSuccess, logFail and AutoMigrateAndLog() functions to print the migrations that ran. This will come in very handy for remote deployments.

It's recommended to declare your models within each migration (separately from the models package) to keep track of the database schema change through time. You can add or delete columns, rename columns, and execute raw SQL in migrations.

A general good practice would be to flatten your migrations once your application achieves version 1, leaving only neat table creation in each migration.

Seeds are very similar to migrations, but seeds do not implement the Rollback function.

Just like migrations, seeds are applied once and tracked using their unique identifier ID by GoMigrate.

Seeds are part of the whole package which allows you to access models, clients and other components directly to configure the application, and perhaps provide dummy data to help with development.

Here's a seed skeleton to get you started. Copy the following structure into a new file under seeds and change the s.ID property.

func init() {

	var s = &gormigrate.Migration{}
	s.ID = "2022081801_new_seed"

	s.Migrate = func(db *gorm.DB) error {

		logSuccess(s.ID)
		return nil
	}

	AddSeed(s)
}

And here's a sample seed to give an idea of how you can utilise seeds.

func init() {

	var s = &gormigrate.Migration{}
	s.ID = "2022081801_seed_cats_data"

	s.Migrate = func(db *gorm.DB) error {

		var err error
    var cats []*models.Cat

		cats = append(cats, &models.Cat{
			Name: "Kitty",
			Type: "Persian",
		})

		cats = append(cats, &models.Cat{
			Name: "Tom",
			Type: "Siamese",
		})

    for _, cat := range cats {
      err = cat.Save()
      if err != nil {
        logFail(s.ID, err)
        return err
      }
    }

		logSuccess(s.ID)
		return nil
	}

	AddSeed(s)
}

Models can sometimes be a complex aspect of any application. In this section, you'll find a rundown on how you can compose your models or database entities.

Model Structure

The first thing is to create a struct that matches your database schema. Almost all models should embed the ModelBase struct that provides the ID, CreatedAt and UpdatedAt properties. Exceptions may include a many-to-many table where you only need to store 2 identifiers. To learn more about model declarations, you can refer to Gorm's official comprehensive documentation.

Here's a Cat model that should correspond to a Cats table that contains 5 properties i.e. ID, CreatedAt, UpdatedAt, Name and Type in a database.

type Cat struct {
	ModelBase
	Name string `gorm:"size:255"`
	Type string `gorm:"size:255"`
}

Notice how every property contains a gorm decoration to specify things like field size, uniqueness or foreign keys etc. For more details, please refer to Gorm's documentation.

Your model may sometimes contain properties that do not correspond to a database column. To do that, you simply need to use the gorm:"-" decoration.

Note: Given that this package is designed to work with multiple database servers like MySQL or Postgres, some data types may be available in some servers and not others. It's worth testing your application with different servers from time to time to accomodate easy switching of database server, unless your use case relies on a specific data type - in which case you're making a calculated decision to lock your application to that server.

Common Basic Functionality

Now that you have a structure that corresponds to a table in your database, some common functionality is in order. Generally, one would at least expect the basic CRUD functionality. Here's a basic CRUD implementation that is required for any model:

• FindAll(), for retrieving all records in the table

func (model *Cat) FindAll() (models []*Cat, err error) {
	result := db.Model(model).Find(&models)
	return models, result.Error
}

• FindMany(), for retrieving many items given an array of IDs

func (model *Cat) FindMany(ids []string) (models []*Cat, err error) {
	result := db.Model(model).Find(&models, ids)
	return models, result.Error
}

• Find(), for retrieving a single item with a given ID

func (model *Cat) Find(id string) (m *Cat, err error) {
	result := db.Model(model).Where("ID=?", id).First(&m)
	return m, result.Error
}

• Save(), for creating a new record in the database and assigning a new ID to it

func (model *Cat) Save() error {
	return db.Model(model).Create(&model).Error
}

• 'Update(), for updating a record in the database given an existing ID

func (model *Cat) Update() error {
	return db.Model(model).Updates(&model).Error
}

• Delete(), for deleting a record in the database given an existing ID

func (model *Cat) Delete(id string) error {
	return db.Model(model).Where("ID=?", id).Delete(&model).Error
}

All of the above functions will return an error if they cannot perform what they're supposed to. That's useful to inform users if the data they're looking for exists or is stored. For detailed utilisation of these functions, check out the handlers folder.

These functions are not abstracted to allow granular control over each model, as each individual model can quickly morph into something very large with child elements, preload functions and pagination.

Model Accessibility

Given the basic functionality defined in the previous section, we've created the ability to do something like the following:

...
catModel := &Cat{}

myCat, err := catModel.Find(catID)
if err != nil {
	fmt.Println("couldn't find cat with ID", catID)
}
...

The problem with the code above is that you will need to instantiate a new struct catModel from &Cat{} in order to have a pointer receiver that can call the Find() function. You can avoid that by using the following common getter structure for all models, right at the top of the model before its declaration to maintain consistency.

var cat *Cat = &Cat{}

func CatModel() *Cat {
	return cat
}

The above will now create a singleton pattern that you can access from any component within the package like models.CatModel().Find().

Note: the CatModel() method should only be used to fetch data from the database. Saving, updating and deleting data should be applied to an actual instance that has been returned through a Find(), FindAll() or FindMany() function.

Working with JSON Forms

Once you have retrieved the records needed from the database, you may want to send those records as a response. To do that, you can use forms. Every model is expected to have at least one method named MapToForm() that returns a JSON representation of that model.

Forms are basic structures that may or may not exactly match all the properties that a model has. The reason it has been done this way is to enable multiple forms where one can contain all model properties e.g. intended for an admin user to view, while another may contain a sanitised version of that model e.g. intended only for a read-only user.

For more details on creating a form, scroll down to the forms section below. Here you'll find a sample implementation of MapToForm() function.

func (model *Cat) MapToForm() *CatForm {
	form := &CatForm{
		Name: model.Name,
		Type: model.Type,
	}
	form.ID = model.ID
	form.CreatedAt = model.CreatedAt
	form.UpdatedAt = model.UpdatedAt
	return form
}

Complete Code

Here's a complete code as a model sample that you can copy as a base model.

var cat *Cat = &Cat{}

func CatModel() *Cat {
	return cat
}

type Cat struct {
	ModelBase
	Name string `gorm:"size:255"`
	Type string `gorm:"size:255"`
}

func (model *Cat) MapToForm() *CatForm {
	form := &CatForm{
		Name: model.Name,
		Type: model.Type,
	}
	form.ID = model.ID
	form.CreatedAt = model.CreatedAt
	form.UpdatedAt = model.UpdatedAt
	return form
}

func (model *Cat) FindAll() (models []*Cat, err error) {
	result := db.Model(model).Find(&models)
	return models, result.Error
}

func (model *Cat) FindMany(ids []string) (models []*Cat, err error) {
	result := db.Model(model).Find(&models, ids)
	return models, result.Error
}

func (model *Cat) Find(id string) (m *Cat, err error) {
	result := db.Model(model).Where("ID=?", id).First(&m)
	return m, result.Error
}

func (model *Cat) Save() error {
	return db.Model(model).Create(&model).Error
}

func (model *Cat) Update() error {
	return db.Model(model).Updates(&model).Error
}

func (model *Cat) Delete(id string) error {
	return db.Model(model).Where("ID=?", id).Delete(&model).Error
}

Copy the code above and replace the name Cat to get started.

Forms are data contracts that are used to send responses to clients and receive/ bind user input.

Each model can have many forms to enable sending specific values with different endpoints. An example scenario would be having an admin with full access to all data in a record whereas a customer has access only to a subset of that data.

Data validation is applied to fields in forms. Here's a sample form to get you started.

type CatForm struct {
	FormBase
	Name string `json:"name" validate:"required,min=2,max=50"`
	Type string `json:"type" validate:"required,min=2,max=80"`
}

func (form *CatForm) MapToModel() *Cat {
	return &Cat{
		Name: form.Name,
		Type: form.Type,
	}
}

The FormBase struct provides the ID, CreatedAt and UpdatedAt fields.

Each field should specify the name mapping in JSON format along with validation rules. For more on validations check out the Playground Validator documentation. To skip validations all together, use validate:"-".

Finally, each form should have a MapToModel() function that returns a model, so it can be stored after it has been validated. Note that forms do not set a model's ID property as that is the job of the model. Instead, it must be set manually prior to a database operation. Think of this like an actual form you fill up that has a section "for office use only".

Note: This go boilerplate uses Echo. If you're ever in doubt, you can refer to Echo's documentation for more details on what's possible with routers.

A handler is any function with the func (c echo.Context) error signature. All handlers should be stored under pkg/api/handlers and categorized in directories following their entity name in plural form. For readability and maintainability, we encourage maintaining a single handler in a single file as we all know that Go files can quickly grow.

Handlers should also be nested, which means a Cats handlers directory can contain a sub directory, such as cats/tags, that helps avoid long file names and improve readability.

How handlers look will largely depend on your project's business logic and requirements. For reference, here's a quick sample to give an idea on how you should construct your handler.

func Get(c echo.Context) error {

	id := c.Param("id")

	if id == "" {
		return helpers.Error(c, constants.ERROR_ID_NOT_FOUND, nil)
	}

	m, err := models.CatModel().Find(id)

	if err != nil {
		return helpers.Error(c, err, nil)
	}

	return c.JSON(http.StatusOK, handlers.Success(m.MapToForm()))

}

And perhaps another example to demonstrate how to receive user input and store a model.

func Post(c echo.Context) error {

	f := &models.CatForm{}

	if err := c.Bind(f); err != nil {
		return helpers.Error(c, constants.ERROR_BINDING_BODY, err)
	}

	if err := helpers.Validate(f); err != nil {
		return c.JSON(http.StatusBadRequest, handlers.ValidationErrors(err))
	}

	m := f.MapToModel()

	if err := m.Save(); err != nil {
		return helpers.Error(c, err, nil)
	}

	return c.JSON(http.StatusOK, handlers.Success(m.MapToForm()))

}

Note: This go boilerplate uses Echo. If you're ever in doubt, you can refer to Echo's documentation for more details on what's possible with routers.

This boilerplate is shipped with 3 routers, public, privdate and hidden routers - all of which ollow the same structure and procedure with slight differences in what is registered within each.

Why have 3 routers? Well, some projects may have public and protected routes, and such use case is straightforward. The latter implements an authentication middleware while the first does not. Attempting to achieve such behaviour within a single router can be tricky, so isolated routers running on different ports are used instead. The third "hidden" router is provided to enable a pattern commonly used to allow one microservice to communicate with another without exposing those routes to the public internet. With that said, wiring those 3 routers can easily be achieved through a different service like Kubernetes or NGINX.

All routers should go through the following process:

1. Initialisation
2. Checking for DevMode and enabling related middlewares
3. Register common middleware
4. Register health routes
5. Register security middleware
6. Register user-defined routes
7. Register error handler

Have a look at pkg/api/routers/protectedApi.go to familiarise yourself with router initialisation process. If you've already created your handlers from the previous section, all you need is to add your new route to this file as such:

func registerProtectedAPIRoutes() {
	cats := protectedApiRouter.Echo.Group("/cats") // Your new REST resource
	cats.GET("", catsHandlers.Index)               // GET "/cats/" route and handler 
	cats.GET("/:id", catsHandlers.Get)             // GET "/cats/:id" route and handler
	cats.POST("", catsHandlers.Post)               // POST "/cats/" route and handler
	cats.PUT("/:id", catsHandlers.Put)             // PUT "/cats/:id" route and handler
	cats.DELETE("/:id", catsHandlers.Delete)       // DELETE "/cats/:id" route and handler

	// add more routes here ...
}

Tasks is a way to extend the command line CLI without having to go through the trouble of understanding the initialisation process.

To create a new task, simply add the following sample into a new file ./pkg/tasks/<myTask>.go:

func init() {
	var t = &Task{
		Name:        "myTask",
		Description: "This is my task",
		RequiredArgs: []string{"key1", "key2"}, // add args here
		Run:         execMyTask,
	}
	Tasks.AddTask(t)
}

func execMyTask(env *TaskEnv, args map[string]string) error {
	// task implementation goes here...
	fmt.Println("My task is executed!")
	return nil
}

Tasks are automatically injected with an env object that contains the environment. They are also injected with an args map containing any values added to the exec command, as long as they're separated by '=' e.g. key1=value1 key2=value2 key3=value3 .

You can also set the required arguments in myTask.RequiredArgs = []string{"key1", "key2"} to prevent execution until all arguments are provided.

To execute the task above, simply run go run . task exec myTask and you should get the "My first task is executed!" message.

This boilerplate automates error handling and error responses.

First, let's talk about logging errors. When using proper logging mechanics and log levels, you can then leave all your logs in the code and have them printed depending on their severity. The package is shipped with the function helpers.Error(), a wrapper that's intended to log an error and return it. These logs will only be visible if the LOG_LEVEL env var permits. Avoid using fmt.Println() at all times, instead, use logger.Debug() or if you're within a handler, you can use the c.Logger().Debug() helper.

Given that each handler can return an error, the router is configured to handle the error using pkg/api/handlers/errors/automatedHttpError.go which will unwrap the error and match it with a list of registered errors under pkg/api/handlers/errors/errors.go. Finally, it will construct an error response and respond to that request.

Validation errors are no different, except they're unwrapped further and sent to the user as individual form inputs so they can be displayed.

You're encouraged to register and maintain as many errors as you can in the same way. It's useful to have a specific error code mapped to each error, that way we can determine exactly what went wrong in each user flow.

All environment variables reside in pkg/config/features. They're categorised within their respective features such as database.go or service.go. Each env var must have a mapstructure: decoration that spells it in caps when parsing an ENV. You can add your own, it's as simple as adding a new line in any of these files, or create your own.

Below is a sample of pkg/config/features/service.go:

type ServiceConfig struct {
	Host                   string `mapstructure:"HOST"`
	ProtectedApiPort       string `mapstructure:"PROTECTED_API_PORT"`
	PublicApiPort          string `mapstructure:"PUBLIC_API_PORT"`
	HiddenApiPort          string `mapstructure:"HIDDEN_API_PORT"`
	LogLevel               string `mapstructure:"LOG_LEVEL"`
	RequestTimeoutDuration string `mapstructure:"REQUEST_TIMEOUT_DURATION"`
	WatcherSleepInterval   string `mapstructure:"WATCHER_SLEEP_INTERVAL"`
}

var service = &Feature{
	Name:       constants.FEATURE_SERVICE,
	Config:     &ServiceConfig{},
	enabled:    true,
	configured: false,
	ready:      false,
	requirements: []string{
		"Host",
		"ProtectedApiPort",
		"PublicApiPort",
		"HiddenApiPort",
		"LogLevel",
		"RequestTimeoutDuration",
		"WatcherSleepInterval",
	},
}

func init() {
	Features.Add(service)
}

From the example above, you can find a type ServiceConfig that states what env vars are to be expected. These are automatically read from the environment. Env vars must belong to a feature which can be toggled on or off. A feature can also define which env vars are required for it to start.

If you wish to disable a feature, you can mention it in the list of DISABLE_FEATURES var in run-time.

Reading the env vars is the job of pkg/config/envVars.go. Each config struct must be registered in envVars.go. The config struct is then automatically injected to its respective feature after initialisation.

It is possible to set a default value for each variable, this can be done in pkg/config/envVars.go under setDefaults().

By the time the CMD calls the Proc, all env vars should have already been read and injected into their features, making them available for the rest of the package.

Root Folder Structure

The package is split into 3 directories

pkg Folder Structure

+- pkg
|  +- api
|  |  +- handlers
|  |  |  +- healthz
|  |  |  +- errors
|  |  |  +- cats            <--- example handlers
|  |  |  +-                 <--- additional handlers
|  |  +- helpers
|  |  |  +- helpers.go
|  |  |  +-                 <--- additional helpers
|  |  +- middlewares
|  |  |  +- authentication.go
|  |  |  +- authorization.go
|  |  |  +-                 <--- additional middleware
|  |  +- routers
|  |  |  +- router.go
|  |  |  +- hiddenApi.go
|  |  |  +- protectedApi.go
|  |  |  +- publicApi.go
|  |  |  +-                 <--- additional routers
|  |  +- api.go
|  +- clients
|  |  +- db
|  |  +- fhttp
|  |  +- keto
|  |  +- kratos
|  |  +- logger
|  |  +- redis
|  +- config
|  |  +- autoEnv.go
|  |  +- config.go
|  |  +- feature.go
|  |  +- features.go
|  |  +- flags.go
|  |  +- service.go
|  +- db
|  |  +- migrations
|  |  |  +- migrations.go   <--- list of migrations to run, be sure to add yours here
|  |  |  +-                 <--- additional migrations
|  |  +- models
|  |  |  +- models.go
|  |  |  +- forms.go
|  |  |  +- cat.go          <--- example model
|  |  |  +- catForm.go      <--- example form
|  |  |  +-                 <--- additional routers
|  |  +- seeds
|  |  |  +- seeds.go        <--- list of seeds to run, be sure to add yours here
|  |  |  +-                 <--- additional seeds
|  +- proc                  <--- entry point to all processes
|  |  +- proc.go
|  |  +- hiddenApi.go
|  |  +- protectedApi.go
|  |  +- publicApi.go
|  |  +- watcher.go
|  +- tasks
|  |  +- myFirstTask.go
|  |  +-                    <--- additional tasks
|  +- utils
|  |  +- constants
|  |  |  +- constants.go    <--- all literal values
|  |  +- init.go
|  |  +- utils.go           <--- reusable functions that don't belong anywhere else
+- go.mod
+- main.go