Hopefully Surprising Blog RSS Feed

AI-First, Engineer-Led

Tue, 16 Jun 2026 12:00:00 GMT

Now that the machine writes the code well, the engineer's craft moves to the floor the machine works on.

I became an engineer because I like building things that work — and, hopefully, things that are a little remarkable. The code was always how I got there; it was never the point of it. So when a machine arrives that can write that code, and write it well, it doesn't take the craft away. It gives me a better question: now that the machine is this good, how do I get the most out of it — more built, and built better?

This is exactly the kind of problem I enjoy most. I design the thing that works — the structure, the constraints, the way the parts fit. When the machine takes over the writing, that instinct doesn't go away; it moves to where it matters most now, which is the environment the machine works in.

There's a name people are starting to use for this work — harness engineering — and the idea behind it is good: an agent does better when you design the environment it builds inside, instead of just giving it a prompt and hoping for the best. Give it the real shape of the code, clear constraints, and a way to check itself, and the output stops being a gamble. If a machine fails, it's my problem.

A picture from the real world helps here. Think of the engineer who runs a factory floor. The machines do the making; the engineer owns the line — its layout, its power, the materials feeding each station, the order the work moves through. Nobody measures that engineer by how fast they can turn a single bolt. They are measured by how well the whole line runs once they step back from it. That is the move I'm making: from turning bolts to designing the floor.

And a floor works only if two things are true. Most of the talk is about the first, and skips the second.

I have to see it clearly enough to shape it, and the machine has to reach into it freely, without a locked door at every turn.

The first keeps me in charge — I can't improve a station I've never looked at. The second is the one that matters most to me. If the agent can only reach a tool through a gate, a wrapper, or a rate limit, that tool might as well not be there at all. An agent boxed into a corner is a weak one; a strong one can move freely, with the same reach I'd give a trusted colleague.

In software, the busiest part of that floor is the command line. It's where the most capable machines plug in — open, scriptable, asking no one for permission — and it's also the hardest part to see all at once: hundreds of tools, each one a maze of subcommands and flags, most of it behind a help page you have to go looking for. The agent reaches all of it freely; I can barely remember it all. Closing that small gap is what made me build a little tool (that I called "Bract"), which I've written about separately.

The direction is clear, and I'll be honest about how I feel about it: for me, the craft has never been more exciting. The machine will always type faster than I can — and that is exactly what frees me for the bigger, longer-lasting work of designing the place where good software gets made. Build the floor well, keep the right doors unlocked, and let the machines run. It's a lot of fun.

Meet Bract

Tue, 16 Jun 2026 11:00:00 GMT

A small terminal tool that makes your CLI toolchain explorable — and the one fortunate fact behind it.

In the previous post I argued that the environment an AI works in has to stay visible to whoever shapes it — and that, in software, the command line is one of the harder parts to see. Let me show you the small tool I built for exactly that.

You know the wall I mean. You open a powerful command you don't use every day — kubectl, say, or the Azure CLI — and to find the one thing you need, you climb down through help page after help page, subcommand into subcommand, putting the right line together one flag at a time. The power is all there. Seeing it is the hard part.

Bract is my attempt to make it visible: you browse your toolchain like a file tree, fill a command in through a form, and copy or run it.

It doesn't run just anywhere, and that's on purpose. Bract reads your mise environment — and if you haven't met mise yet, you're in for a treat: it's a tool manager that pins the exact CLIs, runtimes, and tasks a project needs, so the toolchain is the same for everyone who opens the repo. mise decides what's on your factory floor; Bract is how you walk it.

Now, the obvious worry about a tool like this is coverage. There are thousands of command-line programs out there, all different, and teaching one little browser to understand each of them by hand would be hopeless. The fortunate fact is that it doesn't have to.

A lot of modern CLIs aren't hand-built anymore; they're generated by a few frameworks, and every tool built on the same framework has the same help-text shape. So you teach the framework once, and every tool built on it comes into view together. A single parser for Cobra doesn't just read kubectl — it reads helm, gh, and the hundreds of others built the same way.

I reckon this luck is recent, and it doesn't cover everything. The older tools were each written from scratch, so this approach can't reach them — even though some of them are the most loved tools on the system. It's uneven in another way, too: Cobra alone opens a huge family of tools, while a parser for Microsoft's Knack covers little more than the Azure CLI, and Usage covers mise itself and a small, hand-picked set. The idea only works where a framework became popular — but where it did, it works very well.

Bract has been really useful in my own work, and I'm putting it out because I honestly believe you could find it useful. I recommend giving it a run; the repository has everything else, including where to tell me what you find. I think you'll enjoy finally seeing the whole floor from a new perspective.

Tyk Gateway with OpenAPI Architecture: Comprehensive Guide for Free Tyk & PostgREST Setup

Fri, 16 Feb 2024 00:00:00 GMT

Recently, in one of my projects, I have been working on an API solution providing access to data through REST endpoints. Since data has been structured in a relational database (Postgres) and there is a requirement to be compliant with OpenAPI specification, I was naturally inclined towards choosing PostgREST as API server solution. That would provide quite flexible methods of interacting with the underlying dataset without writing much (if any) code for endpoints. Additionally, OAS compliance is available out of the box.

While a PostgREST instance would cover majority of our needs, some functionality would be quite challenging to implement with a single PostgREST instance. To leave some room for future expansions, I have chosen to use Tyk gateway to be my reverse proxy so that it orchestrates routing the requests to a correct upstream API node. It's available in open-source version, and while lion's share of their documentation is focused on paid Tyk Dashboard, from the functionality perspective, open-source Tyk gateway is complete.

In this article, I'm sharing experience of how to use advantages of OpenAPI specification for configuring a functional API gateway. Also, I will demonstrate how to overcome challenges that occur when doing so for PostgREST (or other OAS-based) instances.

As a bonus, I'm showing the process using an example of two separate REST APIs. A solution that fully benefits from using an API gateway.

Your use case might be different but the target architecture could be similar.

Vision

To leverage the advantages of conformity with OpenAPI specification (especially, predefined API schema), I decided to use OAS definitions as the source of truth for publishing API capabilities to external parties and to the API gateway. Luckily, Tyk gateway has recently started supporting OpenAPI specification as API definition language. But, unfortunately, I haven't found how to make Tyk gateway serve the merged OAS definition. So, there is a separate process for serving a unified API definition using a standalone Swagger instance. See the deployment diagram above.

I'm not covering the separate Swagger instance in this article but taking a look at openapi-merge-cli might give you an idea how to get a single OAS to serve.

The process of setting up such a solution and the challenges inherent in the process are described below.

Solution

Let's consider this example: we store data about the orders placed in our application in one Postgres database while master data of the corporate partners of ours is stored in another. For those two we want to create separate PostgREST API instances. We want to provide centralised access to the both instances of our data API so that from the user standpoint there is no difference where data is coming from.

When there is a request GET /transactions/orders coming to API gateway, it should behave like this:

and when a request GET /partners/companies is coming, it should work like that:

We will achieve it by doing the following things:

Start two Postgres databases and two PostgREST instances
Get available OpenAPI specification for the instances
Convert the provided OpenAPI specifications from version 2 to version 3
Start Tyk gateway
Import OpenAPI specifications of version 3 into Tyk gateway
Check how it works

Start Postgres databases and PostgREST instances

As PostgREST setup is very nicely described in the official documentation, I'm not explaining it here in detail. If you want to play with the overall solution on a good example, please, follow the instructions on PostgREST website. Otherwise, feel free to experiment with your OpenAPI compliant server or use the sample application I pushed to GitHub.

Please note that Postgres databases require some configuration to be PostgREST-ready. Check out the initialisation script if you want to see how the sample application is prepared.

If you are going to bring your own OpenAPI-compliant server, check the version of the specification. You can easily verify the version by checking either swagger or openapi value in the root of your OAS definition.

If you're on the version 3, feel free jump straight to "Import OAS 3.0 specification" section. Otherwise, follow the conversion steps below.

Get available OpenAPI specification for instances

PostgREST doesn’t support the static generation of OAS because it learns the schema of the database at the moment of startup. So, if specification needs to be extracted, at least one PostgREST instance should be up and running to introspect the database and return OAS definition.

While I referred to PostgREST documentation in the first step, here, I want to explicitly mention that by default the OpenAPI schema considers permissions and presents available endpoints only. To get the whole schema, you need to change openapi-mode for the PostgREST instance to ignore-privileges.

If you've done it right, you should be able to access the schema of PostgREST server at the root path. Detailed information is available in documentation.

Convert PostgREST OAS from version 2.0 to 3.0.0

As I mentioned in the beginning, PostgREST currently provides OpenAPI specification of version 2 while Tyk supports versions 3.0.x. It's not a big problem because OAS 2 can be converted to OAS 3. It can be done with an official Swagger Converter. The converter can be started locally in a Docker container but it's also available online - https://converter.swagger.io/.

Outdated OAS version is a recognized issue of PostgREST. As of February'24, the work on making PostgREST providing OAS of version 3 is in progress in a separate repository.

For simplicity, let's use the online version of the converter.

What you need to do is to copy OAS 2 specification and use it in [POST] /convert. If you're using Swagger interface, just click "Try it out" and paste specification as body of the request.

If you hit "Execute", the converter will respond with the specification of version 3.0.1.

This version of specification is perfectly compatible with Tyk gateway. Convert specifications for both instances and it will be good to import to Tyk.

Specification examples for version 2.0 and converted to 3.0.1 can be found in the sample code.

Prepare Tyk gateway

The best way to start a local instance of Tyk is using the Docker image. How it's done is explained by official examples of Tyk with Docker Compose but for the purpose of this experiment you might take the trivial Docker Compose setup from the sample Git repository that I have prepared.

Please, note that not the entire definition is shown in the snippet! Navigate to the source to see the whole configuration.

For our case, no special setup is needed but ensure that unauthenticated access is not blocked. Auth is not in the scope of this tutorial.

Import PostgREST as OAS to Tyk gateway

Now, when we have OAS of version 3.0.1, it can be statically imported as API definition to Tyk.

I want to note that it's possible to import OAS dynamically by calling the dedicated Tyk API endpoint. But to simplify the deployment process and to be able to version gateway as a whole with Git I do this import statically.

Another possible option might be Tyk Sync but it's in early access now so I'm not considering it for production while it's not strictly necessary.

We will create the files in the apps folder and provide this folder as a volume to Tyk gateway container to be available at app_path from the Tyk config.

You need to create two Tyk API definitions per upstream API - one is classic Tyk API definition to store overall API configuration (such as auth, rate limitation and so on) and one for OAS definition with available endpoints.

For classic Tyk API definition, here is the sample configuration (apps/postgrest1.json):

The points of interest here are:

api_id must match the one in the respective OAS definition
is_oas is set to true
proxy.listen_path and proxy.target_url specify at what subpath to receive the requests for the upstream API and at what URL to redirect to

proxy.target_url is set as an environment variable. In the sample Docker Compose setup, it's passed to Tyk gateway as a secret. If you are interested in how it's done, check out this recipe in Tyk docs.

For Tyk OAS API definition, documentation is not very clear which is understandable for a feature which is still pretty new. To understand how to work with it, read the following paragraph carefully.

For the respective OAS part, we need to take the OAS definition of version 3, add additional information for Tyk by placing a configuration object x-tyk-gateway in the root of OAS JSON and save this file next to classic Tyk API definition. In our case, it's postgrest1-oas.json.

You need to copy some information that you have placed in the classic definition earlier.

x-tyk-api-gateway.info.id should match api_id
x-tyk-api-gateway.upstream.url should match proxy.target_url
x-tyk-api-gateway.server.listen_path should match proxy.listen_path

The same should be repeated for the second set of API endpoints that are provided by the second PostgREST instance.

That's basically it. Congratulations! We can check how it works now.

Try it out

If you followed the example and used docker-compose setup from my repo, the gateway is running on the port 8888. You might give it a try.

curl --request GET --url http://localhost:8888/transactions/orders

and

curl --request GET --url http://localhost:8888/partners/companies

should provide you with data that is saved to the database in the initialisation script.

If you see some response there, awesome! You've made it working. Now, you might do something special with the configuration to get something that fits your needs the most.

[
	{
		"createdAt": "2024-02-11T16:56:07.810486+00:00",
		"id": 1,
		"orderNumber": "1",
		"status": "Placed",
		"updatedAt": "2024-02-11T16:56:07.810486+00:00"
	},
	{
		"createdAt": "2024-02-11T16:56:23.915413+00:00",
		"id": 2,
		"orderNumber": "2",
		"status": "Delivered",
		"updatedAt": "2024-02-11T16:56:23.915413+00:00"
	}
]

Summary

If you have reached this point, I'm really grateful for you. Your choice of technologies for your next project opens endless possibilities for something special.

With PostgREST, you might create scalable and configurable API access to your database. Just check what you might do with JWT issued by an IdP. Having RBAC and ABAC for users without any user table - that sounds nice, doesn't it?

With Tyk, you have a great opportunity to be future-proof with a Gartner's leader solution. Rate limiting, onboarding new APIs easily, JWT validation before even going to an upstream API. These will be very much welcome in your next API project. And with support of static API definitions and versioning you might be ambitious enough to target multiple nines.

And if you're compliant with OpenAPI, developers integrating with your API will appreciate your system. It's not a straightforward thing merging multiple OASs and serving it as a single Swagger but it's also not overly challenging. While the result is exciting.

As usual, it would be awesome to hear about your vision of this design and about what immediate improvements or enhancements you are tempted to try in your version of such a setup.

Good luck. It will be fun.

Beyond the Scaffold: Evolving Software Projects with Yeoman Generators

Fri, 29 Dec 2023 00:00:00 GMT

Starting anew always brings a lot of excitement. Every new project gives a mix of confidence from making first steps you've trained a million times and anticipation that this time the results will be greater than ever before because you'll try some new answers to old challenges. At every new beginning, you're stronger than ever before.

It's not surprising that software developers like starting new applications, right?

Nowadays, we create more software packages than ever before. Microservices, microfrontends, microapps, monorepos and tools for them... You need a new module - you isolate it radically, define the communication interface and let it ripen.

Reasonably, the scaffolding tools are more important than ever before.

They help speed up project creation, ensure it's aligned with the best practices and swiftly bring you to the point of creating something that truly creates value - new automations.

Recently, I have been experimenting with extreme modularisation in my Node.js-based project. My objective is to keep it maintainable long term. In respect to it, I need to make each part of the ecosystem genuinely independent - every part should be ready for growth that doesn't impact other applications. To support creating microapplications in a sustainable manner, I need to ensure one thing - these numerous separate projects will be maintained well.

How quickly a project is created doesn't matter much in the long run if maintainability is compromised.

If my scaffolding tool doesn't help me keep settings consistent between dozens of packages long term, I don't need such a tool. I need a tool that will help me "make up" a project whenever I need it.

Vision

My goal was to find or create a tool that will not only take the burden of scaffolding a new project quickly and in a consistent manner (configuration, packages, package versions and so on) but that will also be capable of bringing a project up to advancing standards at any point in future.

If I use traditional approach. I run the generator that promotes current project standards in an empty folder and I get a project that meets the requirements today.

But the standards change over time. Let two months pass after the initial project creation, and a new fast TypeScript compiler reaches production readiness, TypeScript itself get a new version, license of choice has changed for your open source projects and a million other things get updated. And you change the standards because you must.

When running the standards ambassador (the scaffolding tool) in the project again, I want my project to get up to the height of the new standards (ideally, without breaking the application).

That would help take care of one of the most important aspects of why we need a tool - ensure consistency of standards between the projects. And do that continuously.

Solution

Platform

While creating a scaffolding engine from scratch might be a good idea, I decided to review ready solutions that have the following capabilities:

Generator scripts should be easy to create
Generator scripts should be easy to compose
At every step, there should be freedom to decide if and how an atomic change should be done
The tool should be aware of Node.js context and provide relevant utilities (adding dependencies, for example)

Less important but good to have capabilities are:

The tool should have built-in ability to collect user input during the generation stage
The tool should have ability to work with file templates

I have reviewed a few solutions:

I have to say the mentioned tools can provide a platform for flexible and composable generators but doing that in Plop and Hygen was quite an effort since the first one sets the generator parameters at the beginning of the execution (statically) and the second one is more template-oriented while I needed to put significant pieces of code in places to control what and how to do. I found it very challenging to do what I want the way I want with those two tools. Yeoman appeared much more permissive in this regard.

That's why I chose Yeoman.

I have implemented my own generator and published source code on GitHub and npmjs. Below, I'll cover the main challenges I had when implementing the logic I described above and how I approached them. Also, I'll mention what else would be great to have in a scaffolding tool like this.

Challenges

Composability

One of the requirements was composability of generators and I achieved it by introducing two general types of generators:

A "Feature" generator. They execute a set of atomic operations aiming to make up a particular aspect of the project.
A "Makeup" generator. They execute multiple feature generators in a particular order.

For example, to ensure that there is an npm package initialised and the package name, author and license statements are populated is a "feature" (feature-initiate-npm-package). Initialising Jest for unit tests and configuring it (feature-initiate-jest) is also considered a "feature".

Running the first feature and the second feature is responsibility of a "makeup" generator.

The list of "feature" and "makeup" generators and what they do can be found in the repo README. All of them are supposed to be able to run on their own.

Atomic changes

To stay flexible about what and how to do, I employed the feature of Yeoman to define the separate changes in separate generator methods that are ran in a particular order (see "Adding your own functionality" in their docs). All changes are small, independent adjustments that collectively contribute to the overall project structure.

A good example is feature-initiate-typescript generator. For every mini-change, it has a separate safe-to-run-independently method. For example, enableEsModuleInterop gently processes the options and checks that one particular setting matches the expectations around it (it should be true for having ESM support in Jest). If the current value doesn't match, in this and only this case, we change it.

By creating these atomic independent flexible brush strokes, we can construct the project carefully, having flexibility to do something extra if needed.

A good example of something extra can be found in another generator - feature-initiate-npm-package. When creating a LICENSE file, we also check if there is any existing LICENSE.md or LICENSE.txt in the folder already. If there is some, an additional cleanup should be done. And with these atomic changes we can do these changes in an optimal way whenever is needed.

Profile data

My goal is to keep multiple projects generated with the tool consistent. That means, the decisions that have been made before should be taken into consideration at every run of the generator. Let's consider 3 different situations:

1. Setting project name

Every project has a name. And it's one of the first fields you fill in package.json. When you start a project and fill this field, why would you ever change it using a generator? Once user input is received, it can be stored in the project itself. And at every next run of the generator, if name is already stored in a project file, we can make an assumption - nothing to do here, it's named already.

In such cases, package.json seems to be the most natural place to store such information between the generator runs. So, the project configuration is considered generator configuration in such cases.

2. Choosing distribution type

Every project makes assumptions about the model of code distribution. Some are open source, some are proprietary and all of them should have right license information filled. But the way of specifying license and terms of use for a category of projects can change over time. Today, you use "LGPL-3.0" for your open source projects, tomorrow, you might decide that "MIT" is a better choice.

In such case, you can't judge by presence of a license file that the project is all set. The project needs remember the distribution type, not what license it meant at the time of running.

For such things, Yeoman provides Storage API that you can use from the code of your generator. Every configuration value is preserved in a special JSON file in the root of your generated project and this configuration can be read from and written to when the generator is running.

Good job, Yeoman.

3. Setting an author of a project

Another type of information that you might want to preserve between generator runs can, in fact, be reused between projects. For example, once you set "author.email" for one project, it might be used in other ones. There shouldn't be a need to ask for it again and again. It should be profile-level configuration.

Unfortunately, this feature is not well documented in Yeoman (I haven’t found any mention of it in the documentation. I found the hints of it in source code of Yeoman-Generator). Also, this global .yo-rc-global.json file created in the home directory that contains a profile JSON file saves prompts per-project. It’s not enough for my project, so I have created a separate global config storage. Check out file globalConfigurationUtilities.ts, if you’re curious to learn more.

Three types of configuration storages are enough for now

By choosing the right place for storing configuration, I can ensure that necessary information is asked as many times as many times it's strictly needed. That simplifies repetitive runs of the generator for maintenance purpose. I need it to be as straightforward as possible.

Conclusion

The full code of the generator I was experimenting with can be found on GitHub and npmjs. I'm glad that I have the generator as the dedicated place to store design standards for dozens of microapplications that I create. What is especially important in this very case, it is not a one-time activity but a continuous process of keeping projects standards consistent between my projects over long time.

Of course, for this particular generator, it's only beginning. A lot of things need to be considered and addressed in future.

Project types are different. Frontend is not like backend. A REST API microservice significantly differs from a GraphQL microservice. That will bring another challenge of composing general "feature" generators with very specific "feature" generators. That should be possible to do elegantly but that will impact the generator project structure.

Also, to be able to use the generator in different contexts (for your personal projects, for projects of your company, for projects you help a friend with and so on), I would like to introduce proper support of profiles that will allow to use varying for-all-projects presets.

And last but not least, I would be happy to make a generator capable of assisting with possible migrations. For example, a new standard version of a dependency might have breaking changes (some method or argument might be removed) and it's possible to identify the disaster scope automatically. Is this removed method used in any of the project files? Do updated methods receive the right number of arguments? That should be pretty feasible through leveraging text search and AST tools.

These are some challenges I foresee to encounter when working on this generator in future.

I would love to hear about your experiences with scaffolding in multiple projects, your challenges and your solutions. If you have an opinion about the challenges I have mentioned above and especially how they might be approached in an elegant way, do let me know through comments, email messages, GitHub issues and pull requests. The generator is designed to grow - it is using TypeScript for writing the generators and is covered the generators with TypeScript-written Jest tests.

I'll post a story about covering the generator with tests shortly. Thanks for reading.

SonarQube Community Edition: Comprehensive Guide for a Free Personal Setup

Tue, 03 Oct 2023 00:00:00 GMT

When we develop software, we want it to be technically remarkable. It's a common desire to ensure high code quality to benefit both others and your future self. Static code analysers are essential tools that assess your code against predefined rules without running it.

While enterprise-grade solutions for that might be enterprise-level expensive, some of the most advanced solutions - such as SonarQube - can be used for free if all the pieces of the stack are put together by yourself.

In this material, I will walk you through the process of creating personal SonarQube Community Edition instance and setting up the scanner on an example of NodeJS project that I wrote about some time ago (Creating a custom webpack loader for text files, compatible with TypeScript. PlantUML mind map example). In addition to that, I'll share a couple of thoughts about why a particular option is chosen and not an alternative.

Solution

This guide consists of a few parts:

Setting up the stage. Listing system requirements, installing missing dependencies.
Configuring and starting SonarQube CE instance.
Creating a SonarQube project, configuring and running the SonarScanner.
(Bonus). Installing and configuring SonarLint for getting feedback about the code right in IDE.

Setting up the stage

Using Docker as the Runner

We will use Docker for running SonarQube server and SonarScanner.

While it's possible to run the server and SonarScanner from a binary file, I choose the Docker way because of two benefits:

It's much simpler to upgrade SonarQube with Docker: instead of extracting config files, placing binaries somewhere in $PATH again and adding config back, you simply restart the container from a newer image and bind volumes there.
You don't need to have Java in the host machine.

Providing existing database as a storage. Postgres example.

SonarQube needs to store a lot of information about the users, rules, projects, scanning sessions, etc. It uses a connected database instance for that.

SonarQube server supports different databases so we can use this advantage. The key for me is to use the database that you're used to and that is already running in your computer. That helps reduce the footprint of the running SonarQube server - no need to have an entire database instance for a specific use case.

In my case, it's Postgres. I'm not sharing the instructions on how to install Postgres itself but here is the link to official documentation about installing Postgres 12 on Ubuntu. SonarQube's Docker Compose example refers to PostgreSQL 12 so I'm using this version.

When the database is available, we need to create schema, database and the user that SonarQube server will be using for storing data. Below is the Postgres way:

NOTE: Don't forget to replace SONAR_DB_PASSWORD with the real one you'll be using.

CREATE SCHEMA sonarqube;
CREATE USER sonarqube WITH PASSWORD 'SONAR_DB_PASSWORD';
GRANT USAGE ON SCHEMA sonarqube TO sonarqube;
GRANT CREATE ON SCHEMA sonarqube TO sonarqube;
ALTER DEFAULT PRIVILEGES IN SCHEMA sonarqube GRANT ALL ON TABLES TO sonarqube;
ALTER DEFAULT PRIVILEGES IN SCHEMA sonarqube GRANT ALL ON SEQUENCES TO sonarqube;
ALTER DEFAULT PRIVILEGES IN SCHEMA sonarqube GRANT ALL ON FUNCTIONS TO sonarqube;
CREATE DATABASE sonarqube;
ALTER DATABASE sonarqube OWNER TO sonarqube;

Configuring and starting SonarQube CE instance

SonarQube server

Now, when the database and Docker engine are available in the host machine, we can start the SonarQube server container.

Some troubleshooting notes will follow the happy path of the installation.

Create the Docker volumes

sudo docker volume create --name sonarqube_data
sudo docker volume create --name sonarqube_logs
sudo docker volume create --name sonarqube_extensions

Start the SonarQube server container

sudo docker run --detach --name sonarqube \
	--add-host=host.docker.internal:host-gateway \
	--publish 9000:9000 \
	--env SONAR_JDBC_URL=jdbc:postgresql://host.docker.internal:5432/sonarqube \
	--env SONAR_JDBC_USERNAME=sonarqube \
	--env SONAR_JDBC_PASSWORD=SONAR_DB_PASSWORD \
	--volume sonarqube_data:/opt/sonarqube/data \
	--volume sonarqube_extensions:/opt/sonarqube/extensions \
	--volume sonarqube_logs:/opt/sonarqube/logs \
	sonarqube:community

NOTE 1: Don't forget to put the real SONAR_DB_PASSWORD password from the database created earlier.

NOTE 2: If you use different database, put your specific SONAR_JDBC_URL.

Common issues

If the Docker container is not starting, check the logs. The issues I encountered are listed below:

If Elasticsearch complains about the heap size in Linux, update this setting: vm.max_map_count to allow using more memory.
If the Postgres DB is not reachable, you might need to update pg_hba.conf to allow incoming connections for sonarqube user.

Access SonarQube UI

When the container with SonarQube server is running and is available on the port 9000 in your machine (see --publish parameter), you can access https://localhost:9000 in your browser. You'll be invited to log in. The most secure in the world admin - admin are your credentials. You'll need to change it as soon as you log in for the first time.

Running SonarScanner

Generate token for the scanner

Before creating and scanning projects, we need to generate a token that will allow the scanner to upload the results to SonarQube server. For a simple setup, we will create a global token. You can do that via opening the user dropdown in the top right corner of the screen and going to "My Account" -> "Security".

Other options are explained here.

Copy the token and save it in a secure place. You won't be able to see it in the UI ever again.

Create and configure a SonarQube project

In SonarQube dashboard, select "Create Project" and choose "Manually". Input display name and the project key, choose the default branch.

NOTE: For this setup, we will disable versioning so the branch name here doesn't really matter.

For the baseline, you can choose the global settings which is the default SonarQube preset.

In the opened project page, choose "Locally" for scanning setup.

Provide the global token you generated before and click Next.

You will be given the CLI command to run but you can ignore it. We will be running the scanner in a Docker container. Also, we will provide the parameters via .properties file and not CLI arguments.

You can leave the browser page open, it will refresh automatically when the results of scanning were pushed to SonarQube server.

In the root of the project you're going to scan with SonarQube, create sonar-project.properties file with the following content:

sonar.projectKey=plantuml-mindmap-loader

sonar.sources=.
sonar.sourceEncoding=UTF-8
sonar.scm.disabled=true
sonar.exclusions=**/node_modules/**,**/coverage/**,**/lib/**
sonar.javascript.lcov.reportPaths=./coverage/lcov.info

NOTE: This file contains instructions on unit-test coverage because this project has Jest tests. If you don't have it in the project, exclude sonar.javascript.lcov.reportPaths parameter.

Run the scanner

As mentioned above, we will run the scanner in a Docker container. You need to execute the below command with a correct SONAR_GLOBAL_TOKEN and the path to the folder containing the code you scan (this folder should contain sonar-project.properties you created before).

docker run \
	--rm \
	-e SONAR_HOST_URL="http://localhost:9000" \
	-e SONAR_SCANNER_OPTS="-Dproject.settings=sonar-project.properties" \
	-e SONAR_TOKEN="SONAR_GLOBAL_TOKEN" \
	-v "./software/packages/plantuml-mindmap-loader:/usr/src" \
	sonarsource/sonar-scanner-cli:5

It will take a couple of minutes to execute and after it's completed you can enjoy the code quality assessment results in SonarQube

Review the issues, check suggested solutions, apply a patch and run the scan again. The setup is complete.

Bonus. Setup SonarLint in VS Code

While seeing the results of scanning in SonarQube is super helpful, in software development, shortening the feedback loop for code changes is essential for efficient workflow. For that, there is SonarLint extension available for quite a few modern IDEs, including VS Code.

What makes this extension especially impressive is the connected mode when the rules applied to the project are synchronised with SonarQube server.

To install and configure it:

Find and install SonarLint extension in VS Code.
Go to the "SonarLint" tab and click at the plus button in "Connected Mode" section.
Add details: Server URL (http://localhost:9000), User Token (can use SONAR_GLOBAL_TOKEN from before).
Click "Save Connection".
For this connection, click the plus button for "Add project binding".
Select the workspace.
Select the project created in SonarQube server.
Enjoy the binding.

You can see the syntax highlight right in your IDE now and see the underlines going away as you address the quality concerns.

Results & Beyond

With this SonarQube Community Edition setup, you will be able to ensure outstanding quality in your projects with very little time and resource investments. Variety of benefits attracts:

If your project evolves quickly, you can enjoy sturdier stance of your project against changes. Your project will be harder to break, easier to maintain, more challenging to hack.
If it's a weekend project, future self will be extremely happy to start a new feature from a reliable checkpoint - with cleaner code, fewer bugs.

This guide focuses on the most straightforward setup possible and many possible improvements are not included in the scope not to lose focus on the MVP. A few evident things you might work out yourself:

Integrate SonarQube with CI/CD to ensure the quality gate fully protects your shared branches from code of low quality.
Create an automated local runner for SonarQube scanner to have a quality gate not before the code is merge but before the code is committed.

I hope the length of this guide doesn't intimidate you from trying this local setup. In fact, it's pretty straightforward and even lightweight. My objective was to provide information on the aspects of particular steps so that if in your situation a slightly different approach is more suitable, you make this turn and return to the main track with the solution that fits your needs the best.

And last but not the list, getting quality suggestions from SonarQube is a whole load of fun. You'll enjoy it.

Our Experiment with AI-Powered Dev Tools: The GitHub Copilot Experience of One Company

Wed, 27 Sep 2023 00:00:00 GMT

Introduction

As someone who constantly searches for ways to improve personal coding performance and performance of my development teams, I have been experimenting a lot with AI driven utilities that might assist developers in writing code. TabNine, ChatGPT 3 & 4 and, finally, GitHub Copilot that wraps a special version of OpenAI's CodeX in a tool that is easy-to-integrate with modern IDEs.

All these tools have proven useful in many situations and allowed me to create utilities for automating daily routines with ease I had never experienced before. My initial intention was to provide all the developers in my teams with access to this tool. But to better justify this regular spending to my company, I decided to conduct a little experiment: a pilot group had been given paid access to GitHub Copilot with a duty to share feedback regularly.

I have to say that it was a very right decision because the results weren't what I expected to see. And, sorry for the spoiler, we changed our plans - we're not giving it to everyone.

Before we started: Security

Like any company that considers information security a paramount subject, we reviewed the terms of usage of GitHub Copilot with our legal team. We ensured that running GitHub Copilot in a client machine doesn't involve long-term data collection by Microsoft. The prompts are discarded immediately after providing (see details).

We didn't include Copilot Chat in the scope of the pilot project (that is in beta at the moment of writing). Using the Chat might require accepting more conditions for data collection and processing.

Setup

For the pilot group, we have selected 5 software developers of different experience (from 3 years in IT to 10+ years) and different skillset (React.js in the frontend, Ruby on Rails and Node.js in the backend, GitLab CI for CI/CD pipelines). These 5 people were given GitHub Copilot licenses for 10 weeks and were asked to provide feedback every 2 weeks.

The common themes in the feedback have been collected by me (I also was a part of this program. I wanted to stop paying for it from my own wallet) and are presented in this article.

Observations

It's easy to start using it

What everyone has noticed and mentioned is how easy it is to install in VS Code. Basically, after 2 minutes of installing and configuring GitHub Copilot, the tool is ready for use and provides suggestions in all the files you work in. Convenience of installing is extreme.

But I have to mention that some people found it easier to migrate from Sublime to VS Code to start using it. Technically, it should be possible with Sublime though (this one looks interesting but we didn't spend time on it).

When it's wrong, you are (rightfully) not happy

One of the things noticed by more than one person was the effect of a wrong suggestion. When the suggestion is incorrect, you lose time. At best, you gave a second to Copilot to provide you with something you don't want to see. At worst, you have your code broken and your page turns red. And absolutely always, you have a half-a-second pause to see what it has got for you. You wait. For a fraction of a second but you wait. These micro-pauses are not what we liked.

It might be helpful (and dangerous) for beginners

A couple of people have mentioned that they think GitHub Copilot might be more useful to beginners in technologies. While I partially agree with that, I also have concerns. Any complicated tool requires a skilful operator. The suggestions from AI-powered tools must be validated by a person who knows what the code does and what it should do. Without this validation, it might be like an untamed id without ego. Who might want more chaos in their project?

Tricky things (sometimes) don't get less tricky with Copilot

One of more experienced developers tried to use Copilot for tricky things he is not extremely familiar with. The example is writing PostgreSQL query that filters records based on nested parameters stored in JSONB value. The suggestions given by Copilot were always somewhat imperfect. A solution was found with help of good old ChatGPT after specifying context explicitly (schema, version of PostgreSQL, version of ORM and so on).

It might document your function better than you

We spotted that at times GitHub Copilot helps a lot with documenting a method or a class. It's not lazy, it will write what's needed to full extent. It considers the context of the project, all the implications of underlying logic and it produces a good short summary. Probably, you don't want all of your methods to be documented in plain English. Or do you?

Short takes

Sometimes it surprises you with how much you're guessable.
If you have a typo in your code, it inherits it and can suggest code with this typo again.
Try start typing Database tables walk into a bar for a bit of fun.

Summary & Decision

The results are quite surprising and that's why it's in this blog. Every developer in our experiment came to a simple conclusion: if the company doesn't provide the license, they wouldn't purchase it themselves. But that statement might be a bit too generalised - I personally know a few people who pay for this tool for personal use. Including myself. The drawbacks of having wrong suggestions are usually overweighed by ability to create an Apple Script or getting a suggestion on Jira API protocol without leaving IDE.

Copilot Chat seems to be a more promising feature. A conversation-style tool (like ChatGPT) allows adding context that you realise is lacking. Also, it provides a much more natural environment of chatting with an extremely experienced developer who knows programming inside out but probably unaware of what exactly you're working on right now.

We decided to communicate to the developers that they can request a license for themselves. They will need to write a form, to justify the purchase. In simpler terms, we let developers request a tool from the company, just like a specific IDE or GUI for a database. This approach will help us optimise our software expenses while ensuring that those who genuinely want and need the tool have access to it.

GitHub Copilot is not a one-size-fits-all solution, and it was never intended to be.

Searching for the next one. Stay tuned.

Building a TypeScript-Compatible Webpack Loader: A PlantUML Mind Map Example

Wed, 13 Sep 2023 00:00:00 GMT

Introduction

In the historical book "The Pragmatic Programmer", there has always been an entire chapter dedicated to the importance of the ability to process text files of various formats quickly and efficiently (see "The Basic Tools: Text Manipulation"). The ability to use different text formats in various situations has always been essential and likely will continue to be.

For example, while JSON notation is super useful in many contexts, in some use cases, such as creating mind maps, another format might be much better. For example, PlantUML's mind maps might be a more preferable option for storing serialised trees.

That's what we will implement - store this tree-structured data as text in PlanUML format and use it in a JS app.

Quick links

Custom webpack loader in npm
Source code in GitHub

Vision

Although parsing files at runtime might be a good idea, another option might be even more beneficial. If we can integrate loading with the building process, we will be able to ensure that the data required for running the app is a part of the bundle and doesn't require passing non-common extension files separately.

In this tutorial, we will create a webpack loader that will be taking care of parsing PlantUML mind map file, converting the content to a standard JS object and providing the object to any JS module that imports it by file name.

Solution

We need to do 2 things to achieve the goal: create a loader as an injectable package and integrate this loader with the building process of the project that will be actually importing files with .puml extension.

As a bonus, we will setup basic TypeScript support that will allow importing the text file content with type definitions provided.

Create a custom loader

Project setup

Create a folder:

mkdir plantuml-mindmap-loader && cd plantuml-mindmap-loader

Initialise an npm project (you don't need to change default suggestions as some will be manually updated later in package.json):

npm init

Install dev dependencies:

npm install --save-dev @types/webpack typescript

Add webpack as a peer dependency to the project:

In package.json:

{
	...
	"peerDependencies": {
		"webpack": "^4.0.0 || ^5.0.0"
	}
	...
}

Note: It's a webpack loader, hence, webpack is always installed in a "parent" project.

Update main and types properties in package.json to paths that we will be using for the results of building:

{
	...
	"main": "lib/index.js",
	"types": "index.d.ts",
	...
}

Create tsconfig.json in the root directory of the project with following content:

{
	"compilerOptions": {
		"target": "es2016",
		"module": "commonjs",
		"allowJs": false,
		"outDir": "./lib",
		"esModuleInterop": true,
		"strict": true
	}
}

Note: If you want to use Jest for testing, consider adding the below config to skip building test files, like this:

{
	...
	"include": [
		"**/*"
	],
	"exclude": [
		"**/*.spec.ts",
		"coverage/**/*"
	]
	...
}

Create the entry point for the loader - index.ts - with the following content:

import { validate } from 'schema-utils';
import { LoaderContext } from 'webpack';

import { parseMindMap } from './src/parseMindMap';

interface LoaderProperties {
	test: string;
}

const schema = {
	type: 'object' as 'object', // `as` is for making the type match `JSONSchema7`
	properties: {
		test: {
			type: 'string' as 'string', // `as` is for making the type match `JSONSchema7`
		},
	},
};

// Should match the definition in index.d.ts
export interface PlantUMLMindMapNode {
	id: number; // Order number of the line in the file
	label: string;
	tags: string[];
	children: PlantUMLMindMapNode[];
}

export default function (this: LoaderContext<LoaderProperties>, source: string): string {

	// 1. Get options that can be provided to the loader
	// via webpack loader configuration.
	// Also, validate these options.
	const options = this.getOptions();
	validate(schema, options, {
		name: 'plantuml-mindmap-loader',
		baseDataPath: 'options',
	});

	// 2. Do the core thing of your loader.
	// In our case, we take `source` which is the content of a file being imported
	// and convert it to a JS object, recreating the tree structure
	// defined by PlantUML mind map.
	const result = parseMindMap(source);

	// 3. We return a string that is a valid JS code
	// that will be injected as a result of file loading.
	// In our case, we simply stringify the JSON object.
	return `export default ${JSON.stringify(result)};`;
}

Note: The key thing is the content of the default exported function. Here, we have 3 building blocks. Take a look at the comments in the code for information.

The functionality of converting PlantUML mind map string should be placed in ./src/parseMindMap.ts. Since, it's not the core subject of this article, let me only share the link to public GitHub repository where actual content can be found.
Create types for this plugin in index.d.ts so that the consumers of the loader can use it:

declare module '*.mindmap.puml' {

	// Should match the definition in index.ts
	export interface PlantUMLMindMapNode {
		id: number; // Number of the line in the file
		label: string;
		tags: string[];
		children: PlantUMLMindMapNode[];
	}
	const content: PlantUMLMindMapNode;
	export default content;
}

That should be enough for now. If you're interested in how it might be covered with unit-tests, take a look at the GitHub repo.

Integrate it with your target project

This part is pretty straightforward. All you need to do is a few things:

Install the plugin as a dev dependency (it's needed for building only)

npm install --save-dev ../packages/plantuml-mindmap-loader

Note: This installation is from a local directory. For installing from npm or from another repository, change the package path.

Update webpack.config.js to include this loader:

module.exports = {
	...
	module: {
		rules: [
			...
			{
				test: /\.mindmap.puml$/,
				use: 'plantuml-mindmap-loader'
			},
			...
		]
	},
	...
}

Note 1: The test value includes RegEx that specifies for files of what extension should the loader be used. In our case, we ask webpack to process any encountered **.mindmap.puml with this new loader.

Note 2: The test value should match the module definition in index.d.ts file of the loader. That will ensure that every import is covered with an accompanying type set from module definition.

If the types from the plugin aren't visible for your IDE or webpack complains about missing types for your new imports, update tsconfig.json in the target project to include the type definition like this:

{
	...
	"include": [
		"src/**/*",
		"node_modules/plantuml-mindmap-loader/index.d.ts"
	]
	...
}

Import your custom format file into your code and work with it as if it was a JS object:

import testMindMap from "./test.mindmap.puml";

export function printMindMap() {
	console.log(JSON.stringify(navigationMindMap, null, 2));
}

Results

If everything is done right, for such content of test.mindmap.puml:

@startmindmap

* Root #tag1 #tag2
** Node 1 #tag3
*** Node 1.1
**** Node 1.1.1
** Node 2
*** Node 2.1

@endmindmap

you will get that output in your target project:

{
  "id": 3,
  "label": "Root",
  "tags": [
    "#tag1",
    "#tag2"
  ],
  "children": [
    {
      "id": 4,
      "label": "Node 1",
      "tags": [
        "#tag3"
      ],
      "children": [
        {
          "id": 5,
          "label": "Node 1.1",
          "tags": [],
          "children": [
            {
              "id": 6,
              "label": "Node 1.1.1",
              "tags": [],
              "children": []
            }
          ]
        }
      ]
    },
    {
      "id": 7,
      "label": "Node 2",
      "tags": [],
      "children": [
        {
          "id": 8,
          "label": "Node 2.1",
          "tags": [],
          "children": []
        }
      ]
    }
  ]
}

Result & Beyond

By using webpack loaders in this manner, we can work with any text format in our JS projects. This approach isolates the specifics of parsing distinct text formats to the place that is the most suitable for that: the build process.

Also, the validation of content embedded in a loader helps ensure that the static assets of this type are compatible with the latest version of your code.

But it has at least one significant drawback that worth a mention:

With that approach, you can work with static assets only. If a file should be located in a shared storage and on purpose fetched and processed in runtime, a different approach should be chosen.

Nevertheless, this approach is helpful in many scenarios. That makes almost any static asset functional.

Share other samples of inventive use of custom webpack loaders!

Conclusion

I hope this can serve as a helpful guide when you'll be creating your own custom loaders. It's a powerful tool and it deserves more frequent and inventive use in your projects.

The loader I created is also published to npm.

In future posts, I'll share more info about why I want to make PlantUML mind maps first-class citizens of my codebase. Adding these details here would inflate the article... There will be a day and there will be another story.

Follow me so you don't miss these updates.

Securing Software Development: Integrating InfoSec and Scrum Teams

Thu, 24 Aug 2023 00:00:00 GMT

Introduction

As a manager or a software engineer working with Scrum methodology, you might be very well aware of the underlying principles of an effective workflow that maximises the outcome of efforts by establishing a universal pipeline for feature requests from business to the market and back. But it shouldn't be surprising if not all the parts of your organisation follow the principles of the Agile process and Scrum in particular. Each organisation unit, each part of the overall process should be free to choose the approach that works the best.

In a situation when requirements for a development team using Scrum come from external sources, it's critical to establish correct expectations on input and output of the sub-processes driven by different yet interacting teams.

The key to creating successful process here is understanding that a robust Scrum team operates as a streamlined and effective pipeline, efficiently translating formulated business requests into tangible outcomes. However, when these requests arrive in unfamiliar formats, a disconnect emerges, hindering the realisation of objectives. In the worst cases, that might significantly impact feature delivery (for the whole team) due to prolonged investigation and/or implementation.

In this material, we will consider a specific case of addressing information security-related work in the most effective manner when requests for such work are coming from outside the team. We will set expectations on processing vulnerability reports to seamlessly flow the items to Scrum backlog, facilitating this integration into established operational procedures. Addressing this disparity is essential to enable the team to manage these requests without impeding other items in the backlog.

Vision

The core solution revolves around defining precise communication expectations across multiple stakeholders - InfoSec professionals, IT managers, and the development team itself. This structured approach mirrors the concept of the adapter pattern, transferring its essence from code implementation to real-world procedures.

Clear communication plays a pivotal role, with InfoSec leaders gaining insights into the expected format for business requests, while the development team learns how to communicate changes' outcomes. This mutual understanding empowers InfoSec professionals to give the right initial impulse to effective security remediation work and maintain an updated security state in both internal and external documentation.

We're going to utilise the capabilities of a proficient Scrum team to implement change requests that follow the requirements the team puts to input (in a form of Definition of Ready or a similar principle) by bridging the gap between teams following different approach to work organisation.

Solution

The solution lies in following a comprehensive, step-by-step guide that navigates the journey from a fresh security report to Scrum backlog stories and vice versa. This seamless integration ensures minimal disruption to the team's processes and capitalises on their comfort zone and work habits.

InfoSec team: Initial report analysis. The initial holistic report for one or multiple projects undergoes dissection by an InfoSec specialist, segregating it into independent units of work. This step ensures that vulnerabilities can be prioritised, estimated, and planned independently, fostering several benefits:
- Parallel investigation and resolution of multiple items.
- Prevention of low-criticality issues obstructing more critical tasks.
- Enhanced accuracy in estimating smaller tasks, minimising uncertainty.
InfoSec team: Issue categorisation. Issues are classified into two categories: infrastructure issues and application issues. Infrastructure concerns are directed to DevOps or Ops teams, while application issues progress through the provided guide.
InfoSec team: Story Creation: Drafts for individual vulnerabilities translate into user stories within the chosen task management system. These stories are assigned to a delivery manager or another responsible individual who is familiar with both the system and team processes. Story creation emphasises several key points for the team:
- Background Information: Detailed understanding of the vulnerability's nature, affected components, and contextual conditions.
- Reproducibility: Steps for replicating the issue in a controlled environment, vital for QA and automated testing.
- Description: Recommendations for addressing the problem, suggesting common approaches. When the solution is unclear, the story's objective is to identify a solution.
- Acceptance Criteria: Criteria defining successful vulnerability resolution - what we should see to say that we have achieved the goal.

These stories are systematically labeled, tagged, or grouped for streamlined accessibility.

IT Manager: Oversight. A delivery manager reviews the created stories, ensuring they meet the outlined requirements. Additionally, they assess the need for further task breakdowns, particularly for less atomic requests.
IT Manager: Ensuring System Alignment. The recommendations for vulnerability addressing are evaluated against the system's perspective. If necessary, the selected strategy is adjusted to align with the system's values and patterns. Examples include:
- Addressing verbose error messages in a mobile app by updating the backend API.
- Consolidating updates to a dependency by upgrading to the current version during the current process instead of the minimal recommended one.

The delivery manager ensures this alignment is communicated effectively to the product owner and team, with information enabling the identification of issues throughout the application.

IT Manager. Assignment and Monitoring. Stories are assigned to releases based on vulnerability criticality, and tasks are allocated to corresponding teams. A dashboard or filter that presents tagged vulnerability remediation tasks is required at this stage, providing visibility on statuses and release timelines. This process allows for real-time monitoring and adjustments for both IT manager and InfoSec team.

Result & Beyond

The structured procedure ensures that stakeholders can operate within their preferred formats and methodologies while seamlessly integrating security concerns. This approach minimises the impact on the team's ongoing plans, as security issues are addressed with minimal disruption.

This methodology also minimises the risk of unexpected challenges when addressing vulnerabilities. By tailoring recommendations to the project situation, the likelihood of encountering unforeseen obstacles is significantly decreased.

The InfoSec team benefits from real-time visibility into project plans and progress, enabling swift reactions to deprioritisation and successful deliveries as their capacity allows.

While the process is comprehensive, avenues for improvement remain:

Educating InfoSec team and other peers about Scrum methodologies fosters better story creation.
Educating the Scrum team on InfoSec team's priorities enhances understanding of the requests and more accurate issue addressing.
Involvement of senior developers in processing stories streamlines the process and helps scale the process when the vulnerability reports cover multiple projects.
Measuring lead time of the security-related requests in your projects must lead to well-informed decision on claimed remediation time for security vulnerabilities that might lead to establishing a vulnerability remediation SLA if it hasn't happened yet.

Conclusion

Within our organisation, this process has been helpful in rapidly addressing security concerns while maintaining focus on project innovation. We welcome your thoughts on this approach, proposed enhancements, and any additional recommendations or queries.

Let's save time for beautiful innovations through using well-optimised processes for routine tasks!

Enabling Local Editing for MkDocs Documentation with IDE

Sat, 19 Aug 2023 00:00:00 GMT

Introduction

There are a million good reasons to start writing documentation. Documentation is the key to preserving acquired knowledge indefinitely - for others and for your future self (who might be quite different if things are going well). As essential tool as documentation must promote the idea of keeping it up-to-date by ensuring easy access to editing pages, so that including more details to existing information and putting focus on right things is simple.

While some documentation management systems like Confluence, Notion and others allow in-place content editing, systems with serialisable content stored separately, such as MkDocs, it's not that straightforward. In this article, we will discuss MkDocs that offers an out-of-the-box solution for that.

MkDocs has the ability to enhance pages with an "Edit" button that allows opening the source code of a specific page in Git repository. However, most guides (including official documentation) focus on scenarios involving cloud-based Git repositories. That's a quite cool feature, especially, if documentation maintenance is public but that's not as helpful for maintainers working on pages locally.

In this material, let's explore how to make "Edit" button open pages in modern IDEs using custom URI schemes, allowing for a seamless transition from viewing to editing.

Vision

The objective is simple: equip every page in a MkDocs Material generated documentation site with an "Edit" button that opens the corresponding source markdown file in VS Code.

Solution

Understanding the Configuration: Familiarise yourself with the edit_uri configuration option in the MkDocs documentation.
Unveiling VS Code's Power: Explore the documentation on launching VS Code using custom URLs. This knowledge will empower our "Edit" button to spring into action.
Crafting the Base URL: Create a base URL that directs users to the directory housing your documentation. For instance:

vscode://file/${FULL_PATH_TO_CONTENT_DIRECTORY}

Note: If you're an VS Code Insiders user, kick off the URL with vscode-insiders://.
Configuring edit_uri: Modify your mkdocs.yml file, setting the edit_uri option to your created URL:

edit_uri: vscode-insiders://file/Users/me/Development/my-project/documentation/content/.

Note: If you aren't integrating a Git link, you can omit the optional repo_url setting.
Enabling the "Edit" Button: Activate the "Edit" button feature within MkDocs Material by updating your mkdocs.yml as follows:
```
 theme:
 	features:
 		- content.action.edit
```
Final Touch: Now, perform the documentation build process.

Results and Beyond

With these adjustments, every page of your documentation will get an "Edit" button, making the source file open in VS Code. If this solution helps you in your workflow, consider potential expansions:

Conditional URI Setup: Leverage environment variables to configure the URI dynamically. For local use, it fires up VS Code; for hosting, it points to the Git repository.
Beyond VS Code: If your preferred IDE differs from VS Code, explore available URI schemes that align with your choice.
Theme Compatibility: If you're not using the Material theme, verify whether your theme supports the "Edit" button feature.

Conclusion

This approach streamlines local documentation edits while leaving room for enjoying well-organised navigation between built documentation pages. Join the discussion in the comments if you have insights to share about the potential expansions or the overall concept.

Let's document effectively with minimal effort!