Mkdocs for Projects

After writing many types of documentation and experienced opinions from peers and management, I have started down my path of experimenting with a static site generator which has the goal of making documentation. Throughout my career I have written many documents for helping my peers have a reference for what I have changed or observed to be set in place. I have also use multiple types of formats: MoinMoin, Word, Confluence, SharePoint, MarkDown files. These are all okay for their individual needs, but I have never been really satisfied with any of them.

There are two problems that I want to solve:

1. Improve searching for content
2. Git based PRs include documentation updates

Observations

Currently, there are many projects that exist in git. Some of them have lots of docs in MarkDown while others have less, and most have none. We are encouraged to use Confluence as much as possible to post information for others to consume. But here are some observations:

1. Most of the Developers or Engineers do not like to edit on Confluence
2. Most Project Managers, meeting notes, or non-dev/eng love Confluence

This is very interesting to me. While I completly support and recognize the need for Confluence to empower people to make simple notes and tables, I have to take notice of there being [two groups of people][#groups] and the other is a pattern of changing code.

Groups of people

Most technology-based projects have three major groups of people who are involved with the product(s). One is the creator/maintainer of the product(s), the second is the user/consumer of the product. And the third is the Product Owner who represents the high-level needs between the user/consumer and the creator/maintainer. Lets go with the standard terminology and use: Engineers, Users, and Owners. I suggest that the documents for each of the groups is not entirely the same.

What if the documentation that is intended for the Users that overlapped with information shared with the Engineers were generated by a static site generator? Okay maybe I lost you little. Lets look at three, very basic, types of information for a project:

• Login URL

This piece of information is vital to an application. It also overlaps in both groups of people being the Users and the Engineers. When this information changes (most likely it won’t…but we all know that’s not a good thing to say as things nothing stays the same) there will be more login urls on the Engineering side than the Users side. But they do overlap.

• Features/Expectations

As an application grows the features change with each release. Yah I know that’s super obvious. Engineers are going to make these feature changes and they would also know best how to describe the use of these features. When the PR of the version is merged in to the default branch the documentation should be carried with the change.

• Sell me

Why should I use this product(s) instead of something else? Okay, I like what you’re selling. Will I have to work really hard to get signed up? Any new features coming soon? All of these are about the Product Owner and they sell the product. Is there any overlap of content from the above? Yes and No. This content is meant to sell to Users. A high level of why the produc(s) exist and a timeline of what to expect. This is the only category where using a wiki makes sense. More on this down below.

Why did I mention these attributes of a project? Glad you asked, because I see these categories of information have their place for being visible to different groups of people. Lets go over the attributes again, but this time assign the contributors and the consumers of the content:

Yes, thats right, I am saying that documentation is best served in Git and if helpful, to also publish to the Wiki. The last attribute of the Sell me is in the README.md. Bonus points for publishing to Wiki . The reason for doing this is to allow the publishing of features for a product by the Product Owner.

At the beginning and end of a change, the things that change the most are done by the Engineers. When this happens, the documentation that changes the most is the information about the application, how to support it, how it works, and diagrams for the teams who will be supporting it. This information might reach the end-users, but its not needed.

The “sell me” attribute is for the consumer to

“explain to them: Why do I need this?”

and if they want to see more about the project, then point them to the in-depth docs located on Git for more information about the feature/url/testing/what-ever.

Changing code

(lets go back to the creator/engineer)

The other observation(s) (that has grown over the years) goes like this:

• You read more code than you write.
• You read logs more than you read code.
• You read timestamps more than you read logs.
• You read docs more than you read logs or timestamps.

After you have created code, I could probably assert that the amount of times you have re-read code is way more beyond the amount of times you write code. In order to make good/improved choices on making changes, you should be reading well structured logs in order to know where your change should be placed. Along with well structured logs, I could assert that you look at the timestamp of the logs very often, if not even more than you read logs. At this point you should be looking at documentation for information on what you can expect for the state of the application.

What does this all really mean? What does it matter?

To me, this is one way to measure a well described application where making a change is really easy if we just reverse the order from above:

• Read the documenation
• Read the timestamped logs for where you want to make a change
• Read the code that is associated with the logs
• Write the code you want to change

This process usually is trimmed down during normal work hours and causes for time lost on a daily (perhaps hourly) basis.

Bashing Confluence

Next observation I have comes down pretty harsh on the wiki product named Confluence and that its not meant for all users. Please know that I have used other types of wiki in the past (moinmoin, MediaWiki, RedMine, GitHub Enterprise (wiki) and they all have their weeknesses, but the one I don’t like right now is: Confluence

Editing in Confluence is horrible. Lets break this down into sub-topics.

Versions

Yes, its great you can edit the same document with someone else. But there is a huge negative cost to this when you look at trying to restore your previous version which ends up wiping out someone else’s change. If this were to happen in Software changes, we would have a really hard time making good progress.

HTML

The source code of a page is beyond words of disgust. Using {code} and <data> everywhere makes for a massive insult to doing any kind of decent coding of content. This makes it really difficult to have confidence that what is being rendered is actually clean and not full of silly left over spaces that have no style or are just blank. Left over things make for a very confused and frustrated editor.

Editor

I recognize the toolbar of Confluence covers the basics. The issue is if you are not careful with placement of text, you might be surprized by some change in formatting that just ends up getting deleted by accident because the editor can not tell which way is up or down of the style its currently interpreting.

MkDocs

Enter MkDocs.

Recently I have been using MkDocs as a POC for generating documenation. This has been receving lots of good positive feedback for not only it having a good user interface, but the guidelines seem to also be accepted as well.

Here are my reasons for using MkDocs:

1. Straight forward MarkDown files
2. Git controlled
3. Plugins in your control
4. MarkDown is easy
5. Still renders in GitHub
6. Support for diagrams with text
7. Localized search to project(s)
8. Extendable with Python

Method A

Super simple, follow their instructions for one repository in github (also sprinkle in some of my own tools of choice) :

cd to/best/project/ever
# select python version
pyenv local 3.8.5
# install mkdocs
pipenv install mkdocs
# initialize mkdocs
pipenv run mkdocs new .
# start server
pipenv run mkdocs serve

This gets you going with a very basic version of MkDocs which you can preview at http://localhost:8000 and will be updated on most content changes. Every now and then I have found myself having to stop the server if I change the config file of mkdocs.yml or if a file is not found.

The docs for MkDocs is pretty good but I will share some info here as well. The basic structure is a config file called mkdocs.yml and a ./docs directory with files that contain your content.

If you have an existing README.md file, I suggest moving most of its content to files in the ./docs directory.

Creating content is pretty simple as the process is:

1. create a file in ./docs/goodtopic.md
2. update the nav section in mkdocs.yml (use example below)

site_name: My Cool Docs
nav:
  - home: index.md
  - 'Good Topic': goodtopic.md

A more complex configuration would look like the following which includes a really nice theme:

# install theme
pipenv install mkdocs-material plantuml-markdown mkdocs-exclude-search

then make a really nice long mkdocs.yml file with

site_name: "Awesome Site Name"
repo_url: https://github.com/your/url
repo_name: Awesome Site Name
nav:
  - Home:
    - Index: index.md
    - Operations: operations.md
    - Tools: tools.md
plugins:
  - search
  - exclude-search:
      exclude:
        - projects/*
theme:
  name: material
  features:
    - toc.integrate
    - navigation.instant
    - navigation.tabs
    - navigation.sections
    - navigation.top
    - search.suggest
    - header.autohide
  palette:
    - scheme: default
      toggle:
        icon: material/toggle-switch-off-outline
        name: Switch to dark mode
    - scheme: slate
      toggle:
        icon: material/toggle-switch
        name: Switch to light mode
markdown_extensions:
  - pymdownx.tasklist:
      custom_checkbox: true
  - pymdownx.superfences
  - pymdownx.details
  - admonition
  - pymdownx.highlight:
      linenums: true
  - pymdownx.snippets
  - abbr
  - def_list
  - toc:
      permalink: true
  - plantuml_markdown:
      server: http://www.plantuml.com/plantuml

The README.md and how to move

With regards to MkDocs, I have not had success in bringing in the README.md file into the content. If you try to point the mkdocs.yml file to the ./README.md it gets confused. This is okay because when you decide to move the docs to MkDocs, you should really just take the README.md content and move it over to ./docs/index.md or a split up some of the content to some additional files that matches the kind of content that is needed.

At this time, I have found the following files usually cover a good amount of information for most projects but each one will have its own requirements:

./docs/index.md
./docs/quickstart.md
./docs/accounts.md
./docs/design.md
./docs/secrets.md
./docs/media/img.jpg

But do not get rid of the README.md as it is still very useful to provide a Welcome to the project with a Summary, Quick Start, Additional Docs.

One repo, multiple projects

When you have multiple projects inside of a Git repo, the following options are possible:

projects are not related:

I would put multiple mkdocs.yml and ./docs files and directories under their project folder and manage the static content separately. This is the most straight forward approach to keeping the content separate and assuming there is no links required between the projects.

projects are related:

Keep the mkdocs.yml and ./docs at the root level of the git repo and split up the content as needed by the project. Ultimatly the content and project files are all related and the docs should reflect the relationship in the same mannor.

Method B

I came across the project mkdocs-monorepo which allows the inclusing of other mkdocs.yml files. Actually, I think the only content that is being inluded is the Name and the nav of the YML file. But this is the plugin that is making my first version possible. By grouping projects around a team, you have the content localized for searching.

Here is one way to implement:

1. Create your main repo
2. Create a directory called ./projects
3. Use git submodule to add projects:
   • git submodule add https://github.com/aaronaddleman/newproject ./projects/newproject
   • (repeat for additional projects
4. Install mkdocs for main repo
   • pyenv local 3.8.5
   • pipenv install mkdocs mkdocs-material mkdocs-monorepo-plugin
   • pipenv run mkdocs new .
5. Update your mkdocs.yml file to use the example below
6. Repeat the following steps for your git repos in ./projects
   • cd ./projects/newproject
   • mkdocs new .
   • create content in ./docs/goodtopics.md
   • update ./projects/newproject/mkdocs.yml file with nav entries
   • update ./mkdocs nav with includes like the example below

nav:
  - Home:
    - Index: index.md
    - Operations: operations.md
    - Tools: tools.md
    - Projects:
        - project1:
            - application: '!include projects/p1a/mkdocs.yml'
            - infrastructure: '!include projects/p1b/mkdocs.yml'

Conclusion

Why do this? It all comes down to change. There are many ways to maintain documentation and most of the time is forgetten. Aligning documentation to be as close as possible to the group of people that know the most about the product is very helpful. Way better than asking them to repeat words (possibly twice or even three times) onto a wiki when the source code already exists.

When we have the right documentation, we understand expectation. After we have an understanding of expectation, we have words and pictures (hopefully) of how thigs work and locations of logs. Going to the logs to find out what is happening, we gain clarity of what code is being used. Reading the code, we understand what needs to change. Then we can

1. make a change in the code with a new feature
2. update the logs to show when the feature is used
3. deploy the feature as needed
4. modify the documentation to reflect releases

What I did not cover

What I did not cover is the practice of maintaining multiple versions of documents. I have yet to cross this bridge with most of my projects as it has not been needed. I have confidence in you to decide on how to implement this if you got this far in this post.

Good Luck.