Mkdocs for Projects
Documenting with multiple projects threaded together to support teams and their efforts.
After writing many types of documentation and experienced opinions from peers and management, I have started down my path of experimenting with a static site generator which has the goal of making documentation. Throughout my career I have written many documents for helping my peers have a reference for what I have changed or observed to be set in place. I have also use multiple types of formats: MoinMoin, Word, Confluence, SharePoint, MarkDown files. These are all okay for their individual needs, but I have never been really satisfied with any of them.
There are two problems that I want to solve:
- Improve searching for content
- Git based PRs include documentation updates
Observations
Currently, there are many projects that exist in git. Some of them have lots of docs in MarkDown while others have less, and most have none. We are encouraged to use Confluence as much as possible to post information for others to consume. But here are some observations:
- Most of the Developers or Engineers do not like to edit on Confluence
- Most Project Managers, meeting notes, or non-dev/eng love Confluence
This is very interesting to me. While I completly support and recognize the need for Confluence to empower people to make simple notes and tables, I have to take notice of there being [two groups of people][#groups] and the other is a pattern of changing code.
Groups of people
Most technology-based projects have three major groups of people who are involved with the product(s). One is the creator/maintainer of the product(s), the second is the user/consumer of the product. And the third is the Product Owner who represents the high-level needs between the user/consumer and the creator/maintainer. Lets go with the standard terminology and use: Engineers, Users, and Owners. I suggest that the documents for each of the groups is not entirely the same.
What if the documentation that is intended for the Users that overlapped with information shared with the Engineers were generated by a static site generator? Okay maybe I lost you little. Lets look at three, very basic, types of information for a project:
- Login URL
This piece of information is vital to an application. It also overlaps in both groups of people being the Users and the Engineers. When this information changes (most likely it won’t…but we all know that’s not a good thing to say as things nothing stays the same) there will be more login urls on the Engineering side than the Users side. But they do overlap.
- Features/Expectations
As an application grows the features change with each release. Yah I know that’s super obvious. Engineers are going to make these feature changes and they would also know best how to describe the use of these features. When the PR of the version is merged in to the default branch the documentation should be carried with the change.
- Sell me
Why should I use this product(s) instead of something else? Okay, I like what you’re selling. Will I have to work really hard to get signed up? Any new features coming soon? All of these are about the Product Owner and they sell the product. Is there any overlap of content from the above? Yes and No. This content is meant to sell to Users. A high level of why the produc(s) exist and a timeline of what to expect. This is the only category where using a wiki makes sense. More on this down below.
Why did I mention these attributes of a project? Glad you asked, because I see these categories of information have their place for being visible to different groups of people. Lets go over the attributes again, but this time assign the contributors and the consumers of the content:
Attribute | Contributor | Consumer | Where it belongs |
---|---|---|---|
Login urls | Engineers | Engineers/User | Git - ./docs/resources.md, Wiki |
Features/Expectations | Engineers | Engineers/User | Git - ./README.md, Wiki |
Secrets/Accounts/Operations (etc..) | Engineers | Engineers | Git - ./docs/operations/*.md |
Architecture/Design (detailed diagrams) | Engineers | Engineers | Git - ./docs/design.md |
Sell me (including quick diagrams) | Product Owner | User | Git - ./README.md, Wiki (optional) |
Yes, thats right, I am saying that documentation is best served in Git and
if helpful, to also publish to the Wiki. The last attribute
of the Sell me is in the README.md
. Bonus points for publishing to Wiki . The reason for doing
this is to allow the publishing of features for a product by the Product
Owner.
At the beginning and end of a change, the things that change the most are done by the Engineers. When this happens, the documentation that changes the most is the information about the application, how to support it, how it works, and diagrams for the teams who will be supporting it. This information might reach the end-users, but its not needed.
The “sell me” attribute is for the consumer to
“explain to them: Why do I need this?”
and if they want to see more about the project, then point them to the in-depth docs located on Git for more information about the feature/url/testing/what-ever.
Changing code
(lets go back to the creator/engineer)
The other observation(s) (that has grown over the years) goes like this:
- You read more code than you write.
- You read logs more than you read code.
- You read timestamps more than you read logs.
- You read docs more than you read logs or timestamps.
After you have created code, I could probably assert that the amount of times you have re-read code is way more beyond the amount of times you write code. In order to make good/improved choices on making changes, you should be reading well structured logs in order to know where your change should be placed. Along with well structured logs, I could assert that you look at the timestamp of the logs very often, if not even more than you read logs. At this point you should be looking at documentation for information on what you can expect for the state of the application.
What does this all really mean? What does it matter?
To me, this is one way to measure a well described application where making a change is really easy if we just reverse the order from above:
- Read the documenation
- Read the timestamped logs for where you want to make a change
- Read the code that is associated with the logs
- Write the code you want to change
This process usually is trimmed down during normal work hours and causes for time lost on a daily (perhaps hourly) basis.
Bashing Confluence
Next observation I have comes down pretty harsh on the wiki product named Confluence and that its not meant for all users. Please know that I have used other types of wiki in the past (moinmoin, MediaWiki, RedMine, GitHub Enterprise (wiki) and they all have their weeknesses, but the one I don’t like right now is: Confluence
Editing in Confluence is horrible. Lets break this down into sub-topics.
Versions
Yes, its great you can edit the same document with someone else. But there is a huge negative cost to this when you look at trying to restore your previous version which ends up wiping out someone else’s change. If this were to happen in Software changes, we would have a really hard time making good progress.
HTML
The source code of a page is beyond words of disgust. Using {code}
and <data>
everywhere makes for a
massive insult to doing any kind of decent coding of content. This makes it really difficult to have
confidence that what is being rendered is actually clean and not full of silly left over spaces that
have no style or are just blank. Left over things make for a very confused and frustrated editor.
Editor
I recognize the toolbar of Confluence covers the basics. The issue is if you are not careful with placement of text, you might be surprized by some change in formatting that just ends up getting deleted by accident because the editor can not tell which way is up or down of the style its currently interpreting.
MkDocs
Enter MkDocs.
Recently I have been using MkDocs as a POC for generating documenation. This has been receving lots of good positive feedback for not only it having a good user interface, but the guidelines seem to also be accepted as well.
Here are my reasons for using MkDocs:
- Straight forward MarkDown files
- Git controlled
- Plugins in your control
- MarkDown is easy
- Still renders in GitHub
- Support for diagrams with text
- Localized search to project(s)
- Extendable with Python
Method A
Super simple, follow their instructions for one repository in github (also sprinkle in some of my own tools of choice) :
cd to/best/project/ever
# select python version
pyenv local 3.8.5
# install mkdocs
pipenv install mkdocs
# initialize mkdocs
pipenv run mkdocs new .
# start server
pipenv run mkdocs serve
This gets you going with a very basic version of MkDocs which you can preview at http://localhost:8000 and will
be updated on most content changes. Every now and then I have found myself having to stop the server if I change
the config file of mkdocs.yml
or if a file is not found.
The docs for MkDocs is pretty good but I will share some info here as well. The basic structure
is a config file called mkdocs.yml
and a ./docs
directory with files that contain your content.
If you have an existing README.md
file, I suggest moving most of its content to files
in the ./docs
directory.
Creating content is pretty simple as the process is:
- create a file in
./docs/goodtopic.md
- update the nav section in
mkdocs.yml
(use example below)
site_name: My Cool Docs
nav:
- home: index.md
- 'Good Topic': goodtopic.md
A more complex configuration would look like the following which includes a really nice theme:
# install theme
pipenv install mkdocs-material plantuml-markdown mkdocs-exclude-search
then make a really nice long mkdocs.yml file with
site_name: "Awesome Site Name"
repo_url: https://github.com/your/url
repo_name: Awesome Site Name
nav:
- Home:
- Index: index.md
- Operations: operations.md
- Tools: tools.md
plugins:
- search
- exclude-search:
exclude:
- projects/*
theme:
name: material
features:
- toc.integrate
- navigation.instant
- navigation.tabs
- navigation.sections
- navigation.top
- search.suggest
- header.autohide
palette:
- scheme: default
toggle:
icon: material/toggle-switch-off-outline
name: Switch to dark mode
- scheme: slate
toggle:
icon: material/toggle-switch
name: Switch to light mode
markdown_extensions:
- pymdownx.tasklist:
custom_checkbox: true
- pymdownx.superfences
- pymdownx.details
- admonition
- pymdownx.highlight:
linenums: true
- pymdownx.snippets
- abbr
- def_list
- toc:
permalink: true
- plantuml_markdown:
server: http://www.plantuml.com/plantuml
The README.md and how to move
With regards to MkDocs, I have not had success in bringing in the README.md
file into
the content. If you try to point the mkdocs.yml
file to the ./README.md
it gets confused.
This is okay because when you decide to move the docs to MkDocs, you should
really just take the README.md
content and move it over to ./docs/index.md
or a split up
some of the content to some additional files that matches the kind of content that is needed.
At this time, I have found the following files usually cover a good amount of information for most projects but each one will have its own requirements:
./docs/index.md
./docs/quickstart.md
./docs/accounts.md
./docs/design.md
./docs/secrets.md
./docs/media/img.jpg
But do not get rid of the README.md
as it is still very useful to provide a Welcome to the project
with a Summary, Quick Start, Additional Docs.
One repo, multiple projects
When you have multiple projects inside of a Git repo, the following options are possible:
projects are not related:
I would put multiple mkdocs.yml
and ./docs
files and directories under their project folder
and manage the static content separately. This is the most straight forward approach to keeping
the content separate and assuming there is no links required between the projects.
projects are related:
Keep the mkdocs.yml
and ./docs
at the root level of the git repo and split up the content
as needed by the project. Ultimatly the content and project files are all related and the docs
should reflect the relationship in the same mannor.
Method B
I came across the project mkdocs-monorepo
which allows the inclusing of other mkdocs.yml
files. Actually, I think the only content
that is being inluded is the Name
and the nav
of the YML file. But this is the plugin
that is making my first version possible. By grouping projects around a team, you have the
content localized for searching.
Here is one way to implement:
- Create your main repo
- Create a directory called
./projects
- Use git submodule to add projects:
git submodule add https://github.com/aaronaddleman/newproject ./projects/newproject
- (repeat for additional projects
- Install
mkdocs
for main repopyenv local 3.8.5
pipenv install mkdocs mkdocs-material mkdocs-monorepo-plugin
pipenv run mkdocs new .
- Update your
mkdocs.yml
file to use the example below - Repeat the following steps for your git repos in
./projects
cd ./projects/newproject
mkdocs new .
- create content in
./docs/goodtopics.md
- update
./projects/newproject/mkdocs.yml
file with nav entries - update
./mkdocs
nav with includes like the example below
nav:
- Home:
- Index: index.md
- Operations: operations.md
- Tools: tools.md
- Projects:
- project1:
- application: '!include projects/p1a/mkdocs.yml'
- infrastructure: '!include projects/p1b/mkdocs.yml'
Conclusion
Why do this? It all comes down to change. There are many ways to maintain documentation and most of the time is forgetten. Aligning documentation to be as close as possible to the group of people that know the most about the product is very helpful. Way better than asking them to repeat words (possibly twice or even three times) onto a wiki when the source code already exists.
When we have the right documentation, we understand expectation. After we have an understanding of expectation, we have words and pictures (hopefully) of how thigs work and locations of logs. Going to the logs to find out what is happening, we gain clarity of what code is being used. Reading the code, we understand what needs to change. Then we can
- make a change in the code with a new feature
- update the logs to show when the feature is used
- deploy the feature as needed
- modify the documentation to reflect releases
What I did not cover
What I did not cover is the practice of maintaining multiple versions of documents. I have yet to cross this bridge with most of my projects as it has not been needed. I have confidence in you to decide on how to implement this if you got this far in this post.
Good Luck.