Skip to content

Instantly share code, notes, and snippets.

@mvandermeulen
Forked from sharadhr/home_nss_home.md
Created January 1, 2024 06:19
Show Gist options
  • Select an option

  • Save mvandermeulen/4a8edd8ef78e88eedd8ca0131fa2d45f to your computer and use it in GitHub Desktop.

Select an option

Save mvandermeulen/4a8edd8ef78e88eedd8ca0131fa2d45f to your computer and use it in GitHub Desktop.

Revisions

  1. @sharadhr sharadhr revised this gist Aug 16, 2023. 1 changed file with 14 additions and 13 deletions.
    27 changes: 14 additions & 13 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -85,8 +85,8 @@ Some applications pretend that everything is immaculate, by *hiding* their mess
    It would irritate *anyone*.
    I am *particularly* compulsive about software doing what I'd like it to do, and about following platform conventions.
    Conventions, standards, and protocols exist so everyone is on the same page, and there is a common 'language' for software to communicate with.
    When software authors break them just for the sake of 'opinionation', or because one *feels like it* (a worse justification, in my opinion), it only leads to much exasperation on the users' end, because it comes as a surprise to the user, and software should *not* be surprising.
    It ought to be reliable, reproducible, and follow platform conventions and expectations.
    When software authors break them just for the sake of 'opinionation', or because they *feel like it* (a worse justification, in my opinion), it only leads to much exasperation on the users' end, because it comes as a surprise to them, and software should *not* be surprising.
    It ought to be reliable, reproducible, and follow expectations.

    This post is a detailed discussion into user profiles, their directories, and how they are—to put it bluntly—in total disarray on Windows and Linux (I haven't used a Mac in ages, but I assume the situation is very similar there, too).
    Applications treat the user profile as a dumping ground, and any user with a reasonably wide list of installed software will find their user profile very difficult to traverse after some time in use.
    @@ -262,8 +262,8 @@ The Windows API provides enumerated GUIDs called [`KNOWNFOLDERID`](https://learn
    These may be queried with [`SHGetKnownFolderPath`](https://learn.microsoft.com/en-sg/windows/win32/api/shlobj_core/nf-shlobj_core-shgetknownfolderpath), like so:

    ```c++
    #include <ShlObj.h> // SHGetKnownFolderPath
    #include <cstdio>
    #include <ShlObj.h> // SHGetKnownFolderPath

    #pragma comment(lib, "shell32.lib") // link shell32.lib

    @@ -296,17 +296,17 @@ Hereafter, I will use the [Windows shell shortcuts](https://ss64.com/nt/shell.ht
    ##### 2.2.3.1&emsp;Hidden folders on Windows

    In particular, the parent directory of `shell:AppData` and `shell:Local AppData``%USERPROFILE%\AppData`—is a *hidden* directory.
    Unlike \*nix, 'hidden' on Windows has special semantics, accorded by the filesystem (usually NTFS).
    This can be set either graphically (right click → `Properties``General` tab → `Hidden` checkbox under `Attributes`), or retrieved/set programmatically in the Windows API, using [`GetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw) and [`SetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw).
    These functions return or accept a bitwise-ORed [file attribute constant](https://learn.microsoft.com/en-sg/windows/win32/fileio/file-attribute-constants).
    Unlike \*nix, 'hidden' on Windows has special semantics accorded by the filesystem (usually NTFS).
    This can be set either graphically (right-click → `Properties``General` tab → `Hidden` checkbox under `Attributes`), or retrieved/set programmatically in the Windows API, using [`GetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw) and [`SetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw).
    These functions return or accept a bitwise-ORed [file attribute constant](https://learn.microsoft.com/en-sg/windows/win32/fileio/file-attribute-constants), as demonstrated below:

    ```c++
    #include <cstdio>
    #include <ShlObj.h>
    #include <pathcch.h>
    #include <ShlObj.h>

    #pragma comment(lib, "Shell32.lib")
    #pragma comment(lib, "Pathcch.lib")
    #pragma comment(lib, "Shell32.lib")

    auto main() -> int
    {
    @@ -344,7 +344,7 @@ auto main() -> int
    Windows 2000 and XP used `%SYSTEMDRIVE%\Documents and Settings\<username>` for the home directory, which was moved to `%SYSTEMDRIVE%\Users\<username>` with Vista and later.
    Windows Vista also introduced the above-mentioned `KNOWNFOLDERID`s, which superseded the older [`CSIDL`](https://learn.microsoft.com/en-sg/windows/win32/shell/csidl) enumeration (although the latter is still available, and used by the .NET API).

    Many environment variables like `%APPDATA%` also were changed to point to the new locations (the previous was a mouthful: `%SYSTEMDRIVE%\Documents and Settings\<username>\Local Settings\Application Data`).
    Many environment variables like `%APPDATA%` were also redirected to the current locations (the previous was a mouthful: `%SYSTEMDRIVE%\Documents and Settings\<username>\Local Settings\Application Data`).
    New known folders were added, such as `Downloads` and `Saved Games`; the `My⎵` prefix was removed from `My Documents` and `My Music`; locations such as `Start Menu` were moved (in this case, to `%APPDATA%\Microsoft\Windows\Start Menu`).

    The reason for this move is anyone's guess, but I feel the $\geq$ Vista convention makes a lot more sense than the $\leq$ Windows XP one.
    @@ -547,7 +547,7 @@ The rest (excluding the XDG directories) were either auto-generated by `xdg-user
    The link above discussing the origin of dot-files makes it clear that the current behaviour was possibly *a mistake* made by the original developers of UNIX.
    Even if it wasn't, the result today is a messy litter of dot-files dumped all over a user's home directory.
    The XDG Base Directory (hereafter, XBD) specification is more than a decade old—an eternity in computing terms—and yet, there is a veritable parade of very well-known programs that refuse to follow its guidelines: see the quotes above in [3.2](#32screw-your-conventions-weve-always-done-it-this-way).
    The XDG Base Directory (hereafter, XBD) specification is more than a decade old—an eternity in computing terms—and yet, there is a veritable parade of very well-known programs that refuse to follow its guidelines: see the quotes above in [§3.3](#33screw-your-conventions-weve-always-done-it-this-way).
    The Arch Wiki has [a list of programs](https://wiki.archlinux.org/title/XDG_Base_Directory#Support) that are XBD-compliant by default, may be forced to comply after user intervention, and those with hard-coded non-XBD paths.
    The latter two lists combined is almost *twice* as long as the compliant list, and includes some very prominent \*nix-first software like Bash, Vim, OpenSSH, and Firefox.
    @@ -619,11 +619,12 @@ This is frequently violated by software written for \*nix first—notice `.git`,
    Developers either assume Windows operates the same as \*nix, or don't care about Windows much ('second-class citizen'), and again, couch any changes to fix this as 'introducing unnecessary complexity'.
    Honestly, one would instead think the change is fairly straightforward: test which OS the program is running on using `#ifdef`s and OS-specific macros, and delegate to the appropriate function.
    Better still, use the ecosystem's build system to configure the build setup for different platforms, and shift this test to configure and compile-time instead of run-time.
    [Here's a wiki](https://sourceforge.net/p/predef/wiki/OperatingSystems/) listing some of these macros.
    Better still, use the ecosystem's build system to configure builds for different platforms, and shift this test to configure and compile-time instead of run-time.

    Some semi-compliant software hard-codes library paths, assuming user libraries will remain in the default locations, i.e. sub-directories of `%USERPROFILE%`.
    This is *not* true, and users can change their locations as mentioned above. Notice there's a `Contacts` directory in the listing above: this was created by [KDE Connect](https://kdeconnect.kde.org/).
    This is *not* true, and users can change their locations as mentioned above.
    Notice there's a `Contacts` directory in the listing above: this was created by [KDE Connect](https://kdeconnect.kde.org/).
    That said, not *all* the above dot-directories are created by traditionally *nix programs, which is a good segue to the next section...
    @@ -709,7 +710,7 @@ enc_cert.pfx
    mods.settings
    ```
    *Anno 1404*, what are you doing? *Six* different (localised) directories, all containing the same paths.
    *Anno 1404*, what the hell are you doing? *Six* different (localised) directories, all containing the same paths.
    Honestly speaking, I doubt game developers are going to change to `Saved Games`, and this is partially Microsoft's fault, again.
    The folder is neither listed in the default File Explorer `Libraries` view, nor in the `My PC` view; users have to manually navigate to it.
  2. @sharadhr sharadhr revised this gist Aug 16, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -35,11 +35,11 @@

    <div align="justify">

    ## Preface

    <details>
    <summary>Preface</summary>

    ## Preface

    This was supposed to be a blog post, but I have neither the knowledge, nor the time, nor the energy to set up a nice statically-generated blog like everyone else does on [Hacker News](https://news.ycombinator.com) or [proggit](https://www.reddit.com/r/programming).
    Of course, I *want* one, but I also want to use a fully .NET Core-based static site generator, and therefore have experimented with [Statiq.Dev Web](https://www.statiq.dev/web).
    However, it turns out that I still need to pick up non-trivial HTML and CSS to make it look like anything but a website from the 1990s.
  3. @sharadhr sharadhr revised this gist Aug 16, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -751,7 +751,7 @@ I'd like a nice vacuum cleaner for my home directory, please.
    ## 6&emsp;Relevant blog posts
    Here are several blog posts and articles that inspired me and were useful for my post, some previously linked above:
    Here are several blog posts and articles that inspired me and were useful for my postsome of which were linked above:
    - [*Dotfile madness*](https://0x46.net/thoughts/2019/02/01/dotfile-madness/)
    - [*Everything that uses configuration files should report where they're located*](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/ReportConfigFileLocations)
  4. @sharadhr sharadhr revised this gist Aug 16, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -745,13 +745,13 @@ The specification is clear, some developers clearly know about it, and it is a g
    ### 5.1&emsp;Home, sweet home
    We have so many real-life analogues in our computers: from files, folders, and rubbish bins, to the 'desktop' metaphor.
    Surely we can extend the concepts of 'clutter' and 'cleanup', and apply it to our digital lives too?
    Surely we can extend the concepts of 'clutter' and 'cleanup', too?
    Let's be good citizens of the platforms we develop for, and give our users choices and control over their data, and where it is stored.
    I'd like a nice vacuum cleaner for my home directory, please.
    ## 6&emsp;Relevant blog posts
    Here are several blog posts and articles that inspired me, some previously linked above:
    Here are several blog posts and articles that inspired me and were useful for my post, some previously linked above:
    - [*Dotfile madness*](https://0x46.net/thoughts/2019/02/01/dotfile-madness/)
    - [*Everything that uses configuration files should report where they're located*](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/ReportConfigFileLocations)
  5. @sharadhr sharadhr revised this gist Jul 29, 2023. 1 changed file with 112 additions and 67 deletions.
    179 changes: 112 additions & 67 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -19,15 +19,19 @@
    - [2.3 Linux and Windows equivalents](#23linux-and-windows-equivalents)
    - [3 Conventions, and why they're broken](#3conventions-and-why-theyre-broken)
    - [3.1 Respecting the user's choice and expectations](#31respecting-the-users-choice-and-expectations)
    - [3.2 Screw your conventions, *we've always done it this way*](#32screw-your-conventions-weve-always-done-it-this-way)
    - [3.2 So, what now?](#32so-what-now)
    - [3.3 Screw your conventions, *we've always done it this way*](#33screw-your-conventions-weve-always-done-it-this-way)
    - [3.2.1 Ignorance](#321ignorance)
    - [3.2.2 Arrogance](#322arrogance)
    - [3.2.3 Fear of change and complexity](#323fear-of-change-and-complexity)
    - [4 Platform-specific issues](#4platform-specific-issues)
    - [4.1 Linux](#41linux)
    - [4.2 Windows](#42windows)
    - [4.2.1 The Microsoft irony](#421the-microsoft-irony)
    - [4.2.X Video games](#42xvideo-games)
    - [4.2.2 Video games](#422video-games)
    - [5 Solutions](#5solutions)
    - [5.1 Home, sweet home](#51home-sweet-home)
    - [6 Relevant blog posts](#6relevant-blog-posts)

    <div align="justify">

    @@ -74,27 +78,26 @@ Would you still use it, but live with the compromise that your home is not *exac

    If you answer 'maybe', or even 'yes', then welcome to the world of software, where your home is violated on a regular basis.
    Software that spews configuration files, temporary and cache files, generated and save files, catalogue files, downloaded files *everywhere*.
    There is little rhyme or reason to any of it: many applications never tell the user they are putting these files **HERE** or **THERE**, and if the user wants to move them around, the latter isn't given a choice in the matter.
    There is little rhyme or reason to any of it: many applications never tell the user they are putting these files **HERE** or **THERE**, and if the user wants to move them around, they aren't given a choice in the matter.
    Applications regularly disregard platform conventions (many ironies abound, to be detailed below); even if these conventions are clearly documented, some go *out of their way* to do their own thing.
    Some applications pretend that everything is immaculate, by *hiding* their mess (and they typically don't do a great job of it: look at many Linux-first program ported to Windows).

    It would irritate *anyone*.
    I am *particularly* compulsive about software doing what I'd like it to do, and about following platform conventions.
    Conventions, standards, and protocols exist so everyone is on the same page, and there is a common 'language' for software to communicate with.
    When software authors break these conventions just for the sake of 'opinionation', or because one *feels like it* (a worse justification, in my opinion), it only leads to much exasperation on the users' end, because it comes as a surprise to the user.
    Software should *not* be surprising.
    When software authors break them just for the sake of 'opinionation', or because one *feels like it* (a worse justification, in my opinion), it only leads to much exasperation on the users' end, because it comes as a surprise to the user, and software should *not* be surprising.
    It ought to be reliable, reproducible, and follow platform conventions and expectations.

    This post is a detailed discussion into user profiles, their directories, and how they are—to put it bluntly—in total disarray on Windows and Linux (I haven't used a Mac in ages, but I assume the situation is very similar there, too).
    Applications treat the user profile as a dumping ground, and any user with a reasonably wide list of installed software will find their user profile very difficult to traverse after some time in use.
    There are platform conventions and attempts to standardise things on more open-source platforms, but a lot of developers resolutely refuse to change the behaviour of their software for a variety of reasons (some less valid than others).

    The first part is a deep dive into user profiles on Linux and Windows, and the conventions that have been established on these platforms over the years.
    The second section details *how* these conventions are broken on each platform, and *why* they are broken.
    The second section details *how* they are broken on each platform, and *why* they are broken.

    This is certainly a bit of a soapbox, but I also hope developers read this, and at least *attempt* to fix their software so that home directories are cleaner, and users have easier lives maintaining and using their computers.
    This is a bit of a soapbox, but I hope developers read this, and at least *attempt* to fix their software so that home directories are cleaner, and users have easier lives maintaining and using their computers.

    **If you want to skip all the setup drudgery, go to
    ***If you want to skip all the setup drudgery, go to [§3](#3conventions-and-why-theyre-broken)***.

    ## 2&emsp;Setup

    @@ -105,13 +108,13 @@ I will use the terms 'home directory' and 'user profile directory' somewhat inte

    Linux is famously fragmented, but even so, there exist *some* conventions for user profiles, especially for desktop environments.
    As far as I've seen, Linux user profiles are typically created in `/home/<username>`.
    The directory path may be chosen during the out-of-box experience (OOBE)/first-time setup of a Linux distro, or entirely manually, with [`useradd -d`](https://man7.org/linux/man-pages/man8/useradd.8.html#:~:text=%2Dd%2C%20%2D%2Dhome%2Ddir%20HOME_DIR), which writes to `/etc/passwd`.
    Sometimes, `/home` might occupy a separate partition/sub-volume altogether (if the user is using Btrfs, for instance).
    The directory path may be chosen during the out-of-box experience (OOBE)/first-time setup of a Linux distro or entirely manually, with [`useradd -d`](https://man7.org/linux/man-pages/man8/useradd.8.html#:~:text=%2Dd%2C%20%2D%2Dhome%2Ddir%20HOME_DIR), which writes to `/etc/passwd`.
    Sometimes, `/home` might occupy an altogether separate partition/sub-volume (if the user is using Btrfs, for instance).

    #### 2.1.1&emsp;`$HOME`

    Regardless of its location, the environment variable `$HOME` is set upon login by a login process or graphical display manager (e.g. [`login`](https://man7.org/linux/man-pages/man1/login.1.html), or `gdm`, `sddm`, etc.) based on values previously set in `/etc/passwd`.
    This file may be read using the Linux API:
    Regardless of its location, the environment variable `$HOME` is set by a login process or graphical display manager (e.g. [`login`](https://man7.org/linux/man-pages/man1/login.1.html), `gdm`, `sddm`, etc.) upon login, based on values previously set in `/etc/passwd`.
    This file is plaintext, but it may also be read using the Linux API:

    ```c++
    #include <cstdio>
    @@ -123,11 +126,11 @@ auto main() -> int
    {
    auto const pw = getpwuid(getuid());
    if (pw != nullptr) {
    printf("User name: %s\n", pw->pw_name);
    printf("User ID: %d\n", pw->pw_uid);
    printf("Group ID: %d\n", pw->pw_gid);
    printf("Home directory: %s\n", pw->pw_dir);
    printf("Login shell: %s\n", pw->pw_shell);
    std::printf("User name: %s\n", pw->pw_name);
    std::printf("User ID: %d\n", pw->pw_uid);
    std::printf("Group ID: %d\n", pw->pw_gid);
    std::printf("Home directory: %s\n", pw->pw_dir);
    std::printf("Login shell: %s\n", pw->pw_shell);

    return 0;
    } else {
    @@ -140,12 +143,12 @@ Strictly speaking, though, there is no real concept of a home directory *per se*
    The `pw_dir` member variable is just the initial working directory of the login shell and any subsequent shells started by the corresponding user; it could technically point to *any* directory that has read permissions for said user.
    The `pwd.h` manual [states as much](https://man7.org/linux/man-pages/man0/pwd.h.0p.html#:~:text=char%20%20%20%20*pw_dir%20%20%20%20Initial%20working%20directory).

    Most shells and desktop environments parse `~` as an alias to `$HOME`, and `cd` without any command-line arguments also [navigates to `$HOME`](https://man7.org/linux/man-pages/man1/cd.1p.html#:~:text=2.%20If%20no%20directory%20operand%20is%20given%20and%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20is%20set%20to%20a%20non%2Dempty%20value%2C%20the%20cd%20utility%20shall%0A%20%20%20%20%20%20%20%20%20%20%20behave%20as%20if%20the%20directory%20named%20in%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20was%20specified%20as%20the%20directory%20operand.).
    Most shells and desktop environments parse `~` as an alias to `$HOME`; running `cd` without any command-line arguments also [navigates to `$HOME`](https://man7.org/linux/man-pages/man1/cd.1p.html#:~:text=2.%20If%20no%20directory%20operand%20is%20given%20and%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20is%20set%20to%20a%20non%2Dempty%20value%2C%20the%20cd%20utility%20shall%0A%20%20%20%20%20%20%20%20%20%20%20behave%20as%20if%20the%20directory%20named%20in%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20was%20specified%20as%20the%20directory%20operand.).

    #### 2.1.2&emsp;Dot-files and dot-directories

    On \*nix, prefixing a file or directory with a full-stop (`.`) excludes said path from being listed in userland utilities, such as `ls`, or graphical file managers (at least, according to their defaults).
    These files are considered 'hidden', although no special meaning is given to them by the filesystem itself (also unlike on Windows).
    On \*nix, prefixing a file or directory with a full-stop (`.`) excludes said path from being listed in userland utilities, such as `ls` or graphical file managers by default.
    These files are considered 'hidden', although no special meaning is given to them by the filesystem itself (unlike on Windows).
    This is *convention* dating to the [earliest days of UNIX](https://web.archive.org/web/20190211031031/https://plus.google.com/+RobPikeTheHuman/posts/R58WgWwN9jp).

    Most Linux users' home directories will contain a collection of these *dot-files* and *dot-directories*, and they are typically used to set user-specific configuration for almost all programs on \*nix.
    @@ -160,7 +163,7 @@ If you *do* want to list hidden files with `ls`, use `ls -a` or `ls -A`.
    [XDG stands for 'Cross-Desktop Group'](https://www.freedesktop.org/wiki/#:~:text=an%20acronym%20for%20the%20Cross%2DDesktop%20Group).
    </details>

    The [XDG Base Directory specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) defines several environment variables and subdirectories of the user profile directory, in an attempt to standardise dot-files and dot-directories.
    The [XDG Base Directory specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) defines several environment variables expanding to subdirectories of `$HOME` in an attempt to standardise dot-files and dot-directories.
    I summarise them below:

    | **Variable** | **Default value** | **Details** |
    @@ -176,7 +179,7 @@ Notice the specification provides for the scenario that these environment variab
    #### 2.1.4&emsp;`xdg-user-dirs`

    Since Linux does not provide an 'official' userland environment (unlike Windows), the XDG people have again made effort to to set up 'Windows-style' user directories *with localisation*: [`xdg-user-dirs`](https://www.freedesktop.org/wiki/Software/xdg-user-dirs/).
    Since Linux does not provide an 'official' userland environment (unlike Windows), the XDG people have again made effort to set up 'Windows-style' user directories *with localisation*, called [`xdg-user-dirs`](https://www.freedesktop.org/wiki/Software/xdg-user-dirs/).
    Both GNOME and KDE Plasma Desktop require it.
    This tool is configured with a straightforward script in `$(XDG_CONFIG_HOME)/user-dirs.dirs`.
    The default directories generated on an English-language install are:
    @@ -203,16 +206,15 @@ It also has roughly equivalent functionality to Active Directory and roaming use
    <summary>Windows and Microsoft commentary</summary>

    I hold several very controversial opinions about Microsoft and Windows that would at best, raise many hackers' eyebrows, and at worst, get me called an astroturfer or Microsoft shill.
    I *was* going to type several paragraphs, but I realised that would be better off as a blog-post on its own—coming soon.
    Not that Microsoft needs any advertising...
    These are best left to a separate blog post.
    </details>

    User profiles on Windows are a rather complex matter, but this complexity is not necessarily unwelcome.
    User profiles on Windows are a rather complex matter, which is not necessarily unwelcome.
    Most of it is configurable, fairly well-documented, and there is more than one way to do things, although some are clearly better than others.

    #### 2.2.1&emsp;Environment variables

    Modern Windows (i.e. Vista and later) has several conventions and environment variables for important directories, including those within the user profile.
    Modern Windows (i.e. Vista and later) also has environment variables for important directories, including those within the user profile directory.
    Many are listed [here](https://ss64.com/nt/syntax-variables.html) (Microsoft's [official documentation](https://learn.microsoft.com/en-sg/windows/deployment/usmt/usmt-recognized-environment-variables) is a little less forthcoming), and I reproduce a few in the table below.

    | **Variable** | **Value** | **Details** |
    @@ -232,11 +234,11 @@ These environment variables may be accessed from C or C++ with [`getenv()`](http

    #### 2.2.2&emsp;User profile creation and naming

    This is a mess on Windows. Until Windows 8, there was very little ceremony to user profile creation, which were all local.
    Since then, however, OneDrive (then called SkyDrive) and Microsoft account integration has thoroughly road-rolled over this simplicity.
    Today, creating a local, non-connected user profile in the OOBE in Windows 11 is [a rather involved process](https://www.windowscentral.com/how-set-windows-11-without-microsoft-account#:~:text=Use%20the%20%22Shift%20%2B%20F10%22%20keyboard%20shortcut%20to%20open%20Command%20Prompt), and may require Windows 11 Pro, which is usually not shipped with consumer devices.
    This is a mess on Windows. Until Windows 8, user profile creation was straightforward, and only local accounts could be created (not including Active Directory roaming profiles).
    Since then, however, OneDrive (previously SkyDrive) and Microsoft account integration has thoroughly road-rolled over this simplicity.
    Today, creating a local, non-connected user profile in the OOBE in Windows 11 is [a particularly convoluted process](https://www.windowscentral.com/how-set-windows-11-without-microsoft-account#:~:text=Use%20the%20%22Shift%20%2B%20F10%22%20keyboard%20shortcut%20to%20open%20Command%20Prompt), and may require Windows 11 Pro, which is usually not shipped with consumer devices.

    This digression is to discuss what home directory name users end up with: on a profile connected to a Microsoft account, the user home directory is a truncated (to the first five characters), lower-case version of the user name.
    This digression is to discuss what home directory name users end up with: on a profile connected to a Microsoft account, the user home directory has a name that is the first five characters, in lowercase, of the email address or name provided.
    For instance, in the first case, I would have `%SYSTEMDRIVE%\Users\shara`, but on a 'local account', it would be `%SYSTEMDRIVE%\Users\Sharadh` (mine is *actually* the latter—I am obsessive about this, so I created a local account first, and *then* synced it to OneDrive).
    Like many things Windows, configuring the user profile directory after account creation requires [editing registry keys](https://learn.microsoft.com/en-sg/troubleshoot/windows-client/user-profiles-and-logon/renaming-user-account-not-change-profile-path) using *another* administrator account.

    @@ -267,19 +269,19 @@ These may be queried with [`SHGetKnownFolderPath`](https://learn.microsoft.com/e

    auto main() -> int
    {
    auto path = PWSTR{};
    auto path = PWSTR{};

    // get desktop path; return type HRESULT
    if (auto const result = SHGetKnownFolderPath(FOLDERID_Desktop, KF_FLAG_DEFAULT, nullptr, &path);
    FAILED(result)) {
    return 1;
    }
    wprintf_s(L"%ls\n", path);
    std::wprintf_s(L"%ls\n", path);
    return 0;
    }
    ```

    In .NET, the [`Environment.GetFolderPath`](https://learn.microsoft.com/en-us/dotnet/api/system.environment.getfolderpath?view=net-7.0) method, together with the [`Environment.SpecialFolder`](https://learn.microsoft.com/en-us/dotnet/api/system.environment.specialfolder?view=net-7.0) enum returns the same value, although this is missing the Vista changes detailed below.
    In .NET, the [`Environment.GetFolderPath`](https://learn.microsoft.com/en-sg/dotnet/api/system.environment.getfolderpath?view=net-7.0) method, together with the [`Environment.SpecialFolder`](https://learn.microsoft.com/en-sg/dotnet/api/system.environment.specialfolder?view=net-7.0) enum returns the same value, although this is missing the Vista changes detailed below, because these constants are [set to the `CSIDL` values](https://source.dot.net/#System.Private.CoreLib/src/libraries/System.Private.CoreLib/src/System/Environment.SpecialFolder.cs,63) instead of the `KNOWNFOLDERID`s.
    [There is a proposal to add these to .NET 8](https://github.com/dotnet/runtime/issues/70484).

    ```pwsh
    @@ -288,12 +290,13 @@ D:\Libraries\Desktop
    ```

    I have set all my libraries to point to `D:\Libraries\<library>`, because my `C:` drive is rather small and dedicated to the OS and important programs only, whereas `D:` is significantly larger.
    So, the graphical setting for the various libraries update the locations returned by the native Windows API too (which is eventually also called by .NET).
    So, the graphical setting for the various libraries update the locations returned by the native Windows API too.
    Hereafter, I will use the [Windows shell shortcuts](https://ss64.com/nt/shell.html) to describe library locations.

    ##### 2.2.3.1&emsp;Hidden folders on Windows

    In particular, the parent directory of `%APPDATA%` and `%LOCALAPPDATA%`, `%USERPROFILE%\AppData`, is a *hidden* directory.
    Unlike \*nix, 'hidden' on Windows has special semantics accorded to the file or directory, by the filesystem (usually NTFS).
    In particular, the parent directory of `shell:AppData` and `shell:Local AppData``%USERPROFILE%\AppData`is a *hidden* directory.
    Unlike \*nix, 'hidden' on Windows has special semantics, accorded by the filesystem (usually NTFS).
    This can be set either graphically (right click → `Properties``General` tab → `Hidden` checkbox under `Attributes`), or retrieved/set programmatically in the Windows API, using [`GetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw) and [`SetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw).
    These functions return or accept a bitwise-ORed [file attribute constant](https://learn.microsoft.com/en-sg/windows/win32/fileio/file-attribute-constants).

    @@ -314,7 +317,7 @@ auto main() -> int
    return 1;
    }

    printf("SHGetKnownFolderPath succeeded, path is %ws\n", path);
    std::printf("SHGetKnownFolderPath succeeded, path is %ws\n", path);

    // get parent path of %APPDATA%, i.e. `%USERPROFILE%\AppData`
    if (auto const remove_leaf_result = PathCchRemoveFileSpec(path, MAX_PATH);
    @@ -339,13 +342,13 @@ auto main() -> int

    [*A Brief History of Windows Profiles*](https://archive.ph/eOGDp) is a great article, but I want to focus on one important section.
    Windows 2000 and XP used `%SYSTEMDRIVE%\Documents and Settings\<username>` for the home directory, which was moved to `%SYSTEMDRIVE%\Users\<username>` with Vista and later.
    Windows Vista also introduced the above-mentioned `KNOWNFOLDERID`s, which superseded the older [`CSIDL`](https://learn.microsoft.com/en-sg/windows/win32/shell/csidl) enumeration (although the latter is still available).
    Windows Vista also introduced the above-mentioned `KNOWNFOLDERID`s, which superseded the older [`CSIDL`](https://learn.microsoft.com/en-sg/windows/win32/shell/csidl) enumeration (although the latter is still available, and used by the .NET API).

    Many environment variables like `%APPDATA%` also were changed to point to the new locations (the previous was a mouthful: `%SYSTEMDRIVE%\Documents and Settings\<username>\Local Settings\Application Data`).
    New known folders were added, such as `Downloads` and `Saved Games`; the `My⎵` prefix was removed from `My Documents` and `My Music`; locations such as `Start Menu` were moved (in this case, to `%APPDATA%\Microsoft\Windows\Start Menu`).

    The reason for this move is anyone's guess, but I feel the $\geq$ Vista convention makes a lot more sense than the $\leq$ Windows XP one.
    There is an [old guide](https://web.archive.org/web/20120621131135/http://technet.microsoft.com:80/en-us/library/cc766489(v=ws.10).aspx) for sysadmins migrating from Windows XP to Vista, and there is another blog post for Windows application developers: [*Where Should I Store my Data and Configuration Files if I Target Multiple OS Versions?*](https://learn.microsoft.com/en-sg/archive/blogs/patricka/where-should-i-store-my-data-and-configuration-files-if-i-target-multiple-os-versions#targeting-vista-and-higher)
    There is an [old guide](https://web.archive.org/web/20120621131135/http://technet.microsoft.com:80/en-sg/library/cc766489(v=ws.10).aspx) for sysadmins migrating from Windows XP to Vista, and there is another blog post for Windows application developers: [*Where Should I Store my Data and Configuration Files if I Target Multiple OS Versions?*](https://learn.microsoft.com/en-sg/archive/blogs/patricka/where-should-i-store-my-data-and-configuration-files-if-i-target-multiple-os-versions#targeting-vista-and-higher)

    As an aside, there is [an apocryphal tale](https://news.ycombinator.com/item?id=29187896) for the reason why programs are stored in `Program Files` and not `ProgramFiles` or `Program_Files`: apparently Microsoft wanted to force programmers to write code defensively, and handle spaces in paths without crashing, so they made the program install location itself have spaces in it (this then raises the question: why `ProgramData`?)
    [Windows also has decent localisation](https://archive.ph/Eqvd4) (at least amongst Indo-European languages employing the Latin script): set the system language to German, for instance, and `Program Files` is now `Programme`; it is `Programfiler` in Norwegian.
    @@ -380,7 +383,7 @@ It seems like pointless bike-shedding—discussing where user data ought to be s

    These specifications don't exist merely to 'put things somewhere' for the hell of it. Disparate applications from a wide variety of developers and backgrounds which implement these specifications will have a only single place to write to (and read from), and a single point of backup, which reduces user workload.
    As I mentioned in the introduction, software ought not to be surprising or unnecessarily opinionated.
    Users see a specification, or some clearly-labelled pre-generated folder in their home directory, and *expect* software to write to adhere to these specifications, or write to an equally sensible location.
    Users see a specification, or some clearly-labelled pre-generated folder in their home directory, and *expect* software to write to adhere to these specifications, and write to a sensible location.

    GitHub user Lyle Hanson ([@lhanson](https://github.com/lhanson)) puts it more clearly than I could, [in the issue for Vim](https://github.com/vim/vim/issues/2034#issuecomment-559254363):

    @@ -395,22 +398,24 @@ GitHub user Lyle Hanson ([@lhanson](https://github.com/lhanson)) puts it more cl
    I'd like to tack on to his point about 'respecting users': just because the user's home directory and its contents have full read-write-execute permissions for the user does not mean that software executed by said user should have free rein over the directory.

    ### 3.2&emsp;So, what now?

    The default assumption by a programmer might be:

    > Let's put config and data files in the same directory as the program.
    This assumption immediately breaks on most modern desktop operating systems, because programs are typically installed to locations which require elevated write permissions (e.g. `/bin`, `%PROGRAMFILES%`) which means neither users nor their processes can write there, willy-nilly.
    This assumption immediately breaks on most modern desktop operating systems, because programs are typically installed to locations which require elevated write permissions (e.g. `/bin`, `%PROGRAMFILES%`) which means neither users nor their processes can write there willy-nilly.
    Furthermore, most desktop OSs are multi-user, which means handling several different configurations and data for multiple users.
    Even *Android*, a smartphone OS, [supports multiple user profiles since 5.0 Lollipop](https://www.android.com/versions/lollipop-5-0/#:~:text=Multiple%20users%20for%20phones.), which means providing ways for developers to handle different user profiles.

    From my experiments, apps and their data are completely sandboxed per-user on Android, so apps installed by one user can be neither seen nor accessed by another user.
    This is workable on desktop OSs, except that it still doesn't really solve the backup problem: do we just sync the entire app and its contents to a server?
    Many programs on desktop OSs are several gigabytes large, and naïvely syncing this much data is a waste of bandwidth.
    And it hearkens back to the point mentioned above: any backup utility or the user will have to handle a polynomially large number of application-user combinations.
    And it hearkens back to the point mentioned above: any backup utility or the user will have to handle a polynomially large number of application-user-configuration combinations.

    <img src="https://imgs.xkcd.com/comics/workflow.png" width="200" align="left" hspace="10">

    ### 3.2&emsp;Screw your conventions, *we've always done it this way*
    ### 3.3&emsp;Screw your conventions, *we've always done it this way*

    We come to the real crux of the matter: why and how developers break the above-mentioned conventions and standards.
    Developers cite only a few reasons for not wanting to adhere to conventions, and the following are lifted *verbatim* from their respective issue trackers (many closed as 'won't fix'):
    @@ -449,18 +454,20 @@ Nothing else to say here, honestly; if you don't know the platform conventions,
    This is the least acceptable, and it reveals an ugly superiority complex.
    If developers don't respect their users, then why even bother releasing software publicly, except to flex and collect bragging rights?
    Many open-source authors angrily retort: 'develop it yourself if you want $x$ feature'.
    But even despite the dire warnings in many FOSS licences, the *social contract* is that the developer or maintainer listens to and judges user feedback on their own merits, and implements frequently-asked-for features or fixes.
    Yes, there are dire warnings about 'as-is', and 'no warranty' for 'merchantability' and 'fitness for a purpose', and I'm not claiming that open-source software authors are legally obliged to make quality-of-life changes.
    The *social contract*, however, is that said developers and maintainers listen to and judges user feedback on their own merits, and implements frequently-asked-for features or fixes.

    #### 3.2.3&emsp;Fear of change and complexity

    I'm not going to quote Heraclitus or Benjamin Franklin here ('change is the only constant'), but software should be developed to fit users' needs, and platform conventions.
    As software is developed, it *changes*, doesn't it?

    It seems like this fear is a function of *what* is being changed:
    developers tend to view adding new features relevant to their software with delight, but see more menial tasks like properly handling paths and correct cross-platform behaviour, with disdain.
    In all honesty, it seems like this fear is a function of *what* is being changed:
    developers tend to view adding fancy new features with delight, but see more menial tasks like properly handling paths and correct cross-platform behaviour with disdain.
    If your program is going to be cross-platform, you *will* have to handle the inherent complexity in supporting all those platforms, and again, *be a good citizen on those platforms*.

    Merely using cross-platform frameworks (like Electron or Qt) is not a panacea: if your program writes to a non-canonical location in the user's home directory, it ought to tell the user where it is writing to, and what it is writing.
    Case in point: Visual Studio Code is a fairly big problem (see below).

    ## 4&emsp;Platform-specific issues

    @@ -556,7 +563,7 @@ Windows... Ah, Windows.
    Bastion of backwards compatibility, keeping icons and settings options from Windows 3.1 NT around in Windows 11...
    And the source of all your problems.
    As on Linux, let me provide a listing of my `%USERPROFILE`:
    As on Linux, let me provide a listing of my `%USERPROFILE%`:
    <details>
    <summary>My <code>%USERPROFILE%</code></summary>
    @@ -605,50 +612,53 @@ Sti_Trace.log
    </details>
    <img src="https://bonkersworld.net/img/2011-06-27_organizational_charts.png" width="400" align="right" hspace="10">
    Yep, *dot-files*... in *Windows*.
    Windows does *not* automatically hide dot-files, whether it be in Explorer, `dir.exe`, or `Get-ChildItem`.
    Notice the above listing is missing `AppData`—that's because it is properly hidden by the filesystem as mentioned in [2.2.3.1](#2231hidden-folders-on-windows).
    Notice the above listing is missing `AppData`—that's because it is properly hidden by the filesystem as mentioned in [§2.2.3.1](#2231hidden-folders-on-windows).
    This is frequently violated by software written for \*nix first—notice `.git`, `.ssh`... There's even the full set of XDG base directories, which is a \*nix specification!
    Developers either assume Windows operates the same as \*nix, or don't care about Windows much ('second-class citizen'), and again, couch any changes to fix this as 'introducing unnecessary complexity'.
    Honestly, one would instead think the change is fairly straightforward: test which OS the program is running on using `#ifdef`s and OS-specific macros, and delegate to the appropriate function.
    [Here's a wiki](https://sourceforge.net/p/predef/wiki/OperatingSystems/) listing them all.
    Better still, use the ecosystem's build system to configure the build setup for different platforms, and shift this test to configure and compile-time instead of run-time.
    [Here's a wiki](https://sourceforge.net/p/predef/wiki/OperatingSystems/) listing some of these macros.

    Some semi-compliant software hard-codes paths, assuming user libraries will remain in the default locations, i.e. sub-directories of `%USERPROFILE%`.
    This is *not* true, and users can change their locations. Notice there's a `Contacts` directory in the listing above: this was created by [KDE Connect](https://kdeconnect.kde.org/).
    Some semi-compliant software hard-codes library paths, assuming user libraries will remain in the default locations, i.e. sub-directories of `%USERPROFILE%`.
    This is *not* true, and users can change their locations as mentioned above. Notice there's a `Contacts` directory in the listing above: this was created by [KDE Connect](https://kdeconnect.kde.org/).
    That said, not *all* the above dot-directories are created by traditionally *nix programs, which is a good segue to the next section...
    <img src="https://bonkersworld.net/img/2011-06-27_organizational_charts.png" width="400" align="right" hspace="10" vspace="10">
    #### 4.2.1&emsp;The Microsoft irony
    Notice `.dotnet`, `.nuget`, `.omnisharp`, `.templateengine`, `.vscode`, and `source`.
    Microsoft itself is a lousy citizen of Windows.
    Microsoft *itself* is a lousy citizen of Windows.
    `source` is repeatedly created by Visual Studio (not Code), especially on a fresh install.
    The rest of the projects are open source, and have active/on-going issues related to home directory pollution:
    The rest of the projects are open-source, and have active/on-going issues related to home directory pollution:
    - `dotnet` CLI and `.templateengine`: [issue closed](https://github.com/dotnet/sdk/issues/8678), but requires setting `DOTNET_CLI_HOME` environment variable. Otherwise defaults to previous behaviour, and still doesn't fix `.dotnet`;
    - `nuget`: [issue still open](https://github.com/dotnet/sdk/issues/4650);
    - `omnisharp`: [issue still open](https://github.com/OmniSharp/omnisharp-roslyn/issues/953);
    - `vscode`: [issue still open](https://github.com/microsoft/vscode/issues/3884).

    The comic nearby seems to embody the internal structure of Microsoft: teams are siloed, are pit against each other, and the end-user loses out.
    It feels like developers who have never natively used Windows or its stack, work on and contribute to Microsoft software, without liaising with veteran Windows teams.
    Maybe that's why Windows 11 looks so...*inspired*... by macOS (which is an aberration; the macOS windowing system sucks).
    #### 4.2.X&emsp;Video games
    #### 4.2.2&emsp;Video games
    Video games on Windows deserve a special heading of their own, because they are *particularly* egregious offenders.
    In my opinion, game devs *really* have no excuse: almost all games primarily target Windows, or are ported by dedicated teams who know Windows inside out, and more importantly, develop using an entirely native toolchain.
    Even Unreal and Unity have functionality to call out to native code, and developers *still* don't use it to properly handle save data.
    The issue is such that the PCGamingWiki
    The issue is such that the PCGamingWiki has dedicated 'save file location' and 'config file location' headings for every single game in its database.

    Most video games seem to like writing their save files, screenshots, and settings data into `shell:Documents\` or `shell:Documents\My Games`, which is a holdover from Windows XP.
    Most video games seem to like writing their save files, screenshots, and settings data into `Documents\` or `Documents\My Games`, which is a holdover from Windows XP.
    Look, I get it, many game engines are old; they trace their lineage to engines first written in the 1990s and early 2000s.
    But if developers can add a ray-tracing update to a 7-year-old game (*Witcher 3: Wild Hunt*), surely they can spend a couple hours fixing this too (not that the update itself was great: it broke the game's performance for me; I used to be able to run it at max settings, 4K resolution, and at 100 FPS, which was cut by two-thirds after said update, even *without* ray-tracing... 😕)
    But if developers can add ray-tracing updates to a 16-year-old game (*Portal: RTX*), surely they can spend a couple hours fixing this too.
    Here is a listing of my `Documents` library, which illustrates the problem:
    ``` gci -Name D:\Libraries\Documents
    ```pwsh
    > gci D:\Libraries\Documents -Name
    3DMark
    4A Games
    Anno 1404
    @@ -701,14 +711,49 @@ mods.settings
    *Anno 1404*, what are you doing? *Six* different (localised) directories, all containing the same paths.
    Honestly speaking, though, I doubt game developers are going to change to `shell:Saved Games`, and this is partially Microsoft's fault, again.
    Honestly speaking, I doubt game developers are going to change to `Saved Games`, and this is partially Microsoft's fault, again.
    The folder is neither listed in the default File Explorer `Libraries` view, nor in the `My PC` view; users have to manually navigate to it.
    Its rarity is demonstrated by the ratio of games using either `Documents` or `Documents\My Games` compared to those using `Saved Games`: **32:4** on my computer.
    What a pity.
    Its rarity is demonstrated by the ratio of games using either `Documents` or `Documents\My Games` compared to those using `Saved Games`: **32:4** on my computer (the four games using `Saved Games` are *Cyberpunk 2077*, *Metro: Exodus*, *Kingdom Come: Deliverance*, and **)
    Some games forgo `Documents` altogether, and put the save files directly with the game files, or even in `%APPDATA%`.
    Notice even here that Microsoft is polluting `Documents` with `FeedbackHub`, `My Data Sources`, `Outlook Files`, `Visual Studio 20XX`, `PowerShell`, `WindowsPowershell`.
    The folder contains anything *but* documents now, and it is so cluttered that I have the `_Documents` subfolder for *my* stuff, things I saved via a save dialog box or the command-line.
    The leading underscore is a necessary evil, as it means the directory is listed first when sorted by name.
    [This answer on the Game Development StackExchange](https://gamedev.stackexchange.com/a/108243/157009) is relevant:
    > The best reason I can think of is **to reduce cost of customer support**. The problem is that `%APPDATA%` and `%LOCALAPPDATA%` are hidden directories, and nobody seems to know about `%UserProfile%\Saved Games`. Placing the files in `Documents`, studios can save money on support calls that ask how to backup savegames, or how to migrate the savegames from one machine to another.
    Notice even here that Microsoft is polluting `Documents` with `Custom Office Templates`, `FeedbackHub`, `My Data Sources`, `Outlook Files`, `Visual Studio 20XX`, `PowerShell`, `PowerToys`, and `WindowsPowershell` (yes; there are [*two* PowerShells](https://learn.microsoft.com/en-sg/powershell/scripting/whats-new/differences-from-windows-powershell?view=powershell-7.3)).
    The folder contains everything *but* documents now, and it is so cluttered that I have the `_Documents` subfolder for *my* stuff—files I explicitly created via a save dialog box or the command-line.
    The leading underscore is a necessary evil, as it means the directory is listed first when sorted by name in File Explorer.
    In a nutshell... A hopeless situation.
    There's no saving Windows user profiles when Microsoft itself doesn't adhere to its own conventions.
    ## 5&emsp;Solutions
    So if things are so 'hopeless' as I put it, then why bother with this article?
    I think *any* change towards compliance is better than no change, that's all.
    On *nix, the answer is straightforward: get everyone to adhere to the XDG Base Directory specification.
    Of course, 'get everyone to' is doing a lot of work in that sentence: it involves submitting an issue, convincing maintainers that this is a worthwhile, and possibly writing code to satisfy the specification, correctly (including the fall-back directories), and all the review drudgery before things are finally merged.
    On Windows, the problems arise from within, and I sincerely don't see a solution.
    Thankfully, many non-compliant projects are fully open-source, and contributing to them, or up-voting issues, will hopefully help.
    Game developers are famously opaque (especially given how lucrative the industry is as a whole), and as mentioned above, they do have genuine reasons for saving stuff in `Documents`.
    That being said, there is still value in attempting to fix this on Windows.
    The specification is clear, some developers clearly know about it, and it is a good starting point for new projects (Microsoft is apparently promoting Rust now over C++ for green-field Windows development, so maybe there's hope yet).
    ### 5.1&emsp;Home, sweet home
    We have so many real-life analogues in our computers: from files, folders, and rubbish bins, to the 'desktop' metaphor.
    Surely we can extend the concepts of 'clutter' and 'cleanup', and apply it to our digital lives too?
    Let's be good citizens of the platforms we develop for, and give our users choices and control over their data, and where it is stored.
    I'd like a nice vacuum cleaner for my home directory, please.
    ## 6&emsp;Relevant blog posts
    Here are several blog posts and articles that inspired me, some previously linked above:
    - [*Dotfile madness*](https://0x46.net/thoughts/2019/02/01/dotfile-madness/)
    - [*Everything that uses configuration files should report where they're located*](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/ReportConfigFileLocations)
    - [*Where should I store my data and configuration files if I target multiple OS versions?*](https://learn.microsoft.com/en-sg/archive/blogs/patricka/where-should-i-store-my-data-and-configuration-files-if-i-target-multiple-os-versions#targeting-vista-and-higher)
    - [*A Brief History of Windows Profiles*](https://archive.ph/eOGDp)
  6. @sharadhr sharadhr revised this gist Jul 29, 2023. 1 changed file with 222 additions and 29 deletions.
    251 changes: 222 additions & 29 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@

    - [Preface](#preface)
    - [1 What is `$HOME` to you?](#1what-is-home-to-you)
    - [2 User profile setup and conventions](#2user-profile-setup-and-conventions)
    - [2 Setup](#2setup)
    - [2.1 Linux](#21linux)
    - [2.1.1 `$HOME`](#211home)
    - [2.1.2 Dot-files and dot-directories](#212dot-files-and-dot-directories)
    @@ -17,12 +17,17 @@
    - [2.2.4 Windows XP to Vista changes](#224windows-xp-to-vista-changes)
    - [2.2.5 `HKEY_CURRENT_USER` registry hive](#225hkey_current_user-registry-hive)
    - [2.3 Linux and Windows equivalents](#23linux-and-windows-equivalents)
    - [3 Reasons for conventions, and for breaking them](#3reasons-for-conventions-and-for-breaking-them)
    - [3.1 Screw your conventions, *we've always done it this way*](#31screw-your-conventions-weve-always-done-it-this-way)
    - [3.1  Linux](#31-linux)
    - [3.1.1  My `$HOME`](#311-my-home)
    - [4 Windows](#4windows)
    - [4.x Video games](#4xvideo-games)
    - [3 Conventions, and why they're broken](#3conventions-and-why-theyre-broken)
    - [3.1 Respecting the user's choice and expectations](#31respecting-the-users-choice-and-expectations)
    - [3.2 Screw your conventions, *we've always done it this way*](#32screw-your-conventions-weve-always-done-it-this-way)
    - [3.2.1 Ignorance](#321ignorance)
    - [3.2.2 Arrogance](#322arrogance)
    - [3.2.3 Fear of change and complexity](#323fear-of-change-and-complexity)
    - [4 Platform-specific issues](#4platform-specific-issues)
    - [4.1 Linux](#41linux)
    - [4.2 Windows](#42windows)
    - [4.2.1 The Microsoft irony](#421the-microsoft-irony)
    - [4.2.X Video games](#42xvideo-games)

    <div align="justify">

    @@ -91,9 +96,9 @@ This is certainly a bit of a soapbox, but I also hope developers read this, and

    **If you want to skip all the setup drudgery, go to

    ## 2&emsp;User profile setup and conventions
    ## 2&emsp;Setup

    Before I begin, there are some platform-specific details and wording *conventions* to be discussed.
    Before I begin, there are some platform-specific details and setup to be discussed, as well as phrasing *conventions*.
    I will use the terms 'home directory' and 'user profile directory' somewhat interchangeably in this post, whereas 'user profile' means the home directory itself and its contents—including user-specific configuration and data—*combined*.

    ### 2.1&emsp;Linux
    @@ -366,13 +371,16 @@ So far, I've discussed user profiles on Linux and Windows. There are some rough
    | Per-user program data | `$XDG_DATA_HOME` | `%LOCALAPPDATA%` |
    | Per-user program cache | `$XDG_CACHE_HOME` | `%APPDATA%` |

    ## 3&emsp;Reasons for conventions, and for breaking them
    ## 3&emsp;Conventions, and why they're broken

    Okay; that was a pretty long introduction, but I haven't really gotten into *why* these conventions have been established in the first place.
    These specifications don't exist merely to 'put things somewhere' for the hell of it.
    They are there so that disparate applications from a wide variety of developers and backgrounds have a single place to write to, and a single point of backup, which reduces user workload.
    ### 3.1&emsp;Respecting the user's choice and expectations

    Okay; that was a pretty long introduction, but I haven't really gotten into *why* these user profile conventions have been established in the first place.
    It seems like pointless bike-shedding—discussing where user data ought to be saved—but there are real issues which said conventions attempt to solve.

    These specifications don't exist merely to 'put things somewhere' for the hell of it. Disparate applications from a wide variety of developers and backgrounds which implement these specifications will have a only single place to write to (and read from), and a single point of backup, which reduces user workload.
    As I mentioned in the introduction, software ought not to be surprising or unnecessarily opinionated.
    Users see a specification, or some clearly-labelled pre-generated folder in their home directory, and *expect* software to write to sensible locations.
    Users see a specification, or some clearly-labelled pre-generated folder in their home directory, and *expect* software to write to adhere to these specifications, or write to an equally sensible location.

    GitHub user Lyle Hanson ([@lhanson](https://github.com/lhanson)) puts it more clearly than I could, [in the issue for Vim](https://github.com/vim/vim/issues/2034#issuecomment-559254363):

    @@ -398,16 +406,11 @@ Even *Android*, a smartphone OS, [supports multiple user profiles since 5.0 Loll
    From my experiments, apps and their data are completely sandboxed per-user on Android, so apps installed by one user can be neither seen nor accessed by another user.
    This is workable on desktop OSs, except that it still doesn't really solve the backup problem: do we just sync the entire app and its contents to a server?
    Many programs on desktop OSs are several gigabytes large, and naïvely syncing this much data is a waste of bandwidth.


    In fact, a good rule of thumb is as follows:

    >[!IMPORTANT]
    > Programs should only ever write to the top level home directory (or `Documents`, or `Desktop`, or any of the 'media' directories) when *explicitly asked to by the user*, via a save dialog prompt/box.
    And it hearkens back to the point mentioned above: any backup utility or the user will have to handle a polynomially large number of application-user combinations.

    <img src="https://imgs.xkcd.com/comics/workflow.png" width="200" align="left" hspace="10">

    ### 3.1&emsp;Screw your conventions, *we've always done it this way*
    ### 3.2&emsp;Screw your conventions, *we've always done it this way*

    We come to the real crux of the matter: why and how developers break the above-mentioned conventions and standards.
    Developers cite only a few reasons for not wanting to adhere to conventions, and the following are lifted *verbatim* from their respective issue trackers (many closed as 'won't fix'):
    @@ -429,14 +432,44 @@ OpenSSH ([archived page](https://web.archive.org/web/20190925004614/https://bugz
    [Flatpak](https://github.com/flatpak/flatpak/issues/1651#issuecomment-396370107):
    > There is no actual problem here.
    ### 3.1&emsp; Linux
    In a nutshell, these reasons are:

    - **ignorance**, i.e. the developer doesn't know the specification exists
    - **arrogance**, i.e. 'my way is correct, I don't care what my users say because they're stupid', or even 'it works on my machine';
    - **fear of** introducing **change** for change's sake, and breaking user workflows;
    - **fear of** introducing **complexity** in handling these conventions, especially if the program is multi-platform.

    #### 3.2.1&emsp;Ignorance

    This is more understandable than the rest, and is easily mitigated, especially if the developer is responsive and accepting of changes.
    Nothing else to say here, honestly; if you don't know the platform conventions, you don't; hopefully you pick it up and rewrite your software to be a good citizen of the platform you're targetting.

    #### 3.2.2&emsp;Arrogance

    This is the least acceptable, and it reveals an ugly superiority complex.
    If developers don't respect their users, then why even bother releasing software publicly, except to flex and collect bragging rights?
    Many open-source authors angrily retort: 'develop it yourself if you want $x$ feature'.
    But even despite the dire warnings in many FOSS licences, the *social contract* is that the developer or maintainer listens to and judges user feedback on their own merits, and implements frequently-asked-for features or fixes.

    #### 3.2.3&emsp;Fear of change and complexity

    I'm not going to quote Heraclitus or Benjamin Franklin here ('change is the only constant'), but software should be developed to fit users' needs, and platform conventions.
    As software is developed, it *changes*, doesn't it?

    It seems like this fear is a function of *what* is being changed:
    developers tend to view adding new features relevant to their software with delight, but see more menial tasks like properly handling paths and correct cross-platform behaviour, with disdain.
    If your program is going to be cross-platform, you *will* have to handle the inherent complexity in supporting all those platforms, and again, *be a good citizen on those platforms*.

    #### 3.1.1&emsp; My `$HOME`
    Merely using cross-platform frameworks (like Electron or Qt) is not a panacea: if your program writes to a non-canonical location in the user's home directory, it ought to tell the user where it is writing to, and what it is writing.

    Before I proceed, I provide a listing of my own `$HOME` on my Arch Linux install, which has several thousand packages:
    ## 4&emsp;Platform-specific issues

    ### 4.1&emsp;Linux

    Before I proceed, I provide a listing of my own `$HOME` on my Arch Linux install:

    <details>
    <summary>My `$HOME`</summary>
    <summary>My <code>$HOME</code></summary>

    ```zsh
    % ls -A1 --group-directories-first $HOME
    @@ -507,15 +540,175 @@ The rest (excluding the XDG directories) were either auto-generated by `xdg-user
    The link above discussing the origin of dot-files makes it clear that the current behaviour was possibly *a mistake* made by the original developers of UNIX.
    Even if it wasn't, the result today is a messy litter of dot-files dumped all over a user's home directory.
    The XDG Base Directory (hereafter, XBD) specification is more than a decade old—an eternity in computing terms—and yet, there is a veritable parade of very well-known programs that refuse to follow its guidelines, typically citing 'backwards compatibility' and 'not breaking existing workflows' as reasons.
    The XDG Base Directory (hereafter, XBD) specification is more than a decade old—an eternity in computing terms—and yet, there is a veritable parade of very well-known programs that refuse to follow its guidelines: see the quotes above in [3.2](#32screw-your-conventions-weve-always-done-it-this-way).
    The Arch Wiki has [a list of programs](https://wiki.archlinux.org/title/XDG_Base_Directory#Support) that are XBD-compliant by default, may be forced to comply after user intervention, and those with hard-coded non-XBD paths.
    The latter two lists combined is almost *twice* as long as the compliant list, and includes some very prominent \*nix-first software like Bash, Vim, OpenSSH, and Firefox.
    ### 4&emsp;Windows
    There's little more to say here: there exists a specification, many programs adhere to it, and many don't.
    Fragmentation is only useful in grenades, and not in software specifications.
    For the record, people have found XBD compliance painful enough that someone developed a tool *dedicated* to finding non-compliant programs, and suggesting user-side workarounds: [`xdg-ninja`](https://github.com/b3nj5m1n/xdg-ninja) (written in Haskell, by the way, which is nice).
    ### 4.2&emsp;Windows
    Windows... Ah, Windows.
    Bastion of backwards compatibility, keeping icons and settings options from Windows 3.1 NT around in Windows 11...
    And the source of all your problems.
    As on Linux, let me provide a listing of my `%USERPROFILE`:
    <details>
    <summary>My <code>%USERPROFILE%</code></summary>
    ```pwsh
    > gci $env:USERPROFILE -Name
    .android
    .cache
    .config
    .dlv
    .dotnet
    .eclipse
    .fop
    .gk
    .gnupg
    .gradle
    .librarymanager
    .m2
    .matplotlib
    .ms-ad
    .nuget
    .omnisharp
    .platformio
    .sonarlint
    .ssh
    .templateengine
    .thumbnails
    .vscode
    ansel
    Calibre Library
    Contacts
    dotTraceSnapshots
    Heaven
    recovered
    RenPy
    source
    Tracing
    Zotero
    .cortex-debug
    .git-for-windows-updater
    .gitconfig
    .kdiff3rc
    .lesshst
    Sti_Trace.log
    ```
    </details>
    <img src="https://bonkersworld.net/img/2011-06-27_organizational_charts.png" width="400" align="right" hspace="10">
    Yep, *dot-files*... in *Windows*.
    Windows does *not* automatically hide dot-files, whether it be in Explorer, `dir.exe`, or `Get-ChildItem`.
    Notice the above listing is missing `AppData`—that's because it is properly hidden by the filesystem as mentioned in [2.2.3.1](#2231hidden-folders-on-windows).
    This is frequently violated by software written for \*nix first—notice `.git`, `.ssh`... There's even the full set of XDG base directories, which is a \*nix specification!
    Developers either assume Windows operates the same as \*nix, or don't care about Windows much ('second-class citizen'), and again, couch any changes to fix this as 'introducing unnecessary complexity'.
    Honestly, one would instead think the change is fairly straightforward: test which OS the program is running on using `#ifdef`s and OS-specific macros, and delegate to the appropriate function.
    [Here's a wiki](https://sourceforge.net/p/predef/wiki/OperatingSystems/) listing them all.
    Some semi-compliant software hard-codes paths, assuming user libraries will remain in the default locations, i.e. sub-directories of `%USERPROFILE%`.
    This is *not* true, and users can change their locations. Notice there's a `Contacts` directory in the listing above: this was created by [KDE Connect](https://kdeconnect.kde.org/).

    That said, not *all* the above dot-directories are created by traditionally *nix programs, which is a good segue to the next section...

    #### 4.2.1&emsp;The Microsoft irony

    Notice `.dotnet`, `.nuget`, `.omnisharp`, `.templateengine`, `.vscode`, and `source`.
    Microsoft itself is a lousy citizen of Windows.
    `source` is repeatedly created by Visual Studio (not Code), especially on a fresh install.
    The rest of the projects are open source, and have active/on-going issues related to home directory pollution:

    #### 4.x&emsp;Video games
    - `dotnet` CLI and `.templateengine`: [issue closed](https://github.com/dotnet/sdk/issues/8678), but requires setting `DOTNET_CLI_HOME` environment variable. Otherwise defaults to previous behaviour, and still doesn't fix `.dotnet`;
    - `nuget`: [issue still open](https://github.com/dotnet/sdk/issues/4650);
    - `omnisharp`: [issue still open](https://github.com/OmniSharp/omnisharp-roslyn/issues/953);
    - `vscode`: [issue still open](https://github.com/microsoft/vscode/issues/3884).
    The comic nearby seems to embody the internal structure of Microsoft: teams are siloed, are pit against each other, and the end-user loses out.
    #### 4.2.X&emsp;Video games
    Video games on Windows deserve a special heading of their own, because they are *particularly* egregious offenders.
    In my opinion, game devs *really* have no excuse: almost all games primarily target Windows, or are ported by dedicated teams who know Windows inside out, and more importantly, develop using an entirely native toolchain.
    Even Unreal and Unity have functionality to call out to native code, and developers *still* don't use it to properly handle save data.
    The issue is such that the PCGamingWiki

    Most video games seem to like writing their save files, screenshots, and settings data into `shell:Documents\` or `shell:Documents\My Games`, which is a holdover from Windows XP.
    Look, I get it, many game engines are old; they trace their lineage to engines first written in the 1990s and early 2000s.
    But if developers can add a ray-tracing update to a 7-year-old game (*Witcher 3: Wild Hunt*), surely they can spend a couple hours fixing this too (not that the update itself was great: it broke the game's performance for me; I used to be able to run it at max settings, 4K resolution, and at 100 FPS, which was cut by two-thirds after said update, even *without* ray-tracing... 😕)
    Here is a listing of my `Documents` library, which illustrates the problem:
    ``` gci -Name D:\Libraries\Documents
    3DMark
    4A Games
    Anno 1404
    Anno 1404 Venecia
    Anno 1404 Venedig
    Anno 1404 Venezia
    Anno 1404 Venice
    Anno 1404 Venise
    ANNO 1404 Wenecja
    Anno 1800
    Assassin's Creed Odyssey
    Assassin's Creed Unity
    Battlefield 1
    Blackmagic Design
    Custom Office Templates
    Dell
    en-GB
    FeedbackHub
    Graphics
    Horizon Zero Dawn
    IISExpress
    Larian Studios
    MATLAB
    MAXON
    My Data Sources
    My Digital Editions
    My Games
    My Spore Creations
    My Web Sites
    OnScreen Control
    Outlook Files
    PowerShell
    PowerToys
    Rockstar Games
    Shadow of the Tomb Raider
    Sound recordings
    Steam Cloud
    The Witcher 3
    The Witcher 3 Mod Manager
    Visual Studio 2019
    Visual Studio 2022
    WindowsPowerShell
    Witcher 2
    Wolfram Mathematica
    Zoom
    _Documents
    enc_cert.pfx
    mods.settings
    ```
    *Anno 1404*, what are you doing? *Six* different (localised) directories, all containing the same paths.
    Honestly speaking, though, I doubt game developers are going to change to `shell:Saved Games`, and this is partially Microsoft's fault, again.
    The folder is neither listed in the default File Explorer `Libraries` view, nor in the `My PC` view; users have to manually navigate to it.
    Its rarity is demonstrated by the ratio of games using either `Documents` or `Documents\My Games` compared to those using `Saved Games`: **32:4** on my computer.
    What a pity.
    Notice even here that Microsoft is polluting `Documents` with `FeedbackHub`, `My Data Sources`, `Outlook Files`, `Visual Studio 20XX`, `PowerShell`, `WindowsPowershell`.
    The folder contains anything *but* documents now, and it is so cluttered that I have the `_Documents` subfolder for *my* stuff, things I saved via a save dialog box or the command-line.
    The leading underscore is a necessary evil, as it means the directory is listed first when sorted by name.
    </div>
    In a nutshell... A hopeless situation.
    There's no saving Windows user profiles when Microsoft itself doesn't adhere to its own conventions.
  7. @sharadhr sharadhr revised this gist Jul 28, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -402,8 +402,8 @@ Many programs on desktop OSs are several gigabytes large, and naïvely syncing t

    In fact, a good rule of thumb is as follows:

    [!IMPORTANT]
    Programs should only ever write to the top level home directory (or `Documents`, or `Desktop`, or any of the 'media' directories) when *explicitly asked to by the user*, via a save dialog prompt/box.
    >[!IMPORTANT]
    > Programs should only ever write to the top level home directory (or `Documents`, or `Desktop`, or any of the 'media' directories) when *explicitly asked to by the user*, via a save dialog prompt/box.
    <img src="https://imgs.xkcd.com/comics/workflow.png" width="200" align="left" hspace="10">

  8. @sharadhr sharadhr revised this gist Jul 28, 2023. 1 changed file with 162 additions and 7 deletions.
    169 changes: 162 additions & 7 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -17,7 +17,12 @@
    - [2.2.4 Windows XP to Vista changes](#224windows-xp-to-vista-changes)
    - [2.2.5 `HKEY_CURRENT_USER` registry hive](#225hkey_current_user-registry-hive)
    - [2.3 Linux and Windows equivalents](#23linux-and-windows-equivalents)
    - [3 The broken conventions](#3the-broken-conventions)
    - [3 Reasons for conventions, and for breaking them](#3reasons-for-conventions-and-for-breaking-them)
    - [3.1 Screw your conventions, *we've always done it this way*](#31screw-your-conventions-weve-always-done-it-this-way)
    - [3.1  Linux](#31-linux)
    - [3.1.1  My `$HOME`](#311-my-home)
    - [4 Windows](#4windows)
    - [4.x Video games](#4xvideo-games)

    <div align="justify">

    @@ -84,7 +89,7 @@ The second section details *how* these conventions are broken on each platform,

    This is certainly a bit of a soapbox, but I also hope developers read this, and at least *attempt* to fix their software so that home directories are cleaner, and users have easier lives maintaining and using their computers.

    **If you want to skip all the setup drudgery, go to [section 3](#3the-broken-conventions).**
    **If you want to skip all the setup drudgery, go to

    ## 2&emsp;User profile setup and conventions

    @@ -153,11 +158,11 @@ If you *do* want to list hidden files with `ls`, use `ls -a` or `ls -A`.
    The [XDG Base Directory specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) defines several environment variables and subdirectories of the user profile directory, in an attempt to standardise dot-files and dot-directories.
    I summarise them below:

    | **Variable** | **Default value** | **Details** |
    | **Variable** | **Default value** | **Details** |
    |--------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
    | `$XDG_DATA_HOME` | `$HOME/.local/share` | User-specific data files; e.g. program databases, caches that persist through multiple program runs, search indices, 'Trash' directory for desktop environments. |
    | `$XDG_CONFIG_HOME` | `$HOME/.config` | User-specific configuration files, including `.*rc` and `.*env` files; VS Code `settings.json`. |
    | `$XDG_STATE_HOME` | `$HOME/.local/state` | User-specific state files. |
    | `$XDG_STATE_HOME` | `$HOME/.local/state` | User-specific state files, such as terminal history files. |
    | `$XDG_CACHE_HOME` | `$HOME/.cache` | Caches limited to single runs of a program, but can extend to persistent caches, e.g. user-installed package manager caches for `pip`, `pacman` AUR wrappers, `vcpkg`, etc. |

    Notice the specification provides for the scenario that these environment variables are *not* defined:
    @@ -283,7 +288,7 @@ So, the graphical setting for the various libraries update the locations returne
    ##### 2.2.3.1&emsp;Hidden folders on Windows

    In particular, the parent directory of `%APPDATA%` and `%LOCALAPPDATA%`, `%USERPROFILE%\AppData`, is a *hidden* directory.
    Unlike Linux, 'hidden' on Windows has special semantics accorded to the file or directory, by the filesystem (usually NTFS).
    Unlike \*nix, 'hidden' on Windows has special semantics accorded to the file or directory, by the filesystem (usually NTFS).
    This can be set either graphically (right click → `Properties``General` tab → `Hidden` checkbox under `Attributes`), or retrieved/set programmatically in the Windows API, using [`GetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw) and [`SetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw).
    These functions return or accept a bitwise-ORed [file attribute constant](https://learn.microsoft.com/en-sg/windows/win32/fileio/file-attribute-constants).

    @@ -348,7 +353,7 @@ This hive is synchronised across terminals on roaming user profiles (such as wit

    ### 2.3&emsp;Linux and Windows equivalents

    So far, I've discussed user profiles on Linux and Windows. There are some rough equivalents, which ought to be useful for developers aiming to write *well-behaved* cross-platform applications:
    So far, I've discussed user profiles on Linux and Windows. There are some rough equivalents between the two, which ought to be useful for developers aiming to write *well-behaved* cross-platform applications:

    | **Use-case** | **Linux** | **Windows** |
    |-------------------------------------------|--------------------|----------------------------------------------------------------------------------------------------------------------------|
    @@ -361,6 +366,156 @@ So far, I've discussed user profiles on Linux and Windows. There are some rough
    | Per-user program data | `$XDG_DATA_HOME` | `%LOCALAPPDATA%` |
    | Per-user program cache | `$XDG_CACHE_HOME` | `%APPDATA%` |

    ## 3&emsp;The broken conventions
    ## 3&emsp;Reasons for conventions, and for breaking them

    Okay; that was a pretty long introduction, but I haven't really gotten into *why* these conventions have been established in the first place.
    These specifications don't exist merely to 'put things somewhere' for the hell of it.
    They are there so that disparate applications from a wide variety of developers and backgrounds have a single place to write to, and a single point of backup, which reduces user workload.
    As I mentioned in the introduction, software ought not to be surprising or unnecessarily opinionated.
    Users see a specification, or some clearly-labelled pre-generated folder in their home directory, and *expect* software to write to sensible locations.

    GitHub user Lyle Hanson ([@lhanson](https://github.com/lhanson)) puts it more clearly than I could, [in the issue for Vim](https://github.com/vim/vim/issues/2034#issuecomment-559254363):

    > The benefit of respecting the specification isn't just to put it "somewhere else", it's to put it *where I (the user) want it, without having to repeat myself every time I install anything.*
    >
    > ... I may eventually get around to trying to back up and/or version control my configuration files without hauling around everything else on my drive and I'll be delighted that most of them seem to be in `~/.config`.
    > When I have to add exceptions and regex matches for every program which stores its files in what seem to me like a jumble of random locations, the fact that a given program has done it that way for 30 years doesn't make my job any easier or less frustrating than for a program that was written last week without knowing any better.
    >
    > ... From a UX point of view, I'd rather express my preferences at most once and be able to leverage reasonable assumptions later as my file management habits evolve than to have to express my preferences $n \times m$ times, where $n$ is the number of programs I use and $m$ is the number of unique configuration options each of them exposes to specify where all of their files go.
    >
    > ... It's not about simply appeasing a subset of Vim users who for some reason have glommed onto some new-fangled specification, it's about *respecting users in general and making things easier for them by default.*
    I'd like to tack on to his point about 'respecting users': just because the user's home directory and its contents have full read-write-execute permissions for the user does not mean that software executed by said user should have free rein over the directory.

    The default assumption by a programmer might be:

    > Let's put config and data files in the same directory as the program.
    This assumption immediately breaks on most modern desktop operating systems, because programs are typically installed to locations which require elevated write permissions (e.g. `/bin`, `%PROGRAMFILES%`) which means neither users nor their processes can write there, willy-nilly.
    Furthermore, most desktop OSs are multi-user, which means handling several different configurations and data for multiple users.
    Even *Android*, a smartphone OS, [supports multiple user profiles since 5.0 Lollipop](https://www.android.com/versions/lollipop-5-0/#:~:text=Multiple%20users%20for%20phones.), which means providing ways for developers to handle different user profiles.

    From my experiments, apps and their data are completely sandboxed per-user on Android, so apps installed by one user can be neither seen nor accessed by another user.
    This is workable on desktop OSs, except that it still doesn't really solve the backup problem: do we just sync the entire app and its contents to a server?
    Many programs on desktop OSs are several gigabytes large, and naïvely syncing this much data is a waste of bandwidth.


    In fact, a good rule of thumb is as follows:

    [!IMPORTANT]
    Programs should only ever write to the top level home directory (or `Documents`, or `Desktop`, or any of the 'media' directories) when *explicitly asked to by the user*, via a save dialog prompt/box.

    <img src="https://imgs.xkcd.com/comics/workflow.png" width="200" align="left" hspace="10">

    ### 3.1&emsp;Screw your conventions, *we've always done it this way*

    We come to the real crux of the matter: why and how developers break the above-mentioned conventions and standards.
    Developers cite only a few reasons for not wanting to adhere to conventions, and the following are lifted *verbatim* from their respective issue trackers (many closed as 'won't fix'):

    [Arduino](https://github.com/arduino/arduino-cli/issues/1538#issuecomment-961800324):
    > It's a change to an existing and established behaviour, a breaking change for all existing users too.
    [Bash](https://savannah.gnu.org/support/?108134#comment3):
    > There's no reason to change historical behavior here. All the world is not Linux.
    OpenSSH ([archived page](https://web.archive.org/web/20190925004614/https://bugzilla.mindrot.org/show_bug.cgi?id=2050#c1), because the original bug report is inaccessible to guest users):
    > No. OpenSSH (and it's ancestor `ssh-1.x`) have a 17 year history of using `~/.ssh`. This location is baked into innumerable users' brains, millions of happily working configurations and countless tools.
    > Changing the location of our configuration would require a very strong justification and following a trend of desktop applications (of which OpenSSH is not) is not sufficient.
    [RenderDoc](https://github.com/baldurk/renderdoc/pull/1741#issuecomment-592477184):
    > I've thought about it but I don't want to accept this change. The additional complexity and bug surface is not worth the value added by the feature.
    > ... I don't want to add dependency on those environment variables.
    [Flatpak](https://github.com/flatpak/flatpak/issues/1651#issuecomment-396370107):
    > There is no actual problem here.
    ### 3.1&emsp; Linux

    #### 3.1.1&emsp; My `$HOME`

    Before I proceed, I provide a listing of my own `$HOME` on my Arch Linux install, which has several thousand packages:

    <details>
    <summary>My `$HOME`</summary>

    ```zsh
    % ls -A1 --group-directories-first $HOME
    .android
    .astropy
    .cache
    .cgdb
    .config
    .dotnet
    .enthought
    .gitkraken
    .gnome
    .gnupg
    .icons
    .java
    .local
    .miktex
    .mozilla
    .npm
    .nuget
    .omnisharp
    .pki
    .pytest_cache
    .renpy
    .rustup
    .sonarlint
    .sqlsecrets
    .ssh
    .stellarium
    .swt
    .templateengine
    .thunderbird
    .vcpkg
    .vscode
    .zotero
    Desktop
    Documents
    Downloads
    Music
    OneDrive
    Pictures
    Public
    Templates
    Videos
    VirtualBox VMs'
    Zotero
    bin
    enthought
    .Xauthority
    .bash_history
    .fonts.conf
    .gtkrc-2.0
    .lesshst
    .nvidia-settings-rc
    .ocamlinit
    .python_history
    .utop-history
    .viminfo
    .vimrc
    .wget-hsts
    vkvia.html
    ```
    </details>
    Of the above directories and files, I created exactly *two* manually: `OneDrive`, and `bin`.
    The rest (excluding the XDG directories) were either auto-generated by `xdg-user-dirs` (which is okay), or created and written to by non-compliant software.
    The link above discussing the origin of dot-files makes it clear that the current behaviour was possibly *a mistake* made by the original developers of UNIX.
    Even if it wasn't, the result today is a messy litter of dot-files dumped all over a user's home directory.
    The XDG Base Directory (hereafter, XBD) specification is more than a decade old—an eternity in computing terms—and yet, there is a veritable parade of very well-known programs that refuse to follow its guidelines, typically citing 'backwards compatibility' and 'not breaking existing workflows' as reasons.
    The Arch Wiki has [a list of programs](https://wiki.archlinux.org/title/XDG_Base_Directory#Support) that are XBD-compliant by default, may be forced to comply after user intervention, and those with hard-coded non-XBD paths.
    The latter two lists combined is almost *twice* as long as the compliant list, and includes some very prominent \*nix-first software like Bash, Vim, OpenSSH, and Firefox.
    ### 4&emsp;Windows
    #### 4.x&emsp;Video games
    Video games on Windows deserve a special heading of their own, because they are *particularly* egregious offenders.
    </div>
  9. @sharadhr sharadhr revised this gist Jul 27, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -282,8 +282,8 @@ So, the graphical setting for the various libraries update the locations returne

    ##### 2.2.3.1&emsp;Hidden folders on Windows

    In particular, `%APPDATA%` and `%LOCALAPPDATA%` are hidden directories.
    Unlike Linux, a directory or file that is 'hidden' on Windows has special semantics accorded to it by the filesystem (usually NTFS).
    In particular, the parent directory of `%APPDATA%` and `%LOCALAPPDATA%`, `%USERPROFILE%\AppData`, is a *hidden* directory.
    Unlike Linux, 'hidden' on Windows has special semantics accorded to the file or directory, by the filesystem (usually NTFS).
    This can be set either graphically (right click → `Properties``General` tab → `Hidden` checkbox under `Attributes`), or retrieved/set programmatically in the Windows API, using [`GetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw) and [`SetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw).
    These functions return or accept a bitwise-ORed [file attribute constant](https://learn.microsoft.com/en-sg/windows/win32/fileio/file-attribute-constants).

  10. @sharadhr sharadhr revised this gist Jul 27, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -356,6 +356,7 @@ So far, I've discussed user profiles on Linux and Windows. There are some rough
    | Headers | `/usr/include` | Shipped with the program, or available with Windows SDK |
    | System/program libraries | `/usr/lib` | `%WINDIR%\System32` or shipped with the program; Visual C++ and .NET redistributables pre-installed or installed on-demand |
    | Default/system-wide program configuration | `/etc` | `%PROGRAMDATA%` |
    | System-wide logs | `/var/log` | `%PROGRAMDATA%` |
    | Per-user program configuration | `$XDG_CONFIG_HOME` | `%APPDATA%` or `%LOCALAPPDATA%` |
    | Per-user program data | `$XDG_DATA_HOME` | `%LOCALAPPDATA%` |
    | Per-user program cache | `$XDG_CACHE_HOME` | `%APPDATA%` |
  11. @sharadhr sharadhr revised this gist Jul 27, 2023. 1 changed file with 118 additions and 25 deletions.
    143 changes: 118 additions & 25 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -13,9 +13,11 @@
    - [2.2.1 Environment variables](#221environment-variables)
    - [2.2.2 User profile creation and naming](#222user-profile-creation-and-naming)
    - [2.2.3 Libraries and `KNOWNFOLDERID`s](#223libraries-and-knownfolderids)
    - [2.2.3.1 Hidden folders on Windows](#2231hidden-folders-on-windows)
    - [2.2.4 Windows XP to Vista changes](#224windows-xp-to-vista-changes)
    - [2.2.5 `HKEY_CURRENT_USER` registry hive](#225hkey_current_user-registry-hive)
    - [2.3 Linux and Windows equivalents](#23linux-and-windows-equivalents)
    - [3 The broken conventions](#3the-broken-conventions)

    <div align="justify">

    @@ -82,17 +84,18 @@ The second section details *how* these conventions are broken on each platform,

    This is certainly a bit of a soapbox, but I also hope developers read this, and at least *attempt* to fix their software so that home directories are cleaner, and users have easier lives maintaining and using their computers.

    **If you want to skip all the setup drudgery, go to [section 3](#3the-broken-conventions).**

    ## 2&emsp;User profile setup and conventions

    Before I begin, there are some platform-specific details and wording *conventions* to be discussed.
    I will use the terms 'home directory' and 'user profile directory' somewhat interchangeably in this post, whereas 'user profile' means the home directory itself and its contents, including user-specific configuration and data, *combined*.
    On Windows, the latter also includes the `HKEY_CURRENT_USER` registry hive (abbreviated as `HKCU`), stored in `%USERPROFILE%\NTUSER.dat`.
    I will use the terms 'home directory' and 'user profile directory' somewhat interchangeably in this post, whereas 'user profile' means the home directory itself and its contents—including user-specific configuration and data—*combined*.

    ### 2.1&emsp;Linux

    Linux is famously fragmented, but even so, there exist *some* conventions for user profiles, especially for desktop environments.
    As far as I've seen, Linux user profiles are typically created in `/home/<username>`.
    The directory path may be chosen during the out-of-box experience (OOBE)/first-time setup of a Linux distro, or entirely manually, with [`useradd -d`](https://man7.org/linux/man-pages/man8/useradd.8.html#:~:text=%2Dd%2C%20%2D%2Dhome%2Ddir%20HOME_DIR).
    The directory path may be chosen during the out-of-box experience (OOBE)/first-time setup of a Linux distro, or entirely manually, with [`useradd -d`](https://man7.org/linux/man-pages/man8/useradd.8.html#:~:text=%2Dd%2C%20%2D%2Dhome%2Ddir%20HOME_DIR), which writes to `/etc/passwd`.
    Sometimes, `/home` might occupy a separate partition/sub-volume altogether (if the user is using Btrfs, for instance).

    #### 2.1.1&emsp;`$HOME`
    @@ -123,18 +126,20 @@ auto main() -> int
    }
    ```

    Strictly speaking, though, there is no real concept of `home` directory *per se* on Linux/Unix (hereafter referred to as \*nix), and they aren't treated any differently by the OS (unlike Windows, as seen below).
    Strictly speaking, though, there is no real concept of a home directory *per se* on Linux/UNIX (hereafter referred to as \*nix), and they aren't treated any differently by the OS (unlike Windows, as seen below).
    The `pw_dir` member variable is just the initial working directory of the login shell and any subsequent shells started by the corresponding user; it could technically point to *any* directory that has read permissions for said user.
    The `pwd.h` manual [states as much](https://man7.org/linux/man-pages/man0/pwd.h.0p.html#:~:text=char%20%20%20%20*pw_dir%20%20%20%20Initial%20working%20directory).

    Most shells and desktop environments also parse `~` as an alias to `$HOME`; `cd` without any command-line arguments also [navigates to `$HOME`](https://man7.org/linux/man-pages/man1/cd.1p.html#:~:text=2.%20If%20no%20directory%20operand%20is%20given%20and%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20is%20set%20to%20a%20non%2Dempty%20value%2C%20the%20cd%20utility%20shall%0A%20%20%20%20%20%20%20%20%20%20%20behave%20as%20if%20the%20directory%20named%20in%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20was%20specified%20as%20the%20directory%20operand.).
    Most shells and desktop environments parse `~` as an alias to `$HOME`, and `cd` without any command-line arguments also [navigates to `$HOME`](https://man7.org/linux/man-pages/man1/cd.1p.html#:~:text=2.%20If%20no%20directory%20operand%20is%20given%20and%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20is%20set%20to%20a%20non%2Dempty%20value%2C%20the%20cd%20utility%20shall%0A%20%20%20%20%20%20%20%20%20%20%20behave%20as%20if%20the%20directory%20named%20in%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20was%20specified%20as%20the%20directory%20operand.).

    #### 2.1.2&emsp;Dot-files and dot-directories

    On \*nix, prefixing a file or directory with a full-stop (`.`) excludes said path from being listed in userland utilities, such as `ls`, or graphical file managers (at least, according to their defaults).
    These files are considered 'hidden', although no special meaning is given to them by the filesystem.
    These are called dot-files, and they are typically used to set user-specific configuration for almost all programs on Linux and Unix.
    They typically reside in the top level of the user profile directory; for instance, the Vim configuration is in `$HOME/.vimrc`.
    These files are considered 'hidden', although no special meaning is given to them by the filesystem itself (also unlike on Windows).
    This is *convention* dating to the [earliest days of UNIX](https://web.archive.org/web/20190211031031/https://plus.google.com/+RobPikeTheHuman/posts/R58WgWwN9jp).

    Most Linux users' home directories will contain a collection of these *dot-files* and *dot-directories*, and they are typically used to set user-specific configuration for almost all programs on \*nix.
    They typically reside in the top level of `$HOME`; for instance, the Vim configuration is in `$HOME/.vimrc`.
    If you *do* want to list hidden files with `ls`, use `ls -a` or `ls -A`.

    #### 2.1.3&emsp;XDG Base Directories
    @@ -148,18 +153,46 @@ If you *do* want to list hidden files with `ls`, use `ls -a` or `ls -A`.
    The [XDG Base Directory specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) defines several environment variables and subdirectories of the user profile directory, in an attempt to standardise dot-files and dot-directories.
    I summarise them below:

    | **Variable** | **Default value** | **Details** |
    |--------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
    | `$XDG_DATA_HOME` | `$HOME/.local/share` | User-specific data files; e.g. program databases, caches that persist through multiple program runs, search indices, 'Trash' directory for desktop environments. |
    | `$XDG_CONFIG_HOME` | `$HOME/.config` | User-specific configuration files, including `.*rc` and `.*env` files; VS Code `settings.json`. |
    | `$XDG_STATE_HOME` | `$HOME/.local/state` | User-specific state files. |
    | `$XDG_CACHE_HOME` | `$HOME/.cache` | Caches limited to single runs of a program, but can extend to persistent caches, e.g. user-installed package manager caches for `pip`, `pacman` AUR wrappers, `vcpkg`, etc. |

    Notice the specification provides for the scenario that these environment variables are *not* defined:

    > If `$XDG_CONFIG_HOME` is either not set or empty, a default equal to `$HOME/.config` should be used.
    #### 2.1.4&emsp;`xdg-user-dirs`

    Since Linux does not provide an 'official' userland environment (unlike Windows), the XDG people have again made effort to to set up 'Windows-style' user directories *with localisation*: [`xdg-user-dirs`](https://www.freedesktop.org/wiki/Software/xdg-user-dirs/).
    Both GNOME and KDE Plasma Desktop require it.
    This tool is configured with a straightforward script in `$(XDG_CONFIG_HOME)/user-dirs.dirs`.
    The default directories generated on an English-language install are:

    - Desktop
    - Documents
    - Downloads
    - Music
    - Pictures
    - Public
    - Templates
    - Videos

    As always, the [relevant article at the Arch wiki](https://wiki.archlinux.org/title/XDG_user_directories) is very useful.

    #### 2.1.5&emsp;`systemd-homed`

    [`systemd-homed`](https://www.freedesktop.org/software/systemd/man/systemd-homed.service.html) allows Linux administrators to create and manage user profiles, and optionally encrypt them.
    It also has roughly equivalent functionality to Active Directory and roaming user profiles on Windows, where user profiles may be encrypted and stored remotely on some server, for retrieval by terminals upon login.

    ### 2.2&emsp;Windows

    <details>
    <summary>Windows and Microsoft commentary</summary>

    I hold several very controversial opinions about Windows and Microsoft that would raise many hackers' eyebrows.
    I hold several very controversial opinions about Microsoft and Windows that would at best, raise many hackers' eyebrows, and at worst, get me called an astroturfer or Microsoft shill.
    I *was* going to type several paragraphs, but I realised that would be better off as a blog-post on its own—coming soon.
    Not that Microsoft needs any advertising...
    </details>
    @@ -185,10 +218,6 @@ Many are listed [here](https://ss64.com/nt/syntax-variables.html) (Microsoft's [
    | `%APPDATA%` | `%USERPROFILE%\AppData\Roaming` | User-specific *roaming* application data that may be exported between PCs; e.g. Google Chrome user profiles, Steam accounts; likely anything that requires an internet sign-in |
    | `%LOCALAPPDATA%` | `%USERPROFILE%\AppData\Local` | User-specific, PC-specific application data; sometimes includes the entire applications themselves if installed only for the user. e.g. Google Chrome and MiKTeX installed by a non-admin user. |

    In particular, `%APPDATA%` and `%LOCALAPPDATA%` are hidden directories.
    Unlike Linux, a directory or file that is 'hidden' on Windows has special semantics accorded to it by the filesystem (usually NTFS).
    This can be set either graphically (right click → `Properties``General` tab → Hidden` checkbox under `Attributes`), or set programmatically, using

    These environment variables may be accessed from C or C++ with [`getenv()`](https://learn.microsoft.com/en-sg/cpp/c-runtime-library/reference/getenv-wgetenv?view=msvc-170), or any other language-specific API.

    #### 2.2.2&emsp;User profile creation and naming
    @@ -217,7 +246,7 @@ Upon user profile creation, Windows automatically sets up several directories, w
    - Videos

    These libraries are initially created under `%USERPROFILE%`, but may be moved (right click → `Properties``Location` tab → `Move...`) by the user at their own discretion.
    The Windows API also provides enumerated GUIDs called [`KNOWNFOLDERID`](https://learn.microsoft.com/en-sg/windows/win32/shell/knownfolderid)s that map to these directories, as well as *most* of the system directories listed in [Environment variables](#221environment-variables).
    The Windows API provides enumerated GUIDs called [`KNOWNFOLDERID`](https://learn.microsoft.com/en-sg/windows/win32/shell/knownfolderid)s that map to these directories, as well as *most* of the system directories listed in [Environment variables](#221environment-variables).
    These may be queried with [`SHGetKnownFolderPath`](https://learn.microsoft.com/en-sg/windows/win32/api/shlobj_core/nf-shlobj_core-shgetknownfolderpath), like so:

    ```c++
    @@ -229,14 +258,14 @@ These may be queried with [`SHGetKnownFolderPath`](https://learn.microsoft.com/e
    auto main() -> int
    {
    auto path = PWSTR{};
    auto const result = SHGetKnownFolderPath(FOLDERID_Desktop, KF_FLAG_DEFAULT, nullptr,
    &path); // get desktop path; return type HRESULT
    if (SUCCEEDED(result)) {
    wprintf_s(L"%ls\n", path);
    return 0;
    } else {

    // get desktop path; return type HRESULT
    if (auto const result = SHGetKnownFolderPath(FOLDERID_Desktop, KF_FLAG_DEFAULT, nullptr, &path);
    FAILED(result)) {
    return 1;
    }
    wprintf_s(L"%ls\n", path);
    return 0;
    }
    ```

    @@ -251,22 +280,86 @@ D:\Libraries\Desktop
    I have set all my libraries to point to `D:\Libraries\<library>`, because my `C:` drive is rather small and dedicated to the OS and important programs only, whereas `D:` is significantly larger.
    So, the graphical setting for the various libraries update the locations returned by the native Windows API too (which is eventually also called by .NET).

    ##### 2.2.3.1&emsp;Hidden folders on Windows

    In particular, `%APPDATA%` and `%LOCALAPPDATA%` are hidden directories.
    Unlike Linux, a directory or file that is 'hidden' on Windows has special semantics accorded to it by the filesystem (usually NTFS).
    This can be set either graphically (right click → `Properties``General` tab → `Hidden` checkbox under `Attributes`), or retrieved/set programmatically in the Windows API, using [`GetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw) and [`SetFileAttributes`](https://learn.microsoft.com/en-sg/windows/win32/api/fileapi/nf-fileapi-getfileattributesw).
    These functions return or accept a bitwise-ORed [file attribute constant](https://learn.microsoft.com/en-sg/windows/win32/fileio/file-attribute-constants).

    ```c++
    #include <cstdio>
    #include <ShlObj.h>
    #include <pathcch.h>

    #pragma comment(lib, "Shell32.lib")
    #pragma comment(lib, "Pathcch.lib")

    auto main() -> int
    {
    auto path = PWSTR{};

    if (auto const get_path_result = SHGetKnownFolderPath(FOLDERID_RoamingAppData, KF_FLAG_DEFAULT, nullptr, &path);
    FAILED(get_path_result)) {
    return 1;
    }

    printf("SHGetKnownFolderPath succeeded, path is %ws\n", path);

    // get parent path of %APPDATA%, i.e. `%USERPROFILE%\AppData`
    if (auto const remove_leaf_result = PathCchRemoveFileSpec(path, MAX_PATH);
    FAILED(remove_leaf_result)) {
    return 1;
    }

    if (auto const attributes = GetFileAttributesW(path); attributes == INVALID_FILE_ATTRIBUTES) {
    printf("GetFileAttributes failed with error code %d\n", GetLastError());
    return 1;
    } else if (attributes & FILE_ATTRIBUTE_HIDDEN) {
    printf("%%APPDATA%%\\.. is hidden\n");
    return 0;
    } else {
    printf("%%APPDATA%%\\.. is not hidden; something is wrong\n");
    return 1;
    }
    }
    ```

    #### 2.2.4&emsp;Windows XP to Vista changes

    [*A Brief History of Windows Profiles*](https://archive.ph/eOGDp) is a great article on the history of Windows user profiles, but I want to focus on one important section.
    Windows 2000 and XP used `%SYSTEMDRIVE%\Documents and Settings\<username>` for the user profile, which was moved to `%SYSTEMDRIVE%\Users\<username>` with Vista and later.
    Vista also introduced the above-mentioned `KNOWNFOLDERID`s, which superseded the older [`CSIDL`](https://learn.microsoft.com/en-sg/windows/win32/shell/csidl) enumeration (although the latter is still available).
    [*A Brief History of Windows Profiles*](https://archive.ph/eOGDp) is a great article, but I want to focus on one important section.
    Windows 2000 and XP used `%SYSTEMDRIVE%\Documents and Settings\<username>` for the home directory, which was moved to `%SYSTEMDRIVE%\Users\<username>` with Vista and later.
    Windows Vista also introduced the above-mentioned `KNOWNFOLDERID`s, which superseded the older [`CSIDL`](https://learn.microsoft.com/en-sg/windows/win32/shell/csidl) enumeration (although the latter is still available).

    Many environment variables like `%APPDATA%` also were changed to point to the new locations (the previous was a mouthful: `%SYSTEMDRIVE%\Documents and Settings\<username>\Application Data`).
    Many environment variables like `%APPDATA%` also were changed to point to the new locations (the previous was a mouthful: `%SYSTEMDRIVE%\Documents and Settings\<username>\Local Settings\Application Data`).
    New known folders were added, such as `Downloads` and `Saved Games`; the `My⎵` prefix was removed from `My Documents` and `My Music`; locations such as `Start Menu` were moved (in this case, to `%APPDATA%\Microsoft\Windows\Start Menu`).

    The reason for this move is anyone's guess, but I feel the $\geq$ Vista convention makes a lot more sense than the $\leq$ Windows XP one.
    There is an [old guide](https://web.archive.org/web/20120621131135/http://technet.microsoft.com:80/en-us/library/cc766489(v=ws.10).aspx) for sysadmins migrating from Windows XP to Vista.
    There is an [old guide](https://web.archive.org/web/20120621131135/http://technet.microsoft.com:80/en-us/library/cc766489(v=ws.10).aspx) for sysadmins migrating from Windows XP to Vista, and there is another blog post for Windows application developers: [*Where Should I Store my Data and Configuration Files if I Target Multiple OS Versions?*](https://learn.microsoft.com/en-sg/archive/blogs/patricka/where-should-i-store-my-data-and-configuration-files-if-i-target-multiple-os-versions#targeting-vista-and-higher)

    As an aside, there is [an apocryphal tale](https://news.ycombinator.com/item?id=29187896) for the reason why programs are stored in `Program Files` and not `ProgramFiles` or `Program_Files`: apparently Microsoft wanted to force programmers to write code defensively, and handle spaces in paths without crashing, so they made the program install location itself have spaces in it (this then raises the question: why `ProgramData`?)
    [Windows also has decent localisation](https://archive.ph/Eqvd4) (at least amongst Indo-European languages employing the Latin script): set the system language to German, for instance, and `Program Files` is now `Programme`; it is `Programfiler` in Norwegian.

    #### 2.2.5&emsp;`HKEY_CURRENT_USER` registry hive

    On Windows, many user-specific configurations are stored in the `HKEY_CURRENT_USER` registry hive (abbreviated as `HKCU`), stored in `%USERPROFILE%\NTUSER.dat`.
    This includes changes made in both Control Panel and Windows Settings, as well as settings for Microsoft programs like the Office suite.
    This hive is synchronised across terminals on roaming user profiles (such as with Active Directory and domains).

    ### 2.3&emsp;Linux and Windows equivalents

    So far, I've discussed user profiles on Linux and Windows. There are some rough equivalents, which ought to be useful for developers aiming to write *well-behaved* cross-platform applications:

    | **Use-case** | **Linux** | **Windows** |
    |-------------------------------------------|--------------------|----------------------------------------------------------------------------------------------------------------------------|
    | Program install directory | `/bin` | `%PROGRAMFILES%` or `%PROGRAMFILES(X86)%` |
    | Headers | `/usr/include` | Shipped with the program, or available with Windows SDK |
    | System/program libraries | `/usr/lib` | `%WINDIR%\System32` or shipped with the program; Visual C++ and .NET redistributables pre-installed or installed on-demand |
    | Default/system-wide program configuration | `/etc` | `%PROGRAMDATA%` |
    | Per-user program configuration | `$XDG_CONFIG_HOME` | `%APPDATA%` or `%LOCALAPPDATA%` |
    | Per-user program data | `$XDG_DATA_HOME` | `%LOCALAPPDATA%` |
    | Per-user program cache | `$XDG_CACHE_HOME` | `%APPDATA%` |

    ## 3&emsp;The broken conventions

    </div>
  12. @sharadhr sharadhr revised this gist Jul 27, 2023. 1 changed file with 121 additions and 32 deletions.
    153 changes: 121 additions & 32 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -1,14 +1,21 @@
    # `$HOME`, Not So Sweet `$HOME`

    - [Preface](#preface)
    - [What is `$HOME` to you?](#what-is-home-to-you)
    - [User profile setup and conventions](#user-profile-setup-and-conventions)
    - [Linux](#linux)
    - [Windows](#windows)
    - [Environment variables](#environment-variables)
    - [User profile creation and naming](#user-profile-creation-and-naming)
    - [Libraries and `KNOWNFOLDERID`s](#libraries-and-knownfolderids)
    - [Windows XP to Vista changes](#windows-xp-to-vista-changes)
    - [1 What is `$HOME` to you?](#1what-is-home-to-you)
    - [2 User profile setup and conventions](#2user-profile-setup-and-conventions)
    - [2.1 Linux](#21linux)
    - [2.1.1 `$HOME`](#211home)
    - [2.1.2 Dot-files and dot-directories](#212dot-files-and-dot-directories)
    - [2.1.3 XDG Base Directories](#213xdg-base-directories)
    - [2.1.4 `xdg-user-dirs`](#214xdg-user-dirs)
    - [2.1.5 `systemd-homed`](#215systemd-homed)
    - [2.2 Windows](#22windows)
    - [2.2.1 Environment variables](#221environment-variables)
    - [2.2.2 User profile creation and naming](#222user-profile-creation-and-naming)
    - [2.2.3 Libraries and `KNOWNFOLDERID`s](#223libraries-and-knownfolderids)
    - [2.2.4 Windows XP to Vista changes](#224windows-xp-to-vista-changes)
    - [2.2.5 `HKEY_CURRENT_USER` registry hive](#225hkey_current_user-registry-hive)
    - [2.3 Linux and Windows equivalents](#23linux-and-windows-equivalents)

    <div align="justify">

    @@ -21,9 +28,10 @@ This was supposed to be a blog post, but I have neither the knowledge, nor the t
    Of course, I *want* one, but I also want to use a fully .NET Core-based static site generator, and therefore have experimented with [Statiq.Dev Web](https://www.statiq.dev/web).
    However, it turns out that I still need to pick up non-trivial HTML and CSS to make it look like anything but a website from the 1990s.

    For now, though, GitHub-flavoured Markdown handles almost all use-cases I can think of (embedded images, this spoiler you've opened, complicated lists, in-line HTML tags for <kbd>Ctrl</kbd>, in-line $\LaTeX{}$ for $\text{maths} \equiv \text{fun}$...), and GitHub Gists renders Markdown as formatted text anyway, so what else do I need?
    For now, though, GitHub-flavoured Markdown handles almost all use-cases I can think of (embedded images, this spoiler you've opened, complicated lists, in-line HTML for <kbd>Ctrl</kbd>, in-line $\LaTeX{}$ for $\text{maths} \equiv \text{fun}$...), and GitHub Gists renders Markdown as formatted text anyway, so what else do I need?
    I can even pin these gists to my GitHub profile, which is also very nice.
    If I ever *do* prepare a blog, it should be fairly straightforward to migrate these posts to it: they're just Markdown, after all.
    The only thing (so far) I've noticed that is missing is auto-numbered headings, although that's apparently a matter of CSS styling.

    ___

    @@ -33,7 +41,7 @@ What *really* inspired me to get started was [*Everything that uses configuratio
    I resonate strongly with the author's sentiment, and more importantly, dislike 'opinionated' software that likes to do what its author thinks is best instead of following system conventions.
    </details>

    ## What is `$HOME` to you?
    ## 1&emsp;What is `$HOME` to you?

    Home.
    It is where you're supposed to be most comfortable in.
    @@ -65,33 +73,101 @@ When software authors break these conventions just for the sake of 'opinionation
    Software should *not* be surprising.
    It ought to be reliable, reproducible, and follow platform conventions and expectations.

    This post is a detailed discussion into how user profile directories on Windows and Linux (I haven't used a Mac in ages, but I assume the situation is very similar there, too) are—to put it bluntly—in total disarray.
    Applications treat the user profile as a dumping ground, and any user with a reasonably wide list of installed software will find their user profile very difficult to traverse.
    This post is a detailed discussion into user profiles, their directories, and how they are—to put it bluntly—in total disarray on Windows and Linux (I haven't used a Mac in ages, but I assume the situation is very similar there, too).
    Applications treat the user profile as a dumping ground, and any user with a reasonably wide list of installed software will find their user profile very difficult to traverse after some time in use.
    There are platform conventions and attempts to standardise things on more open-source platforms, but a lot of developers resolutely refuse to change the behaviour of their software for a variety of reasons (some less valid than others).

    This is a bit of a soapbox, but I also hope developers read this, and at least *attempt* to fix their software so that home directories are cleaner, and users have easier lives maintaining and using their computers.
    The first part is a deep dive into user profiles on Linux and Windows, and the conventions that have been established on these platforms over the years.
    The second section details *how* these conventions are broken on each platform, and *why* they are broken.

    ## User profile setup and conventions
    This is certainly a bit of a soapbox, but I also hope developers read this, and at least *attempt* to fix their software so that home directories are cleaner, and users have easier lives maintaining and using their computers.

    ## 2&emsp;User profile setup and conventions

    Before I begin, there are some platform-specific details and wording *conventions* to be discussed.
    I will use the terms 'home directory' and 'user profile directory' somewhat interchangeably in this post, whereas 'user profile' means the home directory itself and its contents, including user-specific configuration and data, *combined*.
    On Windows, the latter also includes the `HKEY_CURRENT_USER` registry hive (abbreviated as `HKCU`), stored in `%USERPROFILE%\NTUSER.dat`.

    ### Linux
    ### 2.1&emsp;Linux

    Linux is famously fragmented, but even so, there exist *some* conventions for user profiles, especially for desktop environments.
    As far as I've seen, Linux user profiles are typically created in `/home/<username>`.
    The directory path may be chosen during the out-of-box experience (OOBE)/first-time setup of a Linux distro, or entirely manually, with [`useradd -d`](https://man7.org/linux/man-pages/man8/useradd.8.html#:~:text=%2Dd%2C%20%2D%2Dhome%2Ddir%20HOME_DIR).
    Sometimes, `/home` might occupy a separate partition/sub-volume altogether (if the user is using Btrfs, for instance).

    #### 2.1.1&emsp;`$HOME`

    Regardless of its location, the environment variable `$HOME` is set upon login by a login process or graphical display manager (e.g. [`login`](https://man7.org/linux/man-pages/man1/login.1.html), or `gdm`, `sddm`, etc.) based on values previously set in `/etc/passwd`.
    This file may be read using the Linux API:

    ```c++
    #include <cstdio>
    #include <pwd.h>
    #include <sys/types.h>
    #include <unistd.h>

    auto main() -> int
    {
    auto const pw = getpwuid(getuid());
    if (pw != nullptr) {
    printf("User name: %s\n", pw->pw_name);
    printf("User ID: %d\n", pw->pw_uid);
    printf("Group ID: %d\n", pw->pw_gid);
    printf("Home directory: %s\n", pw->pw_dir);
    printf("Login shell: %s\n", pw->pw_shell);

    return 0;
    } else {
    return 1;
    }
    }
    ```

    Strictly speaking, though, there is no real concept of `home` directory *per se* on Linux/Unix (hereafter referred to as \*nix), and they aren't treated any differently by the OS (unlike Windows, as seen below).
    The `pw_dir` member variable is just the initial working directory of the login shell and any subsequent shells started by the corresponding user; it could technically point to *any* directory that has read permissions for said user.
    The `pwd.h` manual [states as much](https://man7.org/linux/man-pages/man0/pwd.h.0p.html#:~:text=char%20%20%20%20*pw_dir%20%20%20%20Initial%20working%20directory).

    Most shells and desktop environments also parse `~` as an alias to `$HOME`; `cd` without any command-line arguments also [navigates to `$HOME`](https://man7.org/linux/man-pages/man1/cd.1p.html#:~:text=2.%20If%20no%20directory%20operand%20is%20given%20and%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20is%20set%20to%20a%20non%2Dempty%20value%2C%20the%20cd%20utility%20shall%0A%20%20%20%20%20%20%20%20%20%20%20behave%20as%20if%20the%20directory%20named%20in%20the%20HOME%20environment%0A%20%20%20%20%20%20%20%20%20%20%20variable%20was%20specified%20as%20the%20directory%20operand.).

    #### 2.1.2&emsp;Dot-files and dot-directories

    ### Windows
    On \*nix, prefixing a file or directory with a full-stop (`.`) excludes said path from being listed in userland utilities, such as `ls`, or graphical file managers (at least, according to their defaults).
    These files are considered 'hidden', although no special meaning is given to them by the filesystem.
    These are called dot-files, and they are typically used to set user-specific configuration for almost all programs on Linux and Unix.
    They typically reside in the top level of the user profile directory; for instance, the Vim configuration is in `$HOME/.vimrc`.
    If you *do* want to list hidden files with `ls`, use `ls -a` or `ls -A`.

    #### 2.1.3&emsp;XDG Base Directories

    <details>
    <summary>XDG</summary>

    [XDG stands for 'Cross-Desktop Group'](https://www.freedesktop.org/wiki/#:~:text=an%20acronym%20for%20the%20Cross%2DDesktop%20Group).
    </details>

    The [XDG Base Directory specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) defines several environment variables and subdirectories of the user profile directory, in an attempt to standardise dot-files and dot-directories.
    I summarise them below:



    #### 2.1.4&emsp;`xdg-user-dirs`

    #### 2.1.5&emsp;`systemd-homed`

    ### 2.2&emsp;Windows

    <details>
    <summary>Windows and Microsoft commentary</summary>

    I hold several very controversial opinions about Windows and Microsoft that would raise many hackers' eyebrows.
    I was going to type several paragraphs, but I realised that would be better off as a blog-post on its own, coming soon.
    I *was* going to type several paragraphs, but I realised that would be better off as a blog-post on its owncoming soon.
    Not that Microsoft needs any advertising...
    </details>

    User profiles on Windows are a fairly complex matter, but this complexity is *good*.
    User profiles on Windows are a rather complex matter, but this complexity is not necessarily unwelcome.
    Most of it is configurable, fairly well-documented, and there is more than one way to do things, although some are clearly better than others.

    #### Environment variables
    #### 2.2.1&emsp;Environment variables

    Modern Windows (i.e. Vista and later) has several conventions and environment variables for important directories, including those within the user profile.
    Many are listed [here](https://ss64.com/nt/syntax-variables.html) (Microsoft's [official documentation](https://learn.microsoft.com/en-sg/windows/deployment/usmt/usmt-recognized-environment-variables) is a little less forthcoming), and I reproduce a few in the table below.
    @@ -105,13 +181,17 @@ Many are listed [here](https://ss64.com/nt/syntax-variables.html) (Microsoft's [
    | `%PROGRAMFILES(X86)%` | `%SYSTEMDRIVE%\Program Files (x86)` | Only on 64-bit installs of Windows. 32-bit-only programs go here. |
    | `%PROGRAMDATA%` | `%SYSTEMDRIVE%\ProgramData` | System-wide program configuration and storage; e.g. default configurations, logs, etc. |
    | `%USERNAME%` | Set in the registry | Can be changed in Control Panel or Settings. |
    | `%USERPROFILE%` | Usually `%SYSTEMDRIVE%\Users\%USERNAME%`; see discussion below | Equivalent to `$HOME` or `~` on Unix/Linux. |
    | `%USERPROFILE%` | Usually `%SYSTEMDRIVE%\Users\%USERNAME%`; see discussion below | |
    | `%APPDATA%` | `%USERPROFILE%\AppData\Roaming` | User-specific *roaming* application data that may be exported between PCs; e.g. Google Chrome user profiles, Steam accounts; likely anything that requires an internet sign-in |
    | `%LOCALAPPDATA%` | `%USERPROFILE%\AppData\Local` | User-specific, PC-specific application data; sometimes includes the entire applications themselves if installed only for the user. e.g. Google Chrome and MiKTeX installed by a non-admin user. |

    In particular, `%APPDATA%` and `%LOCALAPPDATA%` are hidden directories.
    Unlike Linux, a directory or file that is 'hidden' on Windows has special semantics accorded to it by the filesystem (usually NTFS).
    This can be set either graphically (right click → `Properties``General` tab → Hidden` checkbox under `Attributes`), or set programmatically, using

    These environment variables may be accessed from C or C++ with [`getenv()`](https://learn.microsoft.com/en-sg/cpp/c-runtime-library/reference/getenv-wgetenv?view=msvc-170), or any other language-specific API.

    #### User profile creation and naming
    #### 2.2.2&emsp;User profile creation and naming

    This is a mess on Windows. Until Windows 8, there was very little ceremony to user profile creation, which were all local.
    Since then, however, OneDrive (then called SkyDrive) and Microsoft account integration has thoroughly road-rolled over this simplicity.
    @@ -121,7 +201,7 @@ This digression is to discuss what home directory name users end up with: on a p
    For instance, in the first case, I would have `%SYSTEMDRIVE%\Users\shara`, but on a 'local account', it would be `%SYSTEMDRIVE%\Users\Sharadh` (mine is *actually* the latter—I am obsessive about this, so I created a local account first, and *then* synced it to OneDrive).
    Like many things Windows, configuring the user profile directory after account creation requires [editing registry keys](https://learn.microsoft.com/en-sg/troubleshoot/windows-client/user-profiles-and-logon/renaming-user-account-not-change-profile-path) using *another* administrator account.

    #### Libraries and `KNOWNFOLDERID`s
    #### 2.2.3&emsp;Libraries and `KNOWNFOLDERID`s

    <img align="right" width="300" hspace="10" src="https://files.catbox.moe/gh3u24.png">

    @@ -137,12 +217,12 @@ Upon user profile creation, Windows automatically sets up several directories, w
    - Videos

    These libraries are initially created under `%USERPROFILE%`, but may be moved (right click → `Properties``Location` tab → `Move...`) by the user at their own discretion.
    The Windows API also provides enumerated GUIDs called [`KNOWNFOLDERID`](https://learn.microsoft.com/en-sg/windows/win32/shell/knownfolderid)s that map to these directories, as well as *most* of the system directories listed in [Environment variables](#environment-variables).
    The Windows API also provides enumerated GUIDs called [`KNOWNFOLDERID`](https://learn.microsoft.com/en-sg/windows/win32/shell/knownfolderid)s that map to these directories, as well as *most* of the system directories listed in [Environment variables](#221environment-variables).
    These may be queried with [`SHGetKnownFolderPath`](https://learn.microsoft.com/en-sg/windows/win32/api/shlobj_core/nf-shlobj_core-shgetknownfolderpath), like so:

    ```c++
    #include <ShlObj.h> // SHGetKnownFolderPath
    #include <cstdio> // wprintf
    #include <cstdio>

    #pragma comment(lib, "shell32.lib") // link shell32.lib

    @@ -152,32 +232,41 @@ auto main() -> int
    auto const result = SHGetKnownFolderPath(FOLDERID_Desktop, KF_FLAG_DEFAULT, nullptr,
    &path); // get desktop path; return type HRESULT
    if (SUCCEEDED(result)) {
    wprintf(L"%ls\n", path); // prints "Path:\to\Desktop"
    wprintf_s(L"%ls\n", path);
    return 0;
    } else {
    return 1;
    }
    }
    ```

    Compile the above with a Visual Studio-enabled terminal, i.e. either source `vcvarsall.bat`, or use one of the shortcuts installed with Visual Studio, such as `Developer PowerShell for VS 2022`:
    In .NET, the [`Environment.GetFolderPath`](https://learn.microsoft.com/en-us/dotnet/api/system.environment.getfolderpath?view=net-7.0) method, together with the [`Environment.SpecialFolder`](https://learn.microsoft.com/en-us/dotnet/api/system.environment.specialfolder?view=net-7.0) enum returns the same value, although this is missing the Vista changes detailed below.
    [There is a proposal to add these to .NET 8](https://github.com/dotnet/runtime/issues/70484).

    ```pwsh
    > cl.exe /std:c++20 /nologo .\knownfolderid.cpp && .\knownfolderid.exe
    knownfolderid.cpp
    > [System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::Desktop)
    D:\Libraries\Desktop
    ```

    So, the graphical setting for the various libraries update the Windows API locations, too.
    I have set all my libraries to point to `D:\Libraries\<library>`, because my `C:` drive is rather small and dedicated to the OS and important programs only, whereas `D:` is significantly larger.
    So, the graphical setting for the various libraries update the locations returned by the native Windows API too (which is eventually also called by .NET).

    #### Windows XP to Vista changes
    #### 2.2.4&emsp;Windows XP to Vista changes

    [*A Brief History of Windows Profiles*](https://archive.ph/eOGDp) is a great article on the history of Windows user profiles, but I want to focus on one important section.
    Windows 2000 and XP used `%SYSTEMDRIVE%\Documents and Settings\<username>` for the user profile, which was moved to `%SYSTEMDRIVE%\Users\<username>` with Vista and later.
    Vista also introduced the above-mentioned `KNOWNFOLDERID`s, which superseded the older [`CSIDL`](https://learn.microsoft.com/en-sg/windows/win32/shell/csidl) enumeration (although the latter is still available).
    Many environment variables like `%APPDATA%` also were changed to point to the new locations (the previous was a mouthful: `%SYSTEMDRIVE%\Documents and Settings\<username>\Application Data`).

    Many environment variables like `%APPDATA%` also were changed to point to the new locations (the previous was a mouthful: `%SYSTEMDRIVE%\Documents and Settings\<username>\Application Data`).
    New known folders were added, such as `Downloads` and `Saved Games`; the `My⎵` prefix was removed from `My Documents` and `My Music`; locations such as `Start Menu` were moved (in this case, to `%APPDATA%\Microsoft\Windows\Start Menu`).
    The reason for this move is anyone's guess, but I feel the $\geq$ Vista convention makes a lot more sense than the $\leq$ Windows XP one.
    As an aside, there is an apocryphal tale that says the reason why programs are stored in `Program Files` and not `ProgramFiles` or `Program_Files`: apparently Microsoft wanted application developers to write code defensively, and handle spaces in paths without crashing, so they made the program install location itself have spaces in it.
    There is an [old guide](https://web.archive.org/web/20120621131135/http://technet.microsoft.com:80/en-us/library/cc766489(v=ws.10).aspx) for sysadmins migrating from Windows XP to Vista.

    As an aside, there is [an apocryphal tale](https://news.ycombinator.com/item?id=29187896) for the reason why programs are stored in `Program Files` and not `ProgramFiles` or `Program_Files`: apparently Microsoft wanted to force programmers to write code defensively, and handle spaces in paths without crashing, so they made the program install location itself have spaces in it (this then raises the question: why `ProgramData`?)
    [Windows also has decent localisation](https://archive.ph/Eqvd4) (at least amongst Indo-European languages employing the Latin script): set the system language to German, for instance, and `Program Files` is now `Programme`; it is `Programfiler` in Norwegian.

    #### 2.2.5&emsp;`HKEY_CURRENT_USER` registry hive

    ### 2.3&emsp;Linux and Windows equivalents

    </div>
  13. @sharadhr sharadhr revised this gist Jul 26, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -113,7 +113,7 @@ These environment variables may be accessed from C or C++ with [`getenv()`](http

    #### User profile creation and naming

    User account creation is a mess on Windows. Until Windows 8, there was very little ceremony to user profile creation, which were all local.
    This is a mess on Windows. Until Windows 8, there was very little ceremony to user profile creation, which were all local.
    Since then, however, OneDrive (then called SkyDrive) and Microsoft account integration has thoroughly road-rolled over this simplicity.
    Today, creating a local, non-connected user profile in the OOBE in Windows 11 is [a rather involved process](https://www.windowscentral.com/how-set-windows-11-without-microsoft-account#:~:text=Use%20the%20%22Shift%20%2B%20F10%22%20keyboard%20shortcut%20to%20open%20Command%20Prompt), and may require Windows 11 Pro, which is usually not shipped with consumer devices.

  14. @sharadhr sharadhr revised this gist Jul 26, 2023. 1 changed file with 2 additions and 3 deletions.
    5 changes: 2 additions & 3 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -56,7 +56,7 @@ If you answer 'maybe', or even 'yes', then welcome to the world of software, whe
    Software that spews configuration files, temporary and cache files, generated and save files, catalogue files, downloaded files *everywhere*.
    There is little rhyme or reason to any of it: many applications never tell the user they are putting these files **HERE** or **THERE**, and if the user wants to move them around, the latter isn't given a choice in the matter.
    Applications regularly disregard platform conventions (many ironies abound, to be detailed below); even if these conventions are clearly documented, some go *out of their way* to do their own thing.
    Some applications pretend that everything is crystal-clear, by *hiding* their mess (and they typically don't do a great job of it: look at many Linux-first program ported to Windows).
    Some applications pretend that everything is immaculate, by *hiding* their mess (and they typically don't do a great job of it: look at many Linux-first program ported to Windows).

    It would irritate *anyone*.
    I am *particularly* compulsive about software doing what I'd like it to do, and about following platform conventions.
    @@ -83,11 +83,10 @@ On Windows, the latter also includes the `HKEY_CURRENT_USER` registry hive (abbr

    <details>
    <summary>Windows and Microsoft commentary</summary>
    I have several very controversial opinions about Windows and Microsoft that would raise many hackers' eyebrows.
    I hold several very controversial opinions about Windows and Microsoft that would raise many hackers' eyebrows.
    I was going to type several paragraphs, but I realised that would be better off as a blog-post on its own, coming soon.
    Not that Microsoft needs any advertising...
    </details>
    </br>

    User profiles on Windows are a fairly complex matter, but this complexity is *good*.
    Most of it is configurable, fairly well-documented, and there is more than one way to do things, although some are clearly better than others.
  15. @sharadhr sharadhr revised this gist Jul 26, 2023. 1 changed file with 135 additions and 9 deletions.
    144 changes: 135 additions & 9 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -2,33 +2,42 @@

    - [Preface](#preface)
    - [What is `$HOME` to you?](#what-is-home-to-you)
    - [User profile setup and conventions](#user-profile-setup-and-conventions)
    - [Linux](#linux)
    - [Windows](#windows)
    - [Environment variables](#environment-variables)
    - [User profile creation and naming](#user-profile-creation-and-naming)
    - [Libraries and `KNOWNFOLDERID`s](#libraries-and-knownfolderids)
    - [Windows XP to Vista changes](#windows-xp-to-vista-changes)

    <div align="justify">

    <details>
    <summary>Preface</summary>

    ## Preface

    This was supposed to be a blog post, but I have neither the knowledge, nor the time, nor the energy to set up a fancy blog like everyone else on [Hacker News](https://news.ycombinator.com).
    Of course, I *want* a blog, but I also want to use a fully .NET Core-based static site generator (because I dislike JavaScript), and therefore have experimented with [Statiq.Dev Web](https://www.statiq.dev/web).
    However, it turns out that I still need to learn HTML and CSS to make it look like anything but a website from the 1990s.
    This was supposed to be a blog post, but I have neither the knowledge, nor the time, nor the energy to set up a nice statically-generated blog like everyone else does on [Hacker News](https://news.ycombinator.com) or [proggit](https://www.reddit.com/r/programming).
    Of course, I *want* one, but I also want to use a fully .NET Core-based static site generator, and therefore have experimented with [Statiq.Dev Web](https://www.statiq.dev/web).
    However, it turns out that I still need to pick up non-trivial HTML and CSS to make it look like anything but a website from the 1990s.

    At any rate, GitHub-flavoured Markdown handles almost all use-cases I can think of (embedded images, this spoiler you've clicked on, complicated lists, in-line HTML tags for <kbd>Ctrl</kbd>, in-line $\LaTeX$ for $\text{maths} \equiv \text{fun}$...), and GitHub Gists renders Markdown as formatted text anyway, so what else do I need?
    For now, though, GitHub-flavoured Markdown handles almost all use-cases I can think of (embedded images, this spoiler you've opened, complicated lists, in-line HTML tags for <kbd>Ctrl</kbd>, in-line $\LaTeX{}$ for $\text{maths} \equiv \text{fun}$...), and GitHub Gists renders Markdown as formatted text anyway, so what else do I need?
    I can even pin these gists to my GitHub profile, which is also very nice.
    If I ever *do* prepare a blog, it should be fairly straightforward to migrate these posts to it: they're just Markdown, after all.

    I've been meaning to write this post for a *very* long time—more or less since I ever started using Linux properly four years ago, discovered [`ls -A`](https://man7.org/linux/man-pages/man1/ls.1.html#:~:text=entries%20starting%20with%20.-,%2DA%2C%20%2D%2Dalmost%2Dall,-do%20not%20list), and realised what a clutter `~` was.
    At the same time I realised that the situation on my Windows install was even *worse*.
    ___

    I've been meaning to write this for a *very* long time—more or less since I ever started using Linux properly four years ago, discovered [`ls -A`](https://man7.org/linux/man-pages/man1/ls.1.html#:~:text=entries%20starting%20with%20.-,%2DA%2C%20%2D%2Dalmost%2Dall,-do%20not%20list), and realised what a clutter `~` was.
    At the same time I noticed that the situation on my Windows install was even *worse*.
    What *really* inspired me to get started was [*Everything that uses configuration files should report where they're located*](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/ReportConfigFileLocations).
    I resonate strongly with the author's sentiment, and more importantly, dislike 'opinionated' software that likes to do what its author thinks is best instead of following system conventions.

    </details>

    ## What is `$HOME` to you?

    Home.
    It is where you're supposed to be most comfortable in.
    It is your place of refuge, and a sanctum of comfort.
    It is your place of refuge, and a sanctum from the mess and chaos of the outside world.
    It is where you have complete liberty over *everything*: what you do, when you do things, how you do them, what things you have.
    It is where these things are supposed to be where *you* want them, and *how* you want them to be.

    @@ -47,12 +56,129 @@ If you answer 'maybe', or even 'yes', then welcome to the world of software, whe
    Software that spews configuration files, temporary and cache files, generated and save files, catalogue files, downloaded files *everywhere*.
    There is little rhyme or reason to any of it: many applications never tell the user they are putting these files **HERE** or **THERE**, and if the user wants to move them around, the latter isn't given a choice in the matter.
    Applications regularly disregard platform conventions (many ironies abound, to be detailed below); even if these conventions are clearly documented, some go *out of their way* to do their own thing.
    Some applications pretend that everything is crystal-clear, by *hiding* their mess (and usually they don't do a great job of it: look at any Linux-first program ported to Windows).
    Some applications pretend that everything is crystal-clear, by *hiding* their mess (and they typically don't do a great job of it: look at many Linux-first program ported to Windows).

    It would irritate *anyone*.
    I am *particularly* compulsive about software doing what I'd like it to do, and about following platform conventions.
    Conventions, standards, and protocols exist so everyone is on the same page, and there is a common 'language' for software to communicate with.
    When software authors break these conventions just for the sake of 'opinionation', or because one *feels like it* (a worse justification, in my opinion), it only leads to much exasperation on the users' end, because it comes as a surprise to the user.
    Software should *not* be surprising.
    It ought to be reliable, reproducible, and follow platform conventions and expectations.

    This post is a detailed discussion into how user profile directories on Windows and Linux (I haven't used a Mac in ages, but I assume the situation is very similar there, too) are—to put it bluntly—in total disarray.
    Applications treat the user profile as a dumping ground, and any user with a reasonably wide list of installed software will find their user profile very difficult to traverse.
    There are platform conventions and attempts to standardise things on more open-source platforms, but a lot of developers resolutely refuse to change the behaviour of their software for a variety of reasons (some less valid than others).

    This is a bit of a soapbox, but I also hope developers read this, and at least *attempt* to fix their software so that home directories are cleaner, and users have easier lives maintaining and using their computers.

    ## User profile setup and conventions

    Before I begin, there are some platform-specific details and wording *conventions* to be discussed.
    I will use the terms 'home directory' and 'user profile directory' somewhat interchangeably in this post, whereas 'user profile' means the home directory itself and its contents, including user-specific configuration and data, *combined*.
    On Windows, the latter also includes the `HKEY_CURRENT_USER` registry hive (abbreviated as `HKCU`), stored in `%USERPROFILE%\NTUSER.dat`.

    ### Linux

    ### Windows

    <details>
    <summary>Windows and Microsoft commentary</summary>
    I have several very controversial opinions about Windows and Microsoft that would raise many hackers' eyebrows.
    I was going to type several paragraphs, but I realised that would be better off as a blog-post on its own, coming soon.
    Not that Microsoft needs any advertising...
    </details>
    </br>

    User profiles on Windows are a fairly complex matter, but this complexity is *good*.
    Most of it is configurable, fairly well-documented, and there is more than one way to do things, although some are clearly better than others.

    #### Environment variables

    Modern Windows (i.e. Vista and later) has several conventions and environment variables for important directories, including those within the user profile.
    Many are listed [here](https://ss64.com/nt/syntax-variables.html) (Microsoft's [official documentation](https://learn.microsoft.com/en-sg/windows/deployment/usmt/usmt-recognized-environment-variables) is a little less forthcoming), and I reproduce a few in the table below.

    | **Variable** | **Value** | **Details** |
    |-----------------------|----------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
    | `%SYSTEMDRIVE%` | Usually `C:` | `C:` is not necessarily always the Windows install drive. |
    | `%WINDIR%` | `%SYSTEMDRIVE%\Windows` | `Windows` is not necessarily the install directory for Windows itself. There are legacy directories like `WINNT`, `WINNT35`, `WTSRV`. |
    | `%SYSTEMROOT%` | `%WINDIR%` | |
    | `%PROGRAMFILES%` | `%SYSTEMDRIVE%\Program Files` | Default system-wide installation location for programs. Requires administrator permission to modify. |
    | `%PROGRAMFILES(X86)%` | `%SYSTEMDRIVE%\Program Files (x86)` | Only on 64-bit installs of Windows. 32-bit-only programs go here. |
    | `%PROGRAMDATA%` | `%SYSTEMDRIVE%\ProgramData` | System-wide program configuration and storage; e.g. default configurations, logs, etc. |
    | `%USERNAME%` | Set in the registry | Can be changed in Control Panel or Settings. |
    | `%USERPROFILE%` | Usually `%SYSTEMDRIVE%\Users\%USERNAME%`; see discussion below | Equivalent to `$HOME` or `~` on Unix/Linux. |
    | `%APPDATA%` | `%USERPROFILE%\AppData\Roaming` | User-specific *roaming* application data that may be exported between PCs; e.g. Google Chrome user profiles, Steam accounts; likely anything that requires an internet sign-in |
    | `%LOCALAPPDATA%` | `%USERPROFILE%\AppData\Local` | User-specific, PC-specific application data; sometimes includes the entire applications themselves if installed only for the user. e.g. Google Chrome and MiKTeX installed by a non-admin user. |

    These environment variables may be accessed from C or C++ with [`getenv()`](https://learn.microsoft.com/en-sg/cpp/c-runtime-library/reference/getenv-wgetenv?view=msvc-170), or any other language-specific API.

    #### User profile creation and naming

    User account creation is a mess on Windows. Until Windows 8, there was very little ceremony to user profile creation, which were all local.
    Since then, however, OneDrive (then called SkyDrive) and Microsoft account integration has thoroughly road-rolled over this simplicity.
    Today, creating a local, non-connected user profile in the OOBE in Windows 11 is [a rather involved process](https://www.windowscentral.com/how-set-windows-11-without-microsoft-account#:~:text=Use%20the%20%22Shift%20%2B%20F10%22%20keyboard%20shortcut%20to%20open%20Command%20Prompt), and may require Windows 11 Pro, which is usually not shipped with consumer devices.

    This digression is to discuss what home directory name users end up with: on a profile connected to a Microsoft account, the user home directory is a truncated (to the first five characters), lower-case version of the user name.
    For instance, in the first case, I would have `%SYSTEMDRIVE%\Users\shara`, but on a 'local account', it would be `%SYSTEMDRIVE%\Users\Sharadh` (mine is *actually* the latter—I am obsessive about this, so I created a local account first, and *then* synced it to OneDrive).
    Like many things Windows, configuring the user profile directory after account creation requires [editing registry keys](https://learn.microsoft.com/en-sg/troubleshoot/windows-client/user-profiles-and-logon/renaming-user-account-not-change-profile-path) using *another* administrator account.

    #### Libraries and `KNOWNFOLDERID`s

    <img align="right" width="300" hspace="10" src="https://files.catbox.moe/gh3u24.png">

    Upon user profile creation, Windows automatically sets up several directories, which are also [*libraries*](https://learn.microsoft.com/en-sg/windows/win32/shell/library-ovw), inside the home directory:

    - 3D Objects
    - Contacts
    - Desktop
    - Documents
    - Downloads
    - Music
    - Saved Games
    - Videos

    These libraries are initially created under `%USERPROFILE%`, but may be moved (right click → `Properties``Location` tab → `Move...`) by the user at their own discretion.
    The Windows API also provides enumerated GUIDs called [`KNOWNFOLDERID`](https://learn.microsoft.com/en-sg/windows/win32/shell/knownfolderid)s that map to these directories, as well as *most* of the system directories listed in [Environment variables](#environment-variables).
    These may be queried with [`SHGetKnownFolderPath`](https://learn.microsoft.com/en-sg/windows/win32/api/shlobj_core/nf-shlobj_core-shgetknownfolderpath), like so:

    ```c++
    #include <ShlObj.h> // SHGetKnownFolderPath
    #include <cstdio> // wprintf

    #pragma comment(lib, "shell32.lib") // link shell32.lib

    auto main() -> int
    {
    auto path = PWSTR{};
    auto const result = SHGetKnownFolderPath(FOLDERID_Desktop, KF_FLAG_DEFAULT, nullptr,
    &path); // get desktop path; return type HRESULT
    if (SUCCEEDED(result)) {
    wprintf(L"%ls\n", path); // prints "Path:\to\Desktop"
    return 0;
    } else {
    return 1;
    }
    }
    ```

    Compile the above with a Visual Studio-enabled terminal, i.e. either source `vcvarsall.bat`, or use one of the shortcuts installed with Visual Studio, such as `Developer PowerShell for VS 2022`:

    ```pwsh
    > cl.exe /std:c++20 /nologo .\knownfolderid.cpp && .\knownfolderid.exe
    knownfolderid.cpp
    D:\Libraries\Desktop
    ```

    So, the graphical setting for the various libraries update the Windows API locations, too.

    #### Windows XP to Vista changes

    [*A Brief History of Windows Profiles*](https://archive.ph/eOGDp) is a great article on the history of Windows user profiles, but I want to focus on one important section.
    Windows 2000 and XP used `%SYSTEMDRIVE%\Documents and Settings\<username>` for the user profile, which was moved to `%SYSTEMDRIVE%\Users\<username>` with Vista and later.
    Vista also introduced the above-mentioned `KNOWNFOLDERID`s, which superseded the older [`CSIDL`](https://learn.microsoft.com/en-sg/windows/win32/shell/csidl) enumeration (although the latter is still available).
    Many environment variables like `%APPDATA%` also were changed to point to the new locations (the previous was a mouthful: `%SYSTEMDRIVE%\Documents and Settings\<username>\Application Data`).

    The reason for this move is anyone's guess, but I feel the $\geq$ Vista convention makes a lot more sense than the $\leq$ Windows XP one.
    As an aside, there is an apocryphal tale that says the reason why programs are stored in `Program Files` and not `ProgramFiles` or `Program_Files`: apparently Microsoft wanted application developers to write code defensively, and handle spaces in paths without crashing, so they made the program install location itself have spaces in it.

    </div>
  16. @sharadhr sharadhr revised this gist Jul 26, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,7 @@
    - [Preface](#preface)
    - [What is `$HOME` to you?](#what-is-home-to-you)

    <div style="text-align: justify">
    <div align="justify">
    <details>
    <summary>Preface</summary>

    @@ -17,7 +17,7 @@ At any rate, GitHub-flavoured Markdown handles almost all use-cases I can think
    I can even pin these gists to my GitHub profile, which is also very nice.
    If I ever *do* prepare a blog, it should be fairly straightforward to migrate these posts to it: they're just Markdown, after all.

    I've been meaning to write this post for a *very* long time---more or less since I ever started using Linux properly four years ago, discovered [`ls -A`](https://man7.org/linux/man-pages/man1/ls.1.html#:~:text=entries%20starting%20with%20.-,%2DA%2C%20%2D%2Dalmost%2Dall,-do%20not%20list), and realised what a clutter `~` was.
    I've been meaning to write this post for a *very* long timemore or less since I ever started using Linux properly four years ago, discovered [`ls -A`](https://man7.org/linux/man-pages/man1/ls.1.html#:~:text=entries%20starting%20with%20.-,%2DA%2C%20%2D%2Dalmost%2Dall,-do%20not%20list), and realised what a clutter `~` was.
    At the same time I realised that the situation on my Windows install was even *worse*.
    What *really* inspired me to get started was [*Everything that uses configuration files should report where they're located*](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/ReportConfigFileLocations).
    I resonate strongly with the author's sentiment, and more importantly, dislike 'opinionated' software that likes to do what its author thinks is best instead of following system conventions.
  17. @sharadhr sharadhr revised this gist Jul 26, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -55,3 +55,4 @@ Conventions, standards, and protocols exist so everyone is on the same page, and
    When software authors break these conventions just for the sake of 'opinionation', or because one *feels like it* (a worse justification, in my opinion), it only leads to much exasperation on the users' end, because it comes as a surprise to the user.
    Software should *not* be surprising.
    It ought to be reliable, reproducible, and follow platform conventions and expectations.
    </div>
  18. @sharadhr sharadhr revised this gist Jul 26, 2023. 1 changed file with 57 additions and 1 deletion.
    58 changes: 57 additions & 1 deletion home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -1 +1,57 @@
    # `$HOME`, Not So Sweet `$HOME`
    # `$HOME`, Not So Sweet `$HOME`

    - [Preface](#preface)
    - [What is `$HOME` to you?](#what-is-home-to-you)

    <div style="text-align: justify">
    <details>
    <summary>Preface</summary>

    ## Preface

    This was supposed to be a blog post, but I have neither the knowledge, nor the time, nor the energy to set up a fancy blog like everyone else on [Hacker News](https://news.ycombinator.com).
    Of course, I *want* a blog, but I also want to use a fully .NET Core-based static site generator (because I dislike JavaScript), and therefore have experimented with [Statiq.Dev Web](https://www.statiq.dev/web).
    However, it turns out that I still need to learn HTML and CSS to make it look like anything but a website from the 1990s.

    At any rate, GitHub-flavoured Markdown handles almost all use-cases I can think of (embedded images, this spoiler you've clicked on, complicated lists, in-line HTML tags for <kbd>Ctrl</kbd>, in-line $\LaTeX$ for $\text{maths} \equiv \text{fun}$...), and GitHub Gists renders Markdown as formatted text anyway, so what else do I need?
    I can even pin these gists to my GitHub profile, which is also very nice.
    If I ever *do* prepare a blog, it should be fairly straightforward to migrate these posts to it: they're just Markdown, after all.

    I've been meaning to write this post for a *very* long time---more or less since I ever started using Linux properly four years ago, discovered [`ls -A`](https://man7.org/linux/man-pages/man1/ls.1.html#:~:text=entries%20starting%20with%20.-,%2DA%2C%20%2D%2Dalmost%2Dall,-do%20not%20list), and realised what a clutter `~` was.
    At the same time I realised that the situation on my Windows install was even *worse*.
    What *really* inspired me to get started was [*Everything that uses configuration files should report where they're located*](https://utcc.utoronto.ca/~cks/space/blog/sysadmin/ReportConfigFileLocations).
    I resonate strongly with the author's sentiment, and more importantly, dislike 'opinionated' software that likes to do what its author thinks is best instead of following system conventions.

    </details>

    ## What is `$HOME` to you?

    Home.
    It is where you're supposed to be most comfortable in.
    It is your place of refuge, and a sanctum of comfort.
    It is where you have complete liberty over *everything*: what you do, when you do things, how you do them, what things you have.
    It is where these things are supposed to be where *you* want them, and *how* you want them to be.

    You buy things to decorate, maintain, and improve your home: paintings, photographs, vases, light fixtures, sofas, chairs, ovens, vacuum cleaners.
    You have your own volition to put these wherever you want, and set them up however you want.
    Your vacuum cleaner in a wardrobe? Sure.
    Or in the washroom, or in the service balcony, or the backyard. You can put its dust bag anywhere you want, too.

    What if there existed a vacuum cleaner that stopped working if you plugged it into a different socket to the one the manufacturer set out in the manual?
    What if it stopped working if you changed where you put its dust bag, or swapped it out for a *new* dust bag?
    Would you buy this vacuum cleaner?
    Suppose that this vacuum cleaner was given to you for free, anyway.
    Would you still use it, but live with the compromise that your home is not *exactly* how you want it?

    If you answer 'maybe', or even 'yes', then welcome to the world of software, where your home is violated on a regular basis.
    Software that spews configuration files, temporary and cache files, generated and save files, catalogue files, downloaded files *everywhere*.
    There is little rhyme or reason to any of it: many applications never tell the user they are putting these files **HERE** or **THERE**, and if the user wants to move them around, the latter isn't given a choice in the matter.
    Applications regularly disregard platform conventions (many ironies abound, to be detailed below); even if these conventions are clearly documented, some go *out of their way* to do their own thing.
    Some applications pretend that everything is crystal-clear, by *hiding* their mess (and usually they don't do a great job of it: look at any Linux-first program ported to Windows).

    It would irritate *anyone*.
    I am *particularly* compulsive about software doing what I'd like it to do, and about following platform conventions.
    Conventions, standards, and protocols exist so everyone is on the same page, and there is a common 'language' for software to communicate with.
    When software authors break these conventions just for the sake of 'opinionation', or because one *feels like it* (a worse justification, in my opinion), it only leads to much exasperation on the users' end, because it comes as a surprise to the user.
    Software should *not* be surprising.
    It ought to be reliable, reproducible, and follow platform conventions and expectations.
  19. @sharadhr sharadhr created this gist Jul 25, 2023.
    1 change: 1 addition & 0 deletions home_nss_home.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1 @@
    # `$HOME`, Not So Sweet `$HOME`