No more. At least with apple Silicon dtrace is completely unusable.

Interesting. I didn’t know you could do this. I can see how this can come in handy. Apparently there’s dtruss as an alternative for OS X.<a href="https:&#x2F;&#x2F;8thlight.com&#x2F;insights&#x2F;dtrace-even-better-than-strace-for-os-x" rel="nofollow noreferrer">https:&#x2F;&#x2F;8thlight.com&#x2F;insights&#x2F;dtrace-even-better-than-strace...</a>

Can&#x27;t disable DTrace :)Though DTrace is &quot;only&quot; available on Windows, Mac OS X, Solaris, illumos, and FreeBSD.Oracle has relicensed DTrace to be Linux-friendly and even made kernel patches, but it&#x27;ll probably never end up in the mainline kernel.

You can use sysdig instead, which doesn&#x27;t use ptrace[1] and is much faster (as well as generally more pleasant and powerful).[1]It used to have its own kernel extension but is eEBF based these days.

strace uses ptrace, which some programs disable with prctl as a security measure.So while this can be a useful hack, it doesn&#x27;t always work.

Thank you for your efforts!These days Proton makes a lot of this unnecessary. I hope Trails games will continue to at least function on Linux. :)

+1 for putting together that blog post on Cold Steel. Greetings from PH3 ;)

All that time I spent dicking around with lsof...

On Windows, Process Monitor does the same thing --- with the additional feature that, since configuration is often stored in the registry there, it will also show which registry keys it touches.

I do this with Brendan Gregg&#x27;s opensnoop, which is standard on macOS.<a href="https:&#x2F;&#x2F;www.brendangregg.com&#x2F;blog&#x2F;2014-07-25&#x2F;opensnoop-for-linux.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.brendangregg.com&#x2F;blog&#x2F;2014-07-25&#x2F;opensnoop-for-l...</a><a href="https:&#x2F;&#x2F;github.com&#x2F;brendangregg&#x2F;perf-tools&#x2F;blob&#x2F;master&#x2F;opensnoop">https:&#x2F;&#x2F;github.com&#x2F;brendangregg&#x2F;perf-tools&#x2F;blob&#x2F;master&#x2F;opens...</a>

I do this much more frequently than I would like. But occasionally you have issues where the program lists a directory then only acts on specific names. So the strace output won&#x27;t tell you what the expected name is.

I use quick and dirty greps all the time, even when there is a &quot;better&quot; option available. It just works and is very intuitive in interactive contexts. Probably GP works in a similar way.

<pre><code> ps wxa | grep &lt;cmd&gt; # or pgrep
 strace -f -f -e trace=file &lt;cmd&gt; | grep -v &#x27;-1 ENOENT (No such file or directory)$&#x27;
</code></pre>
IIRC there&#x27;s some way to filter ENOENT messages with strace instead of grep?Strace: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Strace" rel="nofollow noreferrer">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Strace</a>

A quick and dirty way to do this is run the program through `strace` and find out what syscall it issues to read it.I had to do this to figure out where Cold Steel was reading save files from years back: <a href="https:&#x2F;&#x2F;vaughanhilts.me&#x2F;blog&#x2F;2018&#x2F;02&#x2F;16&#x2F;playing-trails-of-cold-steel-on-linux-wine-guide.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;vaughanhilts.me&#x2F;blog&#x2F;2018&#x2F;02&#x2F;16&#x2F;playing-trails-of-co...</a>

It&#x27;s more common to have to guess in blackbox &#x2F; proprietary software, like games.

But that is exactly the information you need in order to determine which config file is being read in all situations. A single location doesn&#x27;t cut it if the reality is more complicated.If the goal is to find out which particular file is being used at this particular time, on this particular system, the maximum verbosity log might have that information, and otherwise strace can help, as described in the other comments.

Doesn&#x27;t the man page often list several potential locations where it looks for the file and the ordering behavior, leaving you to investigate and find which is actually loaded? It&#x27;s still annoying.

my manpath includes $HOME&#x2F;.local&#x2F;share&#x2F;man; and that is with $MANPATH unset, so it&#x27;s a distribution default

Out of interest, I checked out man man on Linux and found the search logic documented in man 5 manpath. In addition to system directories specified in &#x2F;etc&#x2F;manpath.config, the man command checks directories in the user&#x27;s PATH and maps them to related man directories: pathdir, pathdir&#x2F;..&#x2F;man, pathdir&#x2F;man, pathdir&#x2F;..&#x2F;share&#x2F;man, and pathdir&#x2F;share&#x2F;man are all considered. The MANPATH environment variable overrides all other configuration and should not be necessary.So, following current practices, installing binaries in ~&#x2F;.local&#x2F;bin and man pages in ~&#x2F;.local&#x2F;share&#x2F;man should work without any additional configuration.

&gt; With the exception of New Age software like Go, that for some reason doesn&#x27;t use manpagesPrimarily because man pages need to go in the MANPATH to be useful but these tools tend to dwell somewhere in ~&#x2F;.

A short way in to TFA:&gt; Ideally I&#x27;d like to avoid scanning all the way through the manual page or other documentation for the program to find out, because that&#x27;s slow and annoying.

I don&#x27;t think I&#x27;ve ever encountered a widely used program that doesn&#x27;t have that information in its manpage. The manpage is the canonical source the article calls for.With the exception of New Age software like Go, that for some reason doesn&#x27;t use manpages, I&#x27;m not convinced that the problem implied here actually exists.

All parts of systemd are configured entirely using plain text config files and symlinks.

don&#x27;t give the systemd crowd any ideas, the Windows Registry is right up their alley

When you give people the freedom to use the KV store as they wish, you inevitably run into a meta-problem whereby the way they write their data structures in to the KV store is varied across applications. I don&#x27;t know if you could standardise this structure beyond enforcing applications to use an OS-assigned private store, but I don&#x27;t see how that would work with portable executables etc.I suppose you&#x27;d need to be very opinionated for it to work, but people will still bodge it to make it work with their needs that your system doesn&#x27;t natively satisfy (people putting JSON strings in the values, for example).

A system-wide key-value store, that&#x27;s distinct from file storage, seems like a pretty great idea. The Windows Registry is icky, we all know that, it somehow appeared out of nowhere in Windows 95 already crusty as hell. But if it were designed by people with taste and discretion, would it be good?

Let me introduce you to the nightmare that is the Windows Registry...

I am familiar with how software is configured, and the implied context is not clear from only the headline, hence the headline needs fixing. &quot;Everything&quot; is massive overstatement: bank websites use configuration files, so if hackers try to log in to the bank, the website should report where the config files are located? Nope. Headline way too broad.

Possibly if you are not familiar with how software is configured. It was immediately clear to me.

The title read without context sounds like it&#x27;s suggesting they should report which IP address&#x2F;country&#x2F;jurisdiction they&#x27;re in. To any random user on the internet. Which would be insane. But it&#x27;s really only saying &quot;which system path they&#x27;re in, to e.g. a sys admin&quot;.

I have this tool, which is used in a few different environments, so I gave it a “xxx config” command which tells which config files were read, which environment variables were used, which environment variables from the tool’s XXX_ namespace are unknown, what configuration values are effective etc.I don’t think anyone ever used that function.

Configuration files are like parents’ advice - carefully thought out, well-intentioned, but often overwritten by the ‘environment variables’ of real life experiences!

Programs reconfigurable at runtime are by far rarity, as that&#x27;s generally pretty hard thing to do. But yes, just have a config file, ENVs are nice as a backup for small apps, but never as only way

You can’t change env vars after a program is running, but I can change a file and send a SIGHUP.

Indeed, I only had &quot;simple&quot; cases in mind. But hey... shouldn&#x27;t the complex config files paths be specified through env vars ? ;)

I&#x27;ve had rare cases of needing nested data structures in the environment. My solution was to specify them as JSON strings and load the value in code. It works.<pre><code> USER=&#x27;{
 &quot;name&quot;: &quot;joe&quot;,
 &quot;email&quot;:&quot;joe@example.com&quot;
 }&#x27;</code></pre>

Sometimes configuration is fairly complex (e.g. nested map data structures) which gets pretty complex with simple key&#x2F;value env variables.I&#x27;m thinking of things like config for nginx, Apache, Kubernetes, etc.

Yes. Env vars beat constants but have stunted our thinking because we know deep down it&#x27;s not a great idea to have too too many.Config files are better, but can lead to the mess described here and typically only give us crude axes for overriding.It&#x27;s time for us to embrace &amp; tame the complexity. If you use a feature flag tool already, you know what it&#x27;s like to target different values based on rules.If I want to experiment with changing the http timeout for specific calls from a single service in a single region I should be able to roll that out.It&#x27;s time to expand our brains and bring this feature flag style thinking to the rest of our configuration.

Coz it&#x27;s entirely miserable experience when you get like... more than 4 config options

For anything more than trivial configurations it can be nice to commit them to source control, giving some reversion capabilities, and easy replication on other projects.

- If your configuration has more than 5-10 options then env vars become a mess while a configuration file is more maintainable
- Nested configuration &#x2F; scoping is a mature advantage of configuration files
- You can reload configuration files whereas you can&#x27;t reload environment variables during runtime
- A configuration file is a transparent record of your configuration and easier to backup and restore than env variables. Env variables can be loaded from many different locations depending on your os.
- In configuration files you can have typed values that are parsed automatically with env variables you need to parse them yourself. This is just a difference not that bad for env variables per se.

All the software I write is configured through env vars... why use anything else ?

System environment variables&#x2F;system registry sounds like the best way to do that from a pragmatic perspective

Also, I do believe that programs should be capable of self-configurationlike if you need LLVM and didnt manage to find it under&quot;app_folder&#x2F;llvm&quot;then let&#x27;s go find it somewhere on the disk(s) and ask user if he wants to use it

You can change the commandline string at runtime. You could inject a fake &quot;-c correct&#x2F;path&quot; even if it&#x27;s not there. (That&#x27;s useful for other things too, like injecting the git revision of the app into the commandline)

It&#x27;s not a bad idea but it&#x27;s not applicable  to every piece of software. I don&#x27;t think that passing a config file for every git command would be convenient.

&#x2F;nix&#x2F;store&#x2F;9ii&lt;tab&gt;nginx.conf

How long does it take you to type out that path?

&gt; This looks like a global variable in a codebase (something we generally try to avoid).Aren&#x27;t they more like global constants than variables? Loaded at startup, and never change during that run of the program. (With the exception of only explicitly being re-read on SIGUSR1 for daemon-like programs.)And global consts, or #defines, or whatever, are things we generally don&#x27;t try to avoid?

The main issue is to use configuration files residing somewhere in the filesystem. This looks like a global variable in a codebase (something we generally try to avoid).
Instead, the configuration file should be explicitly provided as a command line argument. Systemd sandboxing can also be useful to ensure the program only uses the expected set of files.For instance, on my NixOS machine, the Nginx configuration is not in `&#x2F;etc&#x2F;nginx` but explicitly provided and can then be known with ps:$ ps aux | grep nginxnginx: master process &#x2F;nix&#x2F;store&#x2F;9iiqv6c3q8zlsqfp75vd2r7f1grwbxh7-nginx-1.24.0&#x2F;bin&#x2F;nginx -c &#x2F;nix&#x2F;store&#x2F;9ffljwl3nlw4rkzbyhhirycb9hjv89lr-nginx.conf

Redis has consitently been the nicest program to admin everytime i have to interact with it as a sysadmin

<pre><code> $ redis-cli INFO
 ...
 executable:&#x2F;Users&#x2F;antirez&#x2F;hack&#x2F;redis&#x2F;src&#x2F;.&#x2F;redis-server
 config_file:&#x2F;etc&#x2F;redis&#x2F;redis.conf
 ...
</code></pre>
In the case of Redis, it is possible to do much more. With the CONFIG command you can check specific configuration parameters at runtime, and even change most of them.There are commands to get reports about latency and memory usage, performing a self-analysis of the instance condition. You can see counters for commands called, and even obtian a live stream of the commands the instance is executing.This should all be obvious features to make system software approachable, observable, and to make people able to act easily when shit happens, without having to guess where the configuration file is from the system startup scripts or alike. UX is not only about GUIs.

Seconding dry-run, while looking intently aws-cli for haveing the worst possible dry-run option imaginable.

As a system administrator I want 2 things from config files:* loading a directory of config files, so various Configuration Management tools have easier time deploying config (especially for multi-use apps like say syslog daemon, where multiple disparate apps might want to put a part of the config in)* a function to dry-run config validation, so CM can alert about that instead of restarting an app to a wrong configFinding where is one has never been a problem.

Not sure I get the reasoning here. That requires you to actually go and proofread the configuration of each and every running process, at which point it&#x27;s easier to just verify the effective configuration on disk and restart the service.SSH is a special case because it leaves active sessions alive when it restarts, but that doesn&#x27;t generalize to other daemons. Just restart if in doubt.

Also: processes &#x2F; running programs should provide a straightforward way to show their configuration.While &quot;sshd -T&quot; shows the effective configuration from all configuration file parsing, there is no straightforward way to see the configuration of the&#x2F;a currently running sshd process.This has footgun potential, as someone could think their sshd is secured and hardened by checking with &quot;sshd -T&quot;, while the actually running sshd process still uses old configurations.Apache also makes it needlessly hard to find out its effective (and active &#x2F; currently running) configuration. I don&#x27;t know of any better way then awkwardly using mod_info ( <a href="https:&#x2F;&#x2F;httpd.apache.org&#x2F;docs&#x2F;2.4&#x2F;mod&#x2F;mod_info.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;httpd.apache.org&#x2F;docs&#x2F;2.4&#x2F;mod&#x2F;mod_info.html</a> ) to get such insights.

You might want to read on specificity because it’s a huge spanner in your assumptions.

The rules of CSS specificity are a bit more involved than &#x27;!important&#x27; or not, and I frequently see beginners trip there, but apart from that I agree that the CSS rule set might be worth incorporating.

Fun trick to use if you use a css post processing (less, sass, etc.) is &quot;&amp;&amp;&amp;&quot;, it will resolve to a very specific rule which overwrites the other rules

I’m sure you know this, but for readers who don’t, it’s much more complex than “last defined wins with a few override options”. Based on “specificity”, rules based on the selector used.Some examples:1. #mydiv overrides .mydiv - IDs are more specific than classes2. .outerdiv .innerdiv overrides .innerdiv - nested selectors are more specific3. .mydiv overrides div - classes are more specific than elementsAnd a lot of other rules.I can’t imagine I’d want this level of complexity in config files.

Not true at all. There is a very much non-intuitive weighting system applied to CSS selectors which is what ultimately determines which gets applied: <a href="https:&#x2F;&#x2F;developer.mozilla.org&#x2F;en-US&#x2F;docs&#x2F;Web&#x2F;CSS&#x2F;Specificity" rel="nofollow noreferrer">https:&#x2F;&#x2F;developer.mozilla.org&#x2F;en-US&#x2F;docs&#x2F;Web&#x2F;CSS&#x2F;Specificity</a>

&gt; Two overlapping overrides revert back to the &quot;last read&quot; approach.IIRC, there&#x27;s an important (heh) exception to that: user stylesheets. Without the &quot;!important&quot; override on the user stylesheet, the site stylesheet wins; with the &quot;!important&quot; override on the user stylesheet, the user stylesheet wins, even if the site stylesheet also had an &quot;!important&quot; for that rule.

The domain of style and configuration are very, very different. I&#x27;m not sure what works okay for displaying a website in different layouts on vastly different unknown devices share a lot of the same concerns of configuring programs on a machine

It&#x27;s a little bit away from the domain of config files, but I think CSS is the most graceful example of handling this as a generalised problem. Style configurations cascade, and the last one read is the one that gets applied. There are overrides to that flow, but you have to very overtly add a piece of syntax to the overriding line like `!important`. Two overlapping overrides revert back to the &quot;last read&quot; approach. The approach is so fundamental to CSS that the entire language is named after it, so there&#x27;s no ambiguity or forwards compatibility concerns in this regard whatsoever.

systemd configuration lookups are lightweights and straightforward compared to ansible variables lookups rules :
--
 Understanding variable precedence<pre><code> Ansible does apply variable precedence, and you might have a use for it. Here is the order of precedence from least to greatest (the last listed variables override all other variables):

 1 command line values (for example, -u my_user, these are not variables)

 2 role defaults (defined in role&#x2F;defaults&#x2F;main.yml) 1

 3 inventory file or script group vars 2

 4 inventory group_vars&#x2F;all 3

 5 playbook group_vars&#x2F;all 3

 6 inventory group_vars&#x2F;* 3

 7 playbook group_vars&#x2F;* 3

 8 inventory file or script host vars 2

 9 inventory host_vars&#x2F;* 3

 10 playbook host_vars&#x2F;* 3

 11 host facts &#x2F; cached set_facts 4

 12 play vars

 13 play vars_prompt

 14 play vars_files

 15 role vars (defined in role&#x2F;vars&#x2F;main.yml)

 16 block vars (only for tasks in block)

 17 task vars (only for the task)

 18 include_vars

 19 set_facts &#x2F; registered vars

 20 role (and include_role) params

 21 include params

 22 extra vars (for example, -e &quot;user=my_user&quot;)(always win precedence)</code></pre>
‐‐that is a RedHat-isim

That doesn&#x27;t look too bad to me.&quot;.d&quot; directories are a well-established pattern to split a complicated config file into logical units. In the case of something like systemd this is absolutely essential, as you wouldn&#x27;t want the config for your web server in the same file as the mail server.&#x2F;usr&#x2F;lib is the location where the package manager will drop config files. This is the default, distro-provided configuration. &#x2F;etc&#x2F; is where the administrator can place config files, providing system-specific overrides. &#x2F;run&#x2F; is for files &quot;created at runtime&quot;, I am not quite sure what the point of those is.

&gt;Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.&gt;Read the manpage for all the hilarity.If you find it hilarious it&#x27;s because you don&#x27;t understand the reason. &#x2F;usr belongs to the distribution and &#x2F;etc belongs to the user. The distribution doesn&#x27;t touch &#x2F;etc and the user doesn&#x27;t touch &#x2F;usr.Upstream ships a unit with its own opinion of the defaults. That unit goes in &#x2F;usr.Distribution would like to override a default. It can patch the file but that is somewhat harder to discover (it would require reading the package source). Alternatively it can add a drop-in. That drop-in goes into &#x2F;usr too since it&#x27;s a distribution file.User would like to override a default. They should not edit &#x2F;usr because that belongs to the distribution. They drop their override in &#x2F;etc.Some units are created at runtime on demand and thus ephemeral, like mount units generated by generators. These go in &#x2F;run. These are functionally &quot;system-provided units&quot; because users must be able to override them, so they&#x27;re below &#x2F;etc in the hierarchy.The &quot;defaults&quot; and &quot;overrides&quot; we&#x27;re talking about are both at the level of the individual setting as well as at the level of the whole file. If a user wants one of the settings to be different, they can add a new 99-foo.conf file that unsets &#x2F; changes it. If they want a whole file to be different, they can add a new file with the same name that is empty &#x2F; has different values for its settings.BTW `systemctl cat whatever.unit` will print all the files that systemd considered in the order that it considered them.&gt;And this is just an excerpt which doesn&#x27;t cover the details.The only detail your excerpt doesn&#x27;t include is the consideration of `service.d&#x2F;`, `socket.d&#x2F;` etc directories which provide overrides for all units of that type. It&#x27;s just one point, not commonly used, and not that big of a deal either.

Drop in files (config in .d folders) are a core extension point of systemd and not a symptom of an overly engineered config system. These are how services are meant to be custom tailored to specific machines and even user&#x27;s needs.Let&#x27;s say you need to pass a special flag to sshd on your system, a bad approach would be to go edit the distro shipped sshd.service file. Your change will work until the distro ships an updated file and you forget to hack in your change again.Instead place a sshd.service drop-in config file in the appropriate place to customize the service as necessary. Your config tweak takes precedence over the distro shipped config, but doesn&#x27;t live inside it or conflict with upstream changes.

And yet another gotcha with configuration file loading:Over-elaborated hierarchies where configuration files may reside, and their precedence schemes.E.g. from systemd.unit manpage ( <a href="https:&#x2F;&#x2F;www.freedesktop.org&#x2F;software&#x2F;systemd&#x2F;man&#x2F;systemd.unit.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.freedesktop.org&#x2F;software&#x2F;systemd&#x2F;man&#x2F;systemd.uni...</a> ):&gt;...&gt; In addition to &#x2F;etc&#x2F;systemd&#x2F;system, the drop-in &quot;.d&#x2F;&quot; directories for system services can be placed in &#x2F;usr&#x2F;lib&#x2F;systemd&#x2F;system or &#x2F;run&#x2F;systemd&#x2F;system directories. Drop-in files in &#x2F;etc&#x2F; take precedence over those in &#x2F;run&#x2F; which in turn take precedence over those in &#x2F;usr&#x2F;lib&#x2F;. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.&gt; ...And this is just an excerpt which doesn&#x27;t cover the details. Read the manpage for all the hilarity.

I would look for something like this in the help output, i.e. --help or -h. And of course the manpage if you do supply one.

This is good advice and my own software is following it:$ oq info cfgLooking at the following paths (the last wins)&#x2F;opt&#x2F;openquake&#x2F;oq-engine&#x2F;openquake&#x2F;engine&#x2F;openquake.cfg&#x2F;opt&#x2F;openquake&#x2F;venv&#x2F;openquake.cfgHowever, I do not think there is any standard API to follow, I used &quot;info cfg&quot; but many alternatives were possible. So sysadmins have to explore to find out which is the introspection command to give. Not ideal, but I don&#x27;t think there is a solution.

Could you explain a little more? I am not sure what exactly you mean by &quot;exposing an internal endpoint&quot; - do you mean that as a secured API endpoint which you can call? Isn&#x27;t that hard to secure?

This is not an endorsement, but Spring Boot has this built in (but not enabled by default):<a href="https:&#x2F;&#x2F;docs.spring.io&#x2F;spring-boot&#x2F;docs&#x2F;current&#x2F;reference&#x2F;html&#x2F;actuator.html#actuator.endpoints" rel="nofollow noreferrer">https:&#x2F;&#x2F;docs.spring.io&#x2F;spring-boot&#x2F;docs&#x2F;current&#x2F;reference&#x2F;ht...</a>I have been leaning towards a different approach. Rather than reading multiple config sources at runtime, read them at build time, and emit a single config file with everything fully resolved. Put that on the filesystem, and have the app read it. You have an easy canonical reference for the configuration, with no need to expose anything. The lifecycle of the config doesn&#x27;t have to match that off the app binary, so it could (and probably should) have a separate build and release process.

While at it I’ve made it a practice to expose an internal endpoint for every service to output its config. It has saved me on numerous occasions to catch config related bugs.

Logging keys only should be sufficient for this.

I&#x27;d rather have some config reserialization capability with provenance (perhaps as an opt-in). A good config reserialization can make itself useful in so many situations, if it has the right features. Like pointing out key-value pairs that don&#x27;t have any effect, or simply writing out some documentation.Logging has its merits, like discoverability when you stare at it without suspecting configuration to be the culprit, but I believe that there can and should be more.

Agreed, logging in config retrieval code should log all this stuff, requests, where they are filled from, that sort of thing.The problems with this approach when it&#x27;s enabled is that log aggregators (e.g splunk) can end up including stuff like passwords which shouldn&#x27;t be visible to the group able to view those logs, so some care is needed.

logs always go to stdout&#x2F;stderrhopefully!

Also, they should report where logs are written.

First&#x2F;last is ambiguous anyway and implementation dependant.

At least with the spaces&#x2F;tabs my IDE can automatically convert it for me.

First wins makes some sense from a security pov. You want to load the more &quot;trusted&quot; configuration files first because they must be able to further restrict the less trusted files, but don&#x27;t want to let the less trusted files override the settings set by the trusted files.Rejecting multiple definitions is better, unless you care about DoS (I would be pissed if I got locked out of ssh access due to an option being defined twice, for instance).

I would prefer a convention… but first wins is backwards IMO. It really sucks when a bad convention gets adopted, like this first wins, or spaces instead of tabs. ;)

I had the distinct pleasure (?) of owning the C++, Python, and Java implementations of that config for a while and ... yeah, it had some positives, but the fact that it had two orthogonal hierarchies of configuration in addition to implementation-specific feature gaps and nuances made it something of a nightmare in practice.

Amazon had a hierarchical configuration system for loading config that could override other config (or add to it, depending on the implementation). The Java framework would provide an endpoint to show you the final resolved config along with which file and line that value came from. Very helpful when trying to figure out why config was different than expected.At least the inspector in Safari (and maybe other WebKit browsers) does something similar for CSS.I’m all for it. It’s a pain when writing config systems (no longer just key&#x2F;value, but key+value+meta), but very helpful. It can be a pain for things like JSON where libraries don’t give you that type of diagnostic information easily, however.

Config file overrides the defaults, Command line flags overrides the configuration file.Don&#x27;t forget &quot;user configuration file overrides system configuration file (but is overridden by command line)&quot;.In tarsnap the priority (lowest wins) is:0: Command line1: ~&#x2F;.tarsnaprc2: &#x2F;etc&#x2F;tarsnap.conf (may be in a different system etc depending on platform standards)3: tarsnap defaults.

If you pass -verbose switch to any of my programs you&#x27;ll definitely see a line similar to this:<pre><code> verbose: Configuration file found at $PATH.
</code></pre>
There are a couple of places I look for my config files: &#x2F;etc, ~&#x2F;.config and &quot;.&#x2F;conf&quot;, at that order. The last one accelerates testing a great deal.And every configuration that overrides the built-in defaults are reported one by one.Also, I started to detail how configuration file works in README.md file, under its section now.Lastly, I decided to fix all of my tools to work on same specs from now on, regardless of the programming language they&#x27;re written in:1. Use TOML, and TOML only.2. Config file overrides the defaults, Command line flags overrides the configuration file.

I prefer the &quot;setting it twice to different values is an error&quot; mode, which isn&#x27;t especially common.

Reading in general to more specific configuration directories in order and using the last value read makes the most sense to me. This is usually how I write my own applications.Anyway, I thoroughly document the order and precedence, but the suggestion from Chris&#x27; blog to report more detail about the process is a good one.

Nice example, which opened a small rabbit hole for me: what does &quot;first occurrence wins&quot; mean for options that are meant to be used multiple times, such as &quot;Port&quot; in sshd_config. The man page only says &quot;Multiple options of this type are permitted.&quot;, but now I wonder what happens when I put my own Port into &#x2F;etc&#x2F;ssh&#x2F;sshd_config.d&#x2F;whatever.conf but leave &quot;Port 22&quot; in sshd_config? Or is the (new since bookworm) default sshd_config commenting it out for this reason?More generally, for multi-options do we maybe want &quot;first file wins&quot;?

Ant also had first writer wins. I tried to explain this so, so many times to so many people, and only rarely did it stick. On that project I spent way too much time fixing build errors, and ended up rearranging things to make the behavior more obvious rather than less, because without it people would fool themselves into thinking they had it right, because sometimes it would appear to work.And the thing with developers is that they make lousy scientists. If they want something to work, they’ll take any information that supports that theory and run straight to the repository with it.The trick is not to fool yourself, and you’re the easiest person to fool.

It&#x27;s crazy that a year from now, no one is ever going to be thinking about this garbage ever again. By then, enough programmers will have discovered how insanely good GPT4 is at doing software configuration, and in the absence of the right answer, reading these verbose docs to finally just tell you what to do.

It would also help if an active process said if it was running using the current config.For example, if a change has been introduced to a config file, but the process has not been restarted to incorporate those changes, this should be conveniently mentioned.

When discussing, let&#x27;s keep in mind what really mature configuration can look like:<a href="https:&#x2F;&#x2F;docs.spring.io&#x2F;spring-boot&#x2F;docs&#x2F;2.1.13.RELEASE&#x2F;reference&#x2F;html&#x2F;boot-features-external-config.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;docs.spring.io&#x2F;spring-boot&#x2F;docs&#x2F;2.1.13.RELEASE&#x2F;refer...</a>This is a big ridiculous, but the fact is that the entire order of processing there is a result of practical need evolved over likely thousands of organizations.There are unit testing concerns, dev&#x2F;test&#x2F;stage&#x2F;prod concerns, CLI args, OS vars, config files, IDE vs non-ide concerns, and more. And even some references to databases (if you squint and see JNDI as a database).So yeah, finding out where property value X was set can be a bit maddening oftentimes.&quot;Configuration&quot; is an afterthought in most languages and frameworks in initial iterations. So bespoke &#x2F; one-off solutions will explode early on. As the original complaint details, &quot;configuration&quot; isn&#x27;t just a simple property file. It gets complex pretty quickly, and a good configuration subsystem would be nice:- supports most of the concerns above- provides a means for logging the final config value set and WHERE IT SOURCED FROMBut... oh wait, config values aren&#x27;t just key-value. Wait until you get to trees of values or other more complicated config. Then value overlaying gets pretty hard, and resolution code gets complicated too.Think of infrastructure as code frameworks. The configuration is insanely deep and complicated. Many complicated systems have &quot;configuration languages&quot;, and they are probably Turing Complete.And once your configuration system is complicated enough ... your configuration system kind of implicitly needs ITS OWN configuration system.So maybe the first config layer is actually a bootstrap for loading all the configuration info.And.... then you have config values you need to secure. Hooo boy, now you have all the complexity of high security systems interfaces: authentication and authorization, decryption and hashing.And invariably there are binary blobs encoded in text friendliness.It gets crazy. Fast.BUT THE AMOUNT OF SIMILARITY IS NUTS. NUTS. This is begging for a common approach or standard or library.

I am incredibly torn on if this is brilliant or would have me cursing the developer&#x27;s name.

That would actually be my preference TBH.  Failing quickly at startup is usually far better than unpredictable behaviour later on, particularly if it&#x27;s a long-running service.
Though I&#x27;ve always thought that the behaviour of tools when launched interactively by the user, vs launched via an automated background process should probably be different - it may well be that if a user&#x2F;administrator isn&#x27;t aware it failed on machine startup the consequences of it not being running at all are worse than an ambiguous configuration setting.

&gt;&gt;&gt;Some programs use a &quot;last occurrence wins&quot; approach, while other programs (e.g. sshd) use a &quot;first occurrence wins&quot; approach.A bunch of applications at my job use a config-loading library that will fail on startup if the same option is set to two different value (on different lines). It&#x27;s certainly frustrating at times...but it&#x27;s also prevented a huge amount of ambiguity.

There should be an option to report where any currently active non-default configuration settings actually came from, individually. (including if that&#x27;s shell environment vars, etc).I forget what tool I use regularly that actually does this. Is it git?

Programs should not only report where the configuration files are located, but also report how configuration options contained therein are read in:Some programs use a &quot;last occurrence wins&quot; approach, while other programs (e.g. sshd) use a &quot;first occurrence wins&quot; approach.Buried in <a href="https:&#x2F;&#x2F;man.freebsd.org&#x2F;cgi&#x2F;man.cgi?sshd_config(5)" rel="nofollow noreferrer">https:&#x2F;&#x2F;man.freebsd.org&#x2F;cgi&#x2F;man.cgi?sshd_config(5)</a> and awkwardly expressed:&quot;Unless noted otherwise, for each keyword, the first obtained value will be used.&quot;That&#x27;s why e.g. Debian&#x27;s &#x2F;etc&#x2F;ssh&#x2F;sshd_config ( <a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;ssh-team&#x2F;openssh&#x2F;-&#x2F;blob&#x2F;debian&#x2F;1%259.3p1-1&#x2F;sshd_config?ref_type=tags#L13" rel="nofollow noreferrer">https:&#x2F;&#x2F;salsa.debian.org&#x2F;ssh-team&#x2F;openssh&#x2F;-&#x2F;blob&#x2F;debian&#x2F;1%25...</a> ) starts with<pre><code> Include &#x2F;etc&#x2F;ssh&#x2F;sshd_config.d&#x2F;*.conf
</code></pre>
, so that custom configurations in that directory win over the distribution-provided defaults set up after this Include directive inside &#x2F;etc&#x2F;ssh&#x2F;sshd_config.

I maintained a CLI tool for a long time. I came to the conclusion that good config is a bit harder and more subtle than most people think. I agree with the premise of the article, but there are more layers to it, literally.Depending on how much config you have, sooner rather than later you’ll have multiple sources: system defaults, user config files, sometimes profiles, env vars, and flags. Configuration may be sourced from a combination of these, so overrides need to be handled and naming needs to be consistent. Another source of mess is deeply nested config, such as for a user-defined project. Those are very awkward to add flags for, in which case it’s better to enforce a file (like kubernetes config).To me, tools like git strikes a good balance. And by that I mean that project-local config is in a project root directory that’s not necessarily checked in to VCS, but picks up the config even if you’re in a subdir. It rarely causes any unwanted surprises, but at the cost of being somewhat flag heavy, which is then alleviated with user-defined aliases.

Its got my vote. Speaking as someone contemplating a Solr installation left without any documentation by a previous employee.

and every configurtion file should be read-writable by some kind of UI.Especially those out-of-context .yaml files with lots of DSLs embedded.

Yeah but then at least include a cli interface that runs on that machine which allows you to edit the config.

not supporting configuration files on disk allows a service to work without needing read or worse write access to the disk. storing configuration in the database is a benefit. (potentially even for security)

I&#x27;d add: everything should use configuration files that are stored on the machine. I know services that use configuration files and you are expected to upload them into the service using an IDE and a plugin to then store the config in it&#x27;s database.

Lately I&#x27;ve written all my tools to not only announce where they&#x27;re reading their configs from (which isn&#x27;t precisely what OP is asking for but is close), but also to have an option to generate the default config that will be presumed if a config file is not found. &quot;mytool --gen-config &#x2F;tmp&#x2F;newconfig.txt&quot; should produce a fully-formed config with all defaults explicitly set which can be moved into &#x2F;etc&#x2F;mytool.conf or wherever it&#x27;s supposed to go, all ready for me to tweak without having to look up settings names.

I ran into this the other day as well.You can override the REQUESTS_CA_BUNDLE env var to point to your own certs.<a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;37447847" rel="nofollow noreferrer">https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;37447847</a>

For once I would say it isn&#x27;t Python&#x27;s fault either. There are at least 10 different locations that certificates can be stored at on various Unix variants and Linux distros. Go checks them all, which is probably the only sane solution, but seriously come on Unix devs... Is it that hard to pick a standard location?

Yesterday I was wondering why the self signed root certificate and the certificates I added to my debian install didn&#x27;t work with httpie (ssl warning) but did work with curl (and the browsers I added the RootCA to).I found the explanation: <a href="https:&#x2F;&#x2F;github.com&#x2F;httpie&#x2F;httpie&#x2F;issues&#x2F;480">https:&#x2F;&#x2F;github.com&#x2F;httpie&#x2F;httpie&#x2F;issues&#x2F;480</a> httpie doesn&#x27;t look into &#x2F;usr&#x2F;local&#x2F;share&#x2F;ca-certificates but apparently it&#x27;s not httpie&#x27;s fault, it&#x27;s the python requests library that doesn&#x27;t check that folder.I don&#x27;t know what to think of it and who is supposed to fix it. But I am back to curl with my tail between the legs (because I oversold httpie to my coworkers) for now. There&#x27;s only one place where the bucket stops with curl.

How should this be done instead? Are there any good docs&#x2F;tutorials?

My favourite is a very bad pattern common in Docker images. Just configure it with env vars, and then a script plucks them into a config file somewhere.And then document it misleadingly badly. These options are mandatory! (lists 5 of the 10 actual mandatory options).Then a script branches on a env var value, which isn&#x27;t one that&#x27;s documented, and then fails opaquely if it took the wrong branch because you didn&#x27;t know you needed to set that env var.Best ones are the ones that consume your env vars and set _other_ env vars, it&#x27;s great!

&quot;Submit a change and they&#x27;ll include it&quot; is the demo&#x2F;promotional version of FOSS

If any of the frequent offenders are open source, I wonder if ot would be worth your while to submit (or get someone else to) a change to add this functionality.

In my day job I do a lot of troubleshooting. That&#x27;s most of my job... helping other IT folks figure out what the heck is happening with something.I&#x27;d say that in 70% of things I work on, part of that time is figuring out what config files are used because the app&#x2F;service&#x2F;process&#x2F;whatever is doing something the owner&#x2F;tech&#x2F;whatever doesn&#x27;t expect, yet is rational, and they &quot;didn&#x27;t tell it to do that&quot;.It&#x27;d be great if everything reported as the article mentions -- amazing, in fact -- but I feel like it&#x27;s a pie in the sky wish. Like asking that all programs always exit cleanly or something. I feel like there&#x27;ll always be inevitable edge cases -- libraries that themselves pick up config files or registry values or environment variables that aren&#x27;t expected, who knows -- that&#x27;ll need to be discovered. If things are already not going right with a program, I&#x27;m less likely to trust what it says it&#x27;s doing and more likely to just watch it and see what it&#x27;s actually reading and doing.

this will work 0% of the time if app closes config after reading

just try using the unix command lsofif you know the PID aka Process ID for the application yr running try lsof -p PID_NUM_HEREit will give you a list of every open file your application is using along with it&#x27;s full pathfinally, if that gave you too many lines try lsof -p PID_NUM_HERE | grep -E &#x27;txt|properties|config&#x27;I know you wanted the files listed it in a help file however this will work 100% of the timeBest,
Stephen

macOS has something similar also with defaults [1].<a href="https:&#x2F;&#x2F;macos-defaults.com&#x2F;#%F0%9F%99%8B-what-s-a-defaults-command" rel="nofollow noreferrer">https:&#x2F;&#x2F;macos-defaults.com&#x2F;#%F0%9F%99%8B-what-s-a-defaults-c...</a>

The problem with the Windows registry, at least back in the day, was that it was a single file that could be corrupted and it would wreck your whole system.I think having a standard utility API for *nix configs makes a lot of sense. I&#x27;m surprised it doesn&#x27;t exist.I tried to find one, and there are some libraries for reading and parsing in every language, but nothing that seems to cover everything.This bash script seems to be a fairly &quot;built in&quot; way to parse them: <a href="https:&#x2F;&#x2F;unix.stackexchange.com&#x2F;questions&#x2F;441076&#x2F;which-is-the-standard-configuration-parser-library-used-in-linux" rel="nofollow noreferrer">https:&#x2F;&#x2F;unix.stackexchange.com&#x2F;questions&#x2F;441076&#x2F;which-is-the...</a>

You&#x27;re basically suggesting the windows registry.

I think the best way to achieve this is by providing an OS API that results in the files always being created in the same place. Applications&#x2F;libraries could still choose their own filenames and syntax, just the location would be OS controlled. I think there is room for a new desktop&#x2F;laptop OS to emerge and one good idea from mobile OS design I would like to see is having everything be an API call that allows the OS to impose standardisation, permissions and user preferences rather than the free-for-all desktop OSes have (though I propose letting the user run non-compliant applications, and not porting the iOS app store only model into the OS).

A good and underused tool to figure this out when the tool doesn&#x27;t help is strace.It allows you to trace system calls, and, crucially, file opens.

The config should also come with a short explanation on how to use it on it!

I just implemented a &quot;--config&quot; command in my latest code that reports exactly what was used for configuration, files and environment variables, and where the files are actually located.Makes life a lot easier imho.

If you maintain a Python CLI, you can use Click-Extra [1].It provides a ready-to-use --config option which reports how it sources the configuration file [2]:<pre><code> $ my-cli --verbosity DEBUG subcommand
 debug: Load configuration matching ~&#x2F;.config&#x2F;my-cli&#x2F;*.{toml,yaml,yml,json,ini,xml}
 debug: Pattern is not an URL.
 debug: Search local file system.
 debug: No configuration file found.
 (...)
</code></pre>
It also adds an auto-magic --show-params [3] to help you see where parameters are coming from:<pre><code> $ cli --int-param1 3 --show-params
 ╭─────────────────┬─────────────────────────────────────────┬─────────────────────────────────────────┬──────┬──────────────────┬─────────┬─────────────────┬─────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────┬─────────────╮
 │ ID │ Class │ Spec. │ Type │ Allowed in conf? │ Exposed │ Env. vars. │ Default │ Value │ Source │
 ├─────────────────┼─────────────────────────────────────────┼─────────────────────────────────────────┼──────┼──────────────────┼─────────┼─────────────────┼─────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────┼─────────────┤
 │ cli.color │ click_extra.colorize.ColorOption │ --color, --ansi &#x2F; --no-color, --no-ansi │ bool │ │ │ CLI_COLOR │ True │ True │ DEFAULT │
 │ cli.config │ click_extra.config.ConfigOption │ -C, --config CONFIG_PATH │ str │ │ │ CLI_CONFIG │ &#x2F;home&#x2F;runner&#x2F;.config&#x2F;cli&#x2F;*.{toml,yaml,yml,json,ini,xml} │ &#x2F;home&#x2F;runner&#x2F;.config&#x2F;cli&#x2F;*.{toml,yaml,yml,json,ini,xml} │ DEFAULT │
 │ cli.help │ click_extra.colorize.HelpOption │ -h, --help │ bool │ │ │ CLI_HELP │ False │ False │ DEFAULT │
 │ cli.int_param1 │ cloup._params.Option │ --int-param1 INTEGER │ int │ │ │ CLI_INT_PARAM1 │ 10 │ 3 │ COMMANDLINE │
 │ cli.int_param2 │ cloup._params.Option │ --int-param2 INTEGER │ int │ │ │ CLI_INT_PARAM2 │ 555 │ 555 │ DEFAULT │
 │ cli.show_params │ click_extra.parameters.ShowParamsOption │ --show-params │ bool │ │ │ CLI_SHOW_PARAMS │ False │ True │ COMMANDLINE │
 │ cli.time │ click_extra.timer.TimerOption │ --time &#x2F; --no-time │ bool │ │ │ CLI_TIME │ False │ False │ DEFAULT │
 │ cli.verbosity │ click_extra.logging.VerbosityOption │ -v, --verbosity LEVEL │ str │ │ │ CLI_VERBOSITY │ WARNING │ Debug │ COMMANDLINE │
 │ cli.version │ click_extra.version.VersionOption │ --version │ bool │ │ │ CLI_VERSION │ False │ False │ DEFAULT │
 ╰─────────────────┴─────────────────────────────────────────┴─────────────────────────────────────────┴──────┴──────────────────┴─────────┴─────────────────┴─────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────┴─────────────╯
</code></pre>
[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;kdeldycke&#x2F;click-extra">https:&#x2F;&#x2F;github.com&#x2F;kdeldycke&#x2F;click-extra</a>[2]: <a href="https:&#x2F;&#x2F;kdeldycke.github.io&#x2F;click-extra&#x2F;config.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;kdeldycke.github.io&#x2F;click-extra&#x2F;config.html</a>[3]: <a href="https:&#x2F;&#x2F;kdeldycke.github.io&#x2F;click-extra&#x2F;parameters.html#show-params-option" rel="nofollow noreferrer">https:&#x2F;&#x2F;kdeldycke.github.io&#x2F;click-extra&#x2F;parameters.html#show...</a>

This doesn&#x27;t tell you the current running configuration, it tells you the current configuration on disk. These should be the same, but aren&#x27;t always. I personally have a nasty habit of forgetting to reload configs. I make sure to leave them in a syntactically valid state, but I really don&#x27;t care if someone can&#x27;t access my webserver for a while; if I have to go, I add a quote and a curly bracket, save, and close my ssh session. If my 5 minutes away runs into tomorrow, then I no longer remember what&#x27;s been changed (I manage this these days by making &#x2F;etc a git repo with an ignore list and stashing incomplete changes...then forgetting to commit them when I&#x27;m done)

Seconding this, for sure. &quot;&#x2F;usr&#x2F;sbin&#x2F;sshd -T&quot; (T for &quot;test&quot;) has been a life-saver more than once.My strongest version of this is that any application that manages any state should be able to report on that state in a human- and machine-readable way.

I wish more programs had an obvious way of showing what configuration values are currently set. Something like `program --config` with no arguments should print all configuration keys and their current values.

This is a high quality blog post. It raises awareness of a usecase that&#x27;s always been there but it might never surfaced, it goes straight to the point, and provides clear examples.It&#x27;s a blog post where everyone ends up being slightly better after reading it.

thanks for pointing that out, I&#x27;ll make sure to include it next time I write a new application.

Basically the location of everything should be exposed to the user, especially if he is the reader of the source code.

We could use pkg-config for that. It&#x27;s meant for build systems but you could store any metadata in there, in theory. <a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Pkg-config" rel="nofollow noreferrer">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Pkg-config</a> <a href="https:&#x2F;&#x2F;people.freedesktop.org&#x2F;~dbn&#x2F;pkg-config-guide.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;people.freedesktop.org&#x2F;~dbn&#x2F;pkg-config-guide.html</a>

systemd has manual pages. I pointed out some missing configuration file locations back in 2016. They are still undocumented today.* <a href="https:&#x2F;&#x2F;jdebp.uk&#x2F;FGA&#x2F;systemd-documentation-errata.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;jdebp.uk&#x2F;FGA&#x2F;systemd-documentation-errata.html</a>

List them in the FILES section of the man page. You have a man page, right?Also, follow the XDG base directory specification[1].[1] <a href="https:&#x2F;&#x2F;specifications.freedesktop.org&#x2F;basedir-spec&#x2F;basedir-spec-latest.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;specifications.freedesktop.org&#x2F;basedir-spec&#x2F;basedir-...</a>

Especially if there are multiple places to read them from: current dir, PATH, classpath, $PROJECT_HOME, database, env vars... With env vars it would be handy to also know where they come from, is there a way? like having a timestamp when it was set, by what user and process.

I also vastly prefer it when error messages give a hint about what reconfiguration you can do to fix it, including correct syntax and googleable terms.So don&#x27;t do: farblewarble too low.Instead:FarbleWarble is 1234 but need at least 5678. Edit &#x2F;etc&#x2F;whatever.conf and add: FarbleWarble=&quot;5678&quot;There is a special hell for programs that talk abut how the Burble is configured incorrectly , without any mention of FarbleWarble

Are there any good examples of projects that do this really well?

This is how we wind up with three implementations of a “Windows-style Registry” while also having all the other management modes; local dotfiles,system dotfiles, all of &#x2F;etc and whatever is allowed under &#x2F;opt.Be smarter about package management and this is a non-issue; checksums, install-time dumps of file modifications, and the like.It’s why tools like CFEngine and the rest were invented, and I’m upset this is still an issue for people like OP twenty years later.

I like the way that Postgres handles this.There is a system view on top of their configuration file(s) that reports from which file and line number the currently active value was taken from.

Not only that, but past a certain complexity of the config, programs should have a way to dump their runtime config to the command line. Yes, I am talking to you, server daemons that take configs from 2-3 different rc-file locations, hardcoded defaults, envvars, the moon phase and the current market value of dried parsnips.If you don&#x27;t have a way to simply tell me what your current config is, then you have no earthly right to run it.

When standards are not enforced, they became optional and IMO not really a standard.

There are standards. The article even mentions the xdg base dir standard. The problem is that not everyone follows them but a large number still do and then it is easy to find them. I still agree that app containers are still a good idea.

I feel no standards for config files is probably the biggest reason app containers got traction with advanced home users. Save your docker containers nuke your server install move to a new distro and reimport back the app container.

&gt; $CWD&#x2F;openssl EXECUTABLE!!! (why look in $CWD?)debootstrap, maybe?

Details of hidden files opened by Debian&#x27;s update-ca-certificates:<a href="https:&#x2F;&#x2F;egbert.net&#x2F;blog&#x2F;articles&#x2F;ca-certificates-rebuild-on-debian.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;egbert.net&#x2F;blog&#x2F;articles&#x2F;ca-certificates-rebuild-on-...</a>&#x2F;etc&#x2F;ca-certificates.conf file&#x2F;usr&#x2F;share&#x2F;ca-certificates&#x2F;* directory&#x2F;usr&#x2F;share&#x2F;ca-certificates&#x2F;mozilla&#x2F;* directory&#x2F;usr&#x2F;share&#x2F;ca-certificates&#x2F;&#x2F; directory&#x2F;usr&#x2F;local&#x2F;share&#x2F;ca-certificates&#x2F;* directory&#x2F;usr&#x2F;local&#x2F;share&#x2F;ca-certificates&#x2F;&#x2F; directory&#x2F;etc&#x2F;ssl&#x2F;certs directory
$CWD&#x2F;&lt;all-certs-read-before&gt; files&#x2F;usr&#x2F;lib&#x2F;ssl&#x2F;openssl.cnf file&#x2F;etc&#x2F;ca-certificates&#x2F;update.d directory&#x2F;etc&#x2F;ca-certificates&#x2F;update.d&#x2F;jks-keystore directory&#x2F;etc&#x2F;default&#x2F;cacerts directory&#x2F;etc&#x2F;java-11-openjdk&#x2F;security&#x2F;nss.cfg file&#x2F;usr&#x2F;share&#x2F;ca-certificates-java directory&#x2F;usr&#x2F;lib&#x2F;jvm&#x2F;java-11-openjdk-amd64&#x2F;lib&#x2F;jvm.cfg file&#x2F;usr&#x2F;share&#x2F;ca-certificates-java&#x2F;ca-certificates-java.jar file&#x2F;etc&#x2F;ca-certificates&#x2F;update.d&#x2F;mono-keystore directory&#x2F;etc&#x2F;mono&#x2F;4.5&#x2F;machine.config file&#x2F;etc&#x2F;mono&#x2F;assemblies&#x2F;cert-sync&#x2F;cert-sync.config file&#x2F;etc&#x2F;mono&#x2F;assemblies&#x2F;Mono.Security&#x2F;Mono.Security.config file&#x2F;etc&#x2F;mono&#x2F;assemblies&#x2F;mscorlib&#x2F;mscorlib.config file&#x2F;etc&#x2F;mono&#x2F;assemblies&#x2F;System&#x2F;System.config file&#x2F;etc&#x2F;mono&#x2F;config file&#x2F;etc&#x2F;ssl&#x2F;certs&#x2F;ca-certificates.crt file$HOME&#x2F;.mono&#x2F;config file&#x2F;usr&#x2F;lib&#x2F;mono&#x2F;4.5&#x2F;cert-sync.exe.config file&#x2F;usr&#x2F;lib&#x2F;mono&#x2F;4.5&#x2F;cert-sync.exe.config file&#x2F;usr&#x2F;lib&#x2F;mono&#x2F;4.5&#x2F;mscorlib.dll.config file&#x2F;usr&#x2F;lib&#x2F;mono&#x2F;gac&#x2F;Mono.Security&#x2F;4.0.0.0_0738eb9f132ed765&#x2F;Mono.Security.dll.config file&#x2F;usr&#x2F;lib&#x2F;mono&#x2F;gac&#x2F;System&#x2F;4.0.0.0__b77a5c561934e089&#x2F;System.dll.config file&#x2F;usr&#x2F;share&#x2F;.mono&#x2F;certs&#x2F;Trust&#x2F;ski-*.cer file&#x2F;usr&#x2F;share&#x2F;.mono&#x2F;certs&#x2F;new-certs&#x2F;XXXXXXXX.0 file$CWD&#x2F;openssl EXECUTABLE!!! (why look in $CWD?)&#x2F;usr&#x2F;bin&#x2F;openssl&#x2F;usr&#x2F;local&#x2F;bin&#x2F;openssl&#x2F;usr&#x2F;local&#x2F;sbin&#x2F;openssl&#x2F;usr&#x2F;sbin&#x2F;openssl (VERY STRANGE ordering of &#x2F;usr&#x2F;[local&#x2F;][s]bin&#x2F;

Fastest way to identify hidden and dynamic configuration filenames:<pre><code> strace -f &lt;executable-file&gt; | grep openat
</code></pre>
Yeah, I was just looking at the biggest offender that I have encountered: Debian&#x27;s update-ca-certificates, 37 hidden configuration files.

Alternately, every aysadmin should learn how to drive strace

I love this post. Such a crazy diagram for something that intuitively we believe should be simple. You can easily imagine most of the the compromises and feature requests over the years that brought it to this state. All software that survives decades of care and upkeep tends to end up in a similar state.

This is a why Murex (my shell) has a ‘runtime’ builtin which lets you query which script updates what. Every function, variable etc can be queried to find where they were set and even when (time stamped) too.

I wonder if there is any mainstream program with a more convoluted configuration scheme than bash? Just when think you understand the various invocations of .profile, .bash_profile, .bashrc in login&#x2F;non-login&#x2F;interactive&#x2F;non-interactive modes in home and&#x2F;or &#x2F;etc after digesting an article with flowcharts[1], you realize that any invocation can execute arbitrary code from any location, and different distributions (not to mention, macOS, WSL, git-for-Windows, etc) all source different stuff in their default configs (like executing .bashrc in .profile).[1] <a href="https:&#x2F;&#x2F;blog.flowblok.id.au&#x2F;2013-02&#x2F;shell-startup-scripts.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;blog.flowblok.id.au&#x2F;2013-02&#x2F;shell-startup-scripts.ht...</a>

I use Pcgamingwiki so that I can find out where the hell the developers decided to put their config and save game location. 
Windows is wild.

It would also be nice to have an “explain” option where it shows how or why a value was updated. I am specifically thinking of Ansible which has a multitude of locations from which it sources variables. It would be nice to be able to spy on a variable and report every value change and where that change came from.

It would for env vars as well, why not? Just trace program trying to reach out for some var X from env.

Might work for config files but would it work for environment variables?

This seems to be an interesting approach.Why do you suggest this path over documentating the behavior explicitly?What makes one approach better the other? Is it faster to run a trace? Should a developer trace their own code and auto-generate documentation post-build?

I disagree it&#x27;s responsibility of the tool. Program should just reach out for files and some higher order or meta runner should trace what the process does. One of the commenters mentioned `strace`, this is the correct approach, but should be done in well engineered way.Other way is to run program in symbomlic interpreter and trace what it wants to read in any of the branches.

Easier option: sandbox each executable, so every output file will be located in a special folder.

Actually, why not both?Why not have files and an database like interface for them?Even better, databases normally provide a query plan explanation, quite like what is demanded here.

&gt;If you need complex data structures use YAML.Please god no. JSON or TOML at most. YAML is too complicated.

&gt; much simpler to work with.It&#x27;s simple when the task is simple, and it becomes a lot more complex than necessary when problems become more complex.Consider the joy of sysfs &#x2F; procfs and similar. These are filesystems that, through files, expose information that is numbers, strings, lists, even trees. There&#x27;s no uniformity there, no way to predict where and what sort of files with what sort of data will appear... Sometimes you get extra files with &quot;metadata&quot; that describes the format found in other files, sometimes it&#x27;s in the same file, and sometimes it just doesn&#x27;t exist. Sometimes numbers are encoded as strings, and sometimes as bytes. Booleans can be encoded as numbers, (which are either strings or bytes).There&#x27;s plenty of weird behavior that isn&#x27;t described by files &#x2F; their properties. For example, suppose you want to delete a RAID device, but if fails because it turns out that it&#x27;s used by LVM volume, but you didn&#x27;t even know you were supposed to look for that...It gets even worse with proprietary stuff that doesn&#x27;t follow conventions. Take for instance NVidia GPUs. They do appear in sysfs as PCIe devices... but most interesting properties just aren&#x27;t listed there. Instead, NVidia&#x27;s driver creates &quot;devices&quot; in udevfs (i.e. &#x2F;dev) with arbitrary properties that you cannot predict because the format is whatever the vendor wanted it to be at any version of their product.In other words, files are a very poor mechanism for describing data. In order to work with them you need conventions that aren&#x27;t part of any file mechanism. Take network interface names, or block device names, stuff like eth0 or nvme0n1p1 -- you need to know the convention and how to interpret the name (i.e. the first suggests that the device is using Ethernet driver and is the first one connected, the second means that the device is using NVMe driver, it&#x27;s the first device of its kind attached to PCIe bus, first namespace, first partition.) Specifically, in the case of block devices, you could discover the same information given in the file name by traversing the relevant filesystem subtree, but in some cases the information is just encoded as a file name or as it contents, but it doesn&#x27;t utilize the filesystem primitives. It just relies on convention.----&gt; I think most Unix software uses a fairly simple text format for config filesHow did you count?Do you mean software that is part of UNIX standard or software that runs on UNIX?What makes the majority an interesting metric? (I.e. suppose there are 10M of different programs that run on UNIX, and 6M of them are simple enough that they don&#x27;t require complicated configuration, there are still 4M of those that do, and some of them are probably used daily by millions of users.I mean, who cares if simple programs don&#x27;t need complicated configuration? The problem is the configuration of complex programs, and they are indispensible in our daily lives.

&gt; If you need complex data structures use YAMLYAML is a poorly engineered format. It doesn&#x27;t think about storage at all, for example (i.e. in its definition, it&#x27;s just a single text blob. What if you need to split it?The format is awful for random access: you need to read the whole thing to extract a single data item. If you need to query such a configuration multiple times you&#x27;ll do a lot of unnecessary work.YAML poorly defines numerical types (how many digits can a float have? what about integers? Are floats and integers the same thing?) YAML lists... allow mixed types. Not sure it&#x27;s such a good thing from performance and correctness perspective.YAML has unnecessary &quot;hash-table&quot; type. Hash-tables aren&#x27;t a data format, they require function in the reader that makes asymptotic properties of hash-tables work. In fact, YAML has just a weird sub-type of list, where elements come in pairs. Since it inherited all this from JSON, there&#x27;s also the ambiguity related to repeated keys in such hash-tables -- what should an application do with those nobody knows.YAML sucks for transmission due to the ambiguity caused by repeated keys in &quot;hash-tables&quot;. You cannot stream such a format if you have the policy that the last key wins or if you have the policy that repetition is not allowed. Also, since YAML allows references but doesn&#x27;t require that the reference lexically point to a previously defined element, you may have unresolved references which, again, will prevent you from streaming.YAML defines mappings to &quot;native&quot; elements of different other languages... but what should you do if you are parsing it in Ruby, but get a mapping to Python?YAML schema sucks. It would require a separate &quot;expose&quot; to describe why.YAML comes with no concept of users or namespaces which would be necessary for ownership &#x2F; secure sharing of information. It also comes without triggers which would allow an application to respond to changes in data.YAML is very hard to parse for no reason. It&#x27;s very easy to make a typo in YAML that will not make it invalid, but will be interpreted contrary to what was intended.YAML doesn&#x27;t have a canoncial form which makes comparing two elements a non-trivial and in some cases undefined task.YAML doesn&#x27;t have variables, nor does it have forall &#x2F; exists functionality. This results in a lot of tools that work with YAML overlaying it with yet another layer of configuration, which includes variables (eg. Helm charts, Ansible playbooks).----I mean, honestly, people who created YAML probably saw it as an inconsequential, sketchy project that should take maybe a weekend or two. They didn&#x27;t plan for this format to be the best... they just made something... that sort of did something... but not really. I have no idea why something like this received the acclaim that it did.

Having spent many years in the windows land I definitely get where you&#x27;re coming from. However, I find the Unix way where everything is a file, much simpler to work with.Just take a simple task of copying a subset of some system&#x27;s configuration to another one. On Unix it&#x27;s just a matter of copying some files. On Windows (and our imaginary is that provides object storage) we would need to find the relevant objects, then write a script to export&#x2F;import it (or mess with export&#x2F;import tools for a while).Personally I see &quot;everything is a file&quot; as strength not weakness. In addition to my personal preference having a centralised configuration system not only restricts in many ways, presents a potential performance bottleneck, but also is an extra point of failure.Additionally, I think most Unix software uses a fairly simple text format for config files. There really is little reason for using anything else. If you need complex data structures use YAML. I&#x27;ve found many more proprietary binary config file formats on Windows (despite registry) than on Unix. So if the os would make something like this available there would still be no guarantee all apps use it. It would just be an extra thing to deal with for little benefit IMO.

I significantly prefer using DConf over brute forcing some new configuration file format.It works like the Windows registry, but now the fields also have native support for a description that tells you what the settings do.

&gt;Wasn&#x27;t Windows Registry a (failed) attempt at unifying this stuff?The way I see it, Windows Registry &quot;failed&quot; for no reason other than a handful of CLI neckbeards refusing to let go of individual .ini and other such plain text config files written however and scattered wherever they damn well please.The Registry is really well designed, it&#x27;s a central repository of settings for everything that exists in that Windows installation including Windows itself. It can hold data besides plain text. It can be manipulated both by the CLI and GUI and is easily accessed, either individual keys or in batches. It can accomodate multi-user environments independently of the file system.The Registry still exists and is still used by most Windows programs, though unfortunately even Windows is starting to disregard the Registry with its newer developments. The Registry for me is a reminder that the programming industry is, in many ways, one of the most regressive industries ever.

It&#x27;s funny that editing the registry is considered black-magic in Windows world while elsewhere we often edit untyped unchecked text files.

Wasn&#x27;t Windows Registry a (failed) attempt at unifying this stuff?Now, hear me out. The problem is not the location of configuration. The problem is the crippled and anemic way UNIX (and the likes) programs take configuration. These can only predictably take environment variables and command-line arguments. Both ways are crippled due to size limit (especially the command-line arguments) and structure (you cannot even have lists &#x2F; arrays in environment variables). This is the reason why every application author believes they must write their own configuration, with their own (crippled) configuration format and logic for locating their configuration.On the bright side, I think, systemd is moving towards making this mess more uniform. On the not so bright side, systemd doesn&#x27;t appreciate the difficulties associated with the problem and doesn&#x27;t seem to have any sort of plan moving forward. So, unit files are kind of configuration, but because the format is so crippled and anemic, they immediately added a way to source random external files for process environment etc.Finally, systemd, even if they understood the problem well and put a good effort towards solving it are still powerless against the format in which system supplies input to processes. What really needs to change is all the family of execve, execlp and so on. This was always a stop-gap &#x2F; bad idea, but it was OK for a while, before programs started to require more complex input. It should&#x27;ve died in the 80s... but today it is so deeply entrenched that removing it will take a very long time.Another problem here is the approach that &quot;everything is a file&quot;. So, the only way to provide configuration beyond command-line arguments and environment variables is a file... But, imagine an alternative reality in which OS is capable of providing basic data-structures, which can also be persisted and queried? Kind of like you do it with a relational database? I think this idea is so much not novel that it&#x27;s older than most people commenting in this thread... And, even though this would make it so much easier for so many application authors to provide consistent and easy to inspect, better tested interface to their programs... nobody&#x27;s doing that :) Because files are good enough.

That&#x27;s one thing I like about &quot;systemctl cat &lt;service&#x2F;unit name&gt;&quot;: it shows both the content and the path of the config file.

Every new account&#x2F;os installation, I have to hunt around to turn that off. I absolutely dislike colored output in almost all cases. Might be because I&#x27;m color-blind. But they keep moving those settings around. Perhaps that part is done in their spare time by the MS Office menu bar team.

Interestingly `LS_COLORS` is set by default on most distributions... and you definitely wouldn&#x27;t want to type that in every time.

I&#x27;m of the opinion that most programs should get their config files explicitly from the command line. None of that ~&#x2F;.apprc or $APP_PORT=8080. That&#x27;s inconvenient for tools you use often, like ls or grep. If they read config files, an option like --ls-config would be very welcome.

Agreed! Few things anger me more than when a program simply reports, &quot;File not found&quot;... OK, which file? Where were you looking for said file?I know how we get here, people simply report the error from the OS w&#x2F;out adding context. This is why I love anyhow reporting in Rust, you can attach context (like the file name and path) to the error.In log-store.com I report the default name of the config file (you can specify it via cmdline if you want), and the 3 locations and order searched, if the config file isn&#x27;t found. I also have it report the file being used, so if you expected the file in &#x2F;etc, but you accidentally had one in your home directory, you&#x27;ll know on startup.These things seem like table-stakes in 2023.

Have my upvote. So many hours lost to &quot;[software package] config file location&quot;.

strace has its own filtering mechanism, so you can do `strace -e open`. Though you might have to add `,openat,openat2` so maybe your way is easier in practice.

I&#x27;ve found strace | grep &quot;open&quot; particularly useful for locating config files. They&#x27;re usually opened right at the start of the program as well.(For reference, strace prints out every system call used by a program with the arguments that were called; occasionally, a program may use something other than `open` or `fopen` to read in a file, so that should be kept in mind when using this method)

That seems like a really good solution, for the cost of a little more mucking around to begin with.

In Conduit (<a href="https:&#x2F;&#x2F;conduit.rs" rel="nofollow noreferrer">https:&#x2F;&#x2F;conduit.rs</a>) there is no default config path. In order to start Conduit, you need to specify the CONDUIT_CONFIG environment variable, which is the path to your config. This will typically be done in a systemd service.This has multiple benefits:- You can&#x27;t accidentally start Conduit with a wrong config (e.g. the config in the current working directory)- You can have multiple Conduits running at the same time with different config- It&#x27;s easier to understand for the system administrator because there is no hidden information

The popular VLC alternative MPV is very guilty of this. I got shut down when I tried to discuss it in a Github issue

If I need to know where some program is looking for startup files and in what exact order, I just use strace on it.

I try to write all my CLIs this way - to the point of writing my own Go CLI parsing library [0] to do it.Now, when I run `cmd -h`, I get output like the following for all flags (example from [1]):<pre><code> --count , -c : Number of times to dig
 type : int
 default : 1
 configpath : dig.combine.count
 required : true
 currentvalue (set by config) : 1
</code></pre>
It&#x27;s sometimes more verbose than I&#x27;d like, but it&#x27;s quite useful for figuring out how a flag got its current value[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;bbkane&#x2F;warg">https:&#x2F;&#x2F;github.com&#x2F;bbkane&#x2F;warg</a>
[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;bbkane&#x2F;shovel">https:&#x2F;&#x2F;github.com&#x2F;bbkane&#x2F;shovel</a>

this would also really help the CLI in terms of auto completion, I see people go above and beyond writing completion scripts, which could be simplified instead by running --autocomplete or checking some file somewhere.

I would go one step further and say that all sources of runtime configuration should be reportable:* Config files* CLI options* Environment variablesAnd what takes precedence when.

That&#x27;s something I think anyone who remembers what being a newbie linux user is like will agree with.

The first line of resolv.conf is a comment indicating it&#x27;s written by Network Manager. So that part didn&#x27;t trip me up at least.

&gt; Instead you edit either one of the interface configuration files and tell NM (via nmcli) to reload itYeah, and this would be fine if both &#x2F;etc&#x2F;resolv.conf told you what is creating that file and NM didn&#x27;t rewrite its own configuration files all the time.Anyway, it would be not that bad if NM was kept restricted to RedHat, because the entire philosophy of that distro is that configuration files are just suggestions and the system is free to change anything at any time. But instead, that behavior infected everything.

I needed to update &#x2F;etc&#x2F;resolv.conf under RedHat the other day and I was surprised by how much this has changed. You don&#x27;t edit &#x2F;etc&#x2F;resolv.conf directly anymore because it&#x27;s maintained by Network Manager[1]. Instead you edit either one of the interface configuration files and tell NM (via nmcli) to reload it, or you use nmcli to update the interface config file.Fine, I can deal with that.What really surprised me were a couple programs that needed to be restarted to see the updated &#x2F;etc&#x2F;resolv.conf as they apparently don&#x27;t use glibc for DNS resolution. I used lsof to figure out what was sending DNS requests to the old name server IPs.It turned out to be osqueryd and Apache Traffic Server.1. There are various ways to put &#x2F;etc&#x2F;resolv.conf back under manual control if you don&#x27;t want NM to maintain it.

php.ini is my experience of the worst of this, finding which of the fifty five billion config files is the actual one controlling the web server’s php and not the console’s is always a fun journey.Luckily ctrl-r helps the second time.

Compiled languages absolutely can support this.But imperative ones create a bit of a problem. Nothing impossible to deal with, but it becomes harder.But any choice of language, the OS can enforce it too, and maybe in a much easier way than messing with your language.

i wonder whether our current strong compile-time&#x2F;runtime distinction can support this. it may be slightly easier for interpreted and vm-based languages but i’m not sure we can have it more broadly.lately i’ve been playing with prolog and revisiting computer science&#x2F;computer programming before imperative languages drew a curtain on innovation and i think we have lost a lot. our modern programming isn’t the programming the founders had in mind, it seems.

It drives me nuts when systems are unnecessarily built with &quot;magic DI&quot; frameworks rather than initializer&#x2F;constructor DI. Why is it so hard to pass a dep into the initializer? Make the initializer parameter a protocol&#x2F;interface and you can pass a mock in at testing. And you can also grep the whole codebase for callers and find the exact concrete type passed in at runtime.

Object orientation kills traceability. Classes are instantiated in one trace, then methods are called in another trace, and if something goes wrong in the latter, it is not trivial to find the wrong settings passed in the former.I think some functional programming languages solve that by running a good chunk (if not all) of the application in a single trace.

This would be sooooo nice, especially in DI heavy languages like Java. It might not be turtles all the way down, but finding the final turtle can be so difficult.

This should be provided by the PLS for the language, not by an IDE.

So it&#x27;s an interesting thought experiment, but... I suspect the Rust compiler could statically figure this out for a large # of cases .... based on following the borrow&#x2F;copy&#x2F;clone tree for any particular variable. It could in theory be possible to write a tool which traces that graph back to the snippets of code where it originates, though that would usually end up being a command-line args parser, or a config file reader...It&#x27;s mostly feasible in other languages to, but probably not with the same level of rigor.Anybody here from JetBrains?

It seemed very promising until it showed DevOps engineers dragging blocks in a shared (synchronous) canvas like Figma.Am I the only one that thinks pull requests (asynchronous) are a more appropriate medium of collaboration for this kind of work that requires strong consistency between elements?

Sounds a lot like what systeminit is working on: <a href="https:&#x2F;&#x2F;www.systeminit.com&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.systeminit.com&#x2F;</a>

&gt; You click on any value and you see a graph&#x2F;chain of all the transformations this data goes through right back to the source of truth.This kind of reminds me of the Whyline: <a href="https:&#x2F;&#x2F;www.cs.cmu.edu&#x2F;~NatProg&#x2F;whyline.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.cs.cmu.edu&#x2F;~NatProg&#x2F;whyline.html</a>

This is a minor thing which I found useful- The &quot;help&quot; functions of all my bash-defined functions report which file they&#x27;re defined in (such as: `echo &quot;This function is defined in ${BASH_SOURCE[0]}&quot;`). If I could do something similar for my environment-defined values (hmmm, now I&#x27;m thinking...), I would. (I could create a function that tracks at least the changes _I&#x27;VE_ made to these values, which could help...)

In complex prod systems, logging and traceability are generally more important than memory consumption and disk space. Memory and Disk are cheaper than a lawsuit if something goes awry

Check out <a href="https:&#x2F;&#x2F;pernos.co&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;pernos.co&#x2F;</a>.

Yes but wouldn&#x27;t the logging take up quite a bit of memory? Something like O(compute cycles* (log(lines of code)+log(memory allocated))) before compression.

&gt; You click on any value and you see a graph&#x2F;chain of all the transformations this data goes through right back to the source of truthIsn&#x27;t that just a debugger?

FWIW, we do something like this in Spack[1] with `spack config blame`. There are different YAML configs, they can be merged and override each other based on precedence, but we tell you where all the merged values are coming from. Here&#x27;s a simplified example -- you can see that some settings are coming from the user&#x27;s &quot;default&quot; environment, some from Spack&#x27;s defaults, and some from specialized defaults for Darwin:<pre><code> &gt; spack -e default config blame packages
 --- packages:
 spack&#x2F;environments&#x2F;default&#x2F;spack.yaml:59 fftw:
 spack&#x2F;environments&#x2F;default&#x2F;spack.yaml:60 require: ~mpi
 spack&#x2F;etc&#x2F;spack&#x2F;defaults&#x2F;darwin&#x2F;packages.yaml:17 all:
 spack&#x2F;defaults&#x2F;packages.yaml:18 compiler: [apple-clang, gcc, clang]
 spack&#x2F;defaults&#x2F;darwin&#x2F;packages.yaml:20 providers:
 spack&#x2F;defaults&#x2F;darwin&#x2F;packages.yaml:21 unwind: [apple-libunwind]
 spack&#x2F;defaults&#x2F;packages.yaml:20. blas: [openblas, amdblis]
 spack&#x2F;defaults&#x2F;packages.yaml:21 jpeg: [libjpeg-turbo, libjpeg]
 spack&#x2F;defaults&#x2F;packages.yaml:22 lapack: [openblas, amdlibflame]
 spack&#x2F;defaults&#x2F;packages.yaml:23. mpi: [openmpi, mpich]
 spack&#x2F;defaults&#x2F;darwin&#x2F;packages.yaml:22 apple-libunwind:
 spack&#x2F;defaults&#x2F;darwin&#x2F;packages.yaml:23 buildable: False
 spack&#x2F;defaults&#x2F;darwin&#x2F;packages.yaml:24 externals:
 spack&#x2F;defaults&#x2F;darwin&#x2F;packages.yaml:27 - spec: apple-libunwind@35.3
 spack&#x2F;defaults&#x2F;darwin&#x2F;packages.yaml:28 prefix: &#x2F;usr
</code></pre>
The filenames are shown in color in the UI; you can see what it really looks like here: <a href="https:&#x2F;&#x2F;imgur.com&#x2F;uqeiBb5" rel="nofollow noreferrer">https:&#x2F;&#x2F;imgur.com&#x2F;uqeiBb5</a>For the curious, there is more on scopes and merging [2]. This isn&#x27;t exactly easy to do -- we use ruamel.yaml [3], but had to add our own finer-grained source attribution to get this level of detail. There are also tricky things to deal with to maintain provenance on plain old Python dict&#x2F;str&#x2F;list&#x2F;etc. objects [4,5].[1] <a href="https:&#x2F;&#x2F;github&#x2F;spack&#x2F;spack" rel="nofollow noreferrer">https:&#x2F;&#x2F;github&#x2F;spack&#x2F;spack</a>[2] <a href="https:&#x2F;&#x2F;spack.readthedocs.io&#x2F;en&#x2F;latest&#x2F;configuration.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;spack.readthedocs.io&#x2F;en&#x2F;latest&#x2F;configuration.html</a>[3] <a href="https:&#x2F;&#x2F;yaml.readthedocs.io&#x2F;en&#x2F;latest&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;yaml.readthedocs.io&#x2F;en&#x2F;latest&#x2F;</a>[4] <a href="https:&#x2F;&#x2F;github.com&#x2F;spack&#x2F;spack&#x2F;blob&#x2F;95ca9de&#x2F;lib&#x2F;spack&#x2F;spack&#x2F;util&#x2F;spack_yaml.py#L36-L99">https:&#x2F;&#x2F;github.com&#x2F;spack&#x2F;spack&#x2F;blob&#x2F;95ca9de&#x2F;lib&#x2F;spack&#x2F;spack&#x2F;...</a>[5] <a href="https:&#x2F;&#x2F;github.com&#x2F;spack&#x2F;spack&#x2F;blob&#x2F;95ca9de&#x2F;lib&#x2F;spack&#x2F;spack&#x2F;config.py#L1202-L1206">https:&#x2F;&#x2F;github.com&#x2F;spack&#x2F;spack&#x2F;blob&#x2F;95ca9de&#x2F;lib&#x2F;spack&#x2F;spack&#x2F;...</a>

This is the central problem with all programming today.You can think of everything as configuration. Parameters of a function are just configuration for that function.What we need is traceability of every value in a system, not just configuration. You click on any value and you see a graph&#x2F;chain of all the transformations this data goes through right back to the source of truth. This is how we should code. Think about how difficult this is to do today with your current IDE&#x2F;editor.Every task in software is wanting to change what is displayed in a user interface of some sort. Usually you have to make your way through a labyrinth to figure out what to change where, and also what side-effects will occur.If this is not smack bang in front of your face when you are coding, then you are just adding more and more tangled data flows and complexity to your system.

Everything that uses configuration files should report where they're located