Thanks for the pointer! Very interesting. I actually may end up doing a prototype and see how far I can get.

“Announcment”

It used to be quite common on mailing lists to categorise/tag threads by using subject prefixes such as “ANN”, “HELP”, “BUG” and “RESOLVED”.

It’s just an old habit but I feel my messages/posts lack some clarity if I don’t do it 😅

I didn’t like the capitalised names so configured xdg to use all lowercase letters. That’s why ~/opt fits in pretty nicely.

You’ve got a point re ~/.local/opt but I personally like the idea of having the important bits right in my home dir. Here’s my layout (which I’m quite used to now after all these years):

$ ls ~
bin  
desktop  
doc  
downloads  
mnt  
music  
opt 
pictures  
public  
src  
templates  
tmp  
videos  
workspace

where

• bin is just a bunch of symlinks to frequently used apps from opt
• src is where i keep clones of repos (but I don’t do work in src)
• workspace is a where I do my work on git worktrees (based off src)

Thanks! So much for my reading skills/attention span 😂

Which Debian version is it based on?

Something that I’ll definitely keep an eye on. Thanks for sharing!

RE Go: Others have already mentioned the right way, thought I’d personally prefer ~/opt/go over what was suggested.

RE Perl: To instruct Perl to install to another directory, for example to ~/opt/perl5, put the following lines somewhere in your bash init files.

export PERL5LIB="$HOME/opt/perl5/lib/perl5${PERL5LIB:+:${PERL5LIB}}"
export PERL_LOCAL_LIB_ROOT="$HOME/opt/perl5${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"
export PERL_MB_OPT="--install_base \"$HOME/opt/perl5\""
export PERL_MM_OPT="INSTALL_BASE=$HOME/opt/perl5"
export PATH="$HOME/opt/perl5/bin${PATH:+:${PATH}}"

Though you need to re-install the Perl packages you had previously installed.

This is fantastic! 👏

I use Perl one-liners for record and text processing a lot and this will be definitely something I will keep coming back to - I’ve already learned a trick from “Context Matching” (9) 🙂

I couldn’t agree more 😂

Except that, what the author uses is pretty much standard in the Go ecosystem, which is, yes, a shame.

To my knowledge, the only framework which does it quite seamlessly is Spring Boot which, w/ sane and well thought out defaults, gets the tracing done w/o the programmer writing a single line of code to do tracing-related tasks.

That said, even Spring’s solution is pretty heavy-weight compared to what comes OOTB w/ BEAM.

I got to admit that your point about the presentation skills of the author are all correct! Perhaps the reason that I was able to relate to the material and ignore those flaws was that it’s a topic that I’ve been actively struggling w/ in the past few years 😅

That said, I’m still happy that this wasn’t a YouTube video or we’d be having this conversation in the comments section (if ever!) 😂

To your point and @krnpnk@feddit.de’s RE embedded systems:

That’s absolutely true that such a mindset is probably not going to work in an embedded environment. The author, w/o explicitly mentioning it anywhere, is explicitly talking about distributed systems where you’ve got plenty of resources, stable network connectivity and a log/trace ingestion solution (like Sumo or Datadog) alongside your setup.

That’s indeed an expensive setup, esp for embedded software.

The narrow scope and the stylistic problem aside, I believe the author’s view is correct, if a bit radical.
One of major pain points of troubleshooting distributed systems is sifting through the logs produced by different services and teams w/ different takes of what are the important bits of information in a log message.

It get extremely hairy when you’ve got a non-linear lifeline for a request (ie branches of execution.) And even worse when you need to keep your logs free of any type of information which could potentially identify a customer.

The article and the conversation here got me thinking that may be a combo of tracing and structured logging can help simplify investigations.

Thanks for sharing your insights.

Thinking out loud here…

In my experience with traditional logging and distributed systems, timestamps and request IDs do store the information required to partially reconstruct a timeline:

• In the case of a linear (single branch) timeline you can always “query” by a request ID and order by the timestamps and that’s pretty much what tracing will do too.
• Things, however, get complicated when you’ve a timeline w/ multiple branches.
  For example, consider the following relatively simple diagram.
  Reconstructing the causality and join/fork relations between the executions nodes is almost impossible using traditional logs whereas a tracing solution will turn this into a nice visual w/ all the spans and sub-spans.

That said, logs do shine when things go wrong; when you start your investigation by using a stacktrace in the logs as a clue. That (stacktrace) is something that I’m not sure a tracing solution will be able to provide.

they should complement each other

Yes! You nailed it 💯

Logs are indispensable for troubleshooting (and potentially nothing else) while tracers are great for, well, tracing the data/request throughout the system and analyse the mutations.

I’m not sure how this got cross-posted! I most certainly didn’t do it 🤷‍♂️

That was my case until I discovered that GNU tar has got a pretty decent online manual - it’s way better written than the manpage. I rarely forget the options nowadays even though I dont’ use tar that frequently.

I think I understand where RMS was coming from RE “recursive variables”. As I wrote in my blog:

Recursive variables are quite powerful as they introduce a pinch of imperative programming into the otherwise totally declarative nature of a Makefile.

They extend the capabilities of Make quite substantially. But like any other powerful tool, one needs to use them sparsely and responsibly or end up w/ a complex and hard to debug Makefile.

In my experience, most of the times I can avoid using recursive variables and instead lay out the rules and prerequisites in a way that does the same. However, occasionally, I’d have to resort to them and I’m thankful that RMS didn’t win and they exist in GNU Make today 😅 IMO purist solutions have a tendency to turn out impractical.

Uh, I’m not sure I understand what you mean.

TBH I use whatever build tool is the better fit for the job, be it Gradle, SBT or Rebar.

But for some (presumably subjective) reason, I like GNU Make quite a lot. And whenever I get the chance I use it - esp since it’s somehow ubiquitous nowadays w/ all the Linux containers/VMs everywhere and Homebrew on Mac machines.

I just quote my comment on a similar post earlier 😅

A bit too long for my brain but nonetheless it is written in plain English, conveys the message very clearly and is definitely a very good read on the topic. Thanks for sharing.

That single line of Lisp is probably (defmacro generate-compiler (...) ...) which GCC folks call every time they decide to implement a new compiler 😆

Now that I know which endpoints I’m interested in and which arguments I need to pass, exporting them to Prometheus is my next step. Though I wasn’t sure where to begin w/ - I was thinking about writing the HTTP requests in Java or Python and export the results from there.

Blackbox exporter is definitely easier and cleaner. Thanks for the tip 💯