This just popped up on my Mastodon feed a few minutes ago. programmingsimplicity.substack.com/p/what-unix-...

My long-held cancellable view is that Unix pipelines would be much better if programs could distinguish "output for humans" from "output for machines". Keep current formatting for humans, but have a --json/--xml/--sexp tag that produces and consumes semi-structured data instead of parsing text. ↵

I wrote this in my PL textbook in 2003 (hence XML) but and still feel this was a missed opportunity. Semi-structured w/ tags would also make your code more robust to variations in Unix platforms. I disagree with the article's call for "datatypes"; down that road comes mutual incomprehension. ↵

Perlis talks about this in the Foreword to SICP, but I think we could still do a bit better — especially when the "stream of bytes" view is *still available* (just don't ask for the semi-structured output).

I like the idea of self-describing metadata (in this case as little state machines) more than hacking in a few predefined formats. 9p.io/sources/cont...

That's interesting, thanks. Certainly wouldn't mind if data had their own shebang to match that of control. (Racket's `#lang` is basically a modern variant of shebang that applies just as well to data as to control.) ↵

I'm not sold on regular being the way to do it, both since people don't understand their computational limits (StackOverflow: "why doesn't my regexp parse HTML?!?") and because they're hard to write correctly and to read (we actually have a study on this right now). ↵

Trees have proven to be a really good point in the Chomsky hierarchy: almost all of the nice properties of regular, you get context-sensitive for not much work, you can have a nice bicameral syntax, etc., etc.

I'm always impressed by what awk can do. I wonder what percentage of people use it these days. The very minimal time I've spent with powershell I've always left shaking my head thinking awk could have been much more effective with semi-structured text.

I think awk is mostly only used by the 40+ set. It really is great for line-at-a-time. But so often I have to do things that at least slightly cross a line. Imagine an awk that was designed for json, say.

On a case by case basis the data can be made to work better with awk, often boiling down to a few things (including writing emacs keyboard macros or functions). One is that jq -c ...will put a sequence of json objects each on one line.

But yeah, I don't use awk a lot anymore for json. Before json tools were very good I would do a simplistic conversion to xml and use xpath and then work from there. These days I often load json into postgresql and use that excellent toolset. Even that's probably only for the over 40's!

It really frustrates me that I don't know a command pipeline remotely as well as I do the Unix shell for, say, json. I mean, it's fine, I fire up Racket and code away, but there's got to be a better way. I should probably learn jq and friends and quit my bellyaching.

I'm careful to say "I don't know" as opposed to the proper Internet way of speaking, which is to say "there doesn't exist". (-:

I still use sed and awk on the command line, but with JSON I just end up live coding everything with an editor connected Lisp REPL 🤷🏻‍♂️

This opportunity was not missed, it was picked up by #PowerShell :-) Get-ChildItem -Filter *.svg | Select-Xml -XPath //* | Where-Object { $_.Node.Name -eq 'path' } | Foreach-Object { $_.Node.D } _Everything_ is an object in PowerShell, and the object pipeline is overpowered.

Powershell did try that, but perhaps went too far down the typing route and is spectacularly unusable these days without a PhD in defenestration

Yeah, it's tricky getting the right balance. (I don't know what Powershell did, though—maybe there *is* no sweet spot beyond "stream of characters"!) bsky.app/profile/shri...

There *is* a sweet spot beyond "stream of characters" #PowerShell has an object pipeline - things are as structured as you want them and objects are easy to manipulate. Get the square root of a range of odd numbers: 1..1kb | ? { $_ % 2 } | % { [Math]::Sqrt($_) } Don't knock it till you rock it.

What programming language are those terms written in?

Those are all #PowerShell Annotated: 1..1kb | # Give me a range of numbers between 1 and 1024 ? { $_ % 2 } | # Where-Object the is a modulo remainder % { [Math]::Sqrt($_) } # Foreach-Object, call the static method Sqrt on the Math class

...whereas I can just choose my DSL of choice with plain old pipes based on the problem at hand. I still don't get the lure of Powershell I'm afraid

Clarifying one more thing. Because PowerShell is a cli and a language, you can mix/match PowerShell and, say, Python. So it's not "PowerShell or other language" It's "PowerShell _and_ every other language" Is the value proposition more clear now?

So if I want to write these bits in something other than PowerShell-the-language but want to take advantage of PowerShell-the-CLI what would it look like? Can you rewrite this same script in some other language (say Python or OCaml or Racket or…)?

It's the network effect, the interoperability, and the 18 year head start. Basically the PowerShell ecosystem has spent almost two decades building interoperable blocks, and it's easier to put those pieces together than build them from scratch.

Citation wanted. PowerShell can be either strongly or weakly typed. There may be some cmdlets that are too strongly typed, but the language has always supported both. Most of the scripts I run across are weakly typed. Sorry if most that you run across are overtyped.

True! It would be great if most people building cli apps had a --json and --xml tag (or at least one). It would cross-process communication much easier. #PowerShell pipelines already handle this, if the app supports it i.e. docker images ls --format '{{json .}}' | ConvertFrom-Json

I'll have to email the author ;-). Every N months / years someone comes out of the woodwork and is like "wouldn't it be great if we had an object pipeline" And everyone who knows #PowerShell is like: Yeah, it is great, and it already exists. Just try using PowerShell instead of rebuilding it.