Moat
The way I see things, the moat for software products comes from a combination of infrastructure complexity, aggregated taste across every product decision micro to macro, and agility both in the craft of actual creation and in the business handling.
I believe the complexity is a natural order of the world. So if the objective is based on primitive fundamentals and expands holistically, then coming full circle to solve all parts of the problem that come from its root becomes a primal need, and the system will naturally grow in a coherent direction.
Precisely because of that, I believe the idea that proper software will be easily replicated ain't correct.
A model's context window is bounded and frozen in time. A human carries a practically unbounded context window of every empirical experience ever lived, and it never stops growing.
As powerful as we could get language models to be, their state of "being" isn't going to be that of native consciousness. In the sense that you don't have the weights of a model existing as a continuous stream of consciousness without external input. And if external input is required to direct the "intelligence", then whoever provides that input needs to know what they're doing.
It wouldn't be the same to put a general with access to an advanced intelligence war model against a cadet with access to the same gadget. The model amplifies the intent, it doesn't originate it.
A concrete example of this in the software world would be two individuals wanting to build a messaging platform. One will ask themselves: well, I'm going to have multiple users in multiple regions, they need to communicate fast, we need to have group chats, and what would happen if messages land in the same millisecond for a group chat with members from multiple different regions?
An expert one could realistically task the model with something like:
I want you to build me a global low latency, guaranteed message ordering, at-least-once delivery, messaging system
While the naive one could ask:
Build me a fast global messaging system that scales.
But still, either of the two would be quite naive to ask. Because if you ask the genie the complete goal in advance, how do you know the limits of its magic? In the case of agentic use, are you going to just wait an indefinite amount of time, and an indefinite use of compute resources, to know if you had the right result or not? Will you chip in on every micro-decision that will be essential to the macro result in the long run of the task?
So we can see that either approach, for the expert or the naive wisher, converges on the same fact: either way the wish must be very carefully decomposed. Because otherwise, and as an easy reminder from cartoons, you can arrive at the literal genie problem. You can ask to become rich, and that won't stop the genie from making you become a rich bottle of mineral water.
The genie is still powerful, but how do you get the closest to what you truly meant? The only plausible way is knowing how to articulate what you demand. So the expert in this case has the advantage, because of the understanding that wishes won't be perfect. Instead of demanding the whole thing, a.k.a. vibe coding, you can ask the genie for precise advancements in smaller denominations and become a part of the loop, not just an observer.
Another concrete case can be a genie masquerading correctness, showing the form but not the substance. E.g an agent creating db tables with low-cardinality indexes. It will work at the beginning, but as you scale it will shock you. Now this is a very trivial thing, if you know how indexes work this ain't going to happen to you.
But are you expecting a language model to possess empirical knowledge to understand this? And what would empirical even mean if the model is not encoding world-states in the first place? When you have a system of symbols deriving meaning only from themselves, you have a recursive loop with no base case. It is meaning deferred infinitely.
"Fire" means something to you because it's wired to the memory of the sensation, but besides the actual subjective memory state, it also has an objective chemical reaction. In the case of an LLM, where's that encoding? So who's to blame when the genie gives you something that looks correct but ain't it? And in the case of software, what's even the equivalent of empirical states when software tokens are the symbols themselves?