{"id":240,"date":"2024-11-16T18:05:32","date_gmt":"2024-11-16T18:05:32","guid":{"rendered":"http:\/\/who.ioanpopovici.ro\/?p=240"},"modified":"2024-11-18T19:33:36","modified_gmt":"2024-11-18T19:33:36","slug":"the-plateau-powerhouse","status":"publish","type":"post","link":"http:\/\/who.ioanpopovici.ro\/index.php\/2024\/11\/16\/the-plateau-powerhouse\/","title":{"rendered":"Decoupling of scales and the plateau"},"content":{"rendered":"\n

It is a well known fact that LLMs will reach multiple plateaus during their development. Right about now there should be one. Even Bill Gates is talking about it. Even Yan LeCun. (look it up on X). Altough, to be honest, I’m not really counting LeCun as an expert here. He’s more like a Mike Tyson of the field ( What? Too much? Too Early? \ud83d\ude42 )
<\/p>\n\n\n\n

This time, the reason behind these plateaus is not the lack of technology (as it was the case with previous AI waves (50s, 80s – the LeCun Era – 2000s).
This time, the reason is extremely profound and extremely close to humans. It is concerning an interesting aspect of epistemology: the decoupling of scales issue.
What this means is, in a nutshell is, that you cannot use an emergent level as an indicator for an underlying reality without stopping on the road and gathering mode data (experiments and observations) , and redoing your models entirely.<\/p>\n\n\n\n

My argument is that our models, now, at best, they do clever “Unhobbing” (a basic feedbackloop that somewhat emulates a train of tought by iterating answers)<\/p>\n\n\n\n

We have not yet seen, at all a real bake-in of a new “underlying reality model”. <\/p>\n\n\n\n

I don’t care about AGI, and when or if we will truly reach general intelligence! I care about the fact that we are very <\/strong>far away from drawing the real benefits out of these LLMs. We have not yet even battled the first plateau. And there are many coming.<\/p>\n\n\n\n

The Powerhouse<\/h2>\n\n\n\n

Lots of discussions happening right now in relation to what people call nowadays #AI. Some sterile, some insightful, some driven by fear, some driven by enthusiasm, some driven by alcohol \ud83d\ude0a<\/p>\n\n\n\n

I am defining the \u201cpowerhouse\u201d as being that technology that allows us to create the most value. In industry, in sales, in research, in general :). <\/p>\n\n\n\n

In this light, during the\ninformation age, the one we are just barely<\/strong> scratching the surface of, there\nwere multiple powerhouses that we can remember and talk about.<\/p>\n\n\n\n

  1. The internet and the communication age powerhouse for example, that we have not yet exhausted. Not by far, in terms of potential productivity. This is something that we can understand easily. It\u2019s not worth going into details here.
    <\/li>
  2. The data-churn powerhouse. The ability to store, look-through, transform and stream data. I would argue that this is also easily understandable. However, we may stop a bit here and make a few points: