Generative AI evaluation strategyFig: Enterprise LLM use-cases evaluation strategy (Image by Author)We are at a critical juncture in the Generative AI adoption journey, where we have started hearing conflicting views regarding the transformative potential of Gen AI.Large language model (LLM) providers, e.g., Open AI, Mistral, Google, Meta, etc. are rolling out one LLM after another — with every iteration smaller and more efficient than the previous one. But these are generic pre-trained LLMs without a clear business use-case in mind, or let’s say the business specific use-cases still need to be developed on top of these foundational LLMs. So these LLMs are only an enabler and not a measure of business impact by any means. We do of course have the hyperscalers and technology vendors touting the hundreds (or thousands) of LLM-based use-cases that they have already implemented with quantified business value.On the other hand, we are seeing enterprises / experts start to take a more “pessimistic” view on Gen AI. For example, the recent report by Goldman Sachs is a case in point. The title Gen AI: Too much Spend, Too little Benefit? is self-explanatory and I won’t go into detail—suffice it to say that while nobody is dismissing the future potential of Gen AI, they are not seeing Gen AI (as of now) solve any complex…
Enterprise Use Case-Based Evaluation of LLMs | by Debmalya Biswas | Jul, 2024
