Have you ever wondered, “Have I used or derived a metric to measure the impact of Generative AI or Agentic AI in your personal life or in my work life?”
I guess no, we are bound by getting an outcome from AI rather than tracking the outcome or limitations. Ever wondered why?
While you ponder, a metric is a standard measurement. In the IT world we have used various metrics to define our capability, compliance, performance. Some are direct, some are derived.
Let’s take Automation as an example. We often use the below metrics.
Number of Automations – is a direct metric, but it doesn’t explain the capability or performance.
Number of Hours saved – is a derived metric, based on the potential hours saved.
These metrics give meaning in terms of delivery innovation and potential savings in terms of labor effort.
At what point do these discrete numbers lose value?
The Number of Hours saved, this value to be honest is meaningless and it is often inflated.
When the numbers start looking discrete, we often will be tracking trends on automation numbers and potential automation opportunities. While the exercise is itself forward looking,
there is a hesitation: the trends might add burden to the delivery and even expose the delivery maturity.
Can we have a metric better than Hours saved for Automation?
Yes, perhaps adding a new metric, Number of times an automation is used, by deriving an actual usage of an automation. This helps to understand the usability of Automation and the impact it has on our delivery.
To a curious mind the refinement of metrics is not enough but also the story it says needs to be coherent.
Let’s say we publish this metric, within a few months or even days, another question pops, Why is the use of a particular automation high? At this point will the number of hours saved be coherent with the actual FTE we operate with?
It ignites a further chain of thoughts. Is it a design issue? A technical debt that we are managing? Is it the right way to manage it?
You see here the metric is a journey and a good metric always has a coherent story.
Compliance metrics, that are hard metrics to be met, can remain discrete as it by itself conveys a story.
Now let’s think of metrics in Generative AI and Agentic AI as we have explored how the story changes in the above Automation example.
AI, we use it every day for different purposes. It uses us, yes, AI cuts through our life in many layers more than we think.
Perhaps the first metric you can or should think about is how AI intersects us.
While you are still evaluating how AI spins around you, let’s take an inward step on how we interact with AI. Understand how AI is transforming the world daily.
Prompts!
Let’s say we are generating an image, how many prompts it takes us to get the outcome we intend?
This is a solid metric to start. It speaks of your prompting techniques and also the effectiveness of the AI.
The higher number of prompts might also indicate the complexity of the prompt or requirement or even a learning curve.
To differentiate, we can add another metric, probably categorical one based on complexity(Low, Medium, High). Again fewer prompts for a better output isn’t one directional, it might speak about your prompt or the generative AI, maintain a simple score to keep track.
While Generative AI has enhanced its capabilities, prompting still is a relevant skill. The prompt can give the AI the context, it can make the AI play a role, it can allow the AI focus on the job at hand and narrow or broaden its ability to perform the output.
An anecdote, I remember from last year.
I tried an AI model to generate an image of a character named “Siddha Vendhan” from my novel, no matter what prompt I gave to ensure the output is coherent with my want. The model always generated the image as a Sage. It interpreted “Siddha” as “Sidhar” meaning Sage in Tamil.
Should I call it hallucination or overthinking? Neither to be honest. The meaning of the name might be the closest dot, maybe the KNN algorithm pulled it. Future upgrades evolved to fix this issue.
I often wondered what could have happened in the background. Was this discovered by metrics? Did many users get similar outputs based on meanings of names or N.E.R.?
I won’t have the answer to the behind-the-scenes of generative AI. It is a blackbox!
But when Agentic AI is being adopted, it is no longer a blackbox. It is a system, a solution designed to enhance and transform our delivery.t cannot be a blackbox — it gives scope for many explorations for metrics and stories it might convey.
While we prepare for the agentic wave, are we still going to limit ourselves with the regular metrics, ‘x’ number of people trained in agentic AI? Remember this is a discrete number.
What is Agentic AI?
It is an artificial intelligence system that can accomplish a specific goal with limited supervision.
Talking about metrics, the definition itself has a metric – Limited Supervision!
Has that been defined? Is it transparent? Is that a current goal or will it be the end goal? I can’t answer.
Identifying case studies for Agentic AI is critical. Agentic AI differs from Automation and also differs from Self Healing. While you ponder over the difference, spare a thought on the two concepts of Chain of Thoughts & Decision Tree.
How do you see these as concepts and finding a mechanism to implement should start your journey. While on the journey you can add Why, How, When, Where and widen your curiosity.
Ultimately, metrics should not be a scoreboard for reporting, but a compass for understanding. As we step into this era of Agentic AI, we must pivot: move beyond tracking mere output and start measuring the nuance—the synergy, the intent, and the actual impact. Don’t wait for a mandate to define your success or your metrics. Start with your own curiosity today. Ask the ‘why’ behind your interactions, track the ‘how’ of your workflows, and you will find that the most profound metric isn’t a digit on a dashboard, but the insight it reveals about your own evolution.
