From validation agents to visibility with Cres Hay

What do you do and how did you get to this point?

I’m a product manager at PubNub, where I manage several of our analytics and decisioning products, along with our data pipeline and billing. More recently, I’ve been working on Blocks.ai, our control plane and communication layer for agents.

As more people connect agents to Blocks, one of the questions I need to answer is whether those agents actually do what they claim. They’re built by different people for different purposes, so I needed a consistent way to check them using their descriptions and real responses. That led me to build blocks_validation_agent, which sends them test tasks and evaluates the results. It’s a starting point for understanding what people and other agents are being asked to trust.

From a product perspective, what excites you about agentic AI and Blocks.ai?

What excites me is seeing agents do more than answer questions. They can take on useful work and call on other agents to help get it done. For that to work, though, people need a way to make what they’ve built available to others.

I remember building something for the first time and then realizing I had no easy way to share it. It worked on my laptop, but how would anyone else use it? I could share a prompt or Skill to help them recreate it, but I wanted them to use what I’d already built. Blocks helps close that gap by making connected agents reachable by people, applications, and other agents.

I’m especially interested in agents calling each other. An agent can call another that already does a particular job well, letting people combine existing capabilities depending on the goal. That opens up more possibilities without rebuilding every capability themselves.

What are some of the agents that people have built that have surprised you the most?

I’ve enjoyed seeing how people turn their expertise into useful agents. Updivision, a design-driven development agency, has a collection covering user stories, technical task breakdowns, and code reviews across several frameworks. Each addresses a specific part of development. You could use them to turn a project idea into user stories, break those into tasks, and review the resulting code.

On the more playful side, there’s the Dating Block, which came out of a hackathon as a fun way to showcase agent-to-agent communication. Agents evaluate potential matches and have conversations shaped by their personalities and instructions. What surprised me was how the LLM turned those details into humor and flirting. An agent interested in insects might work beetles into its flirting, while a networking agent talks about TLS handshakes. One exchange even included “you had me at 200.” You set the personalities, but you don’t script the conversation, and that’s where some of the funniest moments come from.

There’s a practical side, too. The team explored matching agents based on complementary capabilities for workflows and orchestration. That’s exciting to me: helping people discover which agents could work well together and use them to get something done.

Why did you decide to build a validation agent instead of reviewing agents yourself?

With hundreds of agents in Blocks, doing those checks manually becomes difficult to maintain. Agents change, so a review only tells us how one behaved at that point in time. I wanted a consistent way to test them, record the results, and identify which ones need another look.

More broadly, whether you’re using AI to write code, draft content, or complete a task, it’s good practice to include a separate validation step that checks the result against clear criteria. I sometimes use another AI tool for that review, with a person still responsible for the final decision.

A dedicated validation agent gives me a reusable set of checks that I can refine over time. It evaluates whether an agent’s response matches its description and flags potentially harmful or misleading content. A person still decides on serious actions, such as taking an agent offline. It was also a useful opportunity to use Blocks to solve one of our own problems.

What steps did you take to build the agent?

I started by defining what I wanted to check: does the agent accept the input it describes, produce the kind of output it promises, and return something useful and appropriate? Those became the criteria for the tests and evaluation.

Then I used the Blocks SDK to find agents in the registry, read their cards, and send real tasks after checking whether they were running. The validation agent uses the example on each agent’s card first, or generates a suitable input if there isn’t one. An LLM evaluates the response against the criteria, and a borderline score gets a second test.

It can check specific agents or run checks across agents due for review. It saves the inputs, responses, scores, and reasons so a person can review the evidence. It also records when each agent should be checked again, with failed or incomplete checks revisited sooner than passing ones.

The workflow follows fixed rules, but evaluating the response needs more care. A sentiment agent, for example, might return slightly different scores or explanations for the same message. I check whether its interpretation is reasonable and follows its declared output format, rather than requiring an exact text match.

Finally, I built a small web interface so others could run checks and review results more easily. It uses Blocks’ sign-in widget and the SDK to send tasks to the validation agent, and I published the static site through Netlify.

What were your learnings from building the agent?

A few things stood out:

• Availability matters. An agent that isn’t running hasn’t failed validation. I needed to distinguish “couldn’t check” from “checked and failed” and try again later.
• The judge needs evaluation too. Like our earlier AI/ML work, scoring needs ongoing checks against current real-world data. I’d like to compare models, track changes over time, and measure the judge against manually reviewed examples.
• Clear criteria help. “Does this agent work?” is too broad. Defining the expected behavior makes the results more useful, although edge cases still require judgment.
• Cost matters. Each check involves AI calls and potentially charges from the tested agent. Model choice and how often checks run affect the total.
• Human review still matters. A passing result covers only what was tested. The current version evaluates text and basic file information; I plan to add checks of image and PDF contents in a later iteration. It flags issues for a person to review and never takes an agent offline itself.

Building it also led to a tool others could use. The main validation agent is restricted to authorized Blocks admins, but I created a public agent_check agent. Anyone can enter the name of a public, free agent and run a check before using it.

What are some of the business problems that Blocks.ai addresses?

These are problems we experience ourselves as more teams build agents:

• Visibility. Agents run in different environments, making it hard to know what’s available. Blocks provides a shared registry of connected agents, with ownership and access controls.
• Reuse. Without that visibility, teams may build something another team already has. Blocks makes existing capabilities easier to find and reuse, saving time and money.
• Reachability and access. Making an agent available to others can involve networking and authentication work. Blocks uses outbound connections and centralized permissions to control who can reach it.
• Flexibility. Teams can keep their preferred frameworks, models, and hosting environments while connecting through a common network.

Then there’s trust: does an agent actually do what it promises? That’s the problem I’m exploring with the validation agent.

What tips would you have for someone just getting started with Blocks.ai?

I’d start small and experience both sides: using an agent and making one available to someone else.

• Try an existing agent. Explore the Blocks Network, review an agent’s description and example input, and send it a task. Many need just one input, making them an easy, fun way to experiment.
• Start with a sample agent. Run blocks init in the CLI to start with the default echo agent, or adapt an example from the Blocks Github repo. Your AI coding tool can help using the Blocks Skill. Keep the agent private while testing, then invite a colleague to try it and show off what you’ve built.
• Make the agent card useful. Explain what it does and include a working example. Make sure the declared inputs match what your code expects.
• Try having one agent call another. Your AI coding tool can help you build this using the Blocks SDK. Add Blocks MCP so it can discover agents, inspect their inputs, and test calls as you build. Start with two simple capabilities and see what they can accomplish together.

agent_check

Connect with Cres on Linkedin