Bridging the AI Infrastructure Gap: Lessons from Industry Leaders

The race to integrate AI agents into enterprise workflows is accelerating, but many organizations are discovering that the real bottlenecks lie not in the AI models themselves but in the legacy infrastructure supporting them. At the recent VB Transform 2026 event, prominent leaders from LinkedIn, Walmart, and Zendesk shared their experiences in overcoming these challenges. Their insights reveal a common theme: to harness the full potential of AI, organizations must adapt their infrastructure to meet the unique demands of AI agents.

As enterprises shift from pilot projects to full-scale deployments of AI agents, the differences between human-driven processes and automated systems become starkly apparent. While AI operates at lightning speed, traditional systems, built around human workflows, often lag behind. This article delves into the specific hurdles faced by these industry giants and outlines actionable steps for organizations to modernize their AI infrastructures.

The Real Bottlenecks: Legacy Infrastructure

The findings from the panel discussion highlighted a crucial insight: the bottlenecks hindering AI deployment are not related to the models themselves but rather to outdated infrastructure. Animesh Singh, Senior Director of AI Platform and Infrastructure at LinkedIn, Desiree Gosby, SVP of Corporate Technology Services at Walmart, and Sami Ghoche, VP of Applied AI at Zendesk, each encountered roadblocks that stemmed from legacy systems designed for human workflows.

Walmart's Citizen Developer Dilemma

Walmart's path to AI integration faced its own set of challenges. The company successfully launched an agent harness that empowered employees, dubbed 'citizen developers,' to create their own AI solutions. While this led to rapid innovation, it also resulted in an overwhelming number of overlapping agents that lacked coordination. Gosby emphasized the importance of governance in such scenarios, advocating for frameworks that identify duplicate efforts and promote the best-performing agents. This approach not only streamlines operations but also ensures that engineering resources are not bogged down by redundancy.

Zendesk's Data Infrastructure Investment

Meanwhile, at Zendesk, the bottleneck was rooted in data management. Ghoche highlighted the immense volume of customer interactions—20 billion conversations—stored in the company's repository. The assumption that simply feeding this data to a large language model (LLM) would yield successful agents proved misguided. Instead, Ghoche stressed the necessity of investing in robust data pipelines and infrastructure to ensure that the data fed into AI models is clean, relevant, and actionable. Without this investment, the potential of AI agents remains largely untapped.

Strategies for AI Infrastructure Modernization

As these leaders navigated their respective challenges, they identified several key strategies that other organizations can adopt to modernize their AI infrastructures and avoid common pitfalls.

• Invest in Evaluation Mechanisms: Establish a robust set of evaluation criteria before implementing AI projects. This allows organizations to break down complex problems and assess progress efficiently.
• Empower Citizen Developers: Provide employees with the tools to create AI solutions while ensuring oversight to prevent redundancy and inefficiency.
• Build for Independence: Design systems that are flexible and context-independent. This ensures that organizations can adapt to new AI models and technologies as they emerge.

Evaluations as a Cornerstone

Ghoche's advice to prioritize evaluation mechanisms resonates across all use cases. By developing a comprehensive set of evaluations, organizations can dissect their AI challenges methodically. This not only accelerates the deployment process but also enhances the quality of outcomes. Having a clear framework for evaluations allows businesses to pivot quickly when issues arise, fostering a culture of agility and continuous improvement.

Fostering Innovation Through Governance

Gosby's experience at Walmart underscores the importance of governance in fostering innovation. By putting AI tools directly into the hands of employees, companies can unleash creativity and problem-solving capabilities. However, this must be balanced with governance structures that prevent chaos. Establishing clear guidelines and oversight mechanisms helps ensure that innovation does not lead to duplication of efforts or wasted resources.

Emphasizing Data Infrastructure

Ghoche's emphasis on investing in data infrastructure cannot be overstated. As organizations collect vast amounts of data, having the right systems in place to manage and analyze that data is critical. This investment enables businesses to harness the full potential of their data, ensuring that AI models are trained on high-quality, relevant information. The ability to leverage historical data effectively can significantly enhance the performance of AI agents.

The Role of Open Source in AI Development

Another common thread among the panelists was the role of open source in AI development. Each leader expressed a preference for owning their models and infrastructure whenever possible. This approach not only provides greater control over AI processes but also helps mitigate risks associated with vendor lock-in.

LinkedIn's AI Gateway

LinkedIn's solution includes an AI gateway, a single interface for all outbound model calls. This allows the company to switch between different AI providers seamlessly, ensuring that they are not tied to any single vendor. This flexibility is crucial in an environment where new AI capabilities emerge rapidly.

Walmart's Vendor-Agnostic Approach

Walmart has implemented its own internal gateway that supports various workload types—ranging from deterministic workflows to more complex planning and reasoning tasks. This vendor-agnostic approach enables the company to choose the most effective model for each specific workload, enhancing operational efficiency and compliance.

Key Takeaways

• AI agents require infrastructure that is designed for their operational speed, not human workflows.
• Investing in evaluation mechanisms is crucial for effective AI integration.
• Governance structures are essential for managing citizen developers and preventing redundancy.
• Robust data infrastructure is necessary for leveraging historical data effectively.
• Embracing open source and building vendor-agnostic systems can enhance flexibility and control.

Frequently Asked Questions

What are the most common infrastructure challenges when scaling AI agents?

The most common challenges include outdated legacy systems that are not optimized for AI workloads, slow provisioning processes, and difficulties in managing and analyzing large volumes of data. These hurdles can significantly impede the effectiveness of AI agents, as they operate at speeds far exceeding those of traditional human-centric workflows.

How can organizations empower citizen developers effectively?

Organizations can empower citizen developers by providing them with user-friendly AI tools, training, and resources. Additionally, implementing governance frameworks that monitor their projects can help ensure that innovation aligns with business objectives and does not lead to chaotic, overlapping efforts.

Why is investing in data infrastructure important for AI success?

Investing in data infrastructure is vital because high-quality, well-managed data is the foundation of effective AI models. Without a strong data pipeline, organizations risk feeding AI systems with outdated or irrelevant information, which can lead to poor performance and outcomes.

How can companies maintain flexibility in their AI infrastructure?

To maintain flexibility, companies should build systems that are context-independent and vendor-agnostic. This approach allows organizations to adapt to new AI technologies as they emerge, ensuring that they can leverage the best available solutions for their specific needs, without being locked into a single vendor.