Modern AIOps:
What It Takes to Build Reliable AI Products

Watch this episode as a video on Spotify!

In this episode of the MongoDB Podcast, host Jesse Hall sits down with Karthik Kalyanamaran, Co-Founder and CTO of Langtrace AI, to discuss how engineering teams are building reliable AI products. Moving from traditional, deterministic software engineering to the non-deterministic world of Large Language Models requires an entirely new approach to debugging, testing, and monitoring.

Karthik shares his journey from scaling observability infrastructure at Coinbase to creating Langtrace AI, an open-source LLM application observability platform built on OpenTelemetry standards. We dive deep into what a modern AIOps stack looks like and how developers can eliminate the guesswork of LLM hallucinations, prompt adjustments, and vector database performance.

Key topics discussed in this episode:

• The Shift to Non-Deterministic Software: Why traditional unit tests fail when building with LLMs, and how to adapt your development and production lifecycle.

• The Core Elements of AIOps: A breakdown of modern AI deployment, including runtime tracing, prompt engineering, and context optimization.

• Optimizing Vector Databases: How Langtrace integrates with MongoDB Atlas Vector Search to track aggregate pipelines, embedding queries, and semantic retrieval accuracy.

• Anonymization and Security: Navigating SOC 2 Type 2 compliance and tracing system performance without exposing sensitive customer data.

• The HTML Era of AI: Why starting with primitive, native constructs directly on top of models often yields better design insights than over-relying on complex frameworks.

• Introducing Hey Zest: A sneak peek into Langtrace closed beta agent platform that allows developers to deploy B2B AI bots natively inside Slack.

Timestamps:00:06 Welcome to MongoDB Podcast Live with host Jesse Hall and Karthik Kalyanamaran00:55 Karthik background: From building infrastructure at Coinbase to launching Langtrace AI02:05 What is Langtrace? Solving the non-deterministic nature of LLMs04:17 The Origin Story: Realizing AI needs robust observability while building a crypto chatbot07:38 Transitioning from reactive traditional web2 monitoring to proactive AI Ops10:54 Defining the modern AI Engineer and the art of Context Engineering12:20 Security at scale: Navigating SOC 2 Type 2 compliance across data vendors like MongoDB14:57 Live Demo: Setting up OpenTelemetry tracing on top of a MongoDB Atlas Vector Search script16:44 Tracking latency, token count metrics, and indexing properties at runtime18:22 Implementing automated evaluations using LLM as a Judge22:05 Future Outlook: Mitigating long context window degradation and advanced tool calling23:05 Developer Advice: Why you should build closer to the bare metal model constructs24:52 Closing remarks, GitHub open source contributions