Situation

The initial ML platform (v1) at SambaNova was unstable and causing customer dissatisfaction, particularly around model deployment and releases. I joined the Modelbox project as a representative from the ML infrastructure team to evaluate the existing pipeline, identify the root issues, and recommend improvements. This quickly evolved into a major effort to design an entirely new Kubernetes- and gRPC-based ML platform.

Task

Assess and overhaul the ML platform's deployment pipeline, design the new Kubernetes-based infrastructure, and lead a team to deliver a reliable, scalable model serving system.

Actions

• Collaborated with the product team and VP of Engineering to establish a clear platform vision, reviewing design documents and aligning on infrastructure requirements.
• Rapidly learned Docker, gRPC, Golang, and CI/CD practices with Jenkins — initially implementing a basic FFN/MNIST model to validate the gRPC and Docker-based architecture end-to-end.
• Scaled the infrastructure to handle production-grade models, successfully adapting it to GPT-3 and standardizing the pipeline to support a wide range of model types.
• Led a team of 5 engineers, managing priorities with Jira and coordinating with cross-functional teams to hit project milestones.
• Established CI/CD processes for automated testing, integration, and deployment, ensuring the system was reliable and straightforward to operate.

Result

The revamped Modelbox platform enabled the deployment of over 10 models for Accenture, significantly improving client satisfaction. We also successfully onboarded the first generative AI model onto the new platform — validating the infrastructure's scalability and setting a strong foundation for future growth.