TFServing

About TFServing

TensorFlow Serving is an open-source serving system for machine learning models, intended for production environments. It focuses on inference: it takes models after training, manages their lifetimes, and gives clients versioned access through a high-performance, reference-counted lookup table. It integrates out of the box with TensorFlow models and can be extended to serve other model types and data.

It can serve multiple models or multiple versions of the same model at once, exposes gRPC and HTTP inference endpoints, and lets new model versions be deployed without changing client code, including canarying and A/B testing. Installation is handled through Docker images (the recommended route), Kubernetes, or a build from source; installing without Docker is documented but not recommended. The repository also describes a scheduler that batches inference requests for joint execution on GPU with configurable latency controls, and a modular architecture whose parts can be used individually or extended to new use cases.

Key features

• Serves multiple models or multiple versions of one model simultaneously
• Exposes both gRPC and HTTP inference endpoints
• Deploy new model versions without changing client code
• Supports canarying new versions and A/B testing experimental models
• Scheduler batches inference requests for joint GPU execution with latency controls
• Supports servables including embeddings, vocabularies, feature transformations and non-TensorFlow models

TFServing at a glance