Service-to-service communication - System Design Guide

Overview

Services communicate either synchronously (the caller waits for a response: REST, gRPC, GraphQL) or asynchronously (the caller sends a message and continues: queues, pub/sub, event streams). Synchronous calls are simple and give immediate answers but couple availability and latency; asynchronous messaging decouples services but introduces eventual consistency and more moving parts.

Production systems use both: synchronous calls for queries that need an answer now (get price, check stock), and asynchronous events for side effects and workflows (send email, update search). Cross-cutting concerns such as service discovery, load balancing, timeouts, retries, authentication (mTLS), and tracing are often handled by client libraries or a service mesh.

Phone calls vs text messages

A phone call (synchronous) gets an immediate answer but both people must be available at the same time. A text message (asynchronous) can be read later, so the sender can continue with their day, but the reply is not instant.

When to use it

Designing how microservices call each other.
Choosing REST vs gRPC vs messaging.
Securing internal traffic.
Reducing coupling and latency in call chains.

Where it shows up in interviews

Sync vs async choice

Recognize it when: does the caller need the result immediately?

Design checkout across services
Design a notification pipeline

Internal API design

Recognize it when: high-volume internal calls.

Design a microservices platform with gRPC
Design a service mesh rollout

Where it is used in real software

gRPC internally, REST externally

Many companies (Google, Netflix, Uber) use gRPC between services and REST or GraphQL at the edge.

Service meshes

Istio and Linkerd add mTLS, retries, timeouts, and telemetry to every call via sidecar or ambient proxies.

Kafka for async

Event streams carry domain events between services at LinkedIn, Uber, and many others.

Key terms

Synchronous: Request-response; caller blocks for the result.
Asynchronous: Message-based; caller does not wait.
mTLS: Mutual TLS: both sides authenticate with certificates.
Service mesh: Infrastructure layer handling service communication.
Temporal coupling: Both services must be up at the same time.

How it works, step by step

1
Decide sync or async per interaction
Need an answer now? Sync. Side effect? Async.
2
Pick the protocol
gRPC for internal high-throughput, REST for simplicity, events for fan-out.
3
Discover and balance
DNS/Services, client-side or mesh load balancing.
4
Add resilience
Timeouts, retries, circuit breakers, idempotency.
5
Secure and observe
mTLS, service identities, propagated trace context.

Communication styles

Choose per interaction

Style	Example	Pros	Cons
REST / HTTP JSON	GET /products/42	Simple, universal	Verbose, weaker contracts
gRPC	PricingService.GetPrice	Fast, typed, streaming	Needs tooling, less browser-friendly
Queue	ResizeImage job	Load leveling, retries	No immediate result
Pub/sub events	OrderPlaced	Decoupled fan-out	Eventual consistency

NOWStyle: REST / HTTP JSON | Example: GET /products/42 | Pros: Simple, universal | Cons: Verbose, weaker contracts

Minimize synchronous call depth: each hop adds latency and failure probability.

Implementation

1syntax = "proto3";2package pricing.v1;3 4service PricingService {5  rpc GetPrice(GetPriceRequest) returns (GetPriceResponse);6  rpc StreamPriceChanges(StreamRequest) returns (stream PriceChanged);7}8 9message GetPriceRequest { string sku = 1; string currency = 2; }10message GetPriceResponse { string sku = 1; int64 amount_cents = 2; string currency = 3; }11message StreamRequest { repeated string skus = 1; }12message PriceChanged { string sku = 1; int64 amount_cents = 2; }

Complexity and performance

Sync chain latencySum of hops

Parallelize where possible.

Sync chain availabilityProduct of hops

Deep chains are fragile.

Trade-offs

Sync simplicity vs coupling

Synchronous calls are easy to reason about but couple uptime and latency across services.

Async resilience vs complexity

Messaging decouples but needs brokers, idempotency, and handling eventual consistency.

Variants and related techniques

Backend for frontend (BFF)

An aggregation layer per client type reduces client-to-service calls.

Request-reply over messaging

Async request with a reply queue and correlation ID.

Common mistakes

Deep synchronous chains (A to B to C to D).
Fix: Flatten with async events, caching, or data replication.
Unauthenticated internal traffic.
Fix: Use mTLS and service identities (zero trust).
No deadlines propagated.
Fix: Pass remaining time budget downstream to avoid wasted work.

Interview questions

When do you use synchronous vs asynchronous communication?

Synchronous when the caller needs the result to continue (price, auth, stock check). Asynchronous for side effects, long-running work, and fan-out, where decoupling and resilience matter more than immediate results.

Why use gRPC between services?

Binary Protocol Buffers and HTTP/2 give lower latency and bandwidth, strongly typed contracts with code generation, streaming, and deadlines, which suit high-volume internal calls.

Practice problems

Problem	Difficulty	What it trains
Classify interactions in checkout as sync or async	Easy	Trade-offs.
Design secure internal communication with mTLS	Medium	Service identity.