Azure and Vertex
Bedrock has its own note; this is the other two. Ads name all three, and the useful knowledge is not the feature list — it is what each platform gives you that a raw API key does not, and where each one’s RAG story sits.
What a cloud AI platform is actually selling
| You get | Instead of |
|---|---|
| Identity | an API key in a secret store |
| Data boundary | trusting a vendor’s terms |
| Private networking | egress to the public internet |
| One bill | a separate vendor invoice |
| Managed RAG | building ingestion yourself |
The first three are why regulated buyers use them. In a bank, “the model runs inside our tenancy and the traffic never leaves the VNet” ends a conversation that an OpenAI API key starts.
Azure
Azure OpenAI is the same models with Azure’s control plane: Entra ID identity, private endpoints, regional data residency, and content filtering you configure rather than accept.
The operational fact that bites: capacity is a deployment you provision per model per region, with its own quota. Running out is a deployment-level problem, not an account-level one, and it is the difference between the Azure and OpenAI mental models.
It shows up in the first line of code you write. There is no API key, and
model= is not a model:
from azure.identity import DefaultAzureCredential
from openai import AzureOpenAI
client = AzureOpenAI(
azure_endpoint=ENDPOINT,
api_version=API_VERSION,
azure_ad_token_provider=token_provider,
)
# The *deployment* name you chose, not "gpt-4o".
client.chat.completions.create(
model="prod-chat-westeurope", messages=msgs
)Gotcha: a 429 here means that deployment’s quota, not your account’s. Scaling out is a Terraform change to provisioned throughput, not a support ticket — and the same deployment name in another region is another quota.
| Piece | Role |
|---|---|
| Azure OpenAI | the models, in your tenancy |
| Azure AI Search | retrieval: vector, keyword, hybrid |
| Azure AI Foundry | the build/deploy/evaluate surface |
Azure AI Search is the one to know, because it is a genuinely strong retrieval product independent of the AI branding: vector plus BM25 plus semantic reranking in one service, with filters and faceting. Where teams get value fast is “bring your own data” — point it at a storage account, get chunking, embedding and an index without writing an ingestion pipeline.
The cost is the usual managed-service trade: less control over chunking and retrieval strategy than Chunking Strategies and Retrieval Techniques would have you tune.
Foundry is the umbrella for building, evaluating and deploying agents on Azure — the place model catalogue, prompt flow, evaluation and deployment live. Name it as the surface rather than a distinct technology.
Vertex AI
Google’s equivalent, with the same shape and different names.
| Piece | Role |
|---|---|
| Vertex AI | models, tuning, endpoints |
| Vertex AI Search | managed retrieval over your data |
| Agent Builder / ADK | agent construction and hosting |
Vertex’s distinguishing pitch is that it is a full ML platform rather than an LLM service bolted onto a cloud: training, pipelines, feature store, registry and endpoints predate the generative work. If the team does classical ML as well as LLM work, that matters — see Experiment tracking and the model registry.
Grounding with Google Search is the other thing to name: a managed way to ground answers in web results, which no other platform offers as a first-party feature.
from google import genai
from google.genai import types
client = genai.Client(
vertexai=True, project=PROJECT, location=REGION,
)
resp = client.models.generate_content(
model=MODEL,
contents="What changed in the spec this month?",
config=types.GenerateContentConfig(
tools=[types.Tool(google_search=types.GoogleSearch())],
),
)Note what that replaces: a search API key, a fetcher, an extractor and a citation store. The trade is that the retrieval strategy is theirs, which is the same bargain as managed RAG below.
Choosing, honestly
You usually do not. The cloud is chosen before you arrive, by the existing estate, the enterprise agreement and where the data already sits. The senior answer says that, then names what would change it:
- Data residency or sovereignty requirements that only one region satisfies.
- A specific model only one platform hosts.
- Existing ML platform investment — Vertex or SageMaker — pulling the generative work alongside it.
Gotcha: the platforms are not drop-in equivalents even where the model is identical. Deployment names, quota semantics, content filtering and the error surface all differ, so “we can swap clouds” is only true if you kept a provider abstraction — see Model-provider abstraction and LLM resilience.
Managed RAG versus building it
Every platform now sells a managed RAG product: Bedrock Knowledge Bases, Azure AI Search’s integrated vectorisation, Vertex AI Search. The trade is always the same.
Managed wins on time-to-first-answer and on the parts nobody enjoys — ingestion, scheduling, connectors. Building wins when retrieval quality is the product, because chunking strategy, hybrid weighting, reranking and metadata design are exactly the levers a managed service abstracts away.
The defensible position: start managed to prove the use case, own the pipeline once retrieval quality becomes the constraint. And know that moving means re-embedding, so decide before the corpus is large.
Related
Interview angle 6
- “Why use Azure OpenAI rather than OpenAI directly?” - identity, data boundary and private networking. Entra ID, private endpoints and regional residency are what let a bank say the traffic never leaves its tenancy, and that is the whole reason the product exists.
- “What’s different operationally on Azure?” - capacity is a deployment you provision per model per region, each with its own quota. Running out is a deployment problem rather than an account one, which is a different mental model from a plain API key.
- “What is Azure AI Search?” - a retrieval service that happens to be good independent of the AI branding: vector, BM25 and semantic reranking in one place, with filters. Its integrated vectorisation gives you chunking, embedding and indexing without an ingestion pipeline.
- “What’s Vertex’s pitch?” - it is a full ML platform rather than an LLM service on a cloud — training, pipelines, feature store and registry predate the generative parts. That matters when the team does classical ML alongside LLM work.
- “Managed RAG or build it?” - start managed to prove the use case, own the pipeline when retrieval quality becomes the constraint, because chunking, hybrid weighting and reranking are exactly what a managed service abstracts. Decide before the corpus is large, since moving means re-embedding.
- “How would you choose a platform?” - usually you don’t; the estate and the data location decide. What would change it: residency rules, a model only one platform hosts, or existing investment in one cloud’s ML platform.