Every enterprise is building something on Azure OpenAI right now, and most proofs of concept share the same gap: they talk to a public endpoint with an API key pasted into app settings, and the retrieval half of the RAG pipeline was assembled in an afternoon and never revisited. Moving to production means treating the AI stack with the same network, identity, and cost discipline as everything else this month.

Network and Identity First

Azure OpenAI is a Cognitive Services account, and it hardens the same way: public access off, private endpoint in, local auth (API keys) disabled, callers authenticate with managed identities holding the Cognitive Services OpenAI User role. Your prompts and completions then never traverse the public internet, and there is no key to leak into a git history.

resource "azurerm_cognitive_account" "openai" {
  name                          = "oai-apps-prod-swc"
  location                      = "swedencentral"
  resource_group_name           = azurerm_resource_group.ai.name
  kind                          = "OpenAI"
  sku_name                      = "S0"
  custom_subdomain_name         = "oai-apps-prod-swc"
  public_network_access_enabled = false
  local_auth_enabled            = false
}

resource "azurerm_cognitive_deployment" "gpt" {
  name                 = "gpt-chat"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = "gpt-4o"
    version = "2024-11-20"
  }

  sku {
    name     = "Standard"
    capacity = 100
  }
}

resource "azurerm_private_endpoint" "openai" {
  name                = "pep-oai-apps-prod"
  location            = "swedencentral"
  resource_group_name = azurerm_resource_group.ai.name
  subnet_id           = azurerm_subnet.endpoints.id

  private_service_connection {
    name                           = "psc-openai"
    private_connection_resource_id = azurerm_cognitive_account.openai.id
    subresource_names              = ["account"]
    is_manual_connection           = false
  }
}

The custom subdomain is required for both Entra auth and private endpoints, set it from day one. Model deployments pin an explicit version; auto upgrade convenience is a behavior change in production waiting to happen, so upgrade versions the way you upgrade any dependency, deliberately and tested.

RAG Architecture That Retrieves Well

The standard grounding pattern pairs the model with Azure AI Search. The pipeline: documents land in ADLS Gen2, an indexer or ingestion job chunks them (300 to 500 tokens with overlap, respecting document structure rather than blind splitting), an embedding deployment vectorizes each chunk, and AI Search stores text, vector, and metadata side by side. At query time, use hybrid retrieval, vector similarity plus keyword, fused with semantic ranking on top, which consistently beats either alone because pure vector search misses exact identifiers and pure keyword misses paraphrase. Two details decide answer quality more than model choice: chunking that respects structure, and metadata filters (department, date, security scope) applied at retrieval time. That last one is also your security model: filter search results by the caller’s group membership, because RAG without document level security trimming is a data leak with a chat interface.

Capacity and Cost

Pay as you go Standard is metered per token with per minute rate limits, right for development and spiky low volume. Provisioned throughput (PTU) buys dedicated capacity with predictable latency, right for sustained production traffic, and the crossover point arrives faster than teams expect once a copilot feature ships to a whole workforce. The practical setup: front your deployments with API Management as a gateway, which gives you per consumer quotas, token usage logging per team for chargeback, retry and circuit breaking, and the ability to spillover from a PTU deployment to a pay as you go deployment when you burst past reserved capacity. Log token counts per request in Application Insights from day one, because the finance conversation always comes, and “we do not know which team used it” is a bad opening line.

Cheers
Osama