OCI GPU instances give you access to NVIDIA A10 and A100 GPUs for AI model training, large-scale inference, and GPU-accelerated data science. GPU instances on OCI use the same VCN, IAM, and storage primitives as regular compute, so you can attach Block Volumes for dataset storage, use Object Storage for model checkpoints, and apply the same security controls. This post covers provisioning GPU instances with Terraform, GPU networking for multi-node training, and cost optimization with preemptible instances.

Step 1: GPU Instance Provisioning

resource "oci_core_instance" "a10_inference_server" {
  compartment_id      = var.compartment_id
  availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
  display_name        = "inference-server-a10"
  shape               = "VM.GPU.A10.1"  # 1x NVIDIA A10 24GB GPU

  source_details {
    source_type             = "image"
    source_id               = var.gpu_oracle_linux_image_id
    boot_volume_size_in_gbs = 200
  }

  create_vnic_details {
    subnet_id             = var.gpu_subnet_id
    nsg_ids               = [var.gpu_inference_nsg_id]
    assign_public_ip      = false
    hostname_label        = "inference-server-a10"
  }

  metadata = {
    ssh_authorized_keys = var.ssh_public_key
    user_data           = base64encode(var.gpu_instance_user_data)
  }

  defined_tags = { "Operations.Environment" = "production", "Operations.Application" = "inference", "Operations.ManagedBy" = "terraform" }
}

# Training cluster: A100 with bare metal for maximum throughput
resource "oci_core_instance" "a100_training_node" {
  count               = var.training_node_count  # e.g., 4 for a 32-GPU cluster
  compartment_id      = var.compartment_id
  availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
  display_name        = "training-node-bm-a100-${count.index}"
  shape               = "BM.GPU.A100-v2.8"  # 8x NVIDIA A100 80GB - bare metal

  source_details {
    source_type             = "image"
    source_id               = var.gpu_oracle_linux_image_id
    boot_volume_size_in_gbs = 300
  }

  create_vnic_details {
    subnet_id        = var.rdma_cluster_subnet_id
    nsg_ids          = [var.gpu_cluster_nsg_id]
    assign_public_ip = false
    hostname_label   = "training-node-${count.index}"
  }

  metadata = {
    ssh_authorized_keys = var.ssh_public_key
  }

  defined_tags = { "Operations.Environment" = "production", "Operations.Application" = "training", "Operations.ManagedBy" = "terraform" }
}

output "inference_server_ip" {
  value = oci_core_instance.a10_inference_server.private_ip
}

output "training_node_ips" {
  value = [for n in oci_core_instance.a100_training_node : n.private_ip]
}

Step 2: NFS Dataset Volume with High Performance Block Storage

resource "oci_core_volume" "training_dataset_volume" {
  compartment_id      = var.compartment_id
  availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
  display_name        = "training-dataset-storage"
  size_in_gbs         = 10000  # 10 TB for large model training datasets
  vpus_per_gb         = 30     # Ultra High Performance for fast data loading
  is_auto_tune_enabled = false

  defined_tags = { "Operations.Application" = "training", "Operations.ManagedBy" = "terraform" }
}

resource "oci_core_volume_attachment" "dataset_attachment" {
  attachment_type            = "iscsi"
  instance_id                = oci_core_instance.a100_training_node[0].id
  volume_id                  = oci_core_volume.training_dataset_volume.id
  use_chap                   = true
  is_multipath               = true
  encryption_in_transit_type = "BM_ENCRYPTION_IN_TRANSIT"
}

Step 3: GPU Utilization Alarm

resource "oci_monitoring_alarm" "gpu_underutilized" {
  compartment_id        = var.compartment_id
  display_name          = "gpu-training-underutilized"
  is_enabled            = true
  metric_compartment_id = var.compartment_id
  namespace             = "oci_compute"
  query                 = "GpuMemoryUsage[5m]{instanceId = 'INSTANCE_OCID'}.mean() < 20"
  severity              = "INFORMATIONAL"
  pending_duration      = "PT30M"
  destinations          = [var.ops_notification_topic_id]
  body                  = "GPU instance showing less than 20% GPU memory utilization for 30 minutes. The training job may have stalled. Check training logs."
}

Operational Notes

For multi-node GPU training, use Cluster Networking to get RDMA-capable connectivity between nodes. Bare metal GPU instances connected to a cluster network communicate at up to 1.6 Tbps aggregate bandwidth with sub-microsecond latency, which is required for efficient distributed training with NCCL collective operations. Without RDMA, multi-node training on standard VCN networking has higher latency that degrades scaling efficiency significantly.

Use preemptible GPU instances for training jobs that can checkpoint and resume. Preemptible instances cost approximately 50 percent less than on-demand and are reclaimed by OCI when capacity is needed, with a 30-second notice. Implement checkpoint logic in your training code to save model state to Object Storage every N steps. When a preemptible instance is reclaimed, restart the training job from the last checkpoint on a new instance.

Regards,
Osama

#OCI #OracleCloud #GPU #NVIDIA #AI #MachineLearning #Terraform #IaC #TechBlog #Oracle #PlatformEngineering #DeepLearning #OracleCloudInfrastructure #A10 #A100 #ModelTraining #Inference #HPC #RDMA #CloudAI