Tag: programming

Cloud cost management has evolved from simple monitoring to sophisticated FinOps practices that combine financial accountability with operational efficiency. Oracle Cloud Infrastructure provides powerful cost management capabilities that, when combined with intelligent automation, enable organizations to optimize spending while maintaining performance and availability. This comprehensive guide explores advanced cost optimization strategies, predictive analytics, and automated governance frameworks for enterprise OCI environments.

FinOps Framework and OCI Cost Architecture

Financial Operations (FinOps) represents a cultural shift where engineering, finance, and operations teams collaborate to maximize cloud value. OCI’s cost management architecture supports this collaboration through comprehensive billing analytics, resource tagging strategies, and automated policy enforcement mechanisms.

The cost management ecosystem integrates multiple data sources including usage metrics, billing information, and performance indicators to provide holistic visibility into cloud spending patterns. Unlike traditional cost tracking approaches, modern FinOps implementations use machine learning algorithms to predict future costs and recommend optimization actions proactively.

OCI’s native cost management tools include detailed billing analytics, budget controls with automated alerts, and resource usage tracking at granular levels. The platform supports advanced tagging strategies that enable cost allocation across business units, projects, and environments while maintaining operational flexibility.

Resource lifecycle management becomes critical for cost optimization, with automated policies that right-size instances, schedule non-production workloads, and implement tiered storage strategies based on access patterns and business requirements.

Intelligent Cost Analytics and Forecasting

Advanced cost analytics goes beyond simple billing reports to provide predictive insights and optimization recommendations. Machine learning models analyze historical usage patterns, seasonal variations, and growth trends to forecast future spending with high accuracy.

Anomaly detection algorithms identify unusual spending patterns that may indicate configuration drift, unauthorized resource creation, or inefficient resource utilization. These systems can detect cost anomalies within hours rather than waiting for monthly billing cycles.

Cost attribution models enable accurate allocation of shared resources across business units while maintaining transparency in cross-functional projects. Advanced algorithms can apportion costs for shared networking, storage, and security services based on actual usage metrics rather than static allocation formulas.

Predictive scaling models combine cost forecasting with performance requirements to recommend optimal resource configurations that minimize costs while meeting service level objectives.

Production Implementation with Automated Optimization

Here’s a comprehensive implementation of intelligent cost management with automated optimization and predictive analytics:

Infrastructure Cost Monitoring and Optimization Framework

#!/usr/bin/env python3
"""
Advanced OCI Cost Management and FinOps Automation Platform
Provides intelligent cost optimization, predictive analytics, and automated
governance for enterprise Oracle Cloud Infrastructure environments.
"""

import oci
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
from typing import Dict, List, Tuple, Optional, Any
from dataclasses import dataclass, field
from enum import Enum
import logging
import asyncio
import json
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email import encoders
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.ensemble import IsolationForest
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)

class CostSeverity(Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    CRITICAL = "critical"

class OptimizationAction(Enum):
    RIGHT_SIZE = "right_size"
    SCHEDULE = "schedule"
    MIGRATE_STORAGE = "migrate_storage"
    TERMINATE = "terminate"
    UPGRADE_COMMITMENT = "upgrade_commitment"

@dataclass
class CostAnomaly:
    """Container for cost anomaly detection results"""
    resource_id: str
    resource_type: str
    resource_name: str
    expected_cost: float
    actual_cost: float
    anomaly_score: float
    severity: CostSeverity
    detected_at: datetime
    description: str
    recommended_action: OptimizationAction
    potential_savings: float = 0.0

@dataclass
class OptimizationRecommendation:
    """Container for cost optimization recommendations"""
    resource_id: str
    resource_type: str
    current_config: Dict[str, Any]
    recommended_config: Dict[str, Any]
    current_monthly_cost: float
    projected_monthly_cost: float
    potential_savings: float
    confidence_score: float
    implementation_effort: str
    risk_level: str
    business_impact: str

@dataclass
class BudgetAlert:
    """Container for budget alert information"""
    budget_name: str
    current_spend: float
    budget_amount: float
    utilization_percentage: float
    forecast_spend: float
    days_remaining: int
    severity: CostSeverity
    recommendations: List[str]

class OCICostOptimizer:
    def __init__(self, config_file: str = 'cost_config.yaml'):
        """Initialize the cost optimization system"""
        self.config = self._load_config(config_file)
        self.signer = oci.auth.signers.get_resource_principals_signer()
        
        # Initialize OCI clients
        self.usage_client = oci.usage_api.UsageapiClient({}, signer=self.signer)
        self.compute_client = oci.core.ComputeClient({}, signer=self.signer)
        self.network_client = oci.core.VirtualNetworkClient({}, signer=self.signer)
        self.storage_client = oci.core.BlockstorageClient({}, signer=self.signer)
        self.monitoring_client = oci.monitoring.MonitoringClient({}, signer=self.signer)
        self.budgets_client = oci.budget.BudgetClient({}, signer=self.signer)
        
        # Cost tracking and ML models
        self.cost_history = pd.DataFrame()
        self.anomaly_detector = IsolationForest(contamination=0.1, random_state=42)
        self.cost_forecaster = LinearRegression()
        self.scaler = StandardScaler()
        
        # Cost optimization thresholds
        self.thresholds = {
            'cost_spike_factor': 2.0,
            'utilization_threshold': 20.0,
            'savings_threshold': 50.0,
            'risk_tolerance': 'medium'
        }

    def _load_config(self, config_file: str) -> Dict:
        """Load configuration from file"""
        import yaml
        try:
            with open(config_file, 'r') as f:
                return yaml.safe_load(f)
        except FileNotFoundError:
            logger.warning(f"Config file {config_file} not found, using defaults")
            return {
                'tenancy_id': 'your-tenancy-id',
                'compartment_id': 'your-compartment-id',
                'time_granularity': 'DAILY',
                'forecast_days': 30,
                'optimization_enabled': True
            }

    async def analyze_cost_trends(self, days_back: int = 90) -> Dict[str, Any]:
        """Analyze cost trends and identify patterns"""
        end_date = datetime.utcnow()
        start_date = end_date - timedelta(days=days_back)
        
        try:
            # Get usage data from OCI
            usage_data = await self._fetch_usage_data(start_date, end_date)
            
            if usage_data.empty:
                logger.warning("No usage data available for analysis")
                return {}
            
            # Perform trend analysis
            trends = {
                'total_cost_trend': self._calculate_cost_trend(usage_data),
                'service_cost_breakdown': self._analyze_service_costs(usage_data),
                'daily_cost_variation': self._analyze_daily_patterns(usage_data),
                'cost_efficiency_metrics': self._calculate_efficiency_metrics(usage_data),
                'anomalies': await self._detect_cost_anomalies(usage_data)
            }
            
            # Generate cost forecast
            trends['cost_forecast'] = await self._forecast_costs(usage_data)
            
            return trends
            
        except Exception as e:
            logger.error(f"Failed to analyze cost trends: {str(e)}")
            return {}

    async def _fetch_usage_data(self, start_date: datetime, end_date: datetime) -> pd.DataFrame:
        """Fetch usage and cost data from OCI"""
        try:
            request_details = oci.usage_api.models.RequestSummarizedUsagesDetails(
                tenant_id=self.config['tenancy_id'],
                time_usage_started=start_date,
                time_usage_ended=end_date,
                granularity=self.config.get('time_granularity', 'DAILY'),
                compartment_depth=6,
                group_by=['compartmentName', 'service', 'resource']
            )
            
            response = self.usage_client.request_summarized_usages(
                request_details=request_details
            )
            
            # Convert to DataFrame
            usage_records = []
            for item in response.data.items:
                usage_records.append({
                    'date': item.time_usage_started,
                    'compartment': item.compartment_name,
                    'service': item.service,
                    'resource': item.resource_name,
                    'computed_amount': float(item.computed_amount) if item.computed_amount else 0.0,
                    'computed_quantity': float(item.computed_quantity) if item.computed_quantity else 0.0,
                    'currency': item.currency,
                    'unit': item.unit,
                    'tags': item.tags if item.tags else {}
                })
            
            df = pd.DataFrame(usage_records)
            if not df.empty:
                df['date'] = pd.to_datetime(df['date'])
                df = df.sort_values('date')
            
            return df
            
        except Exception as e:
            logger.error(f"Failed to fetch usage data: {str(e)}")
            return pd.DataFrame()

    def _calculate_cost_trend(self, usage_data: pd.DataFrame) -> Dict[str, Any]:
        """Calculate overall cost trends"""
        if usage_data.empty:
            return {}
        
        # Group by date and sum costs
        daily_costs = usage_data.groupby('date')['computed_amount'].sum().reset_index()
        
        if len(daily_costs) < 7:
            return {'trend': 'insufficient_data'}
        
        # Calculate trend metrics
        days = np.arange(len(daily_costs))
        costs = daily_costs['computed_amount'].values
        
        # Linear regression for trend
        slope, intercept = np.polyfit(days, costs, 1)
        trend_direction = 'increasing' if slope > 0 else 'decreasing'
        
        # Calculate period-over-period growth
        recent_period = costs[-7:].mean()
        previous_period = costs[-14:-7].mean() if len(costs) >= 14 else costs[:-7].mean()
        
        growth_rate = ((recent_period - previous_period) / previous_period * 100) if previous_period > 0 else 0
        
        # Cost volatility
        volatility = np.std(costs) / np.mean(costs) * 100 if np.mean(costs) > 0 else 0
        
        return {
            'trend': trend_direction,
            'growth_rate_percent': round(growth_rate, 2),
            'volatility_percent': round(volatility, 2),
            'average_daily_cost': round(np.mean(costs), 2),
            'total_period_cost': round(np.sum(costs), 2),
            'trend_slope': slope
        }

    def _analyze_service_costs(self, usage_data: pd.DataFrame) -> Dict[str, Any]:
        """Analyze costs by service type"""
        if usage_data.empty:
            return {}
        
        service_costs = usage_data.groupby('service')['computed_amount'].agg([
            'sum', 'mean', 'count'
        ]).round(2)
        
        service_costs.columns = ['total_cost', 'avg_cost', 'usage_count']
        service_costs['cost_percentage'] = (
            service_costs['total_cost'] / service_costs['total_cost'].sum() * 100
        ).round(2)
        
        # Identify top cost drivers
        top_services = service_costs.nlargest(10, 'total_cost')
        
        # Calculate service growth rates
        service_growth = {}
        for service in usage_data['service'].unique():
            service_data = usage_data[usage_data['service'] == service]
            if len(service_data) >= 14:
                recent_cost = service_data.tail(7)['computed_amount'].sum()
                previous_cost = service_data.iloc[-14:-7]['computed_amount'].sum()
                
                if previous_cost > 0:
                    growth = (recent_cost - previous_cost) / previous_cost * 100
                    service_growth[service] = round(growth, 2)
        
        return {
            'service_breakdown': top_services.to_dict('index'),
            'service_growth_rates': service_growth,
            'total_services': len(service_costs),
            'cost_concentration': service_costs['cost_percentage'].iloc[0]  # Top service percentage
        }

    def _analyze_daily_patterns(self, usage_data: pd.DataFrame) -> Dict[str, Any]:
        """Analyze daily usage patterns"""
        if usage_data.empty:
            return {}
        
        usage_data['day_of_week'] = usage_data['date'].dt.day_name()
        usage_data['hour'] = usage_data['date'].dt.hour
        
        # Daily patterns
        daily_avg = usage_data.groupby('day_of_week')['computed_amount'].mean()
        
        # Identify peak and off-peak periods
        peak_day = daily_avg.idxmax()
        off_peak_day = daily_avg.idxmin()
        
        # Weekend vs weekday analysis
        weekends = ['Saturday', 'Sunday']
        weekend_avg = usage_data[usage_data['day_of_week'].isin(weekends)]['computed_amount'].mean()
        weekday_avg = usage_data[~usage_data['day_of_week'].isin(weekends)]['computed_amount'].mean()
        
        weekend_ratio = weekend_avg / weekday_avg if weekday_avg > 0 else 0
        
        return {
            'daily_averages': daily_avg.to_dict(),
            'peak_day': peak_day,
            'off_peak_day': off_peak_day,
            'weekend_to_weekday_ratio': round(weekend_ratio, 2),
            'cost_variation_coefficient': round(daily_avg.std() / daily_avg.mean(), 2) if daily_avg.mean() > 0 else 0
        }

    def _calculate_efficiency_metrics(self, usage_data: pd.DataFrame) -> Dict[str, Any]:
        """Calculate cost efficiency metrics"""
        if usage_data.empty:
            return {}
        
        # Cost per unit metrics
        efficiency_metrics = {}
        
        for service in usage_data['service'].unique():
            service_data = usage_data[usage_data['service'] == service]
            
            if service_data['computed_quantity'].sum() > 0:
                cost_per_unit = (
                    service_data['computed_amount'].sum() / 
                    service_data['computed_quantity'].sum()
                )
                efficiency_metrics[service] = {
                    'cost_per_unit': round(cost_per_unit, 4),
                    'total_units': service_data['computed_quantity'].sum(),
                    'unit_type': service_data['unit'].iloc[0] if len(service_data) > 0 else 'unknown'
                }
        
        # Overall efficiency trends
        total_cost = usage_data['computed_amount'].sum()
        total_quantity = usage_data['computed_quantity'].sum()
        
        return {
            'service_efficiency': efficiency_metrics,
            'overall_cost_per_unit': round(total_cost / total_quantity, 4) if total_quantity > 0 else 0,
            'efficiency_score': self._calculate_efficiency_score(usage_data)
        }

    def _calculate_efficiency_score(self, usage_data: pd.DataFrame) -> float:
        """Calculate overall efficiency score (0-100)"""
        if usage_data.empty:
            return 0.0
        
        # Factors that contribute to efficiency score
        factors = []
        
        # Cost volatility (lower is better)
        daily_costs = usage_data.groupby('date')['computed_amount'].sum()
        if len(daily_costs) > 1:
            volatility = daily_costs.std() / daily_costs.mean()
            volatility_score = max(0, 100 - (volatility * 100))
            factors.append(volatility_score)
        
        # Resource utilization (mock calculation - would need actual metrics)
        # In real implementation, this would come from monitoring data
        utilization_score = 75  # Placeholder
        factors.append(utilization_score)
        
        # Cost trend (stable or decreasing is better)
        if len(daily_costs) >= 7:
            recent_avg = daily_costs.tail(7).mean()
            previous_avg = daily_costs.head(7).mean()
            
            if previous_avg > 0:
                trend_factor = (previous_avg - recent_avg) / previous_avg
                trend_score = min(100, max(0, 50 + (trend_factor * 50)))
                factors.append(trend_score)
        
        return round(np.mean(factors), 1) if factors else 50.0

    async def _detect_cost_anomalies(self, usage_data: pd.DataFrame) -> List[CostAnomaly]:
        """Detect cost anomalies using machine learning"""
        anomalies = []
        
        if usage_data.empty or len(usage_data) < 30:
            return anomalies
        
        try:
            # Prepare data for anomaly detection
            daily_costs = usage_data.groupby(['date', 'service'])['computed_amount'].sum().reset_index()
            
            for service in daily_costs['service'].unique():
                service_data = daily_costs[daily_costs['service'] == service]
                
                if len(service_data) < 14:  # Need sufficient data
                    continue
                
                costs = service_data['computed_amount'].values.reshape(-1, 1)
                
                # Fit anomaly detector
                detector = IsolationForest(contamination=0.1, random_state=42)
                detector.fit(costs)
                
                # Detect anomalies
                anomaly_scores = detector.decision_function(costs)
                is_anomaly = detector.predict(costs) == -1
                
                # Process anomalies
                for i, (anomaly, score) in enumerate(zip(is_anomaly, anomaly_scores)):
                    if anomaly:
                        date = service_data.iloc[i]['date']
                        actual_cost = service_data.iloc[i]['computed_amount']
                        
                        # Calculate expected cost (median of recent normal values)
                        normal_costs = costs[~is_anomaly]
                        expected_cost = np.median(normal_costs) if len(normal_costs) > 0 else actual_cost
                        
                        # Determine severity
                        cost_factor = actual_cost / expected_cost if expected_cost > 0 else 1
                        
                        if cost_factor >= 3:
                            severity = CostSeverity.CRITICAL
                        elif cost_factor >= 2:
                            severity = CostSeverity.HIGH
                        elif cost_factor >= 1.5:
                            severity = CostSeverity.MEDIUM
                        else:
                            severity = CostSeverity.LOW
                        
                        anomaly = CostAnomaly(
                            resource_id=f"{service}-{date.strftime('%Y%m%d')}",
                            resource_type=service,
                            resource_name=service,
                            expected_cost=expected_cost,
                            actual_cost=actual_cost,
                            anomaly_score=abs(score),
                            severity=severity,
                            detected_at=datetime.utcnow(),
                            description=f"Cost spike detected: {actual_cost:.2f} vs expected {expected_cost:.2f}",
                            recommended_action=OptimizationAction.RIGHT_SIZE,
                            potential_savings=actual_cost - expected_cost
                        )
                        
                        anomalies.append(anomaly)
            
            return sorted(anomalies, key=lambda x: x.potential_savings, reverse=True)
            
        except Exception as e:
            logger.error(f"Failed to detect cost anomalies: {str(e)}")
            return []

    async def _forecast_costs(self, usage_data: pd.DataFrame, forecast_days: int = 30) -> Dict[str, Any]:
        """Forecast future costs using machine learning"""
        if usage_data.empty or len(usage_data) < 14:
            return {'status': 'insufficient_data'}
        
        try:
            # Prepare data for forecasting
            daily_costs = usage_data.groupby('date')['computed_amount'].sum().reset_index()
            daily_costs['days'] = (daily_costs['date'] - daily_costs['date'].min()).dt.days
            
            X = daily_costs[['days']].values
            y = daily_costs['computed_amount'].values
            
            # Fit forecasting model
            self.cost_forecaster.fit(X, y)
            
            # Generate forecast
            last_day = daily_costs['days'].max()
            future_days = np.arange(last_day + 1, last_day + forecast_days + 1).reshape(-1, 1)
            forecasted_costs = self.cost_forecaster.predict(future_days)
            
            # Calculate confidence intervals (simplified)
            residuals = y - self.cost_forecaster.predict(X)
            std_error = np.std(residuals)
            
            forecast_dates = [
                daily_costs['date'].max() + timedelta(days=i) 
                for i in range(1, forecast_days + 1)
            ]
            
            forecast_data = []
            for i, (date, cost) in enumerate(zip(forecast_dates, forecasted_costs)):
                forecast_data.append({
                    'date': date.strftime('%Y-%m-%d'),
                    'forecasted_cost': round(max(0, cost), 2),
                    'confidence_lower': round(max(0, cost - 1.96 * std_error), 2),
                    'confidence_upper': round(cost + 1.96 * std_error, 2)
                })
            
            return {
                'status': 'success',
                'forecast_period_days': forecast_days,
                'total_forecasted_cost': round(sum(forecasted_costs), 2),
                'average_daily_cost': round(np.mean(forecasted_costs), 2),
                'forecast_accuracy': round(self.cost_forecaster.score(X, y), 3),
                'daily_forecasts': forecast_data
            }
            
        except Exception as e:
            logger.error(f"Failed to forecast costs: {str(e)}")
            return {'status': 'error', 'message': str(e)}

    async def discover_optimization_opportunities(self) -> List[OptimizationRecommendation]:
        """Discover cost optimization opportunities across resources"""
        recommendations = []
        
        try:
            # Discover compute instances
            compute_recommendations = await self._analyze_compute_costs()
            recommendations.extend(compute_recommendations)
            
            # Discover storage optimization
            storage_recommendations = await self._analyze_storage_costs()
            recommendations.extend(storage_recommendations)
            
            # Discover network optimization
            network_recommendations = await self._analyze_network_costs()
            recommendations.extend(network_recommendations)
            
            # Sort by potential savings
            recommendations.sort(key=lambda x: x.potential_savings, reverse=True)
            
            return recommendations
            
        except Exception as e:
            logger.error(f"Failed to discover optimization opportunities: {str(e)}")
            return []

    async def _analyze_compute_costs(self) -> List[OptimizationRecommendation]:
        """Analyze compute instance costs and recommend optimizations"""
        recommendations = []
        
        try:
            # Get all compute instances
            instances = self.compute_client.list_instances(
                compartment_id=self.config['compartment_id'],
                lifecycle_state='RUNNING'
            ).data
            
            for instance in instances:
                # Get instance metrics (simplified - would use actual monitoring data)
                utilization_data = await self._get_instance_utilization(instance.id)
                
                # Calculate current cost (simplified pricing)
                current_cost = self._calculate_instance_cost(instance)
                
                # Analyze for right-sizing opportunities
                if utilization_data.get('cpu_utilization', 50) < 20:
                    # Recommend smaller shape
                    recommended_shape = self._recommend_smaller_shape(instance.shape)
                    
                    if recommended_shape:
                        projected_cost = current_cost * 0.6  # Approximate cost reduction
                        savings = current_cost - projected_cost
                        
                        recommendation = OptimizationRecommendation(
                            resource_id=instance.id,
                            resource_type='compute_instance',
                            current_config={
                                'shape': instance.shape,
                                'ocpus': getattr(instance.shape_config, 'ocpus', 'unknown'),
                                'memory_gb': getattr(instance.shape_config, 'memory_in_gbs', 'unknown')
                            },
                            recommended_config={
                                'shape': recommended_shape,
                                'action': 'resize_instance'
                            },
                            current_monthly_cost=current_cost,
                            projected_monthly_cost=projected_cost,
                            potential_savings=savings,
                            confidence_score=0.8,
                            implementation_effort='medium',
                            risk_level='low',
                            business_impact='minimal'
                        )
                        
                        recommendations.append(recommendation)
                
                # Check for unused instances
                if utilization_data.get('cpu_utilization', 50) < 5:
                    recommendation = OptimizationRecommendation(
                        resource_id=instance.id,
                        resource_type='compute_instance',
                        current_config={'shape': instance.shape, 'state': 'running'},
                        recommended_config={'action': 'terminate_or_stop'},
                        current_monthly_cost=current_cost,
                        projected_monthly_cost=0,
                        potential_savings=current_cost,
                        confidence_score=0.9,
                        implementation_effort='low',
                        risk_level='medium',
                        business_impact='requires_validation'
                    )
                    
                    recommendations.append(recommendation)
            
            return recommendations
            
        except Exception as e:
            logger.error(f"Failed to analyze compute costs: {str(e)}")
            return []

    async def _get_instance_utilization(self, instance_id: str) -> Dict[str, float]:
        """Get instance utilization metrics (simplified)"""
        try:
            # In a real implementation, this would query OCI Monitoring
            # For demo purposes, returning mock data
            return {
                'cpu_utilization': np.random.uniform(5, 95),
                'memory_utilization': np.random.uniform(10, 90),
                'network_utilization': np.random.uniform(1, 50)
            }
        except Exception as e:
            logger.error(f"Failed to get utilization for {instance_id}: {str(e)}")
            return {}

    def _calculate_instance_cost(self, instance) -> float:
        """Calculate monthly cost for instance (simplified)"""
        # Simplified cost calculation - in reality would use OCI pricing API
        shape_costs = {
            'VM.Standard2.1': 67.0,
            'VM.Standard2.2': 134.0,
            'VM.Standard2.4': 268.0,
            'VM.Standard2.8': 536.0,
            'VM.Standard.E3.Flex': 50.0,  # Base cost
            'VM.Standard.E4.Flex': 45.0   # Base cost
        }
        
        base_cost = shape_costs.get(instance.shape, 100.0)
        
        # Adjust for flex shapes based on OCPUs
        if 'Flex' in instance.shape and hasattr(instance, 'shape_config'):
            if hasattr(instance.shape_config, 'ocpus'):
                base_cost *= float(instance.shape_config.ocpus)
        
        return base_cost

    def _recommend_smaller_shape(self, current_shape: str) -> Optional[str]:
        """Recommend a smaller instance shape"""
        shape_hierarchy = {
            'VM.Standard2.8': 'VM.Standard2.4',
            'VM.Standard2.4': 'VM.Standard2.2',
            'VM.Standard2.2': 'VM.Standard2.1',
            'VM.Standard.E4.Flex': 'VM.Standard.E3.Flex'
        }
        
        return shape_hierarchy.get(current_shape)

    async def _analyze_storage_costs(self) -> List[OptimizationRecommendation]:
        """Analyze storage costs and recommend optimizations"""
        recommendations = []
        
        try:
            # Get block volumes
            volumes = self.storage_client.list_volumes(
                compartment_id=self.config['compartment_id'],
                lifecycle_state='AVAILABLE'
            ).data
            
            for volume in volumes:
                # Analyze volume usage patterns (simplified)
                usage_pattern = await self._analyze_volume_usage(volume.id)
                
                current_cost = volume.size_in_gbs * 0.0255  # Simplified cost per GB
                
                # Check for infrequent access patterns
                if usage_pattern.get('access_frequency', 'high') == 'low':
                    # Recommend moving to lower performance tier
                    projected_cost = current_cost * 0.7  # Lower tier pricing
                    savings = current_cost - projected_cost
                    
                    recommendation = OptimizationRecommendation(
                        resource_id=volume.id,
                        resource_type='block_volume',
                        current_config={
                            'size_gb': volume.size_in_gbs,
                            'vpus_per_gb': getattr(volume, 'vpus_per_gb', 10)
                        },
                        recommended_config={
                            'action': 'change_volume_performance',
                            'new_vpus_per_gb': 0
                        },
                        current_monthly_cost=current_cost,
                        projected_monthly_cost=projected_cost,
                        potential_savings=savings,
                        confidence_score=0.7,
                        implementation_effort='low',
                        risk_level='low',
                        business_impact='minimal'
                    )
                    
                    recommendations.append(recommendation)
                
                # Check for oversized volumes
                if usage_pattern.get('utilization_percent', 50) < 30:
                    # Recommend volume resize
                    new_size = int(volume.size_in_gbs * 0.6)
                    projected_cost = new_size * 0.0255
                    savings = current_cost - projected_cost
                    
                    recommendation = OptimizationRecommendation(
                        resource_id=volume.id,
                        resource_type='block_volume',
                        current_config={'size_gb': volume.size_in_gbs},
                        recommended_config={
                            'action': 'resize_volume',
                            'new_size_gb': new_size
                        },
                        current_monthly_cost=current_cost,
                        projected_monthly_cost=projected_cost,
                        potential_savings=savings,
                        confidence_score=0.6,
                        implementation_effort='medium',
                        risk_level='medium',
                        business_impact='requires_validation'
                    )
                    
                    recommendations.append(recommendation)
            
            return recommendations
            
        except Exception as e:
            logger.error(f"Failed to analyze storage costs: {str(e)}")
            return []

    async def _analyze_volume_usage(self, volume_id: str) -> Dict[str, Any]:
        """Analyze volume usage patterns (simplified)"""
        # In reality, this would analyze metrics from OCI Monitoring
        return {
            'access_frequency': np.random.choice(['high', 'medium', 'low'], p=[0.3, 0.4, 0.3]),
            'utilization_percent': np.random.uniform(10, 95),
            'iops_usage': np.random.uniform(100, 10000)
        }

    async def _analyze_network_costs(self) -> List[OptimizationRecommendation]:
        """Analyze network costs and recommend optimizations"""
        recommendations = []
        
        try:
            # Get load balancers
            load_balancers = self.network_client.list_load_balancers(
                compartment_id=self.config['compartment_id']
            ).data
            
            for lb in load_balancers:
                # Analyze load balancer utilization
                utilization = await self._analyze_lb_utilization(lb.id)
                
                # Calculate current cost (simplified)
                if hasattr(lb, 'shape_details') and lb.shape_details:
                    current_bandwidth = lb.shape_details.maximum_bandwidth_in_mbps
                    current_cost = current_bandwidth * 0.008  # Simplified pricing
                    
                    # Check for over-provisioning
                    if utilization.get('avg_bandwidth_usage', 50) < current_bandwidth * 0.3:
                        recommended_bandwidth = max(10, int(current_bandwidth * 0.5))
                        projected_cost = recommended_bandwidth * 0.008
                        savings = current_cost - projected_cost
                        
                        recommendation = OptimizationRecommendation(
                            resource_id=lb.id,
                            resource_type='load_balancer',
                            current_config={
                                'max_bandwidth_mbps': current_bandwidth,
                                'shape': getattr(lb, 'shape_name', 'flexible')
                            },
                            recommended_config={
                                'action': 'resize_load_balancer',
                                'new_max_bandwidth_mbps': recommended_bandwidth
                            },
                            current_monthly_cost=current_cost,
                            projected_monthly_cost=projected_cost,
                            potential_savings=savings,
                            confidence_score=0.75,
                            implementation_effort='low',
                            risk_level='low',
                            business_impact='minimal'
                        )
                        
                        recommendations.append(recommendation)
            
            return recommendations
            
        except Exception as e:
            logger.error(f"Failed to analyze network costs: {str(e)}")
            return []

    async def _analyze_lb_utilization(self, lb_id: str) -> Dict[str, Any]:
        """Analyze load balancer utilization (simplified)"""
        return {
            'avg_bandwidth_usage': np.random.uniform(5, 100),
            'peak_bandwidth_usage': np.random.uniform(20, 150),
            'avg_requests_per_second': np.random.uniform(10, 1000)
        }

    async def monitor_budgets(self) -> List[BudgetAlert]:
        """Monitor budget usage and generate alerts"""
        alerts = []
        
        try:
            # Get all budgets
            budgets = self.budgets_client.list_budgets(
                compartment_id=self.config['compartment_id']
            ).data
            
            for budget in budgets:
                # Get current spend
                current_spend = await self._get_current_budget_spend(budget.id)
                budget_amount = float(budget.amount)
                
                utilization_percentage = (current_spend / budget_amount * 100) if budget_amount > 0 else 0
                
                # Forecast end-of-period spend
                forecast_spend = await self._forecast_budget_spend(budget.id)
                
                # Calculate days remaining in budget period
                days_remaining = self._calculate_days_remaining(budget)
                
                # Determine severity
                if utilization_percentage >= 90 or forecast_spend > budget_amount * 1.1:
                    severity = CostSeverity.CRITICAL
                elif utilization_percentage >= 75 or forecast_spend > budget_amount:
                    severity = CostSeverity.HIGH
                elif utilization_percentage >= 60:
                    severity = CostSeverity.MEDIUM
                else:
                    severity = CostSeverity.LOW
                
                # Generate recommendations based on severity
                recommendations = []
                if severity in [CostSeverity.HIGH, CostSeverity.CRITICAL]:
                    recommendations = await self._generate_budget_recommendations(budget.id)
                
                alert = BudgetAlert(
                    budget_name=budget.display_name,
                    current_spend=current_spend,
                    budget_amount=budget_amount,
                    utilization_percentage=utilization_percentage,
                    forecast_spend=forecast_spend,
                    days_remaining=days_remaining,
                    severity=severity,
                    recommendations=recommendations
                )
                
                alerts.append(alert)
            
            return alerts
            
        except Exception as e:
            logger.error(f"Failed to monitor budgets: {str(e)}")
            return []

    async def _get_current_budget_spend(self, budget_id: str) -> float:
        """Get current spend for a budget (simplified)"""
        # In reality, this would query actual spend data
        return np.random.uniform(1000, 50000)

    async def _forecast_budget_spend(self, budget_id: str) -> float:
        """Forecast end-of-period spend for budget"""
        current_spend = await self._get_current_budget_spend(budget_id)
        # Simplified forecast - would use actual trend analysis
        growth_factor = np.random.uniform(1.05, 1.3)
        return current_spend * growth_factor

    def _calculate_days_remaining(self, budget) -> int:
        """Calculate days remaining in budget period"""
        # Simplified calculation - would use actual budget period
        return np.random.randint(1, 30)

    async def _generate_budget_recommendations(self, budget_id: str) -> List[str]:
        """Generate recommendations for budget management"""
        recommendations = [
            "Review and optimize underutilized compute instances",
            "Implement automated scheduling for non-production workloads",
            "Consider Reserved Instances for predictable workloads",
            "Review storage usage and archive old data",
            "Optimize load balancer configurations"
        ]
        
        return recommendations[:3]  # Return top 3 recommendations

    async def generate_cost_report(self, trends: Dict[str, Any], 
                                 recommendations: List[OptimizationRecommendation],
                                 budget_alerts: List[BudgetAlert]) -> str:
        """Generate comprehensive cost management report"""
        
        report_time = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC')
        
        # Calculate summary metrics
        total_potential_savings = sum(r.potential_savings for r in recommendations)
        high_impact_recommendations = [r for r in recommendations if r.potential_savings > 100]
        critical_budget_alerts = [a for a in budget_alerts if a.severity == CostSeverity.CRITICAL]
        
        report = f"""
# OCI Cost Management and FinOps Report
**Generated:** {report_time}

## Executive Summary

### Cost Overview
- **Total Potential Monthly Savings:** ${total_potential_savings:.2f}
- **High-Impact Opportunities:** {len(high_impact_recommendations)} recommendations
- **Critical Budget Alerts:** {len(critical_budget_alerts)} budgets requiring attention
- **Overall Cost Efficiency Score:** {trends.get('cost_efficiency_metrics', {}).get('efficiency_score', 'N/A')}

### Key Insights
"""
        
        # Add cost trend insights
        cost_trend = trends.get('total_cost_trend', {})
        if cost_trend:
            report += f"""
- **Cost Trend:** {cost_trend.get('trend', 'Unknown')} ({cost_trend.get('growth_rate_percent', 0):+.1f}% growth)
- **Daily Average Cost:** ${cost_trend.get('average_daily_cost', 0):.2f}
- **Cost Volatility:** {cost_trend.get('volatility_percent', 0):.1f}%
"""
        
        # Service cost breakdown
        service_costs = trends.get('service_cost_breakdown', {})
        if service_costs and service_costs.get('service_breakdown'):
            report += f"""

## Service Cost Analysis

### Top Cost Drivers
"""
            for service, data in list(service_costs['service_breakdown'].items())[:5]:
                report += f"- **{service}:** ${data['total_cost']:.2f} ({data['cost_percentage']:.1f}%)\n"
        
        # Cost anomalies
        anomalies = trends.get('anomalies', [])
        if anomalies:
            report += f"""

## Cost Anomalies Detected

Found {len(anomalies)} cost anomalies requiring investigation:
"""
            for anomaly in anomalies[:5]:  # Show top 5 anomalies
                report += f"""
### {anomaly.resource_name}
- **Severity:** {anomaly.severity.value.upper()}
- **Expected Cost:** ${anomaly.expected_cost:.2f}
- **Actual Cost:** ${anomaly.actual_cost:.2f}
- **Potential Savings:** ${anomaly.potential_savings:.2f}
- **Recommended Action:** {anomaly.recommended_action.value}
"""
        
        # Optimization recommendations
        if recommendations:
            report += f"""

## Cost Optimization Recommendations

### Top Savings Opportunities
"""
            
            for i, rec in enumerate(recommendations[:10], 1):
                report += f"""
#### {i}. {rec.resource_type.replace('_', ' ').title()} Optimization
- **Current Monthly Cost:** ${rec.current_monthly_cost:.2f}
- **Projected Monthly Cost:** ${rec.projected_monthly_cost:.2f}
- **Monthly Savings:** ${rec.potential_savings:.2f}
- **Confidence Score:** {rec.confidence_score:.0%}
- **Implementation Effort:** {rec.implementation_effort}
- **Risk Level:** {rec.risk_level}
"""
        
        # Budget alerts
        if budget_alerts:
            report += f"""

## Budget Monitoring

### Budget Status Overview
"""
            for alert in budget_alerts:
                status_emoji = "🔴" if alert.severity == CostSeverity.CRITICAL else "🟡" if alert.severity == CostSeverity.HIGH else "🟢"
                
                report += f"""
#### {status_emoji} {alert.budget_name}
- **Current Spend:** ${alert.current_spend:.2f} / ${alert.budget_amount:.2f}
- **Utilization:** {alert.utilization_percentage:.1f}%
- **Forecast Spend:** ${alert.forecast_spend:.2f}
- **Days Remaining:** {alert.days_remaining}
"""
                
                if alert.recommendations:
                    report += "- **Recommendations:**\n"
                    for rec in alert.recommendations:
                        report += f"  - {rec}\n"
        
        # Cost forecast
        forecast = trends.get('cost_forecast', {})
        if forecast.get('status') == 'success':
            report += f"""

## Cost Forecast

### Next 30 Days Projection
- **Total Forecasted Cost:** ${forecast.get('total_forecasted_cost', 0):.2f}
- **Average Daily Cost:** ${forecast.get('average_daily_cost', 0):.2f}
- **Forecast Accuracy:** {forecast.get('forecast_accuracy', 0):.1%}
"""
        
        # Action items and recommendations
        report += f"""

## Recommended Actions

### Immediate Actions (Next 7 Days)
1. **Review Critical Budget Alerts** - {len(critical_budget_alerts)} budgets need immediate attention
2. **Implement High-Impact Optimizations** - Focus on recommendations with savings > $100/month
3. **Investigate Cost Anomalies** - {len([a for a in anomalies if a.severity in [CostSeverity.HIGH, CostSeverity.CRITICAL]])} critical anomalies detected

### Short-term Actions (Next 30 Days)
1. **Resource Right-sizing** - Implement compute and storage optimizations
2. **Automation Implementation** - Set up automated scheduling for non-production workloads
3. **Policy Enforcement** - Implement cost governance policies

### Long-term Initiatives (Next Quarter)
1. **Reserved Instance Strategy** - Evaluate commitment-based pricing for predictable workloads
2. **Architecture Optimization** - Review overall architecture for cost efficiency
3. **FinOps Process Maturity** - Enhance cross-team collaboration and cost accountability

## Cost Optimization Priorities

Based on the analysis, focus on these optimization areas:
"""
        
        # Prioritize recommendations by savings and confidence
        priority_areas = {}
        for rec in recommendations:
            resource_type = rec.resource_type
            if resource_type not in priority_areas:
                priority_areas[resource_type] = {
                    'total_savings': 0,
                    'count': 0,
                    'avg_confidence': 0
                }
            
            priority_areas[resource_type]['total_savings'] += rec.potential_savings
            priority_areas[resource_type]['count'] += 1
            priority_areas[resource_type]['avg_confidence'] += rec.confidence_score
        
        # Calculate averages and sort by impact
        for area in priority_areas.values():
            area['avg_confidence'] = area['avg_confidence'] / area['count']
        
        sorted_areas = sorted(
            priority_areas.items(), 
            key=lambda x: x[1]['total_savings'], 
            reverse=True
        )
        
        for i, (area, data) in enumerate(sorted_areas[:5], 1):
            report += f"""
{i}. **{area.replace('_', ' ').title()}** - ${data['total_savings']:.2f} potential monthly savings
   - {data['count']} optimization opportunities
   - {data['avg_confidence']:.0%} average confidence score
"""
        
        return report

# Automated cost optimization workflow
async def run_cost_optimization_workflow():
    """Run comprehensive cost optimization workflow"""
    optimizer = OCICostOptimizer()
    
    try:
        logger.info("Starting cost optimization workflow...")
        
        # Step 1: Analyze cost trends
        logger.info("Analyzing cost trends...")
        trends = await optimizer.analyze_cost_trends(days_back=90)
        
        # Step 2: Discover optimization opportunities
        logger.info("Discovering optimization opportunities...")
        recommendations = await optimizer.discover_optimization_opportunities()
        
        # Step 3: Monitor budgets
        logger.info("Monitoring budget status...")
        budget_alerts = await optimizer.monitor_budgets()
        
        # Step 4: Generate comprehensive report
        logger.info("Generating cost management report...")
        report = await optimizer.generate_cost_report(trends, recommendations, budget_alerts)
        
        # Step 5: Save report and send notifications
        timestamp = datetime.utcnow().strftime('%Y%m%d_%H%M%S')
        report_filename = f"oci_cost_report_{timestamp}.md"
        
        with open(report_filename, 'w') as f:
            f.write(report)
        
        logger.info(f"Cost optimization report saved to {report_filename}")
        
        # Send alerts for critical issues
        critical_issues = []
        critical_issues.extend([a for a in trends.get('anomalies', []) if a.severity == CostSeverity.CRITICAL])
        critical_issues.extend([a for a in budget_alerts if a.severity == CostSeverity.CRITICAL])
        
        if critical_issues:
            await send_critical_cost_alerts(critical_issues, report_filename)
        
        # Return summary for API consumers
        return {
            'status': 'success',
            'report_file': report_filename,
            'summary': {
                'total_potential_savings': sum(r.potential_savings for r in recommendations),
                'optimization_opportunities': len(recommendations),
                'critical_budget_alerts': len([a for a in budget_alerts if a.severity == CostSeverity.CRITICAL]),
                'cost_anomalies': len(trends.get('anomalies', [])),
                'efficiency_score': trends.get('cost_efficiency_metrics', {}).get('efficiency_score', 0)
            }
        }
        
    except Exception as e:
        logger.error(f"Cost optimization workflow failed: {str(e)}")
        return {'status': 'error', 'message': str(e)}

async def send_critical_cost_alerts(critical_issues: List, report_file: str):
    """Send alerts for critical cost issues"""
    try:
        # Prepare alert message
        alert_message = f"""
CRITICAL COST ALERT - OCI Environment

{len(critical_issues)} critical cost issues detected requiring immediate attention.

Issues:
"""
        for issue in critical_issues[:5]:  # Limit to top 5
            if hasattr(issue, 'resource_name'):
                alert_message += f"- {issue.resource_name}: ${getattr(issue, 'potential_savings', 0):.2f} potential savings\n"
            else:
                alert_message += f"- {issue.budget_name}: {issue.utilization_percentage:.1f}% budget utilization\n"
        
        alert_message += f"\nFull report available in: {report_file}"
        
        # Send to configured notification channels
        # Implementation would depend on your notification preferences
        logger.warning(f"CRITICAL COST ALERT: {len(critical_issues)} issues detected")
        
    except Exception as e:
        logger.error(f"Failed to send critical cost alerts: {str(e)}")

if __name__ == "__main__":
    # Run the cost optimization workflow
    import asyncio
    result = asyncio.run(run_cost_optimization_workflow())
    print(f"Cost optimization completed: {result}")

Automated Cost Governance and Policy Enforcement

Advanced FinOps implementations require automated governance mechanisms that prevent cost overruns before they occur. Policy-as-code frameworks enable organizations to define spending rules, approval workflows, and automated remediation actions that maintain cost discipline across development teams.

Budget enforcement policies can automatically halt resource provisioning when spending thresholds are exceeded, while notification workflows ensure appropriate stakeholders receive timely alerts about budget utilization. These policies integrate with existing CI/CD pipelines to provide cost validation during infrastructure deployments.

Resource tagging policies ensure consistent cost allocation across business units and projects, with automated compliance checking that flags untagged resources or incorrect tag values. This standardization enables accurate chargebacks and cost center reporting.

Automated resource lifecycle management implements policies for non-production environments, automatically stopping development instances outside business hours and deleting temporary resources after predefined periods.

Real-time Cost Monitoring and Alerting

Production FinOps requires real-time cost monitoring that provides immediate visibility into spending changes. Integration with OCI Events service enables automatic notifications when resource costs exceed predefined thresholds or when unusual spending patterns are detected.

Custom dashboards aggregate cost data across multiple dimensions including service type, environment, project, and business unit. These dashboards provide executives with high-level spending trends while giving engineers detailed cost attribution for their specific resources.

Anomaly detection algorithms continuously monitor spending patterns and automatically alert teams when costs deviate significantly from established baselines. Machine learning models learn normal spending patterns and adapt to seasonal variations while maintaining sensitivity to genuine cost anomalies.

Predictive cost modeling uses historical data and planned deployments to forecast future spending with confidence intervals, enabling proactive budget management and capacity planning decisions.

Integration with Enterprise Financial Systems

Enterprise FinOps implementations require integration with existing financial systems for seamless cost allocation and reporting. APIs enable automatic synchronization of OCI billing data with enterprise resource planning (ERP) systems and financial management platforms.

Automated chargeback mechanisms calculate costs by business unit, project, or customer based on resource utilization and predefined allocation rules. These calculations integrate with billing systems to generate accurate invoices for internal cost centers or external customers.

Cost center mapping enables automatic allocation of shared infrastructure costs across multiple business units based on actual usage metrics rather than static percentages. This approach provides more accurate cost attribution while maintaining fairness across different usage patterns.

Integration with procurement systems enables automatic validation of spending against approved budgets and purchase orders, with workflow integration for approval processes when costs exceed authorized amounts.

This comprehensive FinOps approach establishes a mature cost management practice that balances financial accountability with operational agility, enabling organizations to optimize cloud spending while maintaining innovation velocity and service quality.

Enjoy the Cloud
Osama Mustafa