I will have two presentation about the DevOps
- Database Automation, Is this even possible ?
- Kuberenetes in Depth but in simple way
You can register here
The hashtag in use is #APACGBT2021
Enjoy
Cheers
APAC Groundbreakers Virtual Tour 2021
Tag: Osama mustafa
Oracle Database 19c SIG November Meeting
About Quest’s product communities
Quest Oracle Community is home to 25,000+ users of JD Edwards, PeopleSoft, Oracle Cloud apps and Oracle Database products. We connect Oracle users to technology leaders and Oracle experts from companies who are driving innovation and leading through their use of Oracle products.
The Quest Oracle Community is dedicated to helping Oracle users develop skills and expand knowledge by connecting with other Oracle users and experts for education and networking.
I will present about the automation
You can register for the event from here
Thank you
Migrating to Serverless
we’ll look at considerations for migrating existing applications to serverless and common ways for extending the serverless
At a high level, there are three migration patterns that you might follow to migrate your legacy your applications to a serverless model.
Leapfrog
As the name suggests, you bypass interim steps and go straight from an on-premises legacy architecture to a serverless cloud architecture
Organic
You move on-premises applications to the cloud in more of a “lift and shift” model. In this model, existing applications are kept intact, either running on Amazon Elastic Compute Cloud (Amazon EC2) instances or with some limited rewrites to container services like Amazon Elastic Kubernetes Service (Amazon EKS)/Amazon Elastic Container Service (Amazon ECS) or AWS Fargate.
Developers experiment with Lambda in low-risk internal scenarios like log processing or cron jobs. As you gain more experience, you might use serverless components for tasks like data transformations and parallelization of processes.
At some point in the adoption curve, you take a more strategic look at how serverless and microservices might address business goals like market agility, developer innovation, and total cost of ownership.
You get buy-in for a more long-term commitment to invest in modernizing your applications and select a production workload as a pilot. With initial success and lessons learned, adoption accelerates, and more applications are migrated to microservices and serverless.
Strangler
With the strangler pattern, an organization incrementally and systematically decomposes monolithic applications by creating APIs and building event-driven components that gradually replace components of the legacy application.
Distinct API endpoints can point to old vs. new components, and safe deployment options (like canary deployments) let you point back to the legacy version with very little risk.
New feature branches can be “serverless first,” and legacy components can be decommissioned as they are replaced. This pattern represents a more systematic approach to adopting serverless, allowing you to move to critical improvements where you see benefit quickly but with less risk and upheaval than the leapfrog pattern.
Migration questions to answer:
- What does this application do, and how are its components organized?
- How can you break your data needs up based on the command query responsibility (CQRS) pattern?
- How does the application scale, and what components drive the capacity you need?
- Do you have schedule-based tasks?
- Do you have workers listening to a queue?
- Where can you refactor or enhance functionality without impacting the current implementation?
Application Load Balancer vs. API Gateway for directing traffic to serverless targets
| Application Load Balancer | Amazon API Gateway |
|---|---|
| Easier to transition existing compute stack where you are already using an Application Load Balancer | Good for building REST APIs and integrating with other services and Lambda functions |
| Supports authorization via OIDC-capable providers, including Amazon Cognito user pools | Supports authorization via AWS Identity and Access Management (IAM), Amazon Cognito, and Lambda authorizers |
| Charged by the hour, based on Load Balancer Capacity Units | Charged based on requests served |
| May be more cost-effective for a steady stream of traffic | May be more cost-effective for spiky patterns |
| Additional features for API management: Export SDK for clients Use throttling and usage plans to control access Maintain multiple versions of an APICanary deployments |
Consider three factors when comparing costs of ownership:
- The infrastructure cost to run your workload (for example, the costs for your provisioned EC2 capacity vs. the per-invocation cost of your Lambda functions)
- The development effort to plan, architect, and provision resources on which the application will run
- The costs of your team’s time to maintain the application once it is in production
Reference
Cheers
Osama
AWS Transit Gateway
AWS Transit Gateway is a highly available and scalable service that provides interconnectivity between VPCs and your on-premises network. Within a Region, AWS Transit Gateway provides a method for consolidating and centrally managing routing between VPCs with a hub-and-spoke network architecture.
Between Regions, AWS Transit Gateway supports inter-regional peering with other transit gateways. It does this to facilitate routing network traffic between VPCs of different Regions over the AWS global backbone. This removes the need to route traffic over the internet. AWS Transit Gateway also integrates with hybrid network configurations when a Direct Connect or AWS Site-to-Site VPN connection is connected to the transit gateway.
AWS Transit Gateway concepts
Attachments
AWS Transit Gateway supports the following connections:
- One or more VPCs
- A compatible Software-Defined Wide Area Network (SD-WAN) appliance
- A Direct Connect gateway
- A peering connection with another transit gateway
- A VPN connection to a transit gateway
AWS Transit Gateway MTU
AWS Transit Gateway supports an MTU of 8,500 bytes for:
- VPC connections
- Direct Connect connections
- Connections to other transit gateways
- Peering connections
AWS Transit Gateway supports an MTU of 1,500 bytes for VPN connections.
AWS Transit Gateway route table
A transit gateway has a default route table and can optionally have additional route tables. A route table includes dynamic and static routes that decide the next hop based on the destination IP address of the packet. The target of these routes can be any transit gateway attachment.
Associations
Each attachment is associated with exactly one route table. Each route table can be associated with zero to many attachments.
Route propagation
A VPC, VPN connection, or Direct Connect gateway can dynamically propagate routes to a transit gateway route table. With a Direct Connect attachment, the routes are propagated to a transit gateway route table by default.
With a VPC, you must create static routes to send traffic to the transit gateway.
With a VPN connection or a Direct Connect gateway, routes are propagated from the transit gateway to your on-premises router using BGP.
With a peering attachment, you must create a static route in the transit gateway route table to point to the peering attachment.
AWS Transit Gateway inter-regional peering
AWS offers two types of peering connections for routing traffic between VPCs in different Regions: VPC peering and transit gateway peering. Both peering types are one-to-one, but transit gateway peering connections have a simpler network design and more consolidated management.
Suppose a customer has multiple VPCs in three different Regions. As the following diagram illustrates, to permit network traffic to route between each VPC requires creating 72 VPC peering connections. Each VPC needs 8 different routing configurations and security policies.
With AWS Transit Gateway, the same environment only needs three peering connections. The transit gateway in each Region facilitates routing network traffic to all the VPCs in its Region. Because all routing can be managed by the transit gateway, the customer only needs to maintain three routing configurations, simplifying management.
Cheers
Osama
AWS Site-to-Site VPN and AWS Client VPN
AWS VPN is comprised of two services:
- AWS Site-to-Site VPN enables you to securely connect your on-premises network to Amazon VPC, for example your branch office site.
- AWS Client VPN enables you to securely connect users to AWS or on-premises networks, for example remote employees.
AWS Site-to-Site VPN
ased on IPsec technology, AWS Site-to-Site VPN uses a VPN tunnel to pass data from the customer network to or from AWS.
One AWS Site-to-Site VPN connection consists of two tunnels. Each tunnel terminates in a different Availability Zone on the AWS side, but it must terminate on the same customer gateway on the customer side.
AWS Site-to-Site VPN components
Customer gateway
A resource you create and configure in AWS that represents your on-premise gateway device. The resource contains information about the type of routing used by the Site-to-Site VPN, BGP, ASN and other optional configuration information.
Customer gateway device
A customer gateway device is a physical device or software application on your side of the AWS Site-to-Site VPN connection.
Virtual private gateway
A virtual private gateway is the VPN concentrator on the Amazon side of the AWS Site-to-Site VPN connection. You use a virtual private gateway or a transit gateway as the gateway for the Amazon side of the AWS Site-to-Site VPN connection.
Transit gateway
A transit gateway is a transit hub that can be used to interconnect your VPCs and on-premises networks. You use a transit gateway or virtual private gateway as the gateway for the Amazon side of the AWS Site-to-Site VPN connection.
AWS Site-to-Site VPN limitations
- IPv6 traffic is partially supported. AWS Site-to-Site VPN supports IPv4/IPv6-Dualstack through separate tunnels for inner traffic. IPv6 for outer tunnel connection not supported.
- AWS Site-to-Site VPN does not support Path MTU Discovery. The greatest Maximum Transmission Unit (MTU) available on the inside tunnel interface is 1,399 bytes.
- Throughput of AWS Site-to-Site VPN connections is limited. When terminating on a virtual private gateway, only one tunnel out of the pair can be active and carry a maximum of 1.25 Gbps. However, real-life throughput will be about 1 Gbps. When terminating on AWS Transit Gateway, both tunnels in the pair can be active and carry an aggregate maximum of 2.5 Gbps. However, real-life throughput will be 2 Gbps. Each flow (for example, TCP stream) will still be limited to a maximum of 1.25 Gbps, with a real-life value of about 1 Gbps.
- Maximum packets per second (PPS) per VPN tunnel is 140,000.
- AWS Site-to-Site VPN terminating on AWS Transit Gateway supports equal-cost multi-path routing (ECMP) and multi-exit discriminator (MED) across tunnels in the same and different connection. ECMP is only supported for Site-to-Site VPN connections activated on an AWS Transit Gateway. MED is used to identify the primary tunnel for Site-to-Site VPN conncetions that use BGP. Note, BFD is not yet supported on AWS Site-to-Site VPN, though it is supported on Direct Connect.
- AWS Site-to-Site VPN endpoints use public IPv4 addresses and therefore require a public virtual interface to transport traffic over Direct Connect. Support for AWS Site-to-Site VPN over private Direct Connect is not yet available.
- For globally distributed applications, the accelerated Site-to-Site VPN option provides a connection to the global AWS backbone through AWS Global Accelerator. Because the Global Accelerator IP space is not announced over a Direct Connect public virtual interface, you cannot use accelerated Site-to-Site VPN with a Direct Connect public virtual interface.
In addition, when you connect your VPCs to a common on-premises network, it’s recommend that you use nonoverlapping CIDR blocks for your networks.
Client VPN
Based on OpenVPN technology, Client VPN is a managed client-based VPN service that lets you securely access your AWS resources and resources in your on-premises network. With Client VPN, you can access your resources from any location using an OpenVPN-based VPN client.
Client VPN components
Client VPN endpoint
Your Client VPN administrator creates and configures a Client VPN endpoint in AWS. Your administrator controls which networks and resources you can access when you establish a VPN connection.
VPN client application
This is the software application that you use to connect to the Client VPN endpoint and establish a secure VPN connection.
Client VPN endpoint configuration file
This is a configuration file that is provided to you by your Client VPN administrator. The file includes information about the Client VPN endpoint and the certificates required to establish a VPN connection. You load this file into your chosen VPN client application.
Client VPN limitations
- Client VPN supports IPv4 traffic only. IPv6 is not supported.
- Security Assertion Markup Language (SAML) 2.0-based federated authentication only works with an AWS provided client v1.2.0 or later.
- SAML integration with AWS Single Sign-On requires a workaround. Better integration is being worked on.
- Client CIDR ranges must have a block size of at least /22 and must not be greater than /12.
- A Client VPN endpoint does not support subnet associations in a dedicated tenancy VPC.
- Client VPN is not compliant with Federal Information Processing Standards (FIPS).
- Client CIDR ranges cannot overlap with the local CIDR of the VPC in which the associated subnet is located. It also cannot overlap any routes manually added to the Client VPN endpoint’s route table.
- A portion of the addresses in the client CIDR range is used to support the availability model of the Client VPN endpoint and cannot be assigned to clients. Therefore, we recommend that you assign a CIDR block that contains twice the number of required IP addresses. This will ensure the maximum number of concurrent connections that you plan to support on the Client VPN endpoint.
- The client CIDR range cannot be changed after you create the Client VPN endpoint.
- The subnets associated with a Client VPN endpoint must be in the same VPC.
- You cannot associate multiple subnets from the same Availability Zone with a Client VPN endpoint.
- AWS Certificate Manager (ACM) certificates are not supported with mutual authentication because you cannot extract the private key. You can use an ACM server as the server-side certificate. But, to add a client certificate to your customer configuration, you cannot use a general ACM certificate because you can’t extract the required private key details. So you must access the keys in one of two ways. Either generate your own certificate where you have the key or use AWS Certificate Manager Private Certificate Authority (ACM PCA), which gives the private keys. If the customer is authenticating based on Active Directory or SAML, they can use a general ACM-generated certificate because only the server certificate is required.
Cheers
Osama
Scaling Pods in Kubernetes
Continue to pervious post of Configure Kubernetes on my blog.
This post will discuss how to scale the pods, I will assume the Kubernetes installed if not back to the above post.
If you did these steps below , you can skip
Initialize the cluster
kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.11.3
As mentioned the command will generate commands like the picture.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config- Install Flannel
Flannel is an open-source virtual network project managed by CoreOS network designed for Kubernetes. Each host in a flannel cluster runs an agent called flanneld . It assigns each host a subnet, which acts as the IP address pool for containers running on the host.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
- Create deployment
vi deployment.ymlapiVersion: apps/v1
kind: Deployment
metadata:
name: httpd-deployment
labels:
app: httpd
spec:
replicas: 3
selector:
matchLabels:
app: httpd
template:
metadata:
labels:
app: httpd
spec:
containers:
- name: httpd
image: httpd:latest
ports:
- containerPort: 80- Spin up the deployment
kubectl create -f deployment.yml
- Create the service
vim service.ymlkind: Service
apiVersion: v1
metadata:
name: service-deployment
spec:
selector:
app: httpd
ports:
- protocol: TCP
port: 80
targetPort: 80
type: NodePortkubectl create -f service.yml
- Scale the deployment up to 5 replicas.
vi deployment.ymlChange the number of replicas to 5:
spec: replicas: 5
- Apply the changes:
kubectl apply -f deployment.ymlEnjoy
Hope it’s useful
Osama
Dockerize a Flask Application
The Flask Application uploaded to my GitHub Here
I will dockerize the above application and show you the steps to do that
Let’s Start 🤞
- First will add some files i don’t want to Dockerignore file
vim .dockerignore
.dockerignore
Dockerfile
.gitignore
Pipfile.lock
migrations/- Write the dockerfile, which is already included to the above Repo vim Dockerfile
FROM python:3
ENV PYBASE /pybase
ENV PYTHONUSERBASE $PYBASE
ENV PATH $PYBASE/bin:$PATH
RUN pip install pipenv
WORKDIR /tmp
COPY Pipfile .
RUN pipenv lock
RUN PIP_USER=1 PIP_IGNORE_INSTALLED=1 pipenv install -d --system --ignore-pipfile
COPY . /app/notes
WORKDIR /app/notes
EXPOSE 80
CMD ["flask", "run", "--port=80", "--host=0.0.0.0"]- Build and Test
docker build -t notesapp:0.1 .
docker run --rm -it --network notes -v /home/Osama/notes/migrations:/app/notes/migrations notesapp:0.1 bash
The above commands build and run the container, once you are inside the container configure the database
- Configure Database
flask db init
flask db migrate
flask db upgrade- Run and Upgrade
docker run --rm -it --network notes -p 80:80 notesapp:0.1- In a web browser, navigate to the public IP address for the server, and log in to your account.
- Verify that you can create a new note.
Perfect , we are done now
Enjoy the learning 👍
Osama
Storing Container Data in Azure Blob Storage
This time how to store your data to Azure Blog Storage 👍
Let’s start
Configuration
- Obtain the Azure login credentials
az login- Copy the code provided by the command.
- Open a browser and navigate to https://microsoft.com/devicelogin.
- Enter the code copied in a previous step and click Next.
- Use the login credentials from the lab page to finish logging in.
- Switch back to the terminal and wait for the confirmation.
Storage
- Find the name of the Storage account
az storage account list | grep name | head -1Copy the name of the Storage account to the clipboard.
- Export the Storage account name
export AZURE_STORAGE_ACCOUNT=<COPIED_STORAGE_ACCOUNT_NAME>- Retrieve the Storage access key
az storage account keys list --account-name=$AZURE_STORAGE_ACCOUNTCopy the key1 “value” for later use.
- Export the key value
export AZURE_STORAGE_ACCESS_KEY=<KEY1_VALUE>- Install blobfuse
sudo rpm -Uvh https://packages.microsoft.com/config/rhel/7/packages-microsoft-prod.rpm
sudo yum install blobfuse fuse -y- Modify the fuse.conf configuration file
sudo sed -ri 's/# user_allow_other/user_allow_other/' /etc/fuse.confUse Azure Blob container Storage
- Create necessary directories
sudo mkdir -p /mnt/Osama /mnt/blobfusetmp- Change ownership of the directories
sudo chown cloud_user /mnt/Osama/ /mnt/blobfusetmp/- Mount the Blob Storage from Azure
blobfuse /mnt/Osama --container-name=website --tmp-path=/mnt/blobfusetmp -o allow_other- Copy What you want to the files into the Blob Storage container for example website files.
cp -r ~/web/* /mnt/Osama/- Verify the copy worked
ll /mnt/Osama/- Verify the files made it to Azure Blob Storage
az storage blob list -c website --output table- Finally, Run a Docker container using the azure blob storage
docker run -d --name web1 -p 80:80 --mount type=bind,source=/mnt/Osama,target=/usr/local/apache2/htdocs,readonly httpd:2.4Enjoy 🎉😁
Osama
Apply for Oracle exam extension
As we already know oracle has been providing free exam and materials for siz track like the following till 15 May 2020: –
- Oracle Autonomous Database Specialist
- Oracle Cloud Infrastructure Foundations Associate
- Oracle Cloud Infrastructure Cloud Operations Associate
- Oracle Cloud Infrastructure Developer Associate
- Oracle Cloud Infrastructure Architect Associate
- Oracle Cloud Infrastructure Architect Professional
and because of the high demand since there are not available slot anymore, Oracle now providing extension BUT you have to apply for this
Follow this Video :-
Enjoy
Osama
Create Your First VM with Azure Cloud In different ways
To create your first server/VM on Azure cloud, you have different ways to do that :-
- Azure Resource Manager
- Azure PowerShell
- Azure CLI
- Azure REST API
- Azure Client SDK
- Azure VM Extensions
- Azure Automation Services
The Azure portal is the easiest way to create resources such as VMs, i will describe each one of them,
The first way which is The Portal here, to do this it’s very simple :-
- Click on the Create a resource option in the top-left corner of the portal page.
- Use the Search the Marketplace search bar to find “Ubuntu Server” for example.
- Press on Create , then new page will be open.
- Configure the VM, by enter the name, the region, The Subscription,Availability options
- There are several other tabs you can explore to see the settings you can influence during the VM creation. Once you’re finished exploring, click Review + create to review and validate the settings.
- On the review screen, Azure will validate your settings. You might need to supply some additional information based on the requirements of the image creator.
This is was the first way to create the VM which is consider the easiet one also.
Azure Resource Manager
assumig you want to create a copy of a VM with the same settings. You could create a VM image, upload it to Azure, and reference it as the basis for your new VM,Azure provides you with the option to create a template from which to create an exact copy of a VM.
You can do this, after create the VM –> Setting –> export template.
Azure PowerShell
Azure PowerShell is ideal for one-off interactive tasks and/or the automation of repeated tasks, note that PowerShell is a cross-platform shell that provides services like the shell window and command parsing.
New-AzVm -ResourceGroupName "TestResourceGroup" -Name "test-wp1-eus-vm" -Location "East US" -VirtualNetworkName "test-wp1-eus-network" -SubnetName "default" -SecurityGroupName "test-wp1-eus-nsg" -PublicIpAddressName "test-wp1-eus-pubip" -OpenPorts 80,3389Azure CLI
The Azure CLI is Microsoft’s cross-platform command-line tool for managing Azure resources such as virtual machines and disks from the command line. It’s available for macOS, Linux, and Windows, this is also found in Different cloud vendor for example For Amazon it’s called aws cli, for Oracle it’s Called OCI-CLI and Google it’s called GCP-CLI.
az vm create --resource-group TestResourceGroup --name test-wp1-eus-vm --image win2016datacenter --admin-username osama --admin-password anythingProgrammatic (APIs)
This is no my expertise so i will no go deep dive with it, But we were talking about Azure CLI and powershell, you can install something called Azure REST API and start using differen programing language to deal with Azure, i did this with python for AWS using Boto3 module, i post about it before here.
The same can be done for Azure or any Cloud vendor.
Azure VM Extensions
Azure VM extensions are small applications that allow you to configure and automate tasks on Azure VMs after initial deployment. Azure VM extensions can be run with the Azure CLI, PowerShell, Azure Resource Manager templates, and the Azure portal.
Thank you
Osama Mustafa
Technology Geek 