Next Event- Speaking Schedule/2020

I was honored this year to speak in different conferences around the world, and Due to Covid-19 most of the conference was Virtual, I miss my friends around the world, but saftey comes first, i hope everyone is safe.

I am writing this post to share with people the schedule of different conferences some of them already done but in case you missed something there is always next time, register in the virtual conferences and don’t miss the chance to learn something new.

My 2020 Schedule

Rule #1 in IT, never, and I do mean NEVER, stop learning.

Conference NameGroupDateLinkNote
Oracle Groundbreakers EMEA Virtual TourKSAOUG10-Oct-2020 1 2:30pmhttp://ogbemea.com/10-oct-2020/
SPOUG20SPOUGOctober 20, 2020 10:50 amhttp://spoug.es/spoug-20/
Oracle Groundbreakers APAC Virtual Tour 2020APAC GroupComing Soonhttps://www.apacgbt.com/

This table will be updated depends on the schedule or any new conferences.

Regards

Osama

configure PMM for Percona MYSQL

Percona Monitoring and Management (PMM) is a best-of-breed open source database monitoring solution. It helps you reduce complexity, optimize performance, and improve the security of your business-critical database environments, no matter where they are located or deployed.

PMM is a free and open-source solution that you can run in your own environment for maximum security and reliability. It provides thorough time-based analysis for MySQL and MongoDB servers to ensure that your data works as efficiently as possible.

PMM, at a high-level, is made up of two basic components: the client and the server. The PMM Client is installed on the database servers themselves and is used to collect metrics. The client contains technology specific exporters (which collect and export data), and an “admin interface” (which makes the management of the PMM platform very simple). The PMM server is a “pre-integrated unit” (Docker, VM or AWS AMI) that contains four components that gather the metrics from the exporters on the PMM client(s). The PMM server contains Consul, Grafana, Prometheus and a Query Analytics Engine that Percona has developed. Here is a graphic from the architecture section of our documentation. In order to keep this post to a manageable length.

In this post i will setup the PMM on docker.

Pulling the PMM Server Docker Image

docker pull percona/pmm-server:2

Create a persistent data container.

docker create --volume /srv \
--name pmm-data percona/pmm-server:2 /bin/true

Run the image to start PMM Server.

docker run --detach --restart always \ --publish 80:80 --publish 443:443 \ --volumes-from pmm-data --name pmm-server \ percona/pmm-server:2

Once you completed the server configuration, you have to install the client on desired one, for example in our case we want to install it to monitor MySQL so we will go to install PMM Client on MySQL server.

To install the PMM client package, follow these steps.

Configure Percona repositories using the percona-release tool

wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb

Note

If you have previously enabled the experimental or testing Percona repository, don’t forget to disable them and enable the release component of the original repository as follows:

sudo percona-release disable all
sudo percona-release enable original release

Install the PMM client package:

sudo apt-get update
sudo apt-get install pmm2-client

Register your Node:

Before doing this, there are some MySQL requirements should be done from Database side.

Enable Logs in MySQL follow these step

SET GLOBAL slow_query_log_file = '/path/to/slow_query.log';

Determine what makes a query “slow”, by setting the limit (in seconds) after which a query is logged to the slow query log. The example below logs every query that exceeds 10 seconds in duration

mysql> SET GLOBAL long_query_time = 10;

Now enable the Slow Query log.

mysql> SET GLOBAL slow_query_log = 'ON';
mysql> FLUSH LOGS;

If you want to make these changes persistent, modify the my.cnf and add these lines to the [mysqld] part of the config.

[mysqld]
...
slow_query_log = /path/to/slow_query.log
long_query_time = 10
log_queries_not_using_indexes = ON

Verify

mysql> SHOW GLOBAL VARIABLES LIKE 'log_queries_not_using_indexes';

Once you are done you have to create username/password for PMM,this user should have necessary privileges for collecting data. If the pmm user already exists, you can grant the required privileges as follows:

CREATE USER 'pmm'@'localhost' IDENTIFIED BY 'pass' WITH MAX_USER_CONNECTIONS 10;

GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD ON *.* TO 'pmm'@'localhost';

Once you done, register your node.

pmm-admin config --server-insecure-tls --server-url=https://admin:admin@<IP Address>:443

You have to wait for couple of minutes till it will be sync.

You should see the following output:

Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Registered.
Configuration file /usr/local/percona/pmm-agent.yaml updated.
Reloading pmm-agent configuration...
Configuration reloaded.

Regards

Osama

Setting up a Jenkins-Based Continuous Delivery Pipeline with Docker

As an important step in agile development, continuous integration is designed to maintain high quality while accelerating product iteration. Every time when the codes are updated, an automatic test is performed to test the codes and function validity. The codes can only be delivered and deployed after they pass the automatic test, This post describes how to combine Jenkins, one of the most popular integration tools, with Alibaba Cloud Container Service to realize automatic test and image building pushing.

1

Deploying Jenkins Applications and the Slave Nodes

1. Create a Jenkins orchestration template.

Create a new template and create the orchestration based on the following content.

jenkins:  image: 'registry.aliyuncs.com/acs-sample/jenkins:latest'  ports:      - '8080:8080'      - '50000:50000'  volumes:      - /var/lib/docker/jenkins:/var/jenkins_home  privileged: true  restart: always   labels:      aliyun.scale: '1'      aliyun.probe.url: 'tcp://container:8080'      aliyun.probe.initial_delay_seconds: '10'      aliyun.routing.port_8080: jenkins  links:      - slave-nodejs slave-nodejs:  image: 'registry.aliyuncs.com/acs-sample/jenkins-slave-dind-nodejs'  restart: always   volumes:      - /var/run/docker.sock:/var/run/docker.sock  labels:      aliyun.scale: '1' 

2. Use the template to create Jenkins applications and slave nodes.

You can also directly use a Jenkins sample template provided by Alibaba Cloud Container Service to create Jenkins applications and slave nodes.

2

3. After the successful creation, Jenkins applications and slave nodes will be displayed in the service list.

3

4. After opening the access endpoint provided by the Container Service, you can use the Jenkins application deployed just now.

4

Realizing Automatic Test and Automatic Build and Push of Image

Configure the slave container as the slave node of the Jenkins application.

Open the Jenkins application and enter the System Settings interface. Select Manage Node > Create Node, and configure corresponding parameters. See the figure below.

5

Note: Label is the only identifier of the slave. The slave container and Jenkins container run on the Alibaba Cloud platform at the same time. Therefore, you can fill in a container node IP address that is inaccessible to the Internet to isolate the test environment.

6

Use the jenkins account and password (the initial password is jenkins) in Dockerfile for the creation of the slave-nodejs image when adding Credential. Image Dockerfile address HERE

1. Create a project to implement the automatic test.

  1. Create an item and choose to build a software project of free style.
  2. Enter the project name and select a node for running the project. In this example, enter the slave-nodejs-ut node created above.
7

Configure the source code management and code branch. In this example, use GitHub to manage source codes.

8

Configure the trigger for building. In this example, automatically trigger project execution by combining GitHub Webhooks and services.

9

Add the Jenkins service hook to GitHub to implement automatic triggering.

Click the Settings tab on the Github project homepage, and click Webhooks & services > Add service and select Jenkins (Git plugin). Enter ${Jenkins IP}/github-webhook/ in the Jenkins hook URL dialog box.

1. http://jenkins.cd****************.cn-beijing.alicontainer.com/github-webhook/
10

Add a build step of Executes shell type and write shell scripts to execute the test.

11

The command in this example is as follows.

1. pwd
2. ls
3. cd chapter2
4. npm test

Create a project to automatically build and push images.

  1. Create an item and choose to build a software project of free style.
  2. Enter the project name and select a node for running the project. In this example, enter the slave-nodejs-ut node created above.
  3. Configure the source code management and code branch. In this example, use GitHub to manage source codes.
  4. Add the following trigger and set it to implement automatic image building only after success of the unit test.
12

Write shell scripts for building and pushing images.

13

The command in this example is as follows.

a.cd chapter2 b.docker build -t registry.aliyuncs.com/qinyujia-test/nodejs-demo . c.docker login -u ${yourAccount} -p ${yourPassword} registry.aliyuncs.com d.docker push registry.aliyuncs.com/qinyujia-test/nodejs-demo 

Automatically Redeploy the Application

Deploy the application for the first time

Use the orchestration template to deploy the image created above to the Container Service and create the nodejs-demo application.

Example

1. 
2. express:
3. image: 'registry.aliyuncs.com/qinyujia-test/nodejs-demo'
4. expose:
5. - '22'
6. - '3000'
7. restart: always
8. labels:
9. aliyun.routing.port_3000: express
10. 

1. Select the application nodejs-demo just created, and create the trigger.

14

 Add a line to the shell scripts you wrote in Realize automatic test and automatic build and push of image. The address is the trigger link given by the trigger created above.

i.curl 'https://cs.console.aliyun.com/hook/trigger?triggerUrl=***==&secret=***' 

Change the Command in the example from Realize automatic test and automatic build and push of image as follows.

i. cd chapter2
ii. docker build -t registry.aliyuncs.com/qinyujia-test/nodejs-demo .
iii. docker login -u ${yourAccount} -p ${yourPassword} registry.aliyuncs.com iv.docker push registry.aliyuncs.com/qinyujia-test/nodejs-demo
v. curl 'https://cs.console.aliyun.com/hook/trigger?triggerUrl=***==&secret=***'

After pushing the image, Jenkins automatically triggers redeployment of the nodejs-demo application.

Configure The Email Notification for the Results

If you want to send the unit test or image configuration results to relevant developers or project execution initiators through email, perform the following configurations.

On the Jenkins homepage, click System Management > System Settings, and configure a Jenkins system administrator email.

15

Install the Extended Email Notification plugin, configure SMTP server and other relevant information, and set the default recipient list. See the figure below.

16

The above example shows the parameter settings of the Jenkins application system. The following example shows the relevant configurations for Jenkins projects whose results are to be pushed through email.

1. Add post-building operation steps in the Jenkins project, select Editable Email Notification, and enter a recipient list.

17

2. Add a mailing trigger.

18

Cheers

Osama

Set Up Your First Server on Alibaba Cloud

I posted before about AWS, Azure and OCI but this time i am trying to cover Alibaba , in this post i will show you how to setup your first Compute instance whatever OS you want CentOS or Ubuntu depeds on what you want

What you need ?

  • A valid Alibaba Cloud account., create free one with 300$ credit
  • An ECS instance running CentOS 7 or whatever OS you want.
  • A root password for your server.

Connect to your Alibaba Cloud

Locate the Internet IP address (Public IP address) associated with your Alibaba Cloud ECS Instance.

If you are running Linux or Mac, use a terminal application to connect to the instance via SSH. If you are on Windows, you can use PuTTy.

Change the Hostname

The hostname is a default identifier when you communicate to a Linux server. It is like a computer name that is associated with your home PC or laptop. Naming your CentOS 7 server with a descriptive hostname helps you to differentiate your machines especially if you are running a bunch of them.

check the update

$ sudo yum update

check your hostname

$ hostname

change your hostname, we need to install nano text editor using the command below:

$ sudo yum install nano

Then, edit the /etc/cloud/cloud.cfg file and find the entry preserve_hostname. Change its value from false to true.

$ sudo nano /etc/cloud/cloud.cfg
preserve_hostname true

Then, edit the /etc/hostname file using a nano editor by typing the command below:

$ sudo nano /etc/hostname

You will need to add two entries on this file just below the 127.0.0.1 localhost entry. The first entry you are adding uses the loopback interface address 127.0.1.1. Please note that this is different from the address 127.0.0.1 which have a ‘localhost’ value in the same file.

Reboot your Alibaba Cloud ECS instance for the changes to take effect by typing the command below:

$ sudo reboot

Also you can create non root user by

$ sudo adduser

For instance, to add a user identified as osama on your server, use the command below:

$ sudo adduser osama

Next, we assign a password to the user we have created above:

$ sudo passwd osama

now assign the administrative task to this user

$ sudo gpasswd -a osama wheel

Cheers

Osama

Create a Serverless Website with Alibaba Cloud Function Compute

Regarding to Wikipedia, Serverless computing is a cloud computing execution model in which the cloud provider runs the server, and dynamically manages the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity

Today i will show you an example how to create serverless website but this time not using Amazon AWS, Azure or OCI but Alibaba Cloud Provider.

Create a Function Compute Service

Go to the console page and click through to Function Compute.

Click the add button beside Services.

In the Service slide out, give your service a name, an optional description, and then slide open the Advanced Settings.

In Advanced Settings you can grant access for Functions to the Internet, to VPC resources, and you can attach storage and a log service to a Function. You can also configure roles.

For our tutorial, we will need Internet access so make sure this configuration is on.

We will leave VPC and Log Configs as they are.

In the Role Config section, select Create New Role, and in the dropdown list pick AliyunOSSReadOnlyAccess as we will be accessing our static webpages from an Object Storage Service bucket.

Click Authorize.

You will see a summary of the Role you created.

Click Confirm Authorization Policy.

You have successfully added the Role to the Service.

Click OK.

ou will see the details of the Function Compute Service you just created.

Now let’s create a Function in the Service. Click the add button next to Functions.

You will see the Create Function process. The first part of the process is Function Template.

There are many Function Templates available, including an empty Function for writing your own bespoke Functions.

Alibaba Cloud-supplied Template Functions are very useful as they have relevant method invocation and demo code for getting started quickly with Function Compute.

let’s choose the flask-web Function written in Python2.7.

Click Select.

We are now at the Configure Triggers section of creating a Function.

Select HTTP Trigger from the dropdown list. Give the Trigger a name and choose Authorization details (anonymous does not require authorization).

Choose your HTTP methods and click Next. We are going to build a simple web-form application so we will need both the GET and POST HTTP methods.

Now we arrive at the Configure Function Settings.

Give the Function a name then scroll down to Code details.

We’ll leave the supplied code for now. Scroll down to below the code sample.

You will see Environment Variable input options and Runtime Environment details.

Click Next.

Click Next at Configure Function Permissions.

Verify the Configuration details and click Create.

You will arrive at the Function’s IDE. Here you can enter new code, edit the code directly, upload code folders, run, test, and fix your code.

Scroll down.

Copy the URL as we will need to add this to our static webpages so they can connect to our Function Compute Service and Function.

Set Up and Configure an OSS Bucket

Click through to Object Storage Service on the Products page.

If you haven’t yet activated Object Storage Service, go ahead and activate it. In the OSS console, click Create Bucket.

Choose a name for the OSS Bucket and pick the region – you cannot change the region later. Select the Storage Class – you also cannot change this later.

We have selected Public Read for the Access Control List.

When you’re ready, click OK.

You will see the Overview page for your bucket. Make a note of the public Internet URL.

In the Files tab, upload your static web files.

I uploaded a simple index.html homepage and a background picture.

<script type="text/javascript">
        const functionURL = '<<Function URL>>';
        const doHome = new XMLHttpRequest();
doHome.open('GET', functionURL, true);
doHome.onload = function () {    
document.getElementById('home_message').innerHTML = doHome.responseText;
        };
        doHome.send();
</script>

In Basic Settings, click Configure to configure your Static Pages.

Add the homepage details and click Save.

Now go to a new browser window and access the OSS URL you saved earlier.

Back in the Function Compute console, you can now test the flask-app paths directly from the code.

We already tested index.html with no Path variable. Next, we test the app route signin with GET and check the Headers and status code.

The signin page code is working correctly. You can also check the Body to make sure the correct HTML will render on the page. Notice that because I entered the path variable, signin is appended to the URL.

Of course, any errors you encounter will show up in the Logs section for easy debugging.

Now, let’s test this page on the Internet.

If you get an error here, implement a soft link for the page in OSS. Go to the OSS bucket and click More dropdown for the HTML file in question and choose Set soft link.

Give the link a name and click OK.

A link file will appear in the list of static files and you will now be able to access the page online with the relevant soft link and it will render as above.

Back in Function Compute, we can test the POST method in the console with the correct username and password details in the same way.

Add the POST variables to the form upload section in the Body tab.

Now you can test this function online.

Cheers

Osama

Alibaba cloud services

I had chance to test alibaba cloud and so far i found it beyond my expectations, i wrote before this blog about it in small introdtion here.

In this one i will write about the services and what this cloud provide :-

Elastic Computing

  • ELASTIC COMPUTE SERVICE (ECS) : Secure, scalable, and high-performing virtual servers.
  • Elastic GPU Service : Powerful parallel computing capabilities
  • Server Load Balancer (SLB) : Distribute traffic across servers
  • Elastic Container Instance: An agile and secure serverless container instance service.
  • ECS Bare Metal Instance : Elastic bare metal computing service
  • Simple Application Server : All-in-one services for fast deployment

Network

  • Virtual Private Cloud Launch a private cloud network
  • Server Load Balancer Distribute traffic across resources
  • NAT Gateway Public Internet gateway
  • Alibaba Cloud PrivateZone A private domain name management service
  • Elastic IP An independent public IP resource
  • Data Transfer Plan Package for your Internet traffic usage
  • Alibaba Cloud CDN Speed up file distribution to end-users
  • Dynamic Route for CDN Provide dynamic acceleration abilities

Storage

  • Object Storage Service Store, backup and archive your data
  • Elastic Block Storage A high-performance, high-reliability, and low-latency block storage service
  • Apsara File Storage NAS File storage for ECS, HPC, Container

Media Services

  • ApsaraVideo for Media Processing A transcoding multimedia cloud service
  • ApsaraVideo VOD An all-in-one VOD solution
  • ApsaraVideo Live Live streaming platform for audio & video

Database

  • ApsaraDB for PolarDB – New generation of “Super MySQL, PostgreSQL or compatible Oracle” supporting up to 100 TB storage.
  • Distributed Relational Database Service Be scalable for big relational databases.
  • ApsaraDB RDS for MariaDB TX Capable and fully managed database
  • ApsaraDB RDS for SQL Server Flexible version control and lower costs
  • ApsaraDB RDS for MySQL Featured stable and capable with MySQL
  • ApsaraDB RDS for PostgreSQL Get low latency with high concurrency
  • PolarDB Box An ApsaraDB for PolarDB database appliance deployed on the premises
  • ApsaraDB for OceanBase Financial-grade database: high stability, high scalability, and high performance
  • ApsaraDB RDS for PPAS Use simple tools to lower migration cost
  • Distributed Relational Database Service Be scalable for big relational databases

Security

  • Security Center Round the clock security and protection
  • Bastionhost A system O&M and operation audit platform
  • Cloud Firewall The first defense of network security
  • ActionTrail Maintain security and compliance
  • Anti-DDoS Basic Basic DDoS protection integrated with ECS
  • SSL Certificates Service Apply, buy, and manage SSL certificates
  • Web Application Firewall Secure your web applications
  • Cloud Config Configuration trace and compliance audit
  • Anti-DDoS Protect against high volume DDoS attacks

Analytics

  • Hologres (Interactive Analytics) Real-time analytics compatible with PostgreSQL
  • E-MapReduce Big Data processing service
  • Realtime Compute A platform for real-time data processing
  • MaxCompute Large-scale data warehousing
  • DataV Insightful big data visualization
  • Elasticsearch Search and analytics with Elasticsearch
  • Log Service An all-in-one service for log-type data

Artificial Intelligence

  • Intelligent Speech Interaction Speech recognition & synthesis platform
  • Machine Translation Bespoken and qualified machine translation
  • Machine Learning Platform For AI An end-to-end machine learning platform
  • Image Search High-precision visual search solution

Enterprise Applications

  • WHOIS Search WHOIS database to look up domain
  • Domains Join the 20 million on Alibaba Cloud
  • Web Hosting Cloud Web Hosting from $5.90 per month
  • Alibaba Cloud DNS Manage your DNS settings
  • Intelligent Robot Chatbot platform for smart interactions
  • Alibaba Mail A public cloud-based email service
  • Direct Mail A simple and efficient email service
  • API Gateway Open your data and service to users

Internet of Things

  • Tracing Analysis Easily debug and analyze applications
  • Application Real-Time Monitoring Service Build business monitoring capabilities
  • CloudMonitor Real-time cloud monitoring service
  • Application High Availability Service Guarantees application high availability

I tried to brief some of the common products for Alibaba cloud, not all the services listed here, you can back to my blog post here.

Cheers
Osama

Alibaba Cloud – Small Introduction

I had chance to work and test alibaba cloud, so i thought it’s good idea to write something about it since i already used AWS, Azure and OCI and this is will be my 4th cloud vendor.

Alibaba Cloud is the subsidiary of the e-commerce hub Alibaba Group. The group launched its cloud services in 2009. Today, cloud is the most ambitious project of Alibaba Group where they are investing their hard efforts to win over AWS.

The company has an exclusive range of cloud computing products and services that are divided into 7 categories of Elastic Computing and Networking, Security and Management, Database, Application Services, Domains and website, Storage and CDN and Analytics. Customers of Alibaba Cloud are eligible to get the benefits of cloud security, record breaking computing power, cloud security, safeguard your data, etc.

I really like the cloud and the portal, it’s very simple and ease of use, include to this, having a lot of different features same as AWS, you can check them from here.

the alibaba cloud known as different name also, Aliyun, Alibaba Cloud has 19 regional data centres globally, including China North, China South, China East, US West, US East, Europe, United Kingdom, Middle East, Japan, Hong Kong, Singapore, Australia, Malaysia, India, and Indonesia, right now the Data Center in Germany is operated by Vodafone Germany

Some of the clients that using this cloud : Ford, Air Aisa, Lazada, and more.

Some of the services that providing by alibaba:-

  • Elastic Computing
  • Storage & CDN
  • Networking
  • Database Services
  • Security

and will discuss each one of them in different post, the next one will be alibaba services.

Cheers

Osama