Commit f909dad3 authored by phuong's avatar phuong

Update Performance Monitoring

parent d2d3e3c3
# Client Library
For monitoring the metrics of your program, for instance, a python program, you can client api provided by prometheus to log all the metrics. Then you should export the information to an endpoint such as a http port (8000) or a web application port (5000), etc. You can follow the instruction below to practice monitor your python program using prometheus.
## Install Prometheus
Go to the download page and download the corresponding prometheus for your system.
......@@ -8,14 +9,6 @@ user@test:~$ wget https://github.com/prometheus/prometheus/releases/download/v2.
```
## Select the language and corresponding client library. Forexample: Python
```console
user@test:~$ python sample2.py
```
## Configure prometheus to monitor the python program sample2.py
Extract the folder prometheus-version.targ.gz and go inside the folder. Then perform the modifications such as follows:
......@@ -32,7 +25,6 @@ scrape_configs:
```
You can also modify the scrape_interval to an arbitrary number upon your demand. Then you can run prometheus
```console
......@@ -40,8 +32,24 @@ user@test:~$ ./prometheus
```
You now go to prometheus http://yourhost:9090, and search for "hello_world_created". You should see now the value created by your program in prometheus.
## Select the language and corresponding client library. Forexample: Python
After configuring prometheus and run it. You can start to run your python program:
```console
user@test:~$ python sample2.py
```
Prometheus will collect all the metrics of your program. Then you now can go to prometheus http://yourhost:9090, and search for the collected metrics, for example, the "hello_world_created". You should see now the value created by your program in prometheus. The sample3.py is an extension of sample2.py, where we add a machine learning pipeline to the code instead of printing only the hello world text. This is an example that let you know how to monitor your code using prometheus.
In order to run the code, you do the same as the aboved command:
```console
user@test:~$ python sample3.py
```
Now, you can go to the local address http://localhost:8001 to see the accuracy of the pipeline. Check in the prometheus, you will see the performance of your pipeline.
# Prometheus
import http.server
from prometheus_client import start_http_server
from prometheus_client import Summary
LATENCY = Summary('pipeline_perf_seconds', 'Time for a request the pipeline.')
class myFloat( float ):
def __str__(self):
return "%.12f"%self
class MyHandler(http.server.BaseHTTPRequestHandler):
@LATENCY.time()
def do_GET(self):
# Set starting time
start = time.time()
# 1. Add dataset
filename = 'pima-indians-diabetes.data.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
# 2. Processing data
dataframe = read_csv(filename, names=names)
array = dataframe.values
X = array[:, 0:8]
Y = array[:, 8]
test_size = 0.33
seed = 7
# 3. Training the data
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=test_size, random_state=seed)
model = LogisticRegression()
model.fit(X_train, Y_train)
# 4. Get the result
result = model.score(X_test, Y_test)
# print("Accuracy: %.3f%%" % (result * 100.0))
x = myFloat(result * 100.0)
message = str(x)
# 5. Send the result to the web
self.send_response(200)
self.end_headers()
self.wfile.write(b"Pipeline Accuracy:\n")
self.wfile.write(message.encode())
# 6. Get the performance of the pipeline
LATENCY.observe(time.time() - start)
if __name__ == "__main__":
start_http_server(8000)
server = http.server.HTTPServer(('localhost', 8001), MyHandler)
server.serve_forever()
# rate(pipeline_perf_seconds_count[5m])
\ No newline at end of file
# Node Exporter
The node exporter is a built-in program that can be used to extract information of your machine. For using a node exporter, you can follow the instruction below:
## Installation
First, user should download the exporter. Select the exporter for your system and execute it like a binary file.
......@@ -9,7 +10,7 @@ user@test:~$ wget https://github.com/prometheus/node_exporter/releases/download/
```
Go to the path "http://yourhost:9100/metrics" to verify that the node exporter is correctly up and running.
Go to the path "http://yourhost:9100/metrics" to verify that the node exporter is correctly up and running. Notably, you should select the right version of node exporter for your machine, for instance, linux, window, etc. After having the node exporter for your machine, you have to configure Prometheus and Grafana.
## Prometheus Configuration
......@@ -19,7 +20,7 @@ Go to the download page and download the corresponding prometheus for your syste
user@test:~$ wget https://github.com/prometheus/prometheus/releases/download/v2.16.0-rc.0/prometheus-2.16.0-rc.0.linux-amd64.tar.gz
```
Extract the folder prometheus-version.targ.gz and go inside the folder. Then perform the modifications such as follows:
Extract the folder prometheus-version.targ.gz and go inside the folder. Then perform the modifications for the prometheus.yml file such as follows:
```properties
static_configs:
- targets: ['localhost:9090', 'localhost:9100']
......@@ -32,7 +33,7 @@ user@test:~$ ./prometheus
```
## Grafana
head over to https://grafana.com/grafana/download and download the corresponding binary file for your system.
head over to https://grafana.com/grafana/download and download the corresponding binary file for your system. Grafana will be used to visualize the data collected by prometheus.
```console
user@test:~$ wget https://dl.grafana.com/oss/release/grafana-6.6.0.linux-amd64.tar.gz
......
# Pushgateway
The Pushgateway is another type of built-in program that you can send the collected metrics to it. For using a pushgateway, you can follow the instruction below:
## Installation
Head over to the download page and download the pushgateway for your system.
......@@ -7,7 +8,7 @@ Head over to the download page and download the pushgateway for your system.
user@test:~$ wget https://github.com/prometheus/pushgateway/releases/download/v1.1.0/pushgateway-1.1.0.linux-amd64.tar.gz
```
Go to the path "http://yourhost:9100/metrics" to verify that the node exporter is correctly up and running.
Go to the path "http://yourhost:9100/metrics" to verify that the pushgateway is correctly up and running.
## Prometheus Configuration
Go to the download page and download the corresponding prometheus for your system.
......@@ -16,12 +17,12 @@ Go to the download page and download the corresponding prometheus for your syste
user@test:~$ wget https://github.com/prometheus/prometheus/releases/download/v2.16.0-rc.0/prometheus-2.16.0-rc.0.linux-amd64.tar.gz
```
Extract the folder prometheus-version.targ.gz and go inside the folder. Then perform the modifications such as follows:
Extract the folder prometheus-version.targ.gz and go inside the folder. Then perform the modifications for the prometheus.yml file such as follows:
```properties
static_configs:
- targets: ['localhost:9090', 'localhost:9091']
```
You can also modify the scrape_interval to an arbitrary number upon your demand. Notably, the port number of pushgateway is different to the exporter's, so you have to change the port for it correctly.
You can also modify the scrape_interval to an arbitrary number upon your demand. Notably, the port number of pushgateway is different to the exporter's, so you have to change the port so prometheus can collect information from it correctly.
```console
user@test:~$ ./prometheus
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment