Home >Backend Development >Python Tutorial >Develop a system that monitors a log file located on a remote server, similar to the Unix command tail -f.
Develop a system that monitors a log file located on a remote server, similar to the Unix command tail -f.
- Barbara StreisandOriginal
- 2024-09-28 22:11:02844browse
The Objective of this problem is:
The Objective is to develop a system that monitors a log file located on a remote server, similar to the Unix command tail -f. The log file is being continuously appended with new data. This system should consist of:
A server application that tracks the ongoing changes to a specified log file situated on the same server. This application should be capable of transmitting the newly added data in real-time to clients.
A web-based client interface, reachable via a URL (e.g., http://localhost/log), designed to display log file updates dynamically as they occur, without requiring the user to reload the page. Initially, upon visiting the page, users should see the most recent 10 lines from the log file.
Also Handled the following scenarios:The server must actively push updates to the clients to ensure minimal delay, achieving as close to real-time updates as possible.
Given that the log file might be very large (potentially several gigabytes), you'll need to develop a strategy for efficiently fetching the last 10 lines without processing the entire file.
The server should transmit only new additions to the file to the clients, rather than resending the file in its entirety.
It's essential that the server supports concurrent connections from multiple clients without performance degradation.
The client's web page should load promptly without staying in a loading state after the initial request, and it should not require reloading to display new updates.
I have created a Flask application with a simple UI that displays the last 10 messages.
i have used the flask-socketio to form a connection also used the some basic concepts of handling files like fileObj.seek(), fileObj.tell() etc.
from flask import Flask, render_template
from flask_socketio import SocketIO, emit
from threading import Lock
app = Flask(__name__)
socketio = SocketIO(app)
thread = None
thread_lock = Lock()
LOG_FILE_PATH = "./static/client.txt"
last_position = 0
position_lock = Lock()
@app.route('/')
def index():
return render_template('index.html')
@socketio.on('connect')
def test_connect():
global thread
with thread_lock:
if thread is None:
print("started execution in background!")
thread = socketio.start_background_task(target=monitor_log_file)
def monitor_log_file():
global last_position
while True:
try:
with open(LOG_FILE_PATH, 'rb') as f:
f.seek(0, 2)
file_size = f.tell()
if last_position != file_size:
buffer_size = 1024
if file_size < buffer_size:
buffer_size = file_size
f.seek(-buffer_size, 2)
lines = f.readlines()
last_lines = lines[-10:]
content = b'\n'.join(last_lines).decode('utf-8')
socketio.sleep(1) # Add a small delay to prevent high CPU usage
socketio.emit('log_updates', {'content': content})
print("Emitted new Lines to Client!")
last_position = file_size
else:
pass
except FileNotFoundError:
print(f"Error: {LOG_FILE_PATH} not found.")
except Exception as e:
print(f"Error while reading the file: {e}")
if __name__ == '__main__':
socketio.run(app, debug=True, log_output=True, use_reloader=False)
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Basics</title>
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.5.4/socket.io.js"></script>
</head>
<body>
<h1>User Updated Files Display it over here:</h1>
<div id="output"></div>
<script>
var socket = io("http://127.0.0.1:5000");
socket.on('connect', function() {
console.log('Connected to the server');
});
socket.on('disconnect', function() {
console.log('Client disconnected');
});
socket.on('log_updates', function(data) {
console.log("data", data);
var div = document.getElementById('output');
var lines = data.content.split('\n');
div.innerHTML = '';
lines.forEach(function(line) {
var p = document.createElement('p');
p.textContent = line;
div.appendChild(p);
});
});
</script>
</body>
</html>
Also create a client.log file under static folder in flask application.
Please feel free to correct me if I did something wrong. Comment below with any corrections!
The above is the detailed content of Develop a system that monitors a log file located on a remote server, similar to the Unix command tail -f.. For more information, please follow other related articles on the PHP Chinese website!