Detailed explanation of python concurrent acquisition of snmp information and performance testing methods

This article mainly introduces the detailed explanationpythonConcurrent acquisition of snmp information and performance testing. The editor thinks it is quite good, so I will share it with you now and give it as a reference. Let’s follow the editor and take a look.

python & snmp

There are many ready-made libraries that can be used to obtain snmp information using python, among which the more commonly used ones arenetsnmp and pysnmp are two libraries. There are many examples of the two libraries on the Internet.

The focus of this article is how to obtain snmp data concurrently, that is, obtain snmp information from multiple machines at the same time.

netsnmp

Let’s talk about netsnmp first. Python's netsnmp actually comes from the net-snmp package.

Python calls the net-snmp interface to obtain data through a c file.

Therefore, when acquiring multiple machines concurrently, coroutines cannot be used to acquire them. Because coroutines are used, when getting data, the coroutines will always wait for the net-snmp interface to return data, instead of switching the CPU to other coroutines while waiting for data like when using sockets. From this point of view, there is no difference between using coroutines and serial fetching.

So how to solve the problem of concurrent acquisition? You can use threads and multi-thread acquisition (of course you can also use multi-process). Multiple threads call the net-snmp interface to obtain data at the same time, and then the CPU continuously switches between multiple threads. After a thread obtains a result, it can continue to call the interface to obtain the next snmp data.

Here I wrote a sample program. First, make all hosts and oids into tasks and put them in the queue, and then start multiple threads to perform the acquisition task. The program sample is as follows:

import threading
import time
import netsnmp
import Queue
start_time = time.time()
hosts = ["192.20.150.109", "192.20.150.110", "192.20.150.111", "192.20.150.112", "192.20.150.113", "192.20.150.114",
     "192.20.150.115", "192.20.150.116", "192.20.150.117", "192.20.150.118", "192.20.150.119", "192.20.150.120",
     "192.20.150.121", "192.20.80.148", "192.20.80.149", "192.20.96.59", "192.20.82.14", "192.20.82.15",
     "192.20.82.17", "192.20.82.19", "192.20.82.12", "192.20.80.139", "192.20.80.137", "192.20.80.136",
     "192.20.80.134", "192.20.80.133", "192.20.80.131", "192.20.80.130", "192.20.81.141", "192.20.81.140",
     "192.20.82.26", "192.20.82.28", "192.20.82.23", "192.20.82.21", "192.20.80.128", "192.20.80.127",
     "192.20.80.122", "192.20.81.159", "192.20.80.121", "192.20.80.124", "192.20.81.151", "192.20.80.118",
     "192.20.80.119", "192.20.80.113", "192.20.80.112", "192.20.80.116", "192.20.80.115", "192.20.78.62",
     "192.20.81.124", "192.20.81.125", "192.20.81.122", "192.20.81.121", "192.20.82.33", "192.20.82.31",
     "192.20.82.32", "192.20.82.30", "192.20.81.128", "192.20.82.39", "192.20.82.37", "192.20.82.35",
     "192.20.81.130", "192.20.80.200", "192.20.81.136", "192.20.81.137", "192.20.81.131", "192.20.81.133",
     "192.20.81.134", "192.20.82.43", "192.20.82.45", "192.20.82.41", "192.20.79.152", "192.20.79.155",
     "192.20.79.154", "192.25.76.235", "192.25.76.234", "192.25.76.233", "192.25.76.232", "192.25.76.231",
     "192.25.76.228", "192.25.20.96", "192.25.20.95", "192.25.20.94", "192.25.20.93", "192.24.163.14",
     "192.24.163.21", "192.24.163.29", "192.24.163.6", "192.18.136.22", "192.18.136.23", "192.24.193.2",
     "192.24.193.19", "192.24.193.18", "192.24.193.11", "192.20.157.132", "192.20.157.133", "192.24.212.232",
     "192.24.212.231", "192.24.212.230"]
oids = [".1.3.6.1.4.1.2021.11.9.0",".1.3.6.1.4.1.2021.11.10.0",".1.3.6.1.4.1.2021.11.11.0",".1.3.6.1.4.1.2021.10.1.3.1",
    ".1.3.6.1.4.1.2021.10.1.3.2",".1.3.6.1.4.1.2021.10.1.3.3",".1.3.6.1.4.1.2021.4.6.0",".1.3.6.1.4.1.2021.4.14.0",
    ".1.3.6.1.4.1.2021.4.15.0"]
myq = Queue.Queue()
rq = Queue.Queue()
#把host和oid组成任务
for host in hosts:
  for oid in oids:
    myq.put((host,oid))
def poll_one_host():
  while True:
    try:
      #死循环从队列中获取任务，直到队列任务为空
      host, oid = myq.get(block=False)
      session = netsnmp.Session(Version=2, DestHost=host, Community="cluster",Timeout=3000000,Retries=0)
      var_list = netsnmp.VarList()
      var_list.append(netsnmp.Varbind(oid))
      ret = session.get(var_list)
      rq.put((host, oid, ret, (time.time() - start_time)))
    except Queue.Empty:
      break
thread_arr = []
#开启多线程
num_thread = 50
for i in range(num_thread):
  t = threading.Thread(target=poll_one_host, kwargs={})
  t.setDaemon(True)
  t.start()
  thread_arr.append(t)
#等待任务执行完毕
for i in range(num_thread):
  thread_arr[i].join()
while True:
  try:
    info = rq.get(block=False)
    print info
  except Queue.Empty:
    print time.time() - start_time
    break

In addition to supporting get operations, netsnmp also supports walk operations, that is, traversing an oid.

But you need to be careful when using walk to avoid problems such as high latency. For details, please refer to a previous blog on snmpwalk high latency problem analysis.

pysnmp

pysnmp is a set of snmp protocol libraries implemented in python. It itself provides support for asynchronous.

import time
import Queue
from pysnmp.hlapi.asyncore import *
t = time.time()
myq = Queue.Queue()
#回调函数。在有数据返回时触发
def cbFun(snmpEngine, sendRequestHandle, errorIndication, errorStatus, errorIndex, varBinds, cbCtx):
   myq.put((time.time()-t, varBinds))
hosts = ["192.20.150.109", "192.20.150.110", "192.20.150.111", "192.20.150.112", "192.20.150.113", "192.20.150.114",
     "192.20.150.115", "192.20.150.116", "192.20.150.117", "192.20.150.118", "192.20.150.119", "192.20.150.120",
     "192.20.150.121", "192.20.80.148", "192.20.80.149", "192.20.96.59", "192.20.82.14", "192.20.82.15",
     "192.20.82.17", "192.20.82.19", "192.20.82.12", "192.20.80.139", "192.20.80.137", "192.20.80.136",
     "192.20.80.134", "192.20.80.133", "192.20.80.131", "192.20.80.130", "192.20.81.141", "192.20.81.140",
     "192.20.82.26", "192.20.82.28", "192.20.82.23", "192.20.82.21", "192.20.80.128", "192.20.80.127",
     "192.20.80.122", "192.20.81.159", "192.20.80.121", "192.20.80.124", "192.20.81.151", "192.20.80.118",
     "192.20.80.119", "192.20.80.113", "192.20.80.112", "192.20.80.116", "192.20.80.115", "192.20.78.62",
     "192.20.81.124", "192.20.81.125", "192.20.81.122", "192.20.81.121", "192.20.82.33", "192.20.82.31",
     "192.20.82.32", "192.20.82.30", "192.20.81.128", "192.20.82.39", "192.20.82.37", "192.20.82.35",
     "192.20.81.130", "192.20.80.200", "192.20.81.136", "192.20.81.137", "192.20.81.131", "192.20.81.133",
     "192.20.81.134", "192.20.82.43", "192.20.82.45", "192.20.82.41", "192.20.79.152", "192.20.79.155",
     "192.20.79.154", "192.25.76.235", "192.25.76.234", "192.25.76.233", "192.25.76.232", "192.25.76.231",
     "192.25.76.228", "192.25.20.96", "192.25.20.95", "192.25.20.94", "192.25.20.93", "192.24.163.14",
     "192.24.163.21", "192.24.163.29", "192.24.163.6", "192.18.136.22", "192.18.136.23", "192.24.193.2",
     "192.24.193.19", "192.24.193.18", "192.24.193.11", "192.20.157.132", "192.20.157.133", "192.24.212.232",
     "192.24.212.231", "192.24.212.230"]
oids = [".1.3.6.1.4.1.2021.11.9.0",".1.3.6.1.4.1.2021.11.10.0",".1.3.6.1.4.1.2021.11.11.0",".1.3.6.1.4.1.2021.10.1.3.1",
    ".1.3.6.1.4.1.2021.10.1.3.2",".1.3.6.1.4.1.2021.10.1.3.3",".1.3.6.1.4.1.2021.4.6.0",".1.3.6.1.4.1.2021.4.14.0",
    ".1.3.6.1.4.1.2021.4.15.0"]
    
snmpEngine = SnmpEngine()
#添加任务
for oid in oids:
  for h in hosts:
    getCmd(snmpEngine,
      CommunityData('cluster'),
      UdpTransportTarget((h, 161), timeout=3, retries=0,),
      ContextData(),
      ObjectType(ObjectIdentity(oid)),
      cbFun=cbFun)
time1 = time.time() - t
#执行异步获取snmp
snmpEngine.transportDispatcher.runDispatcher()
#打印结果
while True:
  try:
    info = myq.get(block=False)
    print info
  except Queue.Empty:
    print time1
    print time.time() - t
    break

pysnmp itself only supports the most basic get and getnext commands, so if you want to use walk, you need to implement it yourself.

Performance test

The performance test was conducted on both in the same environment. The two collected 198 hosts and 10 oids.

Test group	Time consuming (sec)
netsnmp(20 threads )	6.252
netsnmp(50 threads)	3.269
netsnmp(200 threads)	3.265
pysnmp	4.812

##You can see the collection speed and thread of netsnmp Number related. When the number of threads increases to a certain extent, the collection time is no longer shortened. Because opening threads also consumes time. The existing threads are enough to handle.

The performance of pysnmp is slightly worse. Detailed analysis of pysnmp consumes about 1.2 seconds when adding tasks (when executing getCmd), and subsequent collection consumes about 3.3 seconds.

Increasing the number of oids and conducting experiments. There are still 198 hosts and 42 oids.

Test groupTime consuming (sec)netsnmp(20 threads )30.935netsnmp(50 threads)12.914netsnmp(200 threads)4.044##pysnmp##It can be seen that the gap is further widened. When there are enough threads, netsnmp is significantly more efficient than pysnmp.

11.043

Because both support the parallel collection of multiple hosts, netsnmp is simpler in terms of ease of use, and netsnmp supports the walk function. This article recommends netsnmp.

Installation

netsnmp requires the installation of net-snmp. If centos, it will be more convenient to use yum.

The above is the detailed content of Detailed explanation of python concurrent acquisition of snmp information and performance testing methods. For more information, please follow other related articles on the PHP Chinese website!