[python] 連接MySQL,以及多線程、多進程連接MySQL續
之前參照他人的做法,使用DBUtils.PooledDB來建立多個可復用的MySQL連接,部分文章有誤,方法不當,導致我走了很多彎路,專研幾天後,終於找到了正確的使用方法。
網上有很多使用DBUtils.PooledDB模塊建立連接池,再加threading多線程連接MySQL的例子,不僅沒有告訴讀者如何驗證是否為多線程跑,而且大多是使用cursor()來建立多線程連接,這是錯誤的。(驗證是否為多線程方法請見文章最後)
使用cursor()來建立多線程連接,在執行SQL查詢的時候會報錯:2014, "Commands out of sync; you can‘t run this command now"
究其原因是:
1、查詢結果未釋放,然又執行查詢
2、兩次查詢之間沒有存儲結果應該重復使用connection(),而不是cursor()
Mysql文檔:Commands out of sync
If you get Commands out of sync; you can‘t run this command now in your client code, you are calling client functions in the wrong order.
This can happen, for example, if you are using mysql_use_result() and try to execute a new query before you have called mysql_free_result().
It can also happen if you try to execute two queries that return data without calling mysql_use_result() or mysql_store_result() in between.
DBUtils.PooledDB模塊裏關於這兩個函數的說明如下:
def connection(self, shareable=True):
"""Get a steady, cached DB-API 2 connection from the pool.
If shareable is set and the underlying DB-API 2 allows it,
then the connection may be shared with other threads.
"""
def cursor(self, *args, **kwargs):
"""Return a new Cursor Object using the connection."""
return SteadyDBCursor(self, *args, **kwargs)
所以要並發連接MySQL執行SQL,應該是使用connect(),而不是使用cursor。
多線程連接MySQL.demo
#!/usr/bin/env python # -*- coding:utf-8 -*- import MySQLdb import threading import time from DBUtils.PooledDB import PooledDB ‘‘‘ 建立連接池,返回連接池地址 ‘‘‘ def dbpool(ip,port,username,password,dbname,char_set=‘utf8‘): connKwargs = {‘host‘: ip, ‘port‘: port, ‘user‘: username, ‘passwd‘: password, ‘db‘: dbname,‘charset‘: char_set} pool = PooledDB(MySQLdb, mincached=10, maxcached=10, maxshared=10, maxconnections=10, **connKwargs) return pool ‘‘‘ 從連接池中取出一個連接,執行SQL num:用於統計總的影響行數 ‘‘‘ num=0 def dbconnect(db_pool): global num conn = db_pool.connection() cur = conn.cursor() try: cur.execute("select * from table_name") lock.acquire() num += cur.rowcount lock.release() except Exception as e: print e finally: cur.close() conn.close() if __name__ == ‘__main__‘: ‘‘‘ lock:生成全局鎖,用於執行語句中,被影響行數的統計值加鎖使用,每次只允許一個線程修改被鎖變量 ‘‘‘ lock = threading.Lock() st = time.time() db_pool = dbpool(ip=‘127.0.0.1‘,port = 1234,username=‘root‘,password=‘1234‘,dbname =‘try‘,char_set=‘utf8‘) ‘‘‘ 同時連接MySQL執行的線程數要小於等於前面PooledDB中設置的maxconnections,如果大於這個量,會報異常:TooManyConnections。 設置每次只跑10個線程,跑完後再循環。 ‘‘‘ thread_list = [] for i in range(100): t = threading.Thread(target=dbconnect,args=(db_pool,)) thread_list.append(t) while len(thread_list)!=0: if len(thread_list)>10: thread_list_length = 10 else: thread_list_length = len(thread_list) sub_thread_list = [] for n in range(thread_list_length): sub_thread_list.append(thread_list[0]) thread_list.remove(thread_list[0]) for i in sub_thread_list: i.start() for j in sub_thread_list: j.join() et = time.time() print et - st,num
檢查程序是否為多線程在跑:
1、可以在執行主機上,通過”pstree -p PID“查看執行的進程數
2、在MySQL中查看連接數及SQL執行情況:select * from information_schema.processlist where host like ‘192.168.1.1%‘ order by 1
[python] 連接MySQL,以及多線程、多進程連接MySQL續