1. 程式人生 > >[python] 連接MySQL,以及多線程、多進程連接MySQL續

[python] 連接MySQL,以及多線程、多進程連接MySQL續

python mysqldb dbutils pooleddb

之前參照他人的做法,使用DBUtils.PooledDB來建立多個可復用的MySQL連接,部分文章有誤,方法不當,導致我走了很多彎路,專研幾天後,終於找到了正確的使用方法。


網上有很多使用DBUtils.PooledDB模塊建立連接池,再加threading多線程連接MySQL的例子,不僅沒有告訴讀者如何驗證是否為多線程跑,而且大多是使用cursor()來建立多線程連接,這是錯誤的。(驗證是否為多線程方法請見文章最後)

使用cursor()來建立多線程連接,在執行SQL查詢的時候會報錯:2014, "Commands out of sync; you can‘t run this command now"

究其原因是:

1、查詢結果未釋放,然又執行查詢

2、兩次查詢之間沒有存儲結果應該重復使用connection(),而不是cursor()

Mysql文檔:Commands out of sync

If you get Commands out of sync; you can‘t run this command now in your client code, you are calling client functions in the wrong order.

This can happen, for example, if you are using mysql_use_result() and try to execute a new query before you have called mysql_free_result().

It can also happen if you try to execute two queries that return data without calling mysql_use_result() or mysql_store_result() in between.


DBUtils.PooledDB模塊裏關於這兩個函數的說明如下:


def connection(self, shareable=True):

"""Get a steady, cached DB-API 2 connection from the pool.


If shareable is set and the underlying DB-API 2 allows it,

then the connection may be shared with other threads.


"""

def cursor(self, *args, **kwargs):

"""Return a new Cursor Object using the connection."""

return SteadyDBCursor(self, *args, **kwargs)


所以要並發連接MySQL執行SQL,應該是使用connect(),而不是使用cursor。


多線程連接MySQL.demo

#!/usr/bin/env python
# -*- coding:utf-8 -*-
import MySQLdb
import threading
import time
from DBUtils.PooledDB import PooledDB

‘‘‘
建立連接池,返回連接池地址
‘‘‘
def dbpool(ip,port,username,password,dbname,char_set=‘utf8‘):
    connKwargs = {‘host‘: ip, ‘port‘: port, ‘user‘: username, ‘passwd‘: password, ‘db‘: dbname,‘charset‘: char_set}
    pool = PooledDB(MySQLdb, mincached=10, maxcached=10, maxshared=10, maxconnections=10, **connKwargs)
    return pool

‘‘‘
從連接池中取出一個連接,執行SQL
num:用於統計總的影響行數
‘‘‘
num=0
def dbconnect(db_pool):
    global num
    conn = db_pool.connection()
    cur = conn.cursor()
    try:
        cur.execute("select * from table_name")
        lock.acquire()
        num += cur.rowcount
        lock.release()
    except Exception as e:
        print e
    finally:
        cur.close()
        conn.close()

if __name__ == ‘__main__‘:
	‘‘‘
	lock:生成全局鎖,用於執行語句中,被影響行數的統計值加鎖使用,每次只允許一個線程修改被鎖變量
	‘‘‘
    lock = threading.Lock()
    st = time.time()
    db_pool = dbpool(ip=‘127.0.0.1‘,port = 1234,username=‘root‘,password=‘1234‘,dbname =‘try‘,char_set=‘utf8‘)
    ‘‘‘
    同時連接MySQL執行的線程數要小於等於前面PooledDB中設置的maxconnections,如果大於這個量,會報異常:TooManyConnections。
    設置每次只跑10個線程,跑完後再循環。
    ‘‘‘
    thread_list = []
    for i in range(100):
        t = threading.Thread(target=dbconnect,args=(db_pool,))
        thread_list.append(t)
    while len(thread_list)!=0:
        if len(thread_list)>10:
            thread_list_length = 10
        else:
            thread_list_length = len(thread_list)
        sub_thread_list = []
        for n in range(thread_list_length):
            sub_thread_list.append(thread_list[0])
            thread_list.remove(thread_list[0])
        for i in sub_thread_list:
            i.start()
        for j in sub_thread_list:
            j.join()
    et = time.time()
    print et - st,num


檢查程序是否為多線程在跑:

1、可以在執行主機上,通過”pstree -p PID“查看執行的進程數

2、在MySQL中查看連接數及SQL執行情況:select * from information_schema.processlist where host like ‘192.168.1.1%‘ order by 1

[python] 連接MySQL,以及多線程、多進程連接MySQL續