RocketMQ深度解析(二):NameServer
NamerServer
NameServer是一個非常簡單的Topic路由註冊中心,其角色類似Dubbo中的zookeeper,支援Broker的動態註冊與發現。主要包括兩個功能:Broker管理,NameServer接受Broker叢集的註冊資訊並且儲存下來作為路由資訊的基本資料。然後提供心跳檢測機制,檢查Broker是否還存活;路由資訊管理,每個NameServer將儲存關於Broker叢集的整個路由資訊和用於客戶端查詢的佇列資訊。然後Producer和Conumser通過NameServer就可以知道整個Broker叢集的路由資訊,從而進行訊息的投遞和消費。NameServer通常也是叢集的方式部署,各例項間相互不進行資訊通訊。Broker是向每一臺NameServer註冊自己的路由資訊,所以每一個NameServer例項上面都儲存一份完整的路由資訊。當某個NameServer因某種原因下線了,Broker仍然可以向其它NameServer同步其路由資訊,Producer,Consumer仍然可以動態感知Broker的路由的資訊。
原始碼解析
NameServer啟動流程
填充啟動引數
首先需要建立NameServerConfig(NameServer業務引數)、NettyServerConfig(NameServer網路引數)。
// 首先建立NameServer業務引數
final NamesrvConfig namesrvConfig = new NamesrvConfig();
// 然後建立NameServer網路引數
final NettyServerConfig nettyServerConfig = new NettyServerConfig();
// 預設監聽埠
nettyServerConfig.setListenPort(9876);
// 通過檔案來配置引數
if (commandLine.hasOption('c')) {
String file = commandLine.getOptionValue('c');
if (file != null) {
InputStream in = new BufferedInputStream(new FileInputStream(file));
properties = new Properties();
properties.load(in);
MixAll.properties2Object(properties,namesrvConfig);
MixAll.properties2Object(properties,nettyServerConfig);
namesrvConfig.setConfigStorePath(file);
System.out.printf("load config properties file OK,%s%n" ,file);
in.close();
}
}
// 通過命令列引數
if (commandLine.hasOption('p')) {
InternalLogger console = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_CONSOLE_NAME);
MixAll.printObjectProperties(console,namesrvConfig);
MixAll.printObjectProperties(console,nettyServerConfig);
System.exit(0);
}
MixAll.properties2Object(ServerUtil.commandLine2Properties(commandLine),namesrvConfig);
複製程式碼
//rocketMQ主目錄
private String rocketmqHome = System.getProperty(MixAll.ROCKETMQ_HOME_PROPERTY,System.getenv(MixAll.ROCKETMQ_HOME_ENV));
//儲存KV配置屬性的持久化路徑
private String kvConfigPath = System.getProperty("user.home") + File.separator + "namesrv" + File.separator + "kvConfig.json";
//預設配置檔案路徑,不生效。
private String configStorePath = System.getProperty("user.home") + File.separator + "namesrv" + File.separator + "namesrv.properties";
private String productEnvName = "center";
private boolean clusterTest = false;
// 是否支援順序訊息,預設是不支援。
private boolean orderMessageEnable = false;
複製程式碼
// 監聽埠
private int listenPort = 8888;
//Netty業務執行緒池執行緒個數
private int serverWorkerThreads = 8;
//Netty public任務執行緒數 Netty網路設計,根據不同的業務型別會建立不同的執行緒池,比如處理訊息傳送、訊息消費、心跳檢測等。
//如果該業務型別未註冊執行緒池,則有Public執行緒池執行。
private int serverCallbackExecutorThreads = 0;
//IO執行緒池執行緒個數
private int serverSelectorThreads = 3;
// send oneway訊息請求併發度
private int serverOnewaySemaphoreValue = 256;
// 非同步訊息傳送併發度
private int serverAsyncSemaphoreValue = 64;
// 網路連線最大空閒時間
private int serverChannelMaxIdleTimeSeconds = 120;
//Socket傳送緩衝區大小
private int serverSocketSndBufSize = NettySystemConfig.socketSndbufSize;
//Socket接受緩衝區大小
private int serverSocketRcvBufSize = NettySystemConfig.socketRcvbufSize;
//ByteBuffer是否開啟快取
private boolean serverPooledByteBufAllocatorEnable = true;
//是否啟用Epoll IO模型
private boolean useEpollNativeSelector = false;
複製程式碼
根據啟動屬性建立NamesrvController例項,並初始化該例項。
public static NamesrvController start(final NamesrvController controller) throws Exception {
if (null == controller) {
throw new IllegalArgumentException("NamesrvController is null");
}
//初始化controller例項
boolean initResult = controller.initialize();
if (!initResult) {
controller.shutdown();
System.exit(-3);
}
//註冊JVM鉤子函式,在JVM關閉之前先關閉執行緒池。
Runtime.getRuntime().addShutdownHook(new ShutdownHookThread(log,new Callable<Void>() {
@Override
public Void call() throws Exception {
controller.shutdown();
return null;
}
}));
//啟動controller
controller.start();
return controller;
}
複製程式碼
public boolean initialize() {
this.kvConfigManager.load();
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig,this.brokerHousekeepingService);
this.remotingExecutor =
Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(),new ThreadFactoryImpl("RemotingExecutorThread_"));
// 用於處理網路請求
this.registerProcessor();
// 10秒掃描一次BrokerLiveTable,移除處於不啟用狀態的Broker
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.routeInfoManager.scanNotActiveBroker();
}
},5,10,TimeUnit.SECONDS);
// nameServer每隔10分鐘列印一次KV配置
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.kvConfigManager.printAllPeriodically();
}
},1,TimeUnit.MINUTES);
return true;
}
複製程式碼
可以看到,NameServer啟動主要就是載入了兩個配置,然後開啟了兩個心跳檢測執行緒,用於心跳檢測掃描Broker以及列印KV配置。
路由元資訊
NameServer主要作用是為生產者和消費者提供關於Topic的路由資訊,還要能夠管理Broker節點。我們先看一下路由元資料的資料結構。
// Topic訊息佇列路由資訊,訊息傳送時根據路由表進行負載均衡。
private final HashMap<String/* topic */,List<QueueData>> topicQueueTable;
// Broker基礎資訊
private final HashMap<String/* brokerName */,BrokerData> brokerAddrTable;
// Broker叢集基礎資訊
private final HashMap<String/* clusterName */,Set<String/* brokerName */>> clusterAddrTable;
// Broker狀態資訊
private final HashMap<String/* brokerAddr */,BrokerLiveInfo> brokerLiveTable;
// 類訊息過濾
private final HashMap<String/* brokerAddr */,List<String>/* Filter Server */> filterServerTable;
複製程式碼
路由註冊
路由註冊時通過Broker與NameServer的心跳功能實現的。Broker啟動時向叢集中所有的NameServer保持長連線,每隔30秒向所有的NameServer傳送心跳包,NameServer收到心跳包時會更新brokerLiveTable快取中BrokerLiveInfo的lastUpdateTimestamp,然後NameServer每隔10s掃描brokerLiveTable,如果連續120s沒有收到心跳包,NameServer將移除該Broker的路由資訊同時關閉Socket連線。
Broker心跳傳送
// 每隔10s向NameServer傳送心跳包
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.registerBrokerAll(true,false,brokerConfig.isForceRegister());
} catch (Throwable e) {
log.error("registerBrokerAll Exception",e);
}
}
},1000 * 10,Math.max(10000,Math.min(brokerConfig.getRegisterNameServerPeriod(),60000)),TimeUnit.MILLISECONDS);
複製程式碼
// 獲取nameServerAddress列表
List<String> nameServerAddressList = this.remotingClient.getNameServerAddressList();
if (nameServerAddressList != null && nameServerAddressList.size() > 0) {
final RegisterBrokerRequestHeader requestHeader = new RegisterBrokerRequestHeader();
requestHeader.setBrokerAddr(brokerAddr);
requestHeader.setBrokerId(brokerId);
requestHeader.setBrokerName(brokerName);
requestHeader.setClusterName(clusterName);
requestHeader.setHaServerAddr(haServerAddr);
requestHeader.setCompressed(compressed);
RegisterBrokerBody requestBody = new RegisterBrokerBody();
requestBody.setTopicConfigSerializeWrapper(topicConfigWrapper);
requestBody.setFilterServerList(filterServerList);
final byte[] body = requestBody.encode(compressed);
final int bodyCrc32 = UtilAll.crc32(body);
requestHeader.setBodyCrc32(bodyCrc32);
// 使用countDownLatch阻塞當前執行緒,等待心跳同步執行緒
final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
for (final String namesrvAddr : nameServerAddressList) {
brokerOuterExecutor.execute(new Runnable() {
@Override
public void run() {
try {
// 傳送心跳同步資訊
RegisterBrokerResult result = registerBroker(namesrvAddr,oneway,timeoutMills,requestHeader,body);
if (result != null) {
registerBrokerResultList.add(result);
}
log.info("register broker[{}]to name server {} OK",brokerId,namesrvAddr);
} catch (Exception e) {
log.warn("registerBroker Exception,{}",namesrvAddr,e);
} finally {
countDownLatch.countDown();
}
}
});
}
try {
//等待心跳同步執行緒全部執行完畢才往下執行
countDownLatch.await(timeoutMills,TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
}
}
複製程式碼
// 封裝請求頭
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.REGISTER_BROKER,requestHeader);
request.setBody(body);
//傳送訊息
if (oneway) {
try {
this.remotingClient.invokeOneway(namesrvAddr,request,timeoutMills);
} catch (RemotingTooMuchRequestException e) {
// Ignore
}
return null;
}
傳送訊息
RemotingCommand response = this.remotingClient.invokeSync(namesrvAddr,timeoutMills);
複製程式碼
向NameServer伺服器傳送型別為RequestCode.REGISTER_BROKER的請求。
NameServer處理心跳
NameServer處理網路請求在org.apache.rocketmq.namesrv.processor.DefaultRequestProcessor#processRequest,如果請求型別為RequestCode.REGISTER_BROKER,則請求最終轉發到org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager#registerBroker。
// 寫鎖
this.lock.writeLock().lockInterruptibly();
// 判斷叢集是否存在
Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
if (null == brokerNames) {
brokerNames = new HashSet<String>();
this.clusterAddrTable.put(clusterName,brokerNames);
}
brokerNames.add(brokerName);
複製程式碼
boolean registerFirst = false;
BrokerData brokerData = this.brokerAddrTable.get(brokerName);
// 如果是新的
if (null == brokerData) {
registerFirst = true;
brokerData = new BrokerData(clusterName,brokerName,new HashMap<Long,String>());
this.brokerAddrTable.put(brokerName,brokerData);
}
Map<Long,String> brokerAddrsMap = brokerData.getBrokerAddrs();
Iterator<Entry<Long,String>> it = brokerAddrsMap.entrySet().iterator();
// 刪除過時的
while (it.hasNext()) {
Entry<Long,String> item = it.next();
if (null != brokerAddr && brokerAddr.equals(item.getValue()) && brokerId != item.getKey()) {
it.remove();
}
}
String oldAddr = brokerData.getBrokerAddrs().put(brokerId,brokerAddr);
registerFirst = registerFirst || (null == oldAddr);
複製程式碼
//如果是master
if (null != topicConfigWrapper && MixAll.MASTER_ID == brokerId) {
// 如果config發生了改變或者是第一次註冊
if (this.isBrokerTopicConfigChanged(brokerAddr,topicConfigWrapper.getDataVersion())
|| registerFirst) {
ConcurrentMap<String,TopicConfig> tcTable =
topicConfigWrapper.getTopicConfigTable();
if (tcTable != null) {
for (Map.Entry<String,TopicConfig> entry : tcTable.entrySet()) {
// 更新路由元資料
this.createAndUpdateQueueData(brokerName,entry.getValue());
}
}
}
}
複製程式碼
如果Broker為Master,並且BrokerTopic配置資訊發生變化或者是初次註冊,則需要建立或更新Topic路由元資料,填充TopicQueueTable,其實就是為預設主題自動註冊路由資訊,其中包含MixAll.DEFAULT_TOPIC的路由資訊。如果該主題為建立並且BrokerConfig的autoCreateTopicEnable為true時,將返回MixAll.DEFAULT_TOPIC的路由資訊。
private void createAndUpdateQueueData(final String brokerName,final TopicConfig topicConfig) {
QueueData queueData = new QueueData();
queueData.setBrokerName(brokerName);
queueData.setWriteQueueNums(topicConfig.getWriteQueueNums());
queueData.setReadQueueNums(topicConfig.getReadQueueNums());
queueData.setPerm(topicConfig.getPerm());
queueData.setTopicSynFlag(topicConfig.getTopicSysFlag());
List<QueueData> queueDataList = this.topicQueueTable.get(topicConfig.getTopicName());
if (null == queueDataList) {
queueDataList = new LinkedList<QueueData>();
queueDataList.add(queueData);
this.topicQueueTable.put(topicConfig.getTopicName(),queueDataList);
log.info("new topic registered,{} {}",topicConfig.getTopicName(),queueData);
} else {
boolean addNewOne = true;
Iterator<QueueData> it = queueDataList.iterator();
while (it.hasNext()) {
QueueData qd = it.next();
if (qd.getBrokerName().equals(brokerName)) {
if (qd.equals(queueData)) {
addNewOne = false;
} else {
log.info("topic changed,{} OLD: {} NEW: {}",qd,queueData);
it.remove();
}
}
}
if (addNewOne) {
queueDataList.add(queueData);
}
}
}
複製程式碼
根據TopicConfig建立QueueData資料結構,然後更新topicQueueTable。
BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,new BrokerLiveInfo(
System.currentTimeMillis(),topicConfigWrapper.getDataVersion(),channel,haServerAddr));
if (null == prevBrokerLiveInfo) {
log.info("new broker registered,{} HAServer: {}",brokerAddr,haServerAddr);
}
複製程式碼
brokerLiveTable是執行路由刪除的重要依據。
if (filterServerList != null) {
if (filterServerList.isEmpty()) {
this.filterServerTable.remove(brokerAddr);
} else {
this.filterServerTable.put(brokerAddr,filterServerList);
}
}
if (MixAll.MASTER_ID != brokerId) {
String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
if (masterAddr != null) {
BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
if (brokerLiveInfo != null) {
result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
result.setMasterAddr(masterAddr);
}
}
}
複製程式碼
路由註冊總結
路由註冊就是Broker每30s向所有NameServer傳送一次心跳並帶上自己的資訊,然後NameServer處理這些資訊,更新路由元資料的過程。 可以看到,這裡路由表使用了HashMap資料結構實現,同時使用了鎖粒度較小的讀寫鎖來控制併發,允許多個Client併發讀,但是同一時刻只允許處理一個心跳包。
路由刪除
上面已經介紹過,NameServer和Broker保持長連線,Broker狀態儲存在BrokerLiveTable裡,NameServer會每10s掃描一次NameServer,一旦發現已經有120s沒有收到Broker傳送過來的心跳資訊,就移除該Broker並關閉與Broker的連線,同時更新路由元資訊。 還有一種情況是Broker正常關閉,會執行unRegisterBroker指令。
掃描brokerLiveTable
上面說過,在NameServer啟動時會開啟一個執行緒,每10秒呼叫一次scanNotActiveBroker方法。
public void scanNotActiveBroker() {
Iterator<Entry<String,BrokerLiveInfo>> it = this.brokerLiveTable.entrySet().iterator();
// 遍歷brokerLiveTable的lastUpdateTimestamp
while (it.hasNext()) {
Entry<String,BrokerLiveInfo> next = it.next();
long last = next.getValue().getLastUpdateTimestamp();
//超過120秒就刪除
if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
RemotingUtil.closeChannel(next.getValue().getChannel());
it.remove();
log.warn("The broker channel expired,{} {}ms",next.getKey(),BROKER_CHANNEL_EXPIRED_TIME);
//然後關閉Channel
this.onChannelDestroy(next.getKey(),next.getValue().getChannel());
}
}
}
複製程式碼
維護路由表
try {
// 加鎖
this.lock.writeLock().lockInterruptibly();
this.brokerLiveTable.remove(brokerAddrFound);
this.filterServerTable.remove(brokerAddrFound);
String brokerNameFound = null;
boolean removeBrokerName = false;
Iterator<Entry<String,BrokerData>> itBrokerAddrTable =
this.brokerAddrTable.entrySet().iterator();
while (itBrokerAddrTable.hasNext() && (null == brokerNameFound)) {
BrokerData brokerData = itBrokerAddrTable.next().getValue();
Iterator<Entry<Long,String>> it = brokerData.getBrokerAddrs().entrySet().iterator();
while (it.hasNext()) {
Entry<Long,String> entry = it.next();
Long brokerId = entry.getKey();
String brokerAddr = entry.getValue();
if (brokerAddr.equals(brokerAddrFound)) {
brokerNameFound = brokerData.getBrokerName();
it.remove();
log.info("remove brokerAddr[{},{}] from brokerAddrTable,because channel destroyed",brokerAddr);
break;
}
}
if (brokerData.getBrokerAddrs().isEmpty()) {
removeBrokerName = true;
itBrokerAddrTable.remove();
log.info("remove brokerName[{}] from brokerAddrTable,brokerData.getBrokerName());
}
}
if (brokerNameFound != null && removeBrokerName) {
Iterator<Entry<String,Set<String>>> it = this.clusterAddrTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String,Set<String>> entry = it.next();
String clusterName = entry.getKey();
Set<String> brokerNames = entry.getValue();
boolean removed = brokerNames.remove(brokerNameFound);
if (removed) {
log.info("remove brokerName[{}],clusterName[{}] from clusterAddrTable,brokerNameFound,clusterName);
if (brokerNames.isEmpty()) {
log.info("remove the clusterName[{}] from clusterAddrTable,because channel destroyed and no broker in this cluster",clusterName);
it.remove();
}
break;
}
}
}
if (removeBrokerName) {
Iterator<Entry<String,List<QueueData>>> itTopicQueueTable =
this.topicQueueTable.entrySet().iterator();
while (itTopicQueueTable.hasNext()) {
Entry<String,List<QueueData>> entry = itTopicQueueTable.next();
String topic = entry.getKey();
List<QueueData> queueDataList = entry.getValue();
Iterator<QueueData> itQueueData = queueDataList.iterator();
while (itQueueData.hasNext()) {
QueueData queueData = itQueueData.next();
if (queueData.getBrokerName().equals(brokerNameFound)) {
itQueueData.remove();
log.info("remove topic[{} {}],from topicQueueTable,topic,queueData);
}
}
if (queueDataList.isEmpty()) {
itTopicQueueTable.remove();
log.info("remove topic[{}] all queue,topic);
}
}
}
} finally {
this.lock.writeLock().unlock();
}
複製程式碼
路由刪除總結
主要就是NameServer定時執行緒每10s掃描一次BrokerLiveTable,發現120s沒動靜的Broker就移除該Broker相關資訊,維護路由表就不再贅述。
路由發現
路由發現並不是實時的,當Topic路由出現變化後,NameServer不主動推送給客戶端。由客戶端定時拉取Topic最新的路由。
public RemotingCommand getRouteInfoByTopic(ChannelHandlerContext ctx,RemotingCommand request) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(null);
final GetRouteInfoRequestHeader requestHeader =
(GetRouteInfoRequestHeader) request.decodeCommandCustomHeader(GetRouteInfoRequestHeader.class);
// 從路由表中找到主題對應的路由資訊填充topicRouteData
TopicRouteData topicRouteData = this.namesrvController.getRouteInfoManager().pickupTopicRouteData(requestHeader.getTopic());
if (topicRouteData != null) {
if (this.namesrvController.getNamesrvConfig().isOrderMessageEnable()) {
String orderTopicConf =
this.namesrvController.getKvConfigManager().getKVConfig(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG,requestHeader.getTopic());
topicRouteData.setOrderTopicConf(orderTopicConf);
}
byte[] content = topicRouteData.encode();
response.setBody(content);
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
return response;
}
response.setCode(ResponseCode.TOPIC_NOT_EXIST);
response.setRemark("No topic route info in name server for the topic: " + requestHeader.getTopic()
+ FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL));
return response;
}
複製程式碼
#總結
如上圖所示,Broker每30s向NameServer叢集傳送心跳資訊,NameServer收到心跳資訊後更新brokerLiveTable中的資訊,記錄lastUpdateTime,NameServer每隔10s掃描brokerLiveTable,比較brokerLiveTable中每一個brokerLiveInfo裡的lastUpdateTime,如果發現超過120s,則刪除broker相關的所有資訊。 當Topic資訊發生變化時,NameServer並不通知客戶端,而是客戶端定時拉去Topic路由資訊。參考文獻
-
RocketMQ官方Doc
-
RocketMQ技術內幕