MySQL高可用方案MHA如何部署-創(chuàng)新互聯(lián)
這篇文章將為大家詳細(xì)講解有關(guān)MySQL高可用方案MHA如何部署，小編覺(jué)得挺實(shí)用的，因此分享給大家做個(gè)參考，希望大家閱讀完這篇文章后可以有所收獲。
成都創(chuàng)新互聯(lián)是一家專(zhuān)業(yè)提供中衛(wèi)企業(yè)網(wǎng)站建設(shè),專(zhuān)注與成都做網(wǎng)站、網(wǎng)站設(shè)計(jì)、html5、小程序制作等業(yè)務(wù)。10年已為中衛(wèi)眾多企業(yè)、政府機(jī)構(gòu)等服務(wù)。創(chuàng)新互聯(lián)專(zhuān)業(yè)的建站公司優(yōu)惠進(jìn)行中。MHA（Master High Availability）是一套相對(duì)成熟的MySQL高可用方案，能做到在0~30s內(nèi)自動(dòng)完成數(shù)據(jù)庫(kù)的故障切換操作，在master服務(wù)器不宕機(jī)的情況下，基本能保證數(shù)據(jù)的一致性。
它由兩部分組成：MHA Manager（管理節(jié)點(diǎn)）和MHA Node（數(shù)據(jù)節(jié)點(diǎn)）。其中，MHA Manager可以單獨(dú)部署在一臺(tái)獨(dú)立的機(jī)器上管理多個(gè)master-slave集群，也可以部署在一臺(tái)slave上。MHA Node則運(yùn)行在每個(gè)mysql節(jié)點(diǎn)上，MHA Manager會(huì)定時(shí)探測(cè)集群中的master節(jié)點(diǎn)，當(dāng)master出現(xiàn)故障時(shí)，它自動(dòng)將最新數(shù)據(jù)的slave提升為master，然后將其它所有的slave指向新的master。
在MHA自動(dòng)故障切換過(guò)程中，MHA試圖保存master的二進(jìn)制日志，從而大程度地保證數(shù)據(jù)不丟失，當(dāng)這并不總是可行的，譬如，主服務(wù)器硬件故障或無(wú)法通過(guò)ssh訪問(wèn)，MHA就沒(méi)法保存二進(jìn)制日志，這樣就只進(jìn)行了故障轉(zhuǎn)移但丟失了最新數(shù)據(jù)?？山Y(jié)合MySQL 5.5中推出的半同步復(fù)制來(lái)降低數(shù)據(jù)丟失的風(fēng)險(xiǎn)。
MHA軟件由兩部分組成：Manager工具包和Node工具包，具體說(shuō)明如下：
MHA Manager：
1. masterha_check_ssh：檢查MHA的SSH配置狀況
2. masterha_check_repl：檢查MySQL的復(fù)制狀況
3. masterha_manager：?jiǎn)?dòng)MHA
4. masterha_check_status：檢測(cè)當(dāng)前MHA運(yùn)行狀態(tài)
5. masterha_master_monitor：檢測(cè)master是否宕機(jī)
6. masterha_master_switch：控制故障轉(zhuǎn)移（自動(dòng)或手動(dòng)）
7. masterha_conf_host：添加或刪除配置的server信息
8. masterha_stop：關(guān)閉MHA
MHA Node:
save_binary_logs：保存或復(fù)制master的二進(jìn)制日志
apply_diff_relay_logs：識(shí)別差異的relay log并將差異的event應(yīng)用到其它slave中
filter_mysqlbinlog：去除不必要的ROLLBACK事件（MHA已不再使用這個(gè)工具）
purge_relay_logs：消除中繼日志（不會(huì)堵塞SQL線程）
另有如下幾個(gè)腳本需自定義：
1. master_ip_failover：管理VIP
2. master_ip_online_change：
3. masterha_secondary_check：當(dāng)MHA manager檢測(cè)到master不可用時(shí)，通過(guò)masterha_secondary_check腳本來(lái)進(jìn)一步確認(rèn)，減低誤切的風(fēng)險(xiǎn)。
4. send_report：當(dāng)發(fā)生故障切換時(shí)，可通過(guò)send_report腳本發(fā)送告警信息。
集群信息
角色 IP地址 ServerID 類(lèi)型
Master 192.168.244.10 1 寫(xiě)入
Candicate master 192.168.244.20 2 讀
Slave 192.168.244.30 3 讀
Monitor host 192.168.244.40 監(jiān)控集群組
注：操作系統(tǒng)均為RHEL 6.7
其中，master對(duì)外提供寫(xiě)服務(wù)，備選master提供讀服務(wù)，slave也提供相關(guān)的讀服務(wù)，一旦master宕機(jī)，將會(huì)把備選master提升為新的master，slave指向新的master
一、在所有節(jié)點(diǎn)上安裝MHA node
1. 在MySQL服務(wù)器上安裝MHA node所需的perl模塊（DBD:mysql）
# yum install perl-DBD-MySQL -y
2. 在所有的節(jié)點(diǎn)上安裝mha node
下載地址為：https://code.google.com/p/mysql-master-ha/wiki/Downloads?tm=2
由于該網(wǎng)址在國(guó)內(nèi)被墻，相關(guān)文件下載后，放到了個(gè)人網(wǎng)盤(pán)中，http://pan.baidu.com/s/1boS31vT，有需要的童鞋可自行下載。
# tar xvf mha4mysql-node-0.56.tar.gz
# cd mha4mysql-node-0.56
# perl Makefile.PL
Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/Install/Can.pm line 6.BEGIN failed--compilation aborted at inc/Module/Install/Can.pm line 6. Compilation failed in require at inc/Module/Install.pm line 283. Can't locate ExtUtils/MakeMaker.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/Install/Makefile.pm line 4.BEGIN failed--compilation aborted at inc/Module/Install/Makefile.pm line 4. Compilation failed in require at inc/Module/Install.pm line 283. Can't locate ExtUtils/MM_Unix.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/Install/Metadata.pm line 349.
View Code
通過(guò)報(bào)錯(cuò)可以看出，是相關(guān)依賴包沒(méi)有安裝。
# yum install perl-ExtUtils-MakeMaker -y
# perl Makefile.PL
*** Module::AutoInstall version 1.03*** Checking for Perl dependencies... Can't locate CPAN.pm in @INC (@INC contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/Module/AutoInstall.pm line 277.
# yum install perl-CPAN -y
# perl Makefile.PL
*** Module::AutoInstall version 1.03*** Checking for Perl dependencies... [Core Features]- DBI ...loaded. (1.609)- DBD::mysql ...loaded. (4.013)*** Module::AutoInstall configuration finished. Checking if your kit is complete... Looks good Writing Makefile for mha4mysql::node
View Code
# make
# make install
至此，MHA node節(jié)點(diǎn)安裝完畢，會(huì)在/usr/local/bin下生成以下腳本文件
# ll /usr/local/bin/total 44-r-xr-xr-x 1 root root 16367 Jul 20 07:00 apply_diff_relay_logs-r-xr-xr-x 1 root root 4807 Jul 20 07:00 filter_mysqlbinlog-r-xr-xr-x 1 root root 8261 Jul 20 07:00 purge_relay_logs-r-xr-xr-x 1 root root 7525 Jul 20 07:00 save_binary_logs
二、在Monitor host節(jié)點(diǎn)上部署MHA Manager
# tar xvf mha4mysql-manager-0.56.tar.gz
# cd mha4mysql-manager-0.56
# perl Makefile.PL
*** Module::AutoInstall version 1.03*** Checking for Perl dependencies... [Core Features]- DBI ...loaded. (1.609)- DBD::mysql ...loaded. (4.013)- Time::HiRes ...missing.- Config::Tiny ...missing.- Log::Dispatch ...missing.- Parallel::ForkManager ...missing.- MHA::NodeConst ...missing.==> Auto-install the 5 mandatory module(s) from CPAN? [y] y*** Dependencies will be installed the next time you type 'make'.*** Module::AutoInstall configuration finished. Checking if your kit is complete... Looks good Warning: prerequisite Config::Tiny 0 not found. Warning: prerequisite Log::Dispatch 0 not found. Warning: prerequisite MHA::NodeConst 0 not found. Warning: prerequisite Parallel::ForkManager 0 not found. Warning: prerequisite Time::HiRes 0 not found. Writing Makefile for mha4mysql::manager
View Code
# make
# make install
執(zhí)行完畢后，會(huì)在/usr/local/bin下新增以下幾個(gè)文件
# ll /usr/local/bin/total 40-r-xr-xr-x 1 root root 1991 Jul 20 00:50 masterha_check_repl-r-xr-xr-x 1 root root 1775 Jul 20 00:50 masterha_check_ssh-r-xr-xr-x 1 root root 1861 Jul 20 00:50 masterha_check_status-r-xr-xr-x 1 root root 3197 Jul 20 00:50 masterha_conf_host-r-xr-xr-x 1 root root 2513 Jul 20 00:50 masterha_manager-r-xr-xr-x 1 root root 2161 Jul 20 00:50 masterha_master_monitor-r-xr-xr-x 1 root root 2369 Jul 20 00:50 masterha_master_switch-r-xr-xr-x 1 root root 5167 Jul 20 00:50 masterha_secondary_check-r-xr-xr-x 1 root root 1735 Jul 20 00:50 masterha_stop
三、配置SSH登錄無(wú)密碼驗(yàn)證
1. 在manager上配置到所有Node節(jié)點(diǎn)的無(wú)密碼驗(yàn)證
# ssh-keygen
一路按“Enter”
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.10
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.20
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.30
2. 在Master（192.168.244.10）上配置
# ssh-keygen
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.20
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.30
3. 在Candicate master（192.168.244.20）上配置
# ssh-keygen
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.10
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.30
4. 在Slave（192.168.244.30）上配置
# ssh-keygen
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.10
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.244.20
四、搭建主從復(fù)制環(huán)境
1. 在Master上執(zhí)行備份
# mysqldump --master-data=2 --single-transaction -R --triggers -A > all.sql
其中，-R是備份存儲(chǔ)過(guò)程，--triggers是備份觸發(fā)器 -A代表全庫(kù)
2. 在Master上創(chuàng)建復(fù)制用戶
mysql> grant replication slave on *.* to 'repl'@'192.168.244.%' identified by 'repl'; Query OK, 0 rows affected (0.09 sec)
3. 查看備份文件all.sql中的CHANGE MASTER語(yǔ)句
# head -n 30 all.sql
-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=120;
4. 將備份文件復(fù)制到Candicate master和Slave上
# scp all.sql 192.168.244.20:/root/
# scp all.sql 192.168.244.30:/root/
5. 在Candicate master上搭建從庫(kù)
# mysql < all.sql
設(shè)置復(fù)制信息
mysql> CHANGE MASTER TO -> MASTER_HOST='192.168.244.10', -> MASTER_USER='repl', -> MASTER_PASSWORD='repl', -> MASTER_LOG_FILE='mysql-bin.000002', -> MASTER_LOG_POS=120; Query OK, 0 rows affected, 2 warnings (0.19 sec) mysql> start slave; Query OK, 0 rows affected (0.02 sec) mysql> show slave status\G
6. 在Slave上搭建從庫(kù)
7. slave服務(wù)器設(shè)置為read only
mysql> set global read_only=1; Query OK, 0 rows affected (0.04 sec)
8. 在Master中創(chuàng)建監(jiān)控用戶
mysql> grant all privileges on *.* to 'monitor'@'%' identified by 'monitor123'; Query OK, 0 rows affected (0.07 sec)
五、 配置MHA
1. 在Monitor host（192.168.244.40）上創(chuàng)建MHA工作目錄，并且創(chuàng)建相關(guān)配置文件
# mkdir -p /etc/masterha
# vim /etc/masterha/app1.cnf
[server default] manager_log=/masterha/app1/manager.log //設(shè)置manager的日志manager_workdir=/masterha/app1 //設(shè)置manager的工作目錄master_binlog_dir=/var/lib/mysql //設(shè)置master默認(rèn)保存binlog的位置，以便MHA可以找到master的日志master_ip_failover_script= /usr/local/bin/master_ip_failover //設(shè)置自動(dòng)failover時(shí)候的切換腳本master_ip_online_change_script= /usr/local/bin/master_ip_online_change //設(shè)置手動(dòng)切換時(shí)候的切換腳本user=monitor // 設(shè)置監(jiān)控用戶password=monitor123 //設(shè)置監(jiān)控用戶的密碼ping_interval=1 //設(shè)置監(jiān)控主庫(kù)，發(fā)送ping包的時(shí)間間隔，默認(rèn)是3秒，嘗試三次沒(méi)有回應(yīng)的時(shí)候進(jìn)行自動(dòng)failoverremote_workdir=/tmp //設(shè)置遠(yuǎn)端mysql在發(fā)生切換時(shí)binlog的保存位置repl_user=repl //設(shè)置復(fù)制環(huán)境中的復(fù)制用戶名repl_password=repl //設(shè)置復(fù)制用戶的密碼report_script=/usr/local/bin/send_report //設(shè)置發(fā)生切換后發(fā)送的報(bào)警的腳本secondary_check_script= /usr/local/bin/masterha_secondary_check -s 192.168.244.20 -s 192.168.244.30 --user=root --master_host=192.168.244.10 --master_ip=192.168.244.10 --master_port=3306 //一旦MHA到master的監(jiān)控之間出現(xiàn)問(wèn)題，MHA Manager將會(huì)判斷其它兩個(gè)slave是否能建立到master_ip 3306端口的連接shutdown_script="" //設(shè)置故障發(fā)生后關(guān)閉故障主機(jī)腳本（該腳本的主要作用是關(guān)閉主機(jī)防止發(fā)生腦裂）ssh_user=root //設(shè)置ssh的登錄用戶名[server1]hostname=192.168.244.10port=3306[server2]hostname=192.168.244.20port=3306candidate_master=1 //設(shè)置為候選master，如果設(shè)置該參數(shù)以后，發(fā)生主從切換以后將會(huì)將此從庫(kù)提升為主庫(kù)，即使這個(gè)主庫(kù)不是集群中最新的slavecheck_repl_delay=0 //默認(rèn)情況下如果一個(gè)slave落后master 100M的relay logs的話，MHA將不會(huì)選擇該slave作為一個(gè)新的master，因?yàn)閷?duì)于這個(gè)slave的恢復(fù)需要花費(fèi)很長(zhǎng)時(shí)間，通過(guò)設(shè)置check_repl_delay=0,MHA觸發(fā)切換在選擇一個(gè)新的master的時(shí)候?qū)?huì)忽略復(fù)制延時(shí)，這個(gè)參數(shù)對(duì)于設(shè)置了candidate_master=1的主機(jī)非常有用，因?yàn)樗ＷC了這個(gè)候選主在切換過(guò)程中一定是最新的master[server3]hostname=192.168.244.30port=3306
注意：
1> 在編輯該文件時(shí)，后面的注釋切記要去掉，MHA并不會(huì)將后面的內(nèi)容識(shí)別為注釋。
2> 配置文件中設(shè)置了master_ip_failover_script，secondary_check_script，master_ip_online_change_script，report_script，對(duì)應(yīng)的文件見(jiàn)文章末 尾。
2. 設(shè)置relay log清除方式（在每個(gè)Slave上）
mysql> set global relay_log_purge=0; Query OK, 0 rows affected (0.00 sec)
MHA在發(fā)生切換過(guò)程中，從庫(kù)在恢復(fù)的過(guò)程中，依賴于relay log的相關(guān)信息，所以我們這里要將relay log的自動(dòng)清楚設(shè)置為OFF，采用手動(dòng)清楚relay log的方式。
在默認(rèn)情況下，從服務(wù)器上的中繼日志會(huì)在SQL線程執(zhí)行完后被自動(dòng)刪除。但是在MHA環(huán)境中，這些中繼日志在恢復(fù)其它從服務(wù)器時(shí)可能會(huì)被用到，因此需要禁用中繼日志的自動(dòng)清除。改為定期手動(dòng)清除SQL線程應(yīng)用完的中繼日志。
在ext3文件系統(tǒng)下，刪除大的文件需要一定的時(shí)間，這樣會(huì)導(dǎo)致嚴(yán)重的復(fù)制延遲，所以在Linux中，一般都是通過(guò)硬鏈接的方式來(lái)刪除大文件。
3. 設(shè)置定期清理relay腳本
MHA節(jié)點(diǎn)中包含了purge_relay_logs腳本，它可以為relay log創(chuàng)建硬鏈接，執(zhí)行set global relay_log_purge=1，等待幾秒鐘以便SQL線程切換到新的中繼日志，再執(zhí)行set global relay_log_purge=0。
下面看看腳本的使用方法：
# purge_relay_logs --user=monitor --password=monitor123 -disable_relay_log_purge --workdir=/tmp/
2017-04-24 20:27:46: purge_relay_logs script started. Found relay_log.info: /var/lib/mysql/relay-log.info Opening /var/lib/mysql/mysqld-relay-bin.000001 .. Opening /var/lib/mysql/mysqld-relay-bin.000002 .. Opening /var/lib/mysql/mysqld-relay-bin.000003 .. Opening /var/lib/mysql/mysqld-relay-bin.000004 .. Opening /var/lib/mysql/mysqld-relay-bin.000005 .. Opening /var/lib/mysql/mysqld-relay-bin.000006 .. Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL thread can delete older relay log files (if i t keeps up); SET GLOBAL relay_log_purge=0; .. ok.2017-04-24 20:27:50: All relay log purging operations succeeded.
其中，
--user：mysql用戶名
--password：mysql用戶的密碼
--host： mysqlserver地址
--workdir：指定創(chuàng)建relay log的硬鏈接的位置，默認(rèn)的是/var/tmp。由于系統(tǒng)不同分區(qū)創(chuàng)建硬鏈接文件會(huì)失敗，故需要指定具體的硬鏈接的位置。
--disable_relay_log_purge：默認(rèn)情況下，如果relay_log_purge=1，則腳本會(huì)直接退出。通過(guò)設(shè)置這個(gè)參數(shù)，該腳本會(huì)首先將relay_log_purge設(shè)置為1，清除掉relay log后，再將該參數(shù)設(shè)置為0。
設(shè)置crontab來(lái)定期清理relay log
MHA在切換的過(guò)程中會(huì)直接調(diào)用mysqlbinlog命令，故需要在環(huán)境變量中指定mysqlbinlog的具體路徑。
# vim /etc/cron.d/purge_relay_logs
0 4 * * * /usr/local/bin/purge_relay_logs --user=monitor --password=monitor123 -disable_relay_log_purge --workdir=/tmp/ >> /tmp/purge _relay_logs.log 2>&1
注意：最好是每臺(tái)slave服務(wù)器在不同時(shí)間點(diǎn)執(zhí)行該計(jì)劃任務(wù)。
4. 將mysqlbinlog的路徑添加到環(huán)境變量中
六、 檢查SSH的配置
在Monitor host上執(zhí)行
# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Wed Jul 20 14:33:36 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Wed Jul 20 14:33:36 2016 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Wed Jul 20 14:33:36 2016 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Wed Jul 20 14:33:36 2016 - [info] Starting SSH connection tests.. Wed Jul 20 14:33:51 2016 - [debug] Wed Jul 20 14:33:36 2016 - [debug] Connecting via SSH from root@192.168.244.10(192.168.244.10:22) to root@192.168.244.20(192.168.244.20:22).. Wed Jul 20 14:33:48 2016 - [debug] ok. Wed Jul 20 14:33:48 2016 - [debug] Connecting via SSH from root@192.168.244.10(192.168.244.10:22) to root@192.168.244.30(192.168.244.30:22).. Wed Jul 20 14:33:50 2016 - [debug] ok. Wed Jul 20 14:33:55 2016 - [debug] Wed Jul 20 14:33:37 2016 - [debug] Connecting via SSH from root@192.168.244.30(192.168.244.30:22) to root@192.168.244.10(192.168.244.10:22).. Wed Jul 20 14:33:49 2016 - [debug] ok. Wed Jul 20 14:33:49 2016 - [debug] Connecting via SSH from root@192.168.244.30(192.168.244.30:22) to root@192.168.244.20(192.168.244.20:22).. Wed Jul 20 14:33:54 2016 - [debug] ok. Wed Jul 20 14:33:55 2016 - [debug] Wed Jul 20 14:33:36 2016 - [debug] Connecting via SSH from root@192.168.244.20(192.168.244.20:22) to root@192.168.244.10(192.168.244.10:22).. Wed Jul 20 14:33:49 2016 - [debug] ok. Wed Jul 20 14:33:49 2016 - [debug] Connecting via SSH from root@192.168.244.20(192.168.244.20:22) to root@192.168.244.30(192.168.244.30:22).. Wed Jul 20 14:33:54 2016 - [debug] ok. Wed Jul 20 14:33:55 2016 - [info] All SSH connection tests passed successfully.
View Code
七、查看整個(gè)集群的狀態(tài)
在Monitor host上執(zhí)行
# masterha_check_repl --conf=/etc/masterha/app1.cnf
Wed Jul 20 14:44:30 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Wed Jul 20 14:44:30 2016 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Wed Jul 20 14:44:30 2016 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Wed Jul 20 14:44:30 2016 - [info] MHA::MasterMonitor version 0.56. Wed Jul 20 14:44:31 2016 - [info] GTID failover mode = 0Wed Jul 20 14:44:31 2016 - [info] Dead Servers: Wed Jul 20 14:44:31 2016 - [info] Alive Servers: Wed Jul 20 14:44:31 2016 - [info] 192.168.244.10(192.168.244.10:3306) Wed Jul 20 14:44:31 2016 - [info] 192.168.244.20(192.168.244.20:3306) Wed Jul 20 14:44:31 2016 - [info] 192.168.244.30(192.168.244.30:3306) Wed Jul 20 14:44:31 2016 - [info] Alive Slaves: Wed Jul 20 14:44:31 2016 - [info] 192.168.244.20(192.168.244.20:3306) Version=5.6.31 (oldest major version between slaves) log-bin:disabled Wed Jul 20 14:44:31 2016 - [info] Replicating from 192.168.244.10(192.168.244.10:3306) Wed Jul 20 14:44:31 2016 - [info] Primary candidate for the new Master (candidate_master is set) Wed Jul 20 14:44:31 2016 - [info] 192.168.244.30(192.168.244.30:3306) Version=5.6.31 (oldest major version between slaves) log-bin:disabled Wed Jul 20 14:44:31 2016 - [info] Replicating from 192.168.244.10(192.168.244.10:3306) Wed Jul 20 14:44:31 2016 - [info] Current Alive Master: 192.168.244.10(192.168.244.10:3306) Wed Jul 20 14:44:31 2016 - [info] Checking slave configurations.. Wed Jul 20 14:44:31 2016 - [warning] log-bin is not set on slave 192.168.244.20(192.168.244.20:3306). This host cannot be a master. Wed Jul 20 14:44:31 2016 - [warning] log-bin is not set on slave 192.168.244.30(192.168.244.30:3306). This host cannot be a master. Wed Jul 20 14:44:31 2016 - [info] Checking replication filtering settings.. Wed Jul 20 14:44:31 2016 - [info] binlog_do_db= , binlog_ignore_db= Wed Jul 20 14:44:31 2016 - [info] Replication filtering check ok. Wed Jul 20 14:44:31 2016 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln361] None of slaves can be master. Check failover configuration file or log-bin settings in my.cnf Wed Jul 20 14:44:31 2016 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/local/bin/masterha_check_repl line 48. Wed Jul 20 14:44:31 2016 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers. Wed Jul 20 14:44:31 2016 - [info] Got exit code 1 (Not master dead). MySQL Replication Health is NOT OK!
View Code
報(bào)錯(cuò)很明顯，Candicate master和Slave都沒(méi)有啟動(dòng)log-bin，如果沒(méi)有啟動(dòng)的話，后續(xù)就無(wú)法提升為主
設(shè)置log-bin后，重新執(zhí)行：
Wed Jul 20 15:49:58 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Wed Jul 20 15:49:58 2016 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. Wed Jul 20 15:49:58 2016 - [info] Reading server configuration from /etc/masterha/app1.cnf.. Wed Jul 20 15:49:58 2016 - [info] MHA::MasterMonitor version 0.56. Wed Jul 20 15:49:59 2016 - [info] GTID failover mode = 0Wed Jul 20 15:49:59 2016 - [info] Dead Servers: Wed Jul 20 15:49:59 2016 - [info] Alive Servers: Wed Jul 20 15:49:59 2016 - [info] 192.168.244.10(192.168.244.10:3306) Wed Jul 20 15:49:59 2016 - [info] 192.168.244.20(192.168.244.20:3306) Wed Jul 20 15:49:59 2016 - [info] 192.168.244.30(192.168.244.30:3306) Wed Jul 20 15:49:59 2016 - [info] Alive Slaves: Wed Jul 20 15:49:59 2016 - [info] 192.168.244.20(192.168.244.20:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled Wed Jul 20 15:49:59 2016 - [info] Replicating from 192.168.244.10(192.168.244.10:3306) Wed Jul 20 15:49:59 2016 - [info] Primary candidate for the new Master (candidate_master is set) Wed Jul 20 15:49:59 2016 - [info] 192.168.244.30(192.168.244.30:3306) Version=5.6.31-log (oldest major version between slaves) log-bin:enabled Wed Jul 20 15:49:59 2016 - [info] Replicating from 192.168.244.10(192.168.244.10:3306) Wed Jul 20 15:49:59 2016 - [info] Current Alive Master: 192.168.244.10(192.168.244.10:3306) Wed Jul 20 15:49:59 2016 - [info] Checking slave configurations.. Wed Jul 20 15:49:59 2016 - [info] Checking replication filtering settings.. Wed Jul 20 15:49:59 2016 - [info] binlog_do_db= , binlog_ignore_db= Wed Jul 20 15:49:59 2016 - [info] Replication filtering check ok. Wed Jul 20 15:49:59 2016 - [info] GTID (with auto-pos) is not supported Wed Jul 20 15:49:59 2016 - [info] Starting SSH connection tests.. Wed Jul 20 15:50:17 2016 - [info] All SSH connection tests passed successfully. Wed Jul 20 15:50:17 2016 - [info] Checking MHA Node version.. Wed Jul 20 15:50:18 2016 - [info] Version check ok. Wed Jul 20 15:50:18 2016 - [info] Checking SSH publickey authentication settings on the current master.. Wed Jul 20 15:50:20 2016 - [info] HealthCheck: SSH to 192.168.244.10 is reachable. Wed Jul 20 15:50:21 2016 - [info] Master MHA Node version is 0.56. Wed Jul 20 15:50:21 2016 - [info] Checking recovery script configurations on 192.168.244.10(192.168.244.10:3306).. Wed Jul 20 15:50:21 2016 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysqld-bin.000002 Wed Jul 20 15:50:21 2016 - [info] Connecting to root@192.168.244.10(192.168.244.10:22).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /var/lib/mysql, up to mysqld-bin.000002Wed Jul 20 15:50:23 2016 - [info] Binlog setting check done. Wed Jul 20 15:50:23 2016 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Wed Jul 20 15:50:23 2016 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.244.20 --slave_ip=192.168.244.20 --slave_port=3306 --workdir=/tmp --target_version=5.6.31-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx Wed Jul 20 15:50:23 2016 - [info] Connecting to root@192.168.244.20(192.168.244.20:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to mysqld-relay-bin.000004 Temporary relay log file is /var/lib/mysql/mysqld-relay-bin.000004 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Wed Jul 20 15:50:28 2016 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='monitor' --slave_host=192.168.244.30 --slave_ip=192.168.244.30 --slave_port=3306 --workdir=/tmp --target_version=5.6.31-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx Wed Jul 20 15:50:28 2016 - [info] Connecting to root@192.168.244.30(192.168.244.30:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to mysqld-relay-bin.000008 Temporary relay log file is /var/lib/mysql/mysqld-relay-bin.000008 Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Wed Jul 20 15:50:32 2016 - [info] Slaves settings check done. Wed Jul 20 15:50:32 2016 - [info] 192.168.244.10(192.168.244.10:3306) (current master) +--192.168.244.20(192.168.244.20:3306) +--192.168.244.30(192.168.244.30:3306) Wed Jul 20 15:50:32 2016 - [info] Checking replication health on 192.168.244.20.. Wed Jul 20 15:50:32 2016 - [info] ok. Wed Jul 20 15:50:32 2016 - [info] Checking replication health on 192.168.244.30.. Wed Jul 20 15:50:32 2016 - [info] ok. Wed Jul 20 15:50:32 2016 - [info] Checking master_ip_failover_script status: Wed Jul 20 15:50:32 2016 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.244.10 --orig_master_ip=192.168.244.10 --orig_master_port=3306 Wed Jul 20 15:50:32 2016 - [info] OK. Wed Jul 20 15:50:32 2016 - [warning] shutdown_script is not defined. Wed Jul 20 15:50:32 2016 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK.
View Code
檢查通過(guò)~
八、 檢查MHA Manager的狀態(tài)
# masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING).
如果正常，會(huì)顯示“PING_OK”，否則會(huì)顯示“NOT_RUNNING”，代表MHA監(jiān)控還沒(méi)有開(kāi)啟。
九、開(kāi)啟MHA Manager監(jiān)控
# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /masterha/app1/manager.log 2>&1 &
其中，
remove_dead_master_conf：該參數(shù)代表當(dāng)發(fā)生主從切換后，老的主庫(kù)的IP將會(huì)從配置文件中移除。
ignore_last_failover：在默認(rèn)情況下，MHA發(fā)生切換后將會(huì)在/masterha/app1下產(chǎn)生app1.failover.complete文件，下次再次切換的時(shí)候如果發(fā)現(xiàn)該目錄下存在該文件且兩次切換的時(shí)間間隔不足8小時(shí)的話，將不允許觸發(fā)切換。除非在第一次切換后手動(dòng)rm -rf /masterha/app1/app1.failover.complete。該參數(shù)代表忽略上次MHA觸發(fā)切換產(chǎn)生的文件。
查看MHA Manager監(jiān)控是否正常
# masterha_check_status --conf=/etc/masterha/app1.cnf app1 (pid:1873) is running(0:PING_OK), master:192.168.244.10
十、 關(guān)閉MHA Manager監(jiān)控
# masterha_stop --conf=/etc/masterha/app1.cnf Stopped app1 successfully. [1]+ Exit 1 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /masterha/app1/manager.log 2>&1
至此，MHA部分配置完畢，下面，來(lái)配置VIP。
十一、VIP配置
VIP配置可以采用兩種方式，一是通過(guò)引入Keepalived來(lái)管理VIP，另一種是在腳本中手動(dòng)管理。
對(duì)于keepalived管理VIP，存在腦裂情況，即當(dāng)主從網(wǎng)絡(luò)出現(xiàn)問(wèn)題時(shí)，slave會(huì)搶占VIP，這樣會(huì)導(dǎo)致主從數(shù)據(jù)庫(kù)都持有VIP，造成IP沖突，所以在網(wǎng)絡(luò)不是很好的情況下，不建議采用keepalived服務(wù)。
在實(shí)際生產(chǎn)中使用較多的也是第二種，即在腳本中手動(dòng)管理VIP，所以，對(duì)keepalived不感興趣的童鞋可直接跳過(guò)第一種方式。
1. keepalived管理VIP
1> 安裝keepalived
因?yàn)槲疫@里設(shè)置了Candicate master，故只在Master和Candicate master上安裝。
如果沒(méi)有Candicate master，兩個(gè)Slave的地位平等，則兩個(gè)Slave上都需安裝keepalived。
# wget http://www.keepalived.org/software/keepalived-1.2.24.tar.gz
# tar xvf keepalived-1.2.24.tar.gz
# cd keepalived-1.2.24
# ./configure --prefix=/usr/local/keepalived
# make
# make install
# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
# mkdir /etc/keepalived
# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
2> 為keepalived設(shè)置單獨(dú)的日志文件（非必需）
keepalived的日志默認(rèn)是輸出到/var/log/message中
# vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -d -S 0"
設(shè)置syslog
# vim /etc/rsyslog.conf
添加如下內(nèi)容：
local0.* /var/log/keepalived.log
# service rsyslog restart
2> 配置keepalived
在Master上修改
# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
slowtech@qq.com
}
notification_email_from root@localhost.localdomain
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MySQL-HA
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 150
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress { 192.168.244.188/24
}
}View Code
關(guān)于keepalived的參數(shù)的詳細(xì)介紹，可參考：
LVS+Keepalived搭建MyCAT高可用負(fù)載均衡集群
keepalived工作原理和配置說(shuō)明
將配置文件scp到Candicate master上
# scp /etc/keepalived/keepalived.conf 192.168.244.20:/etc/keepalived/
只需將配置文件中的priority設(shè)置為90
注意：我們?yōu)槭裁丛谶@里設(shè)置keepalived為backup模式呢？
在master-backup模式下，如果主庫(kù)宕掉，VIP會(huì)自動(dòng)漂移到Slave上，當(dāng)主庫(kù)修復(fù)，keepalived啟動(dòng)后，還會(huì)將VIP搶過(guò)來(lái)，即使設(shè)置了nopreempt（不搶占）的方
式，該動(dòng)作仍會(huì)發(fā)生。但在backup-backup模式下，當(dāng)主庫(kù)修改，并啟動(dòng)keepalived后，并不會(huì)搶占新主的VIP，即便原主的priority高于新主的。
3> 啟動(dòng)keepalived
先在Master上啟動(dòng)
# service keepalived start
env: /etc/init.d/keepalived: Permission denied
# chmod +x /etc/init.d/keepalived
# service keepalived start
查看綁定情況
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:c6:47:04 brd ff:ff:ff:ff:ff:ff inet 192.168.244.10/24 brd 192.168.244.255 scope global eth0 inet 192.168.244.188/24 scope global secondary eth0 inet6 fe80::20c:29ff:fec6:4704/64 scope link valid_lft forever preferred_lft forever
View Code
可見(jiàn)，VIP（192168.244.188）已經(jīng)綁定到Master的eth0網(wǎng)卡上了。
啟動(dòng)Candicate master的keepalived
# service keepalived start
4> MHA中引入keepalived
編輯/usr/local/bin/master_ip_failover
相對(duì)于原文件，修改地方為93-95行
1 #!/usr/bin/env perl 2 3 # Copyright (C) 2011 DeNA Co.,Ltd. 4 # 5 # This program is free software; you can redistribute it and/or modify 6 # it under the terms of the GNU General Public License as published by 7 # the Free Software Foundation; either version 2 of the License, or 8 # (at your option) any later version. 9 # 10 # This program is distributed in the hope that it will be useful, 11 # but WITHOUT ANY WARRANTY; without even the implied warranty of 12 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 13 # GNU General Public License for more details. 14 # 15 # You should have received a copy of the GNU General Public License 16 # along with this program; if not, write to the Free Software 17 # Foundation, Inc., 18 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 19 20 ## Note: This is a sample script and is not complete. Modify the script based on your environment. 21 22 use strict; 23 use warnings FATAL => 'all'; 24 25 use Getopt::Long; 26 use MHA::DBHelper; 27 my ( 28 $command, $ssh_user, $orig_master_host, 29 $orig_master_ip, $orig_master_port, $new_master_host, 30 $new_master_ip, $new_master_port, $new_master_user, 31 $new_master_password 32 ); 33 34 GetOptions( 35 'command=s' => \$command, 36 'ssh_user=s' => \$ssh_user, 37 'orig_master_host=s' => \$orig_master_host, 38 'orig_master_ip=s' => \$orig_master_ip, 39 'orig_master_port=i' => \$orig_master_port, 40 'new_master_host=s' => \$new_master_host, 41 'new_master_ip=s' => \$new_master_ip, 42 'new_master_port=i' => \$new_master_port, 43 'new_master_user=s' => \$new_master_user, 44 'new_master_password=s' => \$new_master_password, 45 ); 46 47 exit &main(); 48 49 sub main { 50 if ( $command eq "stop" || $command eq "stopssh" ) { 51 52 # $orig_master_host, $orig_master_ip, $orig_master_port are passed. 53 # If you manage master ip address at global catalog database, 54 # invalidate orig_master_ip here. 55 my $exit_code = 1; 56 eval { 57 58 # updating global catalog, etc 59 $exit_code = 0; 60 }; 61 if ($@) { 62 warn "Got Error: $@\n"; 63 exit $exit_code; 64 } 65 exit $exit_code; 66 } 67 elsif ( $command eq "start" ) { 68 69 # all arguments are passed. 70 # If you manage master ip address at global catalog database, 71 # activate new_master_ip here. 72 # You can also grant write access (create user, set read_only=0, etc) here. 73 my $exit_code = 10; 74 eval { 75 my $new_master_handler = new MHA::DBHelper(); 76 77 # args: hostname, port, user, password, raise_error_or_not 78 $new_master_handler->connect( $new_master_ip, $new_master_port, 79 $new_master_user, $new_master_password, 1 ); 80 81 ## Set read_only=0 on the new master 82 $new_master_handler->disable_log_bin_local(); 83 print "Set read_only=0 on the new master.\n"; 84 $new_master_handler->disable_read_only(); 85 86 ## Creating an app user on the new master 87 #print "Creating app user on the new master..\n"; 88 #FIXME_xxx_create_user( $new_master_handler->{dbh} ); 89 $new_master_handler->enable_log_bin_local(); 90 $new_master_handler->disconnect(); 91 92 ## Update master ip on the catalog database, etc 93 my $cmd; 94 $cmd = 'ssh '.$ssh_user.'@'.$orig_master_ip.' service keepalived stop'; 95 system($cmd); 96 $exit_code = 0; 97 }; 98 if ($@) { 99 warn $@;100 101 # If you want to continue failover, exit 10.102 exit $exit_code;103 }104 exit $exit_code;105 }106 elsif ( $command eq "status" ) {107 108 # do nothing109 exit 0;110 }111 else {112 &usage();113 exit 1;114 }115 }116 117 sub usage {118 print119 "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";120 }View Code
2. 通過(guò)腳本的方式管理VIP
編輯/usr/local/bin/master_ip_failover
#!/usr/bin/ software; you can redistribute it and/ not, Franklin Street, Fifth Floor, Boston, MA -=> = = = = = => => => => => => => => => =>& ( $command eq || $command eq = &= access (create user, set read_only== =->=->->->->->&= & $ssh_user\@$new_master_host \ ` $ssh_user\@$new_master_host \ $ssh_user\@$orig_master_host \
實(shí)際生產(chǎn)環(huán)境中，推薦這種方式來(lái)管理VIP，可有效防止腦裂情況的發(fā)生。
至此，MHA高可用環(huán)境基本搭建完畢。
關(guān)于MHA的常見(jiàn)操作，包括自動(dòng)Failover，手動(dòng)Failover，在線切換，可參考另一篇博客：
MHA在線切換的步驟和原理
MHA自動(dòng)Failover與手動(dòng)Failover的實(shí)踐及原理
總結(jié)：
1. 可單獨(dú)調(diào)試master_ip_failover，master_ip_online_change，send_report等腳本
/usr/local/bin/master_ip_online_change --command=stop --orig_master_ip=192.168.244.10 --orig_master_host=192.168.244.10 --orig_master_port=3306 --orig_master_user=monitor --orig_master_password=monitor123 --orig_master_ssh_user=root --new_master_host=192.168.244.20 --new_master_ip=192.168.244.20 --new_master_port=3306 --new_master_user=monitor --new_master_password=monitor123 --new_master_ssh_user=root
/usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.244.10 --orig_master_ip=192.168.244.10 --orig_master_port=3306 --new_master_host=192.168.244.20 --new_master_ip=192.168.244.20 --new_master_port=3306 --new_master_user='monitor' --new_master_password='monitor123'
2. 官方對(duì)于master_ip_failover，master_ip_online_change，send_report腳本，給出的只是sample，切換的邏輯需要自己定義。
很多童鞋對(duì)perl并不熟悉，覺(jué)得無(wú)從下手，其實(shí)，完全可以調(diào)用其它腳本，譬如python，shell等。
如：
[root@node4 ~]# cat test.pl #!/usr/bin/perluse strict; my $cmd='python /root/test.py'; system($cmd); [root@node4 ~]# cat test.py #!/usr/bin/python print "hello,python"[root@node4 ~]# perl test.pl hello,python
關(guān)于“MySQL高可用方案MHA如何部署”這篇文章就分享到這里了，希望以上內(nèi)容可以對(duì)大家有一定的幫助，使各位可以學(xué)到更多知識(shí)，如果覺(jué)得文章不錯(cuò)，請(qǐng)把它分享出去讓更多的人看到。
網(wǎng)站題目：MySQL高可用方案MHA如何部署-創(chuàng)新互聯(lián)
路徑分享：http://www.rwnh.cn/article12/ccghdc.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供網(wǎng)頁(yè)設(shè)計(jì)公司、外貿(mào)建站、響應(yīng)式網(wǎng)站、網(wǎng)站收錄、外貿(mào)網(wǎng)站建設(shè)、網(wǎng)站制作
聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源： 創(chuàng)新互聯(lián)
猜你還喜歡下面的內(nèi)容
- 如何優(yōu)化HTML的輸入框-創(chuàng)新互聯(lián)
- App地推業(yè)績(jī)統(tǒng)計(jì)方法選擇不當(dāng)會(huì)產(chǎn)生什么問(wèn)題？-創(chuàng)新互聯(lián)
- 使用CSS3怎么實(shí)現(xiàn)一個(gè)切片式圖片輪播效果-創(chuàng)新互聯(lián)
- phpmailer用smtp發(fā)送郵件的過(guò)程-創(chuàng)新互聯(lián)
- Java字符串-創(chuàng)新互聯(lián)
- cad偏移命令怎么用AutoCad如何將完整圖形統(tǒng)一偏移？-創(chuàng)新互聯(lián)
- asp.net中怎么利用Zxing庫(kù)輸出條形碼-創(chuàng)新互聯(lián)
- 澳門(mén)鮮花電商APP開(kāi)發(fā)主要業(yè)務(wù)模式分析 2023-03-27
- 上海APP開(kāi)發(fā)如何進(jìn)行優(yōu)先級(jí)開(kāi)發(fā) 2020-12-03
- 上海哪家APP開(kāi)發(fā)外包公司比較專(zhuān)業(yè)？ 2020-11-14
- 平輿代駕app開(kāi)發(fā)功能與方案 2020-12-23
- 武漢手機(jī)修圖APP開(kāi)發(fā)有哪些功能？ 2023-02-23
- 登封APP開(kāi)發(fā)如何做好用戶數(shù)據(jù)分析？ 2020-12-19
- 互聯(lián)網(wǎng)宣傳，成都app開(kāi)發(fā)還是平臺(tái)開(kāi)發(fā)？ 2022-07-13
- 網(wǎng)絡(luò)友好度——APP開(kāi)發(fā)應(yīng)重視的問(wèn)題 2022-07-15
- 慘不忍睹！你確定已跟上市場(chǎng)APP開(kāi)發(fā)的腳步？ 2016-11-09
- 面對(duì)微信小程序的挑戰(zhàn)，APP開(kāi)發(fā)當(dāng)如何面對(duì)？ 2022-06-30
- 成都原生APP開(kāi)發(fā)的成本為何這么貴？ 2022-07-17
- 真實(shí)測(cè)評(píng)：用uni-app開(kāi)發(fā)小程序，比原生開(kāi)發(fā)好在哪里？ 2022-08-23