怎樣進行numaloadbance的死鎖分析

怎樣進行numa loadbance的死鎖分析，很多新手對此不是很清楚，為了幫助大家解決這個難題，下面小編將為大家詳細(xì)講解，有這方面需求的人可以來學(xué)習(xí)下，希望你能有所收獲。

站在用戶的角度思考問題，與客戶深入溝通，找到赤峰林西網(wǎng)站設(shè)計與赤峰林西網(wǎng)站推廣的解決方案，憑借多年的經(jīng)驗，讓設(shè)計與互聯(lián)網(wǎng)技術(shù)結(jié)合，創(chuàng)造個性化、用戶體驗好的作品，建站類型包括：網(wǎng)站建設(shè)、成都做網(wǎng)站、企業(yè)官網(wǎng)、英文網(wǎng)站、手機端網(wǎng)站、網(wǎng)站推廣、國際域名空間、雅安服務(wù)器托管、企業(yè)郵箱。業(yè)務(wù)覆蓋赤峰林西地區(qū)。

背景：這個是在3.10.0-957.el7.x86_64 遇到的一例crash。下面列一下我們是怎么排查并解這個問題的。

一、故障現(xiàn)象

Oppo云智能監(jiān)控發(fā)現(xiàn)機器down機：

 KERNEL: /usr/lib/debug/lib/modules/3.10.0-957.el7.x86_64/vmlinux  ....       PANIC: "Kernel panic - not syncing: Hard LOCKUP"         PID: 14     COMMAND: "migration/1"        TASK: ffff8f1bf6bb9040  [THREAD_INFO: ffff8f1bf6bc4000]         CPU: 1       STATE: TASK_INTERRUPTIBLE (PANIC)
crash> btPID: 14     TASK: ffff8f1bf6bb9040  CPU: 1   COMMAND: "migration/1" #0 [ffff8f4afbe089f0] machine_kexec at ffffffff83863674 #1 [ffff8f4afbe08a50] __crash_kexec at ffffffff8391ce12 #2 [ffff8f4afbe08b20] panic at ffffffff83f5b4db #3 [ffff8f4afbe08ba0] nmi_panic at ffffffff8389739f #4 [ffff8f4afbe08bb0] watchdog_overflow_callback at ffffffff83949241 #5 [ffff8f4afbe08bc8] __perf_event_overflow at ffffffff839a1027 #6 [ffff8f4afbe08c00] perf_event_overflow at ffffffff839aa694 #7 [ffff8f4afbe08c10] intel_pmu_handle_irq at ffffffff8380a6b0 #8 [ffff8f4afbe08e38] perf_event_nmi_handler at ffffffff83f6b031 #9 [ffff8f4afbe08e58] nmi_handle at ffffffff83f6c8fc#10 [ffff8f4afbe08eb0] do_nmi at ffffffff83f6cbd8#11 [ffff8f4afbe08ef0] end_repeat_nmi at ffffffff83f6bd69    [exception RIP: native_queued_spin_lock_slowpath+462]    RIP: ffffffff839121ae  RSP: ffff8f1bf6bc7c50  RFLAGS: 00000002    RAX: 0000000000000001  RBX: 0000000000000082  RCX: 0000000000000001    RDX: 0000000000000101  RSI: 0000000000000001  RDI: ffff8f1afdf55fe8---鎖    RBP: ffff8f1bf6bc7c50   R8: 0000000000000101   R9: 0000000000000400    R10: 000000000000499e  R11: 000000000000499f  R12: ffff8f1afdf55fe8    R13: ffff8f1bf5150000  R14: ffff8f1afdf5b488  R15: ffff8f1bf5187818    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018--- <NMI exception stack> ---#12 [ffff8f1bf6bc7c50] native_queued_spin_lock_slowpath at ffffffff839121ae#13 [ffff8f1bf6bc7c58] queued_spin_lock_slowpath at ffffffff83f5bf4b#14 [ffff8f1bf6bc7c68] _raw_spin_lock_irqsave at ffffffff83f6a487#15 [ffff8f1bf6bc7c80] cpu_stop_queue_work at ffffffff8392fc70#16 [ffff8f1bf6bc7cb0] stop_one_cpu_nowait at ffffffff83930450#17 [ffff8f1bf6bc7cc0] load_balance at ffffffff838e4c6e#18 [ffff8f1bf6bc7da8] idle_balance at ffffffff838e5451#19 [ffff8f1bf6bc7e00] __schedule at ffffffff83f67b14#20 [ffff8f1bf6bc7e88] schedule at ffffffff83f67bc9#21 [ffff8f1bf6bc7e98] smpboot_thread_fn at ffffffff838ca562#22 [ffff8f1bf6bc7ec8] kthread at ffffffff838c1c31#23 [ffff8f1bf6bc7f50] ret_from_fork_nospec_begin at ffffffff83f74c1dcrash>

二、故障現(xiàn)象分析

hardlock一般是由于關(guān)中斷時間過長，從堆?？?，上面的"migration/1" 進程在搶spinlock，由于_raw_spin_lock_irqsave 會先調(diào)用 arch_local_irq_disable,然后再去拿鎖，而arch_local_irq_disable 是常見的關(guān)中斷函數(shù),下面分析這個進程想要拿的鎖被誰拿著。

x86架構(gòu)下，native_queued_spin_lock_slowpath的rdi就是存放鎖地址的

crash> arch_spinlock_t ffff8f1afdf55fe8struct arch_spinlock_t {  val = {    counter = 257  }}

下面，我們需要了解，這個是一把什么鎖。從調(diào)用鏈分析 idle_balance-->load_balance-->stop_one_cpu_nowait-->cpu_stop_queue_work反匯編 cpu_stop_queue_work 拿鎖阻塞的代碼：

crash> dis -l ffffffff8392fc70/usr/src/debug/kernel-3.10.0-957.el7/linux-3.10.0-957.el7.x86_64/kernel/stop_machine.c: 910xffffffff8392fc70 <cpu_stop_queue_work+48>:    cmpb   $0x0,0xc(%rbx)
     85 static void cpu_stop_queue_work(unsigned int cpu, struct cpu_stop_work *work)     86 {     87         struct cpu_stopper *stopper = &per_cpu(cpu_stopper, cpu);     88         unsigned long flags;     89      90         spin_lock_irqsave(&stopper->lock, flags);---所以是卡在拿這把鎖     91         if (stopper->enabled)     92                 __cpu_stop_queue_work(stopper, work);     93         else     94                 cpu_stop_signal_done(work->done, false);     95         spin_unlock_irqrestore(&stopper->lock, flags);     96 }

看起來需要根據(jù)cpu號，來獲取對應(yīng)的percpu變量 cpu_stopper,這個入?yún)⒃?load_balance 函數(shù)中找到的最忙的rq，然后獲取其對應(yīng)的cpu號:

   6545 static int load_balance(int this_cpu, struct rq *this_rq,   6546                         struct sched_domain *sd, enum cpu_idle_type idle,   6547                         int *should_balance)   6548 {....   6735                         if (active_balance) {   6736                                 stop_one_cpu_nowait(cpu_of(busiest),   6737                                         active_load_balance_cpu_stop, busiest,   6738                                         &busiest->active_balance_work);   6739                         }....  6781 }
  crash> dis -l load_balance |grep stop_one_cpu_nowait -B 60xffffffff838e4c4d <load_balance+2045>:  callq  0xffffffff83f6a0e0 <_raw_spin_unlock_irqrestore>/usr/src/debug/kernel-3.10.0-957.el7/linux-3.10.0-957.el7.x86_64/kernel/sched/fair.c: 67360xffffffff838e4c52 <load_balance+2050>:  mov    0x930(%rbx),%edi------------根據(jù)rbx可以取cpu號，rbx就是最忙的rq0xffffffff838e4c58 <load_balance+2056>:  lea    0x908(%rbx),%rcx0xffffffff838e4c5f <load_balance+2063>:  mov    %rbx,%rdx0xffffffff838e4c62 <load_balance+2066>:  mov    $0xffffffff838de690,%rsi0xffffffff838e4c69 <load_balance+2073>:  callq  0xffffffff83930420 <stop_one_cpu_nowait>

然后我們再棧中取的數(shù)據(jù)如下：

最忙的組是：crash> rq.cpu ffff8f1afdf5ab80  cpu = 26

也就是說，1號cpu在等 percpu變量cpu_stopper 的26號cpu的鎖。

然后我們搜索這把鎖在其他哪個進程的棧中，找到了如下：

ffff8f4957fbfab0: ffff8f1afdf55fe8 --------這個在  355608 的棧中crash> kmem ffff8f4957fbfab0    PID: 355608COMMAND: "custom_exporter"   TASK: ffff8f4aea3a8000  [THREAD_INFO: ffff8f4957fbc000]    CPU: 26--------剛好也是運行在26號cpu的進程  STATE: TASK_RUNNING (ACTIVE)

下面，就需要分析，為什么位于26號cpu的進程 custom_exporter 會長時間拿著 ffff8f1afdf55fe8

我們來分析26號cpu的堆棧：

crash> bt -f 355608PID: 355608  TASK: ffff8f4aea3a8000  CPU: 26  COMMAND: "custom_exporter"..... #3 [ffff8f1afdf48ef0] end_repeat_nmi at ffffffff83f6bd69    [exception RIP: try_to_wake_up+114]    RIP: ffffffff838d63d2  RSP: ffff8f4957fbfa30  RFLAGS: 00000002    RAX: 0000000000000001  RBX: ffff8f1bf6bb9844  RCX: 0000000000000000    RDX: 0000000000000001  RSI: 0000000000000003  RDI: ffff8f1bf6bb9844    RBP: ffff8f4957fbfa70   R8: ffff8f4afbe15ff0   R9: 0000000000000000    R10: 0000000000000000  R11: 0000000000000000  R12: 0000000000000000    R13: ffff8f1bf6bb9040  R14: 0000000000000000  R15: 0000000000000003    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000--- <NMI exception stack> --- #4 [ffff8f4957fbfa30] try_to_wake_up at ffffffff838d63d2    ffff8f4957fbfa38: 000000000001ab80 0000000000000086     ffff8f4957fbfa48: ffff8f4afbe15fe0 ffff8f4957fbfb48     ffff8f4957fbfa58: 0000000000000001 ffff8f4afbe15fe0     ffff8f4957fbfa68: ffff8f1afdf55fe0 ffff8f4957fbfa80     ffff8f4957fbfa78: ffffffff838d6705  #5 [ffff8f4957fbfa78] wake_up_process at ffffffff838d6705    ffff8f4957fbfa80: ffff8f4957fbfa98 ffffffff8392fc05  #6 [ffff8f4957fbfa88] __cpu_stop_queue_work at ffffffff8392fc05    ffff8f4957fbfa90: 000000000000001a ffff8f4957fbfbb0     ffff8f4957fbfaa0: ffffffff8393037a  #7 [ffff8f4957fbfaa0] stop_two_cpus at ffffffff8393037a.....    ffff8f4957fbfbb8: ffffffff838d3867  #8 [ffff8f4957fbfbb8] migrate_swap at ffffffff838d3867    ffff8f4957fbfbc0: ffff8f4aea3a8000 ffff8f1ae77dc100 -------棧中的 migration_swap_arg    ffff8f4957fbfbd0: 000000010000001a 0000000080490f7c     ffff8f4957fbfbe0: ffff8f4aea3a8000 ffff8f4957fbfc30     ffff8f4957fbfbf0: 0000000000000076 0000000000000076     ffff8f4957fbfc00: 0000000000000371 ffff8f4957fbfce8     ffff8f4957fbfc10: ffffffff838dd0ba  #9 [ffff8f4957fbfc10] task_numa_migrate at ffffffff838dd0ba    ffff8f4957fbfc18: ffff8f1afc121f40 000000000000001a     ffff8f4957fbfc28: 0000000000000371 ffff8f4aea3a8000 ---這里ffff8f4957fbfc30 就是 task_numa_env 的存放在棧中的地址    ffff8f4957fbfc38: 000000000000001a 000000010000003f     ffff8f4957fbfc48: 000000000000000b 000000000000022c     ffff8f4957fbfc58: 00000000000049a0 0000000000000012     ffff8f4957fbfc68: 0000000000000001 0000000000000003     ffff8f4957fbfc78: 000000000000006f 000000000000499f     ffff8f4957fbfc88: 0000000000000012 0000000000000001     ffff8f4957fbfc98: 0000000000000070 ffff8f1ae77dc100     ffff8f4957fbfca8: 00000000000002fb 0000000000000001     ffff8f4957fbfcb8: 0000000080490f7c ffff8f4aea3a8000 ---rbx壓棧在此，所以這個就是current    ffff8f4957fbfcc8: 0000000000017a48 0000000000001818     ffff8f4957fbfcd8: 0000000000000018 ffff8f4957fbfe20     ffff8f4957fbfce8: ffff8f4957fbfcf8 ffffffff838dd4d3 #10 [ffff8f4957fbfcf0] numa_migrate_preferred at ffffffff838dd4d3    ffff8f4957fbfcf8: ffff8f4957fbfd88 ffffffff838df5b0 .....crash> crash>

整體上看，26號上的cpu也正在進行numa的balance動作，簡單展開介紹一下numa在balance下的動作在 task_tick_fair 函數(shù)中：

static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued){  struct cfs_rq *cfs_rq;  struct sched_entity *se = &curr->se;
  for_each_sched_entity(se) {    cfs_rq = cfs_rq_of(se);    entity_tick(cfs_rq, se, queued);  }
  if (numabalancing_enabled)----------如果開啟numabalancing，則會調(diào)用task_tick_numa    task_tick_numa(rq, curr);
  update_rq_runnable_avg(rq, 1);}

而 task_tick_numa 會根據(jù)掃描情況，將當(dāng)前進程需要numa_balance的時候推送到一個work中。通過調(diào)用change_prot_numa將所有映射到VMA的PTE頁表項該為PAGE_NONE，使得下次進程訪問頁表的時候產(chǎn)生缺頁中斷，handle_pte_fault 函數(shù)會由于缺頁中斷的機會來根據(jù)numa 選擇更好的node，具體不再展開。

在 26號cpu的調(diào)用鏈中，stop_two_cpus-->cpu_stop_queue_two_works-->cpu_stop_queue_work 函數(shù)由于 cpu_stop_queue_two_works 被內(nèi)聯(lián)了，但是 cpu_stop_queue_two_works 調(diào)用cpu_stop_queue_work有兩次，所以需要根據(jù)壓棧地址判斷當(dāng)前是哪次調(diào)用出現(xiàn)問題。

    227 static int cpu_stop_queue_two_works(int cpu1, struct cpu_stop_work *work1,    228                                     int cpu2, struct cpu_stop_work *work2)    229 {    230         struct cpu_stopper *stopper1 = per_cpu_ptr(&cpu_stopper, cpu1);    231         struct cpu_stopper *stopper2 = per_cpu_ptr(&cpu_stopper, cpu2);    232         int err;    233     234         lg_double_lock(&stop_cpus_lock, cpu1, cpu2);    235         spin_lock_irq(&stopper1->lock);---注意到這里已經(jīng)持有了stopper1的鎖    236         spin_lock_nested(&stopper2->lock, SINGLE_DEPTH_NESTING);.....    243         __cpu_stop_queue_work(stopper1, work1);    244         __cpu_stop_queue_work(stopper2, work2);.....    251 }

根據(jù)壓棧的地址:

 #5 [ffff8f4957fbfa78] wake_up_process at ffffffff838d6705    ffff8f4957fbfa80: ffff8f4957fbfa98 ffffffff8392fc05  #6 [ffff8f4957fbfa88] __cpu_stop_queue_work at ffffffff8392fc05    ffff8f4957fbfa90: 000000000000001a ffff8f4957fbfbb0     ffff8f4957fbfaa0: ffffffff8393037a  #7 [ffff8f4957fbfaa0] stop_two_cpus at ffffffff8393037a    ffff8f4957fbfaa8: 0000000100000001 ffff8f1afdf55fe8 
crash> dis -l ffffffff8393037a 2/usr/src/debug/kernel-3.10.0-957.el7/linux-3.10.0-957.el7.x86_64/kernel/stop_machine.c: 2440xffffffff8393037a <stop_two_cpus+394>: lea    0x48(%rsp),%rsi0xffffffff8393037f <stop_two_cpus+399>: mov    %r15,%rdi

說明壓棧的是244行的地址，也就是說目前調(diào)用的是243行的 __cpu_stop_queue_work。

然后分析對應(yīng)的入?yún)ⅲ?/p>

crash> task_numa_env ffff8f4957fbfc30struct task_numa_env {  p = 0xffff8f4aea3a8000,   src_cpu = 26,   src_nid = 0,   dst_cpu = 63,   dst_nid = 1,   src_stats = {    nr_running = 11,     load = 556, ---load高    compute_capacity = 18848, ---容量相當(dāng)    task_capacity = 18,     has_free_capacity = 1  },   dst_stats = {    nr_running = 3,     load = 111, ---load低，且容量相當(dāng)，要遷移過來    compute_capacity = 18847, ---容量相當(dāng)    task_capacity = 18,     has_free_capacity = 1  },   imbalance_pct = 112,   idx = 0,   best_task = 0xffff8f1ae77dc100, ---要對調(diào)的task，是通過 task_numa_find_cpu-->task_numa_compare-->task_numa_assign 來獲取的  best_imp = 763,   best_cpu = 1---最佳的swap的對象對于1號cpu}
crash> migration_swap_arg ffff8f4957fbfbc0 struct migration_swap_arg {  src_task = 0xffff8f4aea3a8000,   dst_task = 0xffff8f1ae77dc100,   src_cpu = 26,   dst_cpu = 1-----選擇的dst cpu為1}

根據(jù) cpu_stop_queue_two_works 的代碼，它在持有 cpu_stopper:26號cpu鎖的情況下，去調(diào)用try_to_wake_up ，wake的對象是用來migrate的 kworker。

static void __cpu_stop_queue_work(struct cpu_stopper *stopper,          struct cpu_stop_work *work){  list_add_tail(&work->list, &stopper->works);  wake_up_process(stopper->thread);//其實一般就是喚醒 migration}

由于最佳的cpu對象為1，所以需要cpu上的migrate來拉取進程。

crash> p cpu_stopper:1per_cpu(cpu_stopper, 1) = $33 = {  thread = 0xffff8f1bf6bb9040, ----需要喚醒的目的task  lock = {    {      rlock = {        raw_lock = {          val = {            counter = 1          }        }      }    }  },   enabled = true,   works = {    next = 0xffff8f4957fbfac0,     prev = 0xffff8f4957fbfac0  },   stop_work = {    list = {      next = 0xffff8f4afbe16000,       prev = 0xffff8f4afbe16000    },     fn = 0xffffffff83952100,     arg = 0x0,     done = 0xffff8f1ae3647c08  }}crash> kmem 0xffff8f1bf6bb9040CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZEffff8eecffc05f00 task_struct             4152       1604      2219    317    32k  SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE  fffff26501daee00  ffff8f1bf6bb8000     1      7          7     0  FREE / [ALLOCATED]  [ffff8f1bf6bb9040]
    PID: 14COMMAND: "migration/1"--------------目的task就是對應(yīng)的cpu上的migration    TASK: ffff8f1bf6bb9040  [THREAD_INFO: ffff8f1bf6bc4000]    CPU: 1  STATE: TASK_INTERRUPTIBLE (PANIC)
      PAGE         PHYSICAL      MAPPING       INDEX CNT FLAGSfffff26501daee40 3076bb9000                0        0  0 6fffff00008000 tail

現(xiàn)在的問題是，雖然我們知道了當(dāng)前cpu26號進程在拿了鎖的情況下去喚醒1號cpu上的migrate進程，那么為什么會遲遲不釋放鎖，導(dǎo)致1號cpu因為等待該鎖時間過長而觸發(fā)了hardlock的panic呢？

下面就分析，為什么它持鎖的時間這么長：

 #3 [ffff8f1afdf48ef0] end_repeat_nmi at ffffffff83f6bd69    [exception RIP: try_to_wake_up+114]    RIP: ffffffff838d63d2  RSP: ffff8f4957fbfa30  RFLAGS: 00000002    RAX: 0000000000000001  RBX: ffff8f1bf6bb9844  RCX: 0000000000000000    RDX: 0000000000000001  RSI: 0000000000000003  RDI: ffff8f1bf6bb9844    RBP: ffff8f4957fbfa70   R8: ffff8f4afbe15ff0   R9: 0000000000000000    R10: 0000000000000000  R11: 0000000000000000  R12: 0000000000000000    R13: ffff8f1bf6bb9040  R14: 0000000000000000  R15: 0000000000000003    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000--- <NMI exception stack> --- #4 [ffff8f4957fbfa30] try_to_wake_up at ffffffff838d63d2    ffff8f4957fbfa38: 000000000001ab80 0000000000000086     ffff8f4957fbfa48: ffff8f4afbe15fe0 ffff8f4957fbfb48     ffff8f4957fbfa58: 0000000000000001 ffff8f4afbe15fe0     ffff8f4957fbfa68: ffff8f1afdf55fe0 ffff8f4957fbfa80 
    crash> dis -l ffffffff838d63d2/usr/src/debug/kernel-3.10.0-957.el7/linux-3.10.0-957.el7.x86_64/kernel/sched/core.c: 17900xffffffff838d63d2 <try_to_wake_up+114>:        mov    0x28(%r13),%eax
   1721 static int   1722 try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)   1723 {.....   1787          * If the owning (remote) cpu is still in the middle of schedule() with   1788          * this task as prev, wait until its done referencing the task.   1789          */   1790         while (p->on_cpu)---------原來循環(huán)在此   1791                 cpu_relax();.....   1814         return success;   1815 }

我們用一個簡單的圖來表示一下這個hardlock：

    CPU1                                    CPU26    schedule(.prev=migrate/1)               <fault>      pick_next_task()                        ...        idle_balance()                          migrate_swap()          active_balance()                        stop_two_cpus()                                                    spin_lock(stopper0->lock)                                                    spin_lock(stopper1->lock)                                                    try_to_wake_up                                                      pause() -- waits for schedule()            stop_one_cpu(1)              spin_lock(stopper26->lock) -- waits for stopper lock

查看上游的補丁

 static void __cpu_stop_queue_work(struct cpu_stopper *stopper,-          struct cpu_stop_work *work)+               struct cpu_stop_work *work,+               struct wake_q_head *wakeq) {   list_add_tail(&work->list, &stopper->works);-  wake_up_process(stopper->thread);+  wake_q_add(wakeq, stopper->thread); }

三、故障復(fù)現(xiàn)

由于這個是一個race condition導(dǎo)致的hardlock，邏輯上分析已經(jīng)沒有問題了，就沒有花時間去復(fù)現(xiàn)，該環(huán)境運行一個dpdk的node，不過為了性能設(shè)置了只在一個numa節(jié)點上運行，可以頻繁造成numa的不均衡，所以要復(fù)現(xiàn)的同學(xué)，可以參考單numa節(jié)點上運行dpdk來復(fù)現(xiàn)，會概率大一些。

四、故障規(guī)避或解決

我們的解決方案是：

1.關(guān)閉numa的自動balance.

2.手工合入 linux社區(qū)的 0b26351b910f 補丁

3.這個補丁在centos的 3.10.0-974.el7 合入了:

[kernel] stop_machine, sched: Fix migrate_swap() vs. active_balance() deadlock (Phil Auld) [1557061]

同時紅帽又反向合入到了3.10.0-957.27.2.el7.x86_64，所以把centos內(nèi)核升級到 3.10.0-957.27.2.el7.x86_64也是一種選擇。

看完上述內(nèi)容是否對您有幫助呢？如果還想對相關(guān)知識有進一步的了解或閱讀更多相關(guān)文章，請關(guān)注創(chuàng)新互聯(lián)行業(yè)資訊頻道，感謝您對創(chuàng)新互聯(lián)的支持。

本文題目：怎樣進行numaloadbance的死鎖分析
標(biāo)題路徑：http://www.rwnh.cn/article12/ipccgc.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供網(wǎng)站制作、動態(tài)網(wǎng)站、外貿(mào)網(wǎng)站建設(shè)、ChatGPT、App設(shè)計、微信公眾號

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請盡快告知，我們將會在第一時間刪除。文章觀點不代表本網(wǎng)站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時需注明來源：創(chuàng)新互聯(lián)

猜你還喜歡下面的內(nèi)容

内射老阿姨1区2区3区4区_久久精品人人做人人爽电影蜜月_久久国产精品亚洲77777_99精品又大又爽又粗少妇毛片

怎樣進行numaloadbance的死鎖分析

一、故障現(xiàn)象

四、故障規(guī)避或解決

一、故障現(xiàn)象