[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470682323 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002008] Switched APIC routing to physical x2apic. [ 0.003010] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006008] pid_max: default: 32768 minimum: 301 [ 0.007119] LSM: Security Framework initializing [ 0.008034] Yama: becoming mindful. [ 0.009024] SELinux: Initializing. [ 0.010043] *** VALIDATE selinux *** [ 0.018783] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022754] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024113] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027132] *** VALIDATE tmpfs *** [ 0.028456] *** VALIDATE proc *** [ 0.029221] *** VALIDATE cgroup *** [ 0.030008] *** VALIDATE cgroup2 *** [ 0.031261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034029] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.038421] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.040155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041650] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042026] ... version: 2 [ 0.043012] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045014] ... value mask: 0000ffffffffffff [ 0.046016] ... max period: 00007fffffffffff [ 0.047009] ... fixed-purpose events: 3 [ 0.047936] ... event mask: 000000070000000f [ 0.048233] rcu: Hierarchical SRCU implementation. [ 0.050466] smp: Bringing up secondary CPUs ... [ 0.051593] x86: Booting SMP configuration: [ 0.052039] .... node #0, CPUs: #1 #2 #3 [ 0.056714] smp: Brought up 1 node, 4 CPUs [ 0.058016] smpboot: Max logical packages: 1 [ 0.059020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145115] node 0 deferred pages initialised in 82ms [ 0.148158] devtmpfs: initialized [ 0.149219] x86/mm: Memory block size: 128MB [ 0.151904] gcov: version magic: 0x41383552 [ 0.153153] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154059] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155185] pinctrl core: initialized pinctrl subsystem [ 0.156145] [ 0.156598] ************************************************************* [ 0.157010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158010] ** ** [ 0.159007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160011] ** ** [ 0.161013] ** This means that this kernel is built to expose internal ** [ 0.162012] ** IOMMU data structures, which may compromise security on ** [ 0.163010] ** your system. ** [ 0.164012] ** ** [ 0.165015] ** If you see this message and you are not debugging the ** [ 0.166014] ** kernel, report this immediately to your vendor! ** [ 0.167029] ** ** [ 0.168017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169013] ************************************************************* [ 0.170669] NET: Registered protocol family 16 [ 0.171448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174495] cpuidle: using governor menu [ 0.176838] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179430] PCI: Using configuration type 1 for base access [ 0.181126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195063] cryptd: max_cpu_qlen set to 1000 [ 0.197243] ACPI: Added _OSI(Module Device) [ 0.198015] ACPI: Added _OSI(Processor Device) [ 0.199011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204868] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206584] ACPI: Interpreter enabled [ 0.207074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208014] ACPI: Using IOAPIC for interrupt routing [ 0.209092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210370] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.219629] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.220043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221042] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.222333] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230366] acpiphp: Slot [2] registered [ 0.231126] acpiphp: Slot [5] registered [ 0.232157] acpiphp: Slot [6] registered [ 0.233123] acpiphp: Slot [7] registered [ 0.234156] acpiphp: Slot [8] registered [ 0.235135] acpiphp: Slot [9] registered [ 0.236085] acpiphp: Slot [10] registered [ 0.237042] acpiphp: Slot [3] registered [ 0.237984] acpiphp: Slot [4] registered [ 0.238062] acpiphp: Slot [11] registered [ 0.239084] acpiphp: Slot [12] registered [ 0.240117] acpiphp: Slot [13] registered [ 0.241146] acpiphp: Slot [14] registered [ 0.242083] acpiphp: Slot [15] registered [ 0.243071] acpiphp: Slot [16] registered [ 0.244068] acpiphp: Slot [17] registered [ 0.245076] acpiphp: Slot [18] registered [ 0.246060] acpiphp: Slot [19] registered [ 0.247076] acpiphp: Slot [20] registered [ 0.248080] acpiphp: Slot [21] registered [ 0.250122] acpiphp: Slot [22] registered [ 0.251105] acpiphp: Slot [23] registered [ 0.252091] acpiphp: Slot [24] registered [ 0.253098] acpiphp: Slot [25] registered [ 0.255083] acpiphp: Slot [26] registered [ 0.256066] acpiphp: Slot [27] registered [ 0.257103] acpiphp: Slot [28] registered [ 0.259095] acpiphp: Slot [29] registered [ 0.260082] acpiphp: Slot [30] registered [ 0.261094] acpiphp: Slot [31] registered [ 0.263074] PCI host bridge to bus 0000:00 [ 0.264016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.267020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.269021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.272025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.274021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.277028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.282174] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285246] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.299116] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.302020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.304021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.306016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.308066] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.310881] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.314038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.316852] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.323016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.336016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.343016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.348236] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.360016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.373014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.404014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.416356] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.421011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.425012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.441021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.446053] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.450011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.454011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.464012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.471236] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.488018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.494016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.515016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.527389] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.531011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.535012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.546021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.552113] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.556020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.560012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.569015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.581820] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.584493] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.587510] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.589365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.592209] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.597133] iommu: Default domain type: Passthrough [ 0.598405] SCSI subsystem initialized [ 0.600130] ACPI: bus type USB registered [ 0.602096] usbcore: registered new interface driver usbfs [ 0.604092] usbcore: registered new interface driver hub [ 0.606078] usbcore: registered new device driver usb [ 0.608160] pps_core: LinuxPPS API ver. 1 registered [ 0.610011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.613076] PTP clock support registered [ 0.615134] EDAC MC: Ver: 3.0.0 [ 0.617128] PCI: Using ACPI for IRQ routing [ 0.618789] NetLabel: Initializing [ 0.619008] NetLabel: domain hash size = 128 [ 0.621008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.622071] NetLabel: unlabeled traffic allowed by default [ 0.624113] vgaarb: loaded [ 0.625256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.626010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.633419] clocksource: Switched to clocksource kvm-clock [ 0.738887] VFS: Disk quotas dquot_6.6.0 [ 0.740069] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.742065] *** VALIDATE ramfs *** [ 0.742962] *** VALIDATE hugetlbfs *** [ 0.744214] pnp: PnP ACPI init [ 0.746676] pnp: PnP ACPI: found 6 devices [ 0.775187] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.777966] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.779613] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.781678] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.783407] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.785116] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.787141] NET: Registered protocol family 2 [ 0.788857] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.792842] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.795851] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.799957] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.802637] TCP: Hash tables configured (established 65536 bind 65536) [ 0.805553] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.807830] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.810013] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.812198] NET: Registered protocol family 1 [ 0.814564] RPC: Registered named UNIX socket transport module. [ 0.816278] RPC: Registered udp transport module. [ 0.817274] RPC: Registered tcp transport module. [ 0.818932] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.821427] NET: Registered protocol family 44 [ 0.822837] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.825092] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.827194] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.829538] PCI: CLS 0 bytes, default 64 [ 0.831173] Unpacking initramfs... [ 2.170825] debug: unmapping init [mem 0xffff9e9e3cc54000-0xffff9e9e3ffbffff] [ 2.174747] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.177070] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.180181] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.653885] Initialise system trusted keyrings [ 2.655318] Key type blacklist registered [ 2.656833] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.666955] zbud: loaded [ 2.670266] *** VALIDATE nfs *** [ 2.671288] *** VALIDATE nfs4 *** [ 2.672527] pstore: using deflate compression [ 2.676280] Platform Keyring initialized [ 2.772174] NET: Registered protocol family 38 [ 2.774198] Key type asymmetric registered [ 2.775054] Asymmetric key parser 'x509' registered [ 2.776858] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.780165] io scheduler mq-deadline registered [ 2.784079] io scheduler kyber registered [ 2.786527] io scheduler bfq registered [ 2.788115] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.790516] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.792506] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.794781] ACPI: Power Button [PWRF] [ 2.799721] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.805547] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.815854] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.821521] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.836412] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.862242] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.889151] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.894545] Non-volatile memory driver v1.3 [ 2.896478] Linux agpgart interface v0.103 [ 2.925335] virtio_blk virtio1: [vda] 139368 512-byte logical blocks (71.4 MB/68.1 MiB) [ 2.928200] vda: detected capacity change from 0 to 71356416 [ 2.947534] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.950290] vdb: detected capacity change from 0 to 1073741824 [ 2.965858] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.967884] vdc: detected capacity change from 0 to 2621440000 [ 2.985377] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.988412] vdd: detected capacity change from 0 to 2621440000 [ 3.008387] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.010899] vde: detected capacity change from 0 to 4294967296 [ 3.025419] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.027532] vdf: detected capacity change from 0 to 4294967296 [ 3.031915] libphy: Fixed MDIO Bus: probed [ 3.040446] usbcore: registered new interface driver usbserial_generic [ 3.042228] usbserial: USB Serial support registered for generic [ 3.043926] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.047196] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.048516] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.050389] mousedev: PS/2 mouse device common for all mice [ 3.055291] rtc_cmos 00:05: RTC can wake from S4 [ 3.057608] rtc_cmos 00:05: registered as rtc0 [ 3.059371] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.061330] intel_pstate: CPU model not supported [ 3.063956] hid: raw HID events driver (C) Jiri Kosina [ 3.065251] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.066275] usbcore: registered new interface driver usbhid [ 3.072309] usbhid: USB HID core driver [ 3.073374] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.074048] drop_monitor: Initializing network drop monitor service [ 3.078152] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.078508] Initializing XFRM netlink socket [ 3.082641] NET: Registered protocol family 10 [ 3.084866] Segment Routing with IPv6 [ 3.085909] NET: Registered protocol family 17 [ 3.087416] mpls_gso: MPLS GSO support [ 3.094262] RAS: Correctable Errors collector initialized. [ 3.095883] AVX version of gcm_enc/dec engaged. [ 3.097200] AES CTR mode by8 optimization enabled [ 3.171726] sched_clock: Marking stable (3171695037, 0)->(4041886893, -870191856) [ 3.174221] registered taskstats version 1 [ 3.175690] Loading compiled-in X.509 certificates [ 3.177270] zswap: loaded using pool lzo/zbud [ 3.196386] Key type big_key registered [ 3.206587] Key type encrypted registered [ 3.207521] ima: No TPM chip found, activating TPM-bypass! [ 3.208902] ima: Allocated hash algorithm: sha1 [ 3.209913] ima: No architecture policies found [ 3.211094] evm: Initialising EVM extended attributes: [ 3.212132] evm: security.selinux [ 3.212818] evm: security.ima [ 3.213447] evm: security.capability [ 3.214208] evm: HMAC attrs: 0x1 [ 3.215637] rtc_cmos 00:05: setting system clock to 2026-02-10 00:47:26 UTC (1770684446) [ 3.220307] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.222086] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.231132] Write protecting the kernel read-only data: 28672k [ 3.233221] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.234826] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.261588] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.268086] systemd[1]: Detected virtualization kvm. [ 3.269347] systemd[1]: Detected architecture x86-64. [ 3.270646] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.297759] systemd[1]: No hostname configured. [ 3.299169] systemd[1]: Set hostname to . [ 3.300613] random: systemd: uninitialized urandom read (16 bytes read) [ 3.302562] systemd[1]: Initializing machine ID from random generator. [ 3.448071] random: systemd: uninitialized urandom read (16 bytes read) [ 3.450814] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.455350] random: systemd: uninitialized urandom read (16 bytes read) [ 3.458117] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.462802] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.970419] device-mapper: uevent: version 1.0.3 [ 3.972113] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.745994] virtio_net virtio0 ens2: renamed from eth0 [ 4.762108] random: fast init done [ 4.814360] scsi host0: ata_piix [ 4.830614] scsi host1: ata_piix [ 4.832418] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.835217] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.315354] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.717848] random: crng init done [ 9.719386] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.170381] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.366722] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.590322] SELinux: Disabled at runtime. [ 11.645900] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.651568] systemd[1]: Detected virtualization kvm. [ 11.652843] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.079844] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.082409] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.085807] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.088700] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.090923] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.097484] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.112693] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 12.211572] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.544047] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.811686] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.888614] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.008741] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.018117] EDAC sbridge: Ver: 1.1.2 [ 14.503906] Key type dns_resolver registered [ 14.816789] NFS: Registering the id_resolver key type [ 14.819410] Key type id_resolver registered [ 14.821261] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg327-server login: [ 42.222689] libcfs: loading out-of-tree module taints kernel. [ 42.261117] Key type ._llcrypt registered [ 42.262865] Key type .llcrypt registered [ 42.368827] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 61.242140] hrtimer: interrupt took 6121980 ns [ 63.704832] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 65.804098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.837764] alg: No test for adler32 (adler32-zlib) [ 67.862986] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 68.854111] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 70.634526] Key type lgssc registered [ 72.982220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.973403] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.885957] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 154.823592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.870926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.328372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.366733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.548304] Lustre: lustre-MDT0000: new disk, initializing [ 156.684772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.744885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.521362] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.368395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 177.544264] Lustre: 6494:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.589695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.591701] Lustre: Skipped 1 previous similar message [ 177.742696] Lustre: lustre-MDT0001: new disk, initializing [ 177.846784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.901736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.926691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.130959] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.503545] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.804678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 199.115630] Lustre: lustre-OST0000: new disk, initializing [ 199.121529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.127736] Lustre: 8401:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.136108] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 199.194555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.302179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.319364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.370234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.240642] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.739305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 221.864766] Lustre: lustre-OST0001: new disk, initializing [ 221.868805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.874388] Lustre: 9457:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.934747] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.155939] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.501855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.520174] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.717491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.774757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.305088] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.471416] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.844857] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing yml_node [ 277.245935] Lustre: DEBUG MARKER: Client: 2.17.50.65 [ 280.905736] Lustre: DEBUG MARKER: MDS: 2.17.50.65 [ 284.424546] Lustre: DEBUG MARKER: OSS: 2.17.50.65 [ 286.497879] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Feb 9 19:52:08 EST 2026 [ 311.133354] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 313.457152] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 331.745646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.751974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.761644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.305245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.307307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.323342] Lustre: Skipped 1 previous similar message [ 334.332649] Lustre: Skipped 1 previous similar message [ 336.268401] LustreError: 12200:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 336.412620] Lustre: server umount lustre-MDT0000 complete [ 340.178208] LustreError: 6485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770684783 with bad export cookie 4520171752608199006 [ 340.183815] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.194435] LustreError: 6485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 344.546513] LustreError: 10678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.547392] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.553391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.553402] Lustre: Skipped 2 previous similar messages [ 344.575366] LustreError: 10678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 344.626747] Lustre: Skipped 2 previous similar messages [ 345.568896] LustreError: 6503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.594662] LustreError: 6503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 346.513858] LustreError: 12402:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 346.525556] LustreError: 12402:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 346.867059] Lustre: server umount lustre-MDT0001 complete [ 351.275258] LustreError: 12605:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 351.279500] LustreError: 12605:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 351.523906] Lustre: server umount lustre-OST0000 complete [ 355.794100] LustreError: 12805:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 355.800183] LustreError: 12805:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 356.069991] Lustre: server umount lustre-OST0001 complete [ 362.436832] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 372.030512] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 415.375967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.866649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 415.915400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.023812] Lustre: lustre-MDT0000: new disk, initializing [ 416.168972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.193417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.216182] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 432.580870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 432.704943] Lustre: 14990:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.773555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.778598] Lustre: Skipped 1 previous similar message [ 432.870096] Lustre: lustre-MDT0001: new disk, initializing [ 432.983825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.017053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.031516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 437.819063] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 445.482462] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 450.369681] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 458.814991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 459.131693] Lustre: lustre-OST0000: new disk, initializing [ 459.135129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 459.149041] Lustre: 16733:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 459.241905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.160099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 461.188957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 461.298833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 465.966678] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 473.953387] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 479.049701] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 479.315414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 484.557445] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 484.791898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 492.004129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.005049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 492.005214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 492.020288] Lustre: Skipped 1 previous similar message [ 492.023534] Lustre: Skipped 3 previous similar messages [ 493.198245] LustreError: 17923:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 493.208581] LustreError: 17923:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 493.557220] Lustre: server umount lustre-OST0000 complete [ 499.682359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.695790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.243130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 502.255877] Lustre: Skipped 3 previous similar messages [ 504.670186] LustreError: 18124:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 504.682449] LustreError: 18124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 505.094652] Lustre: server umount lustre-MDT0000 complete [ 507.362950] LustreError: 14996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.385442] LustreError: 14996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 509.942464] LustreError: 14982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770684953 with bad export cookie 4520171752608200539 [ 509.947527] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.959962] LustreError: 14982:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 510.701901] Lustre: server umount lustre-MDT0001 complete [ 515.399303] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:55:57 (1770684957) [ 518.181919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 520.475620] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:56:02 (1770684962) [ 527.384352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.976345] LustreError: 18996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.039324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 532.464188] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 533.475746] LustreError: 18995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.494115] LustreError: 18995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 541.627800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 547.084833] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 553.344215] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 557.857898] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 565.000959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 570.999985] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 576.150712] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 583.138497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.161664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.173558] Lustre: Skipped 1 previous similar message [ 583.180884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 588.553361] LustreError: 21235:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 588.559751] LustreError: 21235:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 588.691757] Lustre: server umount lustre-OST0000 complete [ 593.386606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 593.395714] LustreError: Skipped 1 previous similar message [ 607.200537] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 607.493736] Lustre: server umount lustre-MDT0000 complete [ 608.738287] LustreError: 21149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.760900] LustreError: 21149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 611.376660] LustreError: 18975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685054 with bad export cookie 4520171752608201421 [ 611.378653] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.390298] LustreError: 18975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.742054] Lustre: server umount lustre-MDT0001 complete [ 624.422744] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 627.158534] Key type lgssc unregistered [ 627.428973] LNet: 22173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.434791] LNetError: 22173:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 628.455303] LNet: Removed LNI 192.168.203.127@tcp [ 629.534564] Key type .llcrypt unregistered [ 629.540109] Key type ._llcrypt unregistered [ 652.003369] Key type ._llcrypt registered [ 652.006200] Key type .llcrypt registered [ 652.147329] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 653.276275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 653.364725] alg: No test for adler32 (adler32-zlib) [ 654.389662] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 654.675035] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 656.416218] Key type lgssc registered [ 658.130699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 666.718847] loop: module loaded [ 666.882912] loop0: detected capacity change from 0 to 204800000 [ 667.247395] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.265865] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.594969] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.436207] loop0: detected capacity change from 0 to 204800000 [ 673.614169] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.630087] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.641469] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.503655] loop0: detected capacity change from 0 to 204800000 [ 684.169419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 684.188070] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.555511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 685.677225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 685.782122] Lustre: lustre-MDT0000: new disk, initializing [ 685.873521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.893822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 690.193752] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 700.563150] loop1: detected capacity change from 0 to 204800000 [ 706.857758] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 707.098296] Lustre: lustre-OST07c6: new disk, initializing [ 707.103106] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 707.110542] Lustre: Skipped 1 previous similar message [ 707.114315] Lustre: 25319:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 707.177180] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 713.818330] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 714.876480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 714.888422] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 714.929810] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 719.999156] Lustre: 24320:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 720.011143] Lustre: 24320:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 720.031079] Lustre: 24320:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 720.045049] Lustre: 24320:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 720.061341] Lustre: 24320:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 720.075229] Lustre: 24320:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 735.901969] Lustre: 24320:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 735.913395] Lustre: 24320:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 735.919552] Lustre: 24320:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.927688] Lustre: 24320:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 735.936525] Lustre: 24320:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 735.945726] Lustre: 24320:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.670520] Lustre: 24322:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 740.677478] Lustre: 24322:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 740.683063] Lustre: 24322:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 740.689330] Lustre: 24322:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.695609] Lustre: 24322:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 740.702050] Lustre: 24322:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.709280] Lustre: 24322:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 740.714414] Lustre: 24322:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.720371] Lustre: 24322:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 740.724761] Lustre: 24322:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.733230] Lustre: 24322:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.739533] Lustre: 24322:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.935725] Lustre: 24322:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 743.942818] Lustre: 24322:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 743.955524] Lustre: 24322:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 743.962805] Lustre: 24322:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.969136] Lustre: 24322:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 743.989572] Lustre: 24322:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.997470] Lustre: 24322:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 744.007827] Lustre: 24322:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.023209] Lustre: 24322:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 744.032112] Lustre: 24322:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.040239] Lustre: 24322:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.050993] Lustre: 24322:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.693691] Lustre: 24320:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 748.704809] Lustre: 24320:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 748.710556] Lustre: 24320:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 748.715026] Lustre: 24320:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.721547] Lustre: 24320:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 748.726945] Lustre: 24320:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.732599] Lustre: 24320:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 748.737883] Lustre: 24320:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.742652] Lustre: 24320:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 748.746426] Lustre: 24320:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.752414] Lustre: 24320:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 748.757733] Lustre: 24320:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.846703] Lustre: 25997:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 759.857246] Lustre: 25997:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 759.869507] Lustre: 25997:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 759.885910] Lustre: 25997:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.898258] Lustre: 25997:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 759.904415] Lustre: 25997:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.909515] Lustre: 25997:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 759.912494] Lustre: 25997:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.916179] Lustre: 25997:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 759.919759] Lustre: 25997:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.922327] Lustre: 25997:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 759.925820] Lustre: 25997:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 807.395632] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 807.407955] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.443436] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 812.002250] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 817.124446] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 821.216079] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 821.309145] LustreError: 26507:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 821.505988] Lustre: server umount lustre-OST07c6 complete [ 831.808449] LustreError: 27023:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 831.818953] LustreError: 27023:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 832.042093] Lustre: server umount lustre-MDT0000 complete [ 850.797579] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 853.419834] Key type lgssc unregistered [ 853.708485] LNet: 27856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.724529] LNetError: 27856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.748641] LNet: Removed LNI 192.168.203.127@tcp [ 854.439539] Key type .llcrypt unregistered [ 854.441486] Key type ._llcrypt unregistered [ 873.645900] Key type ._llcrypt registered [ 873.658158] Key type .llcrypt registered [ 873.764302] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 874.733868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 874.943261] alg: No test for adler32 (adler32-zlib) [ 876.030567] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 876.216318] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 877.880381] Key type lgssc registered [ 878.666778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 884.937513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 884.951686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.313187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.599630] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 896.486125] LustreError: 29105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 896.504645] LustreError: 29105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 897.564234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 897.990899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 902.461168] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 907.758621] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 911.263318] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 917.670596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 918.048851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.256626] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 929.590609] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 931.377593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 948.711318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 948.720304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.737143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.781988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.792232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.801376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.698236] LustreError: 31551:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 954.876450] Lustre: server umount lustre-OST0000 complete [ 959.458848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 959.464156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.489242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 959.498443] Lustre: Skipped 1 previous similar message [ 962.017863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.019296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.583797] LustreError: 31753:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 964.591537] LustreError: 31753:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 964.706776] Lustre: server umount lustre-MDT0000 complete [ 967.140271] LustreError: 29101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.156847] LustreError: 29101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 968.176068] LustreError: 29085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685411 with bad export cookie 14668953142326770436 [ 968.187908] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.317869] LustreError: 31956:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 968.324108] LustreError: 31956:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 968.536249] Lustre: server umount lustre-MDT0001 complete [ 977.944930] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 980.612330] Key type lgssc unregistered [ 980.915744] LNet: 32488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.928348] LNetError: 32488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.938283] LNet: Removed LNI 192.168.203.127@tcp [ 981.635123] Key type .llcrypt unregistered [ 981.637361] Key type ._llcrypt unregistered [ 999.342490] Key type ._llcrypt registered [ 999.347878] Key type .llcrypt registered [ 999.471883] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1000.752151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1000.776798] alg: No test for adler32 (adler32-zlib) [ 1001.972202] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1002.292465] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1004.032269] Key type lgssc registered [ 1005.606952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1013.116688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1013.139347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.737591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.463980] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1023.985627] LustreError: 33738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.010992] LustreError: 33738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1025.921344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1026.258277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.414612] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1035.520798] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1039.378722] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1045.303699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1045.575456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.250868] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1056.623761] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1058.945353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1071.586116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.597383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.607909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.659065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.663539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.668787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.703385] Lustre: Skipped 1 previous similar message [ 1077.053284] LustreError: 36004:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1077.149696] Lustre: server umount lustre-OST0000 complete [ 1082.861615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1082.874350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.892988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.901616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.906647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.350127] LustreError: 36207:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1087.355527] LustreError: 36207:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1087.513573] Lustre: server umount lustre-MDT0000 complete [ 1090.029924] LustreError: 33739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1090.053780] LustreError: 33739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1090.719712] LustreError: 33718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685534 with bad export cookie 12050859355764953045 [ 1090.724879] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.730522] LustreError: 33718:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1090.956503] LustreError: 36408:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1090.963398] LustreError: 36408:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1091.268490] Lustre: server umount lustre-MDT0001 complete [ 1100.782641] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1103.343815] Key type lgssc unregistered [ 1103.598324] LNet: 36940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.601737] LNetError: 36940:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.617930] LNet: Removed LNI 192.168.203.127@tcp [ 1104.392214] Key type .llcrypt unregistered [ 1104.394280] Key type ._llcrypt unregistered [ 1126.799988] Key type ._llcrypt registered [ 1126.802551] Key type .llcrypt registered [ 1126.883098] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1127.654415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1127.699194] alg: No test for adler32 (adler32-zlib) [ 1128.825578] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1129.033875] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1130.771483] Key type lgssc registered [ 1131.935161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1139.509774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1139.543313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.270593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.467579] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1151.461094] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.486796] LustreError: 38338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1154.103152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1154.515446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1158.397548] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1164.032781] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1168.070561] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1174.408095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1174.720111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.654302] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1184.844944] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1185.966324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1195.459532] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1195.489773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1195.496298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.514664] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.518600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.533459] Lustre: Skipped 1 previous similar message [ 1196.545591] Lustre: Skipped 3 previous similar messages [ 1199.846876] LustreError: 40578:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1199.958729] Lustre: server umount lustre-MDT0000 complete [ 1200.550874] LustreError: 39525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.649340] LustreError: 38339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.664177] LustreError: 38339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1203.346789] LustreError: 38319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685646 with bad export cookie 13387996386932385428 [ 1203.350049] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1203.361697] LustreError: 38319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1205.669935] LustreError: 38335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.719615] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [ 1206.755656] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.809820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1208.823803] Lustre: Skipped 1 previous similar message [ 1209.571158] LustreError: 40782:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1209.579581] LustreError: 40782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1209.732301] Lustre: server umount lustre-MDT0001 complete [ 1230.413605] LustreError: 41005:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1230.417968] LustreError: 41005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1230.553653] Lustre: server umount lustre-OST0000 complete [ 1244.183942] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1246.284525] Key type lgssc unregistered [ 1246.541632] LNet: 41842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.552376] LNetError: 41842:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.570094] LNet: Removed LNI 192.168.203.127@tcp [ 1247.213227] Key type .llcrypt unregistered [ 1247.215111] Key type ._llcrypt unregistered [ 1266.382404] Key type ._llcrypt registered [ 1266.387365] Key type .llcrypt registered [ 1266.499888] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1267.653769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1267.673031] alg: No test for adler32 (adler32-zlib) [ 1268.708926] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1268.886983] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1270.528302] Key type lgssc registered [ 1271.597780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1278.580903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1278.608322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.210077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.506448] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1290.724122] LustreError: 43216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.735183] LustreError: 43216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1292.122508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1292.570457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1296.729869] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1301.157719] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1304.901556] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1312.452330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1312.881992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.588941] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1324.200918] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1326.192977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1331.685740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1331.686392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.690524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1331.712599] Lustre: Skipped 1 previous similar message [ 1336.381315] LustreError: 45459:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1336.457607] Lustre: server umount lustre-OST0000 complete [ 1354.212156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1354.223654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.238761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.249680] Lustre: Skipped 2 previous similar messages [ 1357.288970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.291635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.319600] Lustre: Skipped 1 previous similar message [ 1359.156878] LustreError: 45836:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1359.159967] LustreError: 45836:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1359.228768] Lustre: server umount lustre-MDT0000 complete [ 1362.406291] LustreError: 44235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.421964] LustreError: 44235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1362.992424] LustreError: 43198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685806 with bad export cookie 6753405954166918317 [ 1362.996186] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.008558] LustreError: 43198:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1369.195753] LustreError: 46037:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1369.208566] LustreError: 46037:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1369.374900] Lustre: server umount lustre-MDT0001 complete [ 1379.332313] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1381.656838] Key type lgssc unregistered [ 1381.945439] LNet: 46571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.957688] LNetError: 46571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.979988] LNet: Removed LNI 192.168.203.127@tcp [ 1382.789608] Key type .llcrypt unregistered [ 1382.792754] Key type ._llcrypt unregistered [ 1402.740510] Key type ._llcrypt registered [ 1402.742725] Key type .llcrypt registered [ 1402.819298] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1403.636915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.748392] alg: No test for adler32 (adler32-zlib) [ 1404.747265] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1404.928612] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1406.592323] Key type lgssc registered [ 1407.584800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.780946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.805101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.647990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.792539] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1426.981979] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.007343] LustreError: 47947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1430.860580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1431.342470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1436.432032] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1441.941499] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1445.345314] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1452.050483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1452.332633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.689660] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1462.583298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1464.287294] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1470.185350] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1473.421922] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1485.325954] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-02-0920h12m07s [ 1486.687545] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1491.428635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.433248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.437431] Lustre: Skipped 2 previous similar messages [ 1491.447612] Lustre: Skipped 2 previous similar messages [ 1492.518724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.526092] Lustre: Skipped 2 previous similar messages [ 1493.998108] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1494.005487] Lustre: Skipped 1 previous similar message [ 1494.582695] LustreError: 50508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1494.716298] Lustre: server umount lustre-MDT0000 complete [ 1497.821024] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1498.607046] LustreError: 48659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.629751] LustreError: 48659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1503.222414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1503.322936] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1503.524497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.594780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1504.565452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1506.687898] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1509.643628] Lustre: DEBUG MARKER: Wait for df (36155) ... [ 1510.748978] Lustre: DEBUG MARKER: done [ 1514.978624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.980509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1514.993860] Lustre: Skipped 1 previous similar message [ 1520.105914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1520.117686] Lustre: Skipped 2 previous similar messages [ 1520.319351] LustreError: 51564:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1520.322815] LustreError: 51564:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1520.428831] Lustre: server umount lustre-OST0000 complete [ 1525.219639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1525.221339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1525.245081] Lustre: Skipped 1 previous similar message [ 1529.287595] LustreError: 51765:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1529.292868] LustreError: 51765:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1529.377374] Lustre: server umount lustre-MDT0000 complete [ 1531.362265] LustreError: 48659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.373670] LustreError: 48659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1532.287048] LustreError: 47926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770685975 with bad export cookie 13002984074965417214 [ 1532.296650] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1532.303305] LustreError: 47926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1532.444803] LustreError: 51967:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1532.449479] LustreError: 51967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1532.602854] Lustre: server umount lustre-MDT0001 complete [ 1542.052833] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1544.287109] Key type lgssc unregistered [ 1544.519861] LNet: 52498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.528886] LNetError: 52498:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.546601] LNet: Removed LNI 192.168.203.127@tcp [ 1545.434877] Key type .llcrypt unregistered [ 1545.436701] Key type ._llcrypt unregistered [ 1579.114802] Key type ._llcrypt registered [ 1579.122394] Key type .llcrypt registered [ 1579.263933] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1580.257857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1580.367441] alg: No test for adler32 (adler32-zlib) [ 1581.472302] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1581.713460] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1583.424161] Key type lgssc registered [ 1584.325808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1590.477982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1590.500609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.723585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1591.737796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1591.980843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.217977] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1601.577681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1601.736425] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1601.787634] Lustre: 54663:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1601.993842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1605.245036] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1609.645963] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1612.842627] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1618.953276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1619.127843] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1619.272083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.662879] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1625.212100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1629.293815] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1635.684856] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-02-0920h14m38s [ 1636.960320] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1642.336103] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1644.028866] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1644.098370] LustreError: 54672:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1664.468106] Lustre: lustre-MDT0000: Client bcf41830-9ca3-46d4-a97d-26775a8af720 (at 192.168.203.27@tcp) reconnecting [ 1684.109475] LustreError: 54672:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1686.675691] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1688.937255] Lustre: DEBUG MARKER: done [ 1696.082118] LustreError: 57706:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1696.217107] Lustre: server umount lustre-OST0000 complete [ 1702.369882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.383683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.459955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.464747] Lustre: Skipped 2 previous similar messages [ 1709.542156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.552044] Lustre: Skipped 1 previous similar message [ 1709.562682] LustreError: 57909:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1709.575156] LustreError: 57909:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1709.699477] Lustre: server umount lustre-MDT0000 complete [ 1713.632510] Lustre: 53989:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770686140/real 1770686140] req@000000006a7ef734 x1856698868353536/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1770686156 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1713.632808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.676699] Lustre: 53989:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1713.690321] Lustre: Skipped 1 previous similar message [ 1713.751573] LustreError: 54658:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686157 with bad export cookie 11745234157317455927 [ 1713.769524] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1713.960112] LustreError: 58111:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1713.969894] LustreError: 58111:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1714.213499] Lustre: server umount lustre-MDT0001 complete [ 1724.012743] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1726.851250] Key type lgssc unregistered [ 1727.161769] LNet: 58644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.172848] LNetError: 58644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.231371] LNet: Removed LNI 192.168.203.127@tcp [ 1728.866805] Key type .llcrypt unregistered [ 1728.869226] Key type ._llcrypt unregistered [ 1763.615888] Key type ._llcrypt registered [ 1763.618302] Key type .llcrypt registered [ 1763.692721] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1764.545801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1764.584069] alg: No test for adler32 (adler32-zlib) [ 1765.628193] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 1765.807254] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1767.464814] Key type lgssc registered [ 1768.577924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1771.147309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1771.166574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.575580] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1772.616342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1772.853480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.903635] Lustre: Failing over lustre-MDT0000 [ 1774.042510] LustreError: 60850:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1774.225934] Lustre: server umount lustre-MDT0000 complete [ 1779.805678] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:17:02 (1770686222) [ 1785.582928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.067287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1789.732731] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1797.031621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1797.118784] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1797.150628] Lustre: 61406:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1797.326648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.598839] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1804.999144] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1807.723497] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1812.731803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1812.902803] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1813.069085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.357510] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1818.684172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1822.365449] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1828.429932] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1833.957028] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.961048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.976175] Lustre: Skipped 1 previous similar message [ 1833.984489] Lustre: Skipped 1 previous similar message [ 1835.696524] LustreError: 63776:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1835.700169] LustreError: 63776:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1835.900621] Lustre: server umount lustre-MDT0000 complete [ 1838.562222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1838.567682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.887547] LustreError: 61396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686282 with bad export cookie 911305114329654856 [ 1838.894716] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1838.900720] LustreError: 61396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1839.045535] LustreError: 63976:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1839.054103] LustreError: 63976:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1839.080604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1839.233040] Lustre: server umount lustre-MDT0001 complete [ 1842.241632] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1845.987237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1853.521923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.043455] LustreError: 63027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1865.066281] LustreError: 63027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1865.151212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.290387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1868.224149] LustreError: 63027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.411324] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1873.386121] LustreError: 63027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1876.845651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1877.405533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.448662] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1881.012091] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1885.978439] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1888.687811] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1889.294657] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1896.580277] LustreError: 66231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1896.583200] LustreError: 66231:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1896.818947] Lustre: server umount lustre-MDT0000 complete [ 1897.954646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1897.963033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.971321] Lustre: Skipped 1 previous similar message [ 1897.975989] LustreError: 63027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.510772] LustreError: 61396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686343 with bad export cookie 911305114329657138 [ 1900.519915] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.529761] LustreError: 61396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1900.682912] LustreError: 66433:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1900.686824] LustreError: 66433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1900.878844] Lustre: server umount lustre-MDT0001 complete [ 1907.207520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1916.632908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.480557] Lustre: 60330:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770686347/real 1770686347] req@000000001eb44882 x1856699062347392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770686363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1920.529541] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.546026] Lustre: Skipped 2 previous similar messages [ 1921.569051] Lustre: 60330:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770686348/real 1770686348] req@00000000a6bda289 x1856699062347520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770686364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1925.602651] Lustre: 60329:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770686352/real 1770686352] req@0000000096217b52 x1856699062347648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770686368 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1925.892572] LustreError: 63027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.909738] LustreError: 63027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1925.961750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.006693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1929.982940] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1931.104449] Lustre: 60330:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770686358/real 1770686358] req@000000005f0fa6ed x1856699062348032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770686374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1931.125246] Lustre: 60330:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1935.330866] LustreError: 67286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.352357] LustreError: 67286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1936.352185] Lustre: 60331:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770686362/real 1770686362] req@00000000e03a7b52 x1856699062348288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770686378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1937.252969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1939.703919] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1941.582333] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1946.550459] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.658351] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1955.302865] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.313100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1955.313635] Lustre: Skipped 1 previous similar message [ 1955.320978] Lustre: Skipped 1 previous similar message [ 1957.047548] LustreError: 68684:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1957.052587] LustreError: 68684:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1957.234611] Lustre: server umount lustre-MDT0000 complete [ 1960.429123] LustreError: 67288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.447524] LustreError: 67288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1960.548969] LustreError: 62124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686403 with bad export cookie 911305114329658510 [ 1960.554644] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.562049] LustreError: 62124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1960.924694] Lustre: server umount lustre-MDT0001 complete [ 1966.183388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1969.273506] Lustre: DEBUG MARKER: files compared the same [ 1981.116390] LustreError: 69478:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1981.121503] LustreError: 69478:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1981.247650] Lustre: server umount lustre-OST0000 complete [ 1994.699054] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1997.011283] Key type lgssc unregistered [ 1997.292358] LNet: 70316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.297572] LNetError: 70316:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.312119] LNet: Removed LNI 192.168.203.127@tcp [ 1998.062555] Key type .llcrypt unregistered [ 1998.064618] Key type ._llcrypt unregistered [ 2018.931407] Key type ._llcrypt registered [ 2018.932750] Key type .llcrypt registered [ 2019.031977] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2019.979917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.072722] alg: No test for adler32 (adler32-zlib) [ 2021.085503] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2021.282851] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2022.928166] Key type lgssc registered [ 2023.802294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2030.149724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2030.168785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.539125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.446191] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2040.826127] LustreError: 71728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.860374] LustreError: 71728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2042.818826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2043.014995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2047.131509] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2051.952304] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2055.737649] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2062.356420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2062.640247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.441356] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2072.371806] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2075.944748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2083.297963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2083.300959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.310313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2086.370209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.375198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2086.397883] Lustre: Skipped 1 previous similar message [ 2089.156099] LustreError: 73970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2089.270431] Lustre: server umount lustre-OST0000 complete [ 2094.561907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2094.581526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.608968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.609055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.610246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.633601] Lustre: Skipped 1 previous similar message [ 2098.523748] LustreError: 74173:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2098.527062] LustreError: 74173:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2098.579579] Lustre: server umount lustre-MDT0000 complete [ 2101.328324] LustreError: 74174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686544 with bad export cookie 4976313905857238448 [ 2101.328762] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2101.341386] LustreError: 74174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2101.495847] LustreError: 74375:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2101.504345] LustreError: 74375:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2101.658498] Lustre: server umount lustre-MDT0001 complete [ 2111.826191] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2114.774461] Key type lgssc unregistered [ 2115.081590] LNet: 74909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.086916] LNetError: 74909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.099488] LNet: Removed LNI 192.168.203.127@tcp [ 2115.949494] Key type .llcrypt unregistered [ 2115.953950] Key type ._llcrypt unregistered [ 2137.168301] Key type ._llcrypt registered [ 2137.170596] Key type .llcrypt registered [ 2137.270144] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2138.090840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2138.251918] alg: No test for adler32 (adler32-zlib) [ 2139.239829] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2139.399455] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2141.024247] Key type lgssc registered [ 2141.977911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2147.827218] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2147.848151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2163.425839] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2168.544210] LustreError: 76272:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.127@tcp: failed processing log, type 4: rc = -110 [ 2195.232546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.099933] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2204.780183] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2211.245569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2211.322374] Lustre: *** cfs_fail_loc=706, val=0*** [ 2211.584625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2211.639766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2214.595180] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2216.931368] LustreError: 76297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.953434] LustreError: 76297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2221.472476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2221.753144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.688365] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2229.087193] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2231.809972] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2235.364389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.367841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2235.388589] Lustre: Skipped 1 previous similar message [ 2235.395688] Lustre: Skipped 1 previous similar message [ 2239.546133] LustreError: 78590:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2239.656795] Lustre: server umount lustre-OST0000 complete [ 2245.604592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.619841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2245.623500] Lustre: Skipped 1 previous similar message [ 2250.723895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.728987] Lustre: Skipped 1 previous similar message [ 2254.818753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.823593] Lustre: Skipped 1 previous similar message [ 2257.376643] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2257.457608] LustreError: 78792:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2257.462591] LustreError: 78792:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2257.545585] Lustre: server umount lustre-MDT0000 complete [ 2259.938632] LustreError: 77245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.964201] LustreError: 77245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2260.562886] LustreError: 76278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770686703 with bad export cookie 7805467320278444574 [ 2260.577431] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.692865] LustreError: 78994:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2260.706265] LustreError: 78994:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2260.872729] Lustre: server umount lustre-MDT0001 complete [ 2268.683740] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2270.690585] Key type lgssc unregistered [ 2270.961383] LNet: 79525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.968598] LNetError: 79525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.045169] LNet: Removed LNI 192.168.203.127@tcp [ 2272.612437] Key type .llcrypt unregistered [ 2272.614205] Key type ._llcrypt unregistered [ 2290.309907] Key type ._llcrypt registered [ 2290.313712] Key type .llcrypt registered [ 2290.405595] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2291.113415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2291.231473] alg: No test for adler32 (adler32-zlib) [ 2292.189398] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2292.397279] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2294.088217] Key type lgssc registered [ 2295.006869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2301.121920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2301.137268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.541104] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2311.439784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2311.813686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2314.790119] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2317.283111] LustreError: 81570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.293722] LustreError: 81570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2321.168082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2321.384203] LustreError: 81570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.398393] LustreError: 81570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2321.405496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.403366] LustreError: 81573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.426643] LustreError: 81573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2326.358421] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2327.527971] LustreError: 82300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.545294] LustreError: 82300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2332.644329] LustreError: 81574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.666425] LustreError: 81574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2333.056767] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.070908] Lustre: MGS: Client fceda41b-38b2-48e9-b582-fb3e708eabdd (at 0@lo) reconnecting [ 2333.330869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.388666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2337.044203] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2348.514784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2348.520896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.533066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2349.025218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2349.042455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.055883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2349.059730] Lustre: Skipped 1 previous similar message [ 2354.150352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.304155] LustreError: 83529:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2354.435653] Lustre: server umount lustre-OST0000 complete [ 2359.266968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2359.275970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.286609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2359.286618] Lustre: Skipped 1 previous similar message [ 2359.309670] Lustre: Skipped 1 previous similar message [ 2364.396508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.399186] Lustre: Skipped 1 previous similar message [ 2371.041823] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2371.129194] LustreError: 83731:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2371.134723] LustreError: 83731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2371.172282] Lustre: server umount lustre-MDT0000 complete [ 2374.046945] LustreError: 83934:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2374.054104] LustreError: 83934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2374.159654] Lustre: server umount lustre-MDT0001 complete [ 2377.309743] LustreError: 84134:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2377.317036] LustreError: 84134:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2377.379901] Lustre: server umount MGS complete [ 2385.773746] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:27:08 (1770686828) [ 2406.990197] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 2413.832920] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2442.259448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.931760] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2453.513239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2453.683413] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2453.712412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2453.764922] Lustre: lustre-MDT0001: new disk, initializing [ 2453.821192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2453.832864] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2453.835767] Lustre: 88647:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2453.842651] LustreError: 88647:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2456.868433] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2464.333484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2464.507423] Lustre: lustre-OST0000: new disk, initializing [ 2464.509986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2464.517235] Lustre: 89398:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2464.565857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.020553] Lustre: 89435:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000006993a0f5 x1856699613878016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1770686925 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2468.542552] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2474.549319] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2479.944580] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.955955] Lustre: MGS: Client 3047e9e9-9461-4449-9a08-164032513357 (at 0@lo) reconnecting [ 2479.986383] Lustre: 87935:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2480.001538] LustreError: 87935:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2480.028117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2480.097183] Lustre: lustre-MDT0000: new disk, initializing [ 2480.249205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.284580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2480.412935] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2481.313307] Lustre: 90354:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000ca6beebf x1856699613884288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1770686940 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2483.968952] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2486.753840] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2490.414035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2490.427055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2490.557026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2511.330551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.339180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.350626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.357197] Lustre: Skipped 2 previous similar messages [ 2516.288455] LustreError: 90871:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2516.298764] LustreError: 90871:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2516.406867] Lustre: server umount lustre-OST0000 complete [ 2520.039019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.054029] Lustre: Skipped 2 previous similar messages [ 2520.055877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2520.067970] LustreError: Skipped 1 previous similar message [ 2525.284895] LustreError: 91072:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2525.289843] LustreError: 91072:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2525.310220] Lustre: server umount lustre-MDT0000 complete [ 2542.048142] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2542.111468] LustreError: 91273:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2542.115370] LustreError: 91273:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2542.228618] Lustre: server umount lustre-MDT0001 complete [ 2544.753876] Lustre: server umount MGS complete [ 2551.961407] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:29:54 (1770686994) [ 2563.561689] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2565.390572] Key type lgssc unregistered [ 2565.623182] LNet: 92811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.633234] LNetError: 92811:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2565.645824] LNet: Removed LNI 192.168.203.127@tcp [ 2566.264275] Key type .llcrypt unregistered [ 2566.265923] Key type ._llcrypt unregistered [ 2577.585411] Key type ._llcrypt registered [ 2577.587440] Key type .llcrypt registered [ 2577.645750] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2578.207350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2578.234090] alg: No test for adler32 (adler32-zlib) [ 2579.263176] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2579.424283] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2581.048509] Key type lgssc registered [ 2581.841689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2587.650359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2587.685506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.077736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2592.668943] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2599.210107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2599.394717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.428661] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2606.746832] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2609.956499] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2614.743751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2615.040374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.062069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2619.402807] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2623.611344] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2631.660180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2631.668355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.685370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.728626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.730013] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2634.754991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.798305] LustreError: 96232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2636.892424] Lustre: server umount lustre-OST0000 complete [ 2639.846665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.850566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.870969] Lustre: Skipped 1 previous similar message [ 2639.883969] Lustre: Skipped 2 previous similar messages [ 2644.964792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.972025] Lustre: Skipped 1 previous similar message [ 2645.752766] LustreError: 96434:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2645.757426] LustreError: 96434:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2645.827312] Lustre: server umount lustre-MDT0000 complete [ 2648.367454] LustreError: 93974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687091 with bad export cookie 8697168701203297815 [ 2648.373133] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2648.373940] LustreError: 93974:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2648.478313] LustreError: 96635:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2648.483281] LustreError: 96635:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2648.594144] Lustre: server umount lustre-MDT0001 complete [ 2655.356199] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2657.227771] Key type lgssc unregistered [ 2657.454931] LNet: 97167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2657.470161] LNetError: 97167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2657.499109] LNet: Removed LNI 192.168.203.127@tcp [ 2658.073202] Key type .llcrypt unregistered [ 2658.074994] Key type ._llcrypt unregistered [ 2669.548127] Key type ._llcrypt registered [ 2669.549679] Key type .llcrypt registered [ 2669.609759] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2670.168491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2670.336133] alg: No test for adler32 (adler32-zlib) [ 2671.313368] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2671.476438] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2673.096156] Key type lgssc registered [ 2673.959237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2677.872511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2677.877151] LustreError: 98209:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 2677.884722] LustreError: 98210:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 2677.894332] LustreError: 98209:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4989 [ 2677.897956] LustreError: 98209:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2677.908466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.909528] LustreError: 98209:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2679.246977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.528719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2684.719399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.971556] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2691.848267] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing lsmod [ 2693.939298] LustreError: 99372:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 2693.977513] LustreError: 99384:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 2693.980654] LustreError: 99372:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4963 [ 2693.985742] LustreError: 99372:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2693.988706] LustreError: 99372:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2693.995916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2694.239962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.746642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.753371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.770865] Lustre: Skipped 2 previous similar messages [ 2699.776772] Lustre: Skipped 2 previous similar messages [ 2702.354508] LustreError: 99571:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2702.425443] Lustre: server umount lustre-MDT0000 complete [ 2704.865262] LustreError: 98243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.883318] LustreError: 98243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2705.170974] LustreError: 99572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687148 with bad export cookie 17676442304815759966 [ 2705.171830] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.176323] LustreError: 99572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2705.191754] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.200014] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2705.292971] LustreError: 99773:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2705.298697] LustreError: 99773:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2705.403880] Lustre: server umount lustre-MDT0001 complete [ 2708.142653] LustreError: 99974:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2708.145544] LustreError: 99974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2708.216295] Lustre: server umount lustre-OST0000 complete [ 2714.189678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.457283] LustreError: 100448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.478563] LustreError: 100448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2714.518427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.402827] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2719.724299] LustreError: 100448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.833928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2723.097171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.559845] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2729.510440] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2732.225115] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2736.987743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2737.196645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.255982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2740.857069] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2744.714400] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2758.122129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2758.127392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.135970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2758.628292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2763.750923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2763.754167] Lustre: Skipped 2 previous similar messages [ 2768.868551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.878215] Lustre: Skipped 1 previous similar message [ 2769.376331] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2769.418134] LustreError: 102689:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2769.421145] LustreError: 102689:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2769.513187] Lustre: server umount lustre-OST0000 complete [ 2773.986340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.996195] Lustre: Skipped 2 previous similar messages [ 2778.211977] LustreError: 102893:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2778.216551] LustreError: 102893:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2778.303300] Lustre: server umount lustre-MDT0000 complete [ 2779.105673] LustreError: 101582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.110650] LustreError: 101582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2780.806941] LustreError: 100429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687224 with bad export cookie 17676442304815760638 [ 2780.812255] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2780.816322] LustreError: 100429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2781.059953] Lustre: server umount lustre-MDT0001 complete [ 2788.971131] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2791.023320] Key type lgssc unregistered [ 2791.262742] LNet: 103626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2791.274666] LNetError: 103626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2791.286982] LNet: Removed LNI 192.168.203.127@tcp [ 2791.854591] Key type .llcrypt unregistered [ 2791.856795] Key type ._llcrypt unregistered [ 2808.954847] Key type ._llcrypt registered [ 2808.956754] Key type .llcrypt registered [ 2809.042726] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2810.054712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2810.123276] alg: No test for adler32 (adler32-zlib) [ 2811.152463] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2811.334265] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2812.976175] Key type lgssc registered [ 2813.804373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2819.540493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2819.565129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.037060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.474765] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2830.823764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2831.081393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2834.178911] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2838.188201] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2841.426791] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2847.871855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2848.177671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.104429] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2857.238460] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2867.645836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2872.182610] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2874.174903] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2876.902083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2876.910204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.928251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.925712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.926038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2877.936571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.570762] LustreError: 107364:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2882.696826] Lustre: server umount lustre-OST0000 complete [ 2887.657061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2887.663846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.675761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.679706] Lustre: Skipped 1 previous similar message [ 2891.561159] LustreError: 107569:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2891.566519] LustreError: 107569:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2891.665756] Lustre: server umount lustre-MDT0000 complete [ 2893.283730] LustreError: 105783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.299434] LustreError: 105783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2894.379109] LustreError: 104981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687337 with bad export cookie 9449103530781040081 [ 2894.381669] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2894.397046] LustreError: 104981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2894.483699] LustreError: 107772:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2894.489730] LustreError: 107772:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2894.609115] Lustre: server umount lustre-MDT0001 complete [ 2900.753455] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2902.368815] Key type lgssc unregistered [ 2902.528871] LNet: 108302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2902.534512] LNetError: 108302:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2902.546280] LNet: Removed LNI 192.168.203.127@tcp [ 2902.990591] Key type .llcrypt unregistered [ 2902.992629] Key type ._llcrypt unregistered [ 2914.039953] Key type ._llcrypt registered [ 2914.042519] Key type .llcrypt registered [ 2914.156402] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2914.805875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2914.864040] alg: No test for adler32 (adler32-zlib) [ 2915.872482] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2916.059399] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2917.720213] Key type lgssc registered [ 2918.538352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2923.849584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2923.861641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.197894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.965987] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2934.541312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2934.822417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2937.793972] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2941.695433] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2944.071888] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2948.230287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2948.490117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.575242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2952.167618] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2956.276398] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2965.985543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2965.990109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2966.001186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2970.081752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2970.084795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2970.094320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.320117] LustreError: 111723:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2971.445692] Lustre: server umount lustre-OST0000 complete [ 2975.202667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.206850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.208596] Lustre: Skipped 1 previous similar message [ 2975.211323] Lustre: Skipped 1 previous similar message [ 2980.199222] LustreError: 111926:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2980.203778] LustreError: 111926:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2980.271193] Lustre: server umount lustre-MDT0000 complete [ 2980.322887] LustreError: 109479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.334442] LustreError: 109479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2982.935124] LustreError: 109462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687426 with bad export cookie 17862354551671436016 [ 2982.938369] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2982.943927] LustreError: 109462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2983.049428] LustreError: 112128:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2983.053789] LustreError: 112128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2983.152931] Lustre: server umount lustre-MDT0001 complete [ 2989.289367] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2991.136853] Key type lgssc unregistered [ 2991.337628] LNet: 112659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2991.343575] LNetError: 112659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2991.354034] LNet: Removed LNI 192.168.203.127@tcp [ 2991.895361] Key type .llcrypt unregistered [ 2991.897052] Key type ._llcrypt unregistered [ 3006.699186] Key type ._llcrypt registered [ 3006.700525] Key type .llcrypt registered [ 3006.763649] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3007.261210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3007.271170] alg: No test for adler32 (adler32-zlib) [ 3008.231947] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3008.398613] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3010.032393] Key type lgssc registered [ 3010.780734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3016.437647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3016.451619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.870278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3021.798950] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3028.103855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3028.361182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3030.988988] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3034.926176] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3037.376636] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3041.731969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3041.953197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.012916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3045.611975] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3049.117251] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3059.068206] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3061.461204] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3061.464407] Lustre: Skipped 2 previous similar messages [ 3064.419402] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3064.425515] Lustre: Skipped 2 previous similar messages [ 3066.907776] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3066.927870] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3071.348845] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3071.358946] Lustre: Skipped 2 previous similar messages [ 3076.689906] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.27@tcp 192.168.0.[2,10]@tcp [ 3080.607232] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.27/24@tcp in log lustre-MDT0000 [ 3080.612716] Lustre: Skipped 2 previous similar messages [ 3086.909336] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.27/24@tcp [ 3086.916763] Lustre: Skipped 1 previous similar message [ 3093.988199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3093.993045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.994821] LustreError: Skipped 2 previous similar messages [ 3094.016233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3094.502115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3094.514222] LustreError: Skipped 3 previous similar messages [ 3094.518343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.542651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3094.545214] Lustre: Skipped 1 previous similar message [ 3097.853919] LustreError: 117771:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3097.984752] Lustre: server umount lustre-OST0000 complete [ 3104.737771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.742795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3104.754529] Lustre: Skipped 1 previous similar message [ 3106.652036] LustreError: 117973:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3106.662612] LustreError: 117973:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3106.724394] Lustre: server umount lustre-MDT0000 complete [ 3109.348234] LustreError: 114013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687552 with bad export cookie 2243340608127680948 [ 3109.349311] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3109.356226] LustreError: 114013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3109.519921] LustreError: 118175:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3109.523970] LustreError: 118175:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3109.683742] Lustre: server umount lustre-MDT0001 complete [ 3117.308917] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3119.255544] Key type lgssc unregistered [ 3119.453676] LNet: 118707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3119.457767] LNetError: 118707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3119.468064] LNet: Removed LNI 192.168.203.127@tcp [ 3120.012926] Key type .llcrypt unregistered [ 3120.017960] Key type ._llcrypt unregistered [ 3140.005228] Key type ._llcrypt registered [ 3140.007629] Key type .llcrypt registered [ 3140.138495] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3140.903433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3141.002083] alg: No test for adler32 (adler32-zlib) [ 3142.007314] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3142.174202] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3143.800260] Key type lgssc registered [ 3144.496661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3150.048298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3150.066482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.424555] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3151.446282] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.27/24@tcp [ 3151.458208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.107920] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3159.515055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3159.724666] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3159.742714] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.27/24@tcp [ 3159.765626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3162.439367] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3166.291469] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3168.993512] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3173.792404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3174.027206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.109137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3177.579028] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3181.180594] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3191.777899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3191.782925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.794870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3194.849241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3194.852433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3194.855272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.113342] LustreError: 122523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3197.198218] Lustre: server umount lustre-OST0000 complete [ 3199.971749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.974879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.986115] Lustre: Skipped 1 previous similar message [ 3199.994351] Lustre: Skipped 1 previous similar message [ 3205.092157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.095623] Lustre: Skipped 1 previous similar message [ 3205.848936] LustreError: 122724:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3205.854681] LustreError: 122724:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3206.011314] Lustre: server umount lustre-MDT0000 complete [ 3208.505184] LustreError: 120208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687651 with bad export cookie 2886258385413947681 [ 3208.508940] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3208.513388] LustreError: 120208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3208.611662] LustreError: 122925:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3208.617968] LustreError: 122925:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3208.730678] Lustre: server umount lustre-MDT0001 complete [ 3215.619856] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3217.411660] Key type lgssc unregistered [ 3217.597622] LNet: 123457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3217.602247] LNetError: 123457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3217.616742] LNet: Removed LNI 192.168.203.127@tcp [ 3218.123330] Key type .llcrypt unregistered [ 3218.125316] Key type ._llcrypt unregistered [ 3232.537283] Key type ._llcrypt registered [ 3232.538713] Key type .llcrypt registered [ 3232.592894] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 3241.350406] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3241.988406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3241.998630] alg: No test for adler32 (adler32-zlib) [ 3242.926423] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3243.062151] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3244.664161] Key type lgssc registered [ 3245.233639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3263.639231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3263.650699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.826362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3264.849137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3264.901653] Lustre: lustre-MDT0000: new disk, initializing [ 3264.962067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.977790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3266.888545] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3273.171091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3273.231688] Lustre: 126793:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3273.267316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3273.270430] Lustre: Skipped 1 previous similar message [ 3273.311708] Lustre: lustre-MDT0001: new disk, initializing [ 3273.357846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3273.375959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3273.380263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3275.243846] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3279.522212] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3281.263415] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3284.687184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3284.830731] Lustre: lustre-OST0000: new disk, initializing [ 3284.833962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3284.837471] Lustre: 128535:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3284.884818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.480472] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3291.613568] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3292.689387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3292.693690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3292.750820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3293.754794] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3293.852961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3295.758784] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3295.853237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3297.763473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.764614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3297.774285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3297.776882] LustreError: Skipped 1 previous similar message [ 3302.771512] LustreError: 129726:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3302.826491] Lustre: server umount lustre-OST0000 complete [ 3308.003785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.011359] Lustre: Skipped 2 previous similar messages [ 3308.013558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3308.018306] Lustre: Skipped 2 previous similar messages [ 3311.012743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.016889] Lustre: Skipped 1 previous similar message [ 3316.194120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.200261] Lustre: Skipped 1 previous similar message [ 3319.264196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3319.313826] LustreError: 129927:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3319.316829] LustreError: 129927:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3319.351956] Lustre: server umount lustre-MDT0000 complete [ 3321.078987] LustreError: 126786:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687764 with bad export cookie 6583682221386522345 [ 3321.082177] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.083326] LustreError: 126786:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3321.161877] LustreError: 130129:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3321.165496] LustreError: 130129:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3321.248951] Lustre: server umount lustre-MDT0001 complete [ 3325.546866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.894721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.171097] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3331.042201] LustreError: 130605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.050642] LustreError: 130605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3332.646192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3332.855640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3334.825577] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3337.515996] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.664721] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3343.345964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3343.539498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.621405] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3349.375488] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3351.222954] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3352.353751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3354.373375] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3358.566172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3363.856304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3363.948289] Lustre: lustre-OST0001: new disk, initializing [ 3363.951359] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3363.955246] Lustre: 133640:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3364.003704] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.598205] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3370.537742] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3372.035541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3372.040091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3372.073479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3372.671695] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3374.183504] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3374.272688] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3376.000669] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0de0155000.ost_server_uuid 50 [ 3376.935733] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f0de0155000.ost_server_uuid in FULL state after 0 sec [ 3382.245247] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.245306] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3382.246595] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3382.246598] Lustre: Skipped 2 previous similar messages [ 3382.250559] Lustre: Skipped 1 previous similar message [ 3382.253663] LustreError: Skipped 1 previous similar message [ 3385.774564] LustreError: 134895:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3385.776958] LustreError: 134895:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3385.798222] Lustre: server umount lustre-OST0001 complete [ 3387.366726] LustreError: 134187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.373058] LustreError: 134187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3387.873045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3387.876267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3392.484066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3392.485030] LustreError: 132199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.486345] Lustre: Skipped 4 previous similar messages [ 3392.492160] LustreError: 132199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3393.523100] LustreError: 135097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3393.527363] LustreError: 135097:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3393.577248] Lustre: server umount lustre-OST0000 complete [ 3397.601594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.605970] Lustre: Skipped 2 previous similar messages [ 3401.316840] Lustre: server umount lustre-MDT0000 complete [ 3402.721147] LustreError: 130605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.727524] LustreError: 130605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3402.810657] LustreError: 130584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687846 with bad export cookie 6583682221386523220 [ 3402.811446] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3402.819038] LustreError: 130584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3402.887526] LustreError: 135501:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3402.889987] LustreError: 135501:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3402.953593] Lustre: server umount lustre-MDT0001 complete [ 3410.324568] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3411.434464] Key type lgssc unregistered [ 3411.564505] LNet: 136489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3411.567670] LNetError: 136489:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3411.578346] LNet: Removed LNI 192.168.203.127@tcp [ 3411.892989] Key type .llcrypt unregistered [ 3411.894498] Key type ._llcrypt unregistered [ 3427.735344] Key type ._llcrypt registered [ 3427.736621] Key type .llcrypt registered [ 3427.785899] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 3435.247821] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3435.647565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3435.704709] alg: No test for adler32 (adler32-zlib) [ 3436.588710] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3436.682343] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3438.272168] Key type lgssc registered [ 3438.753902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3453.602044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3453.609018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.737543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3454.752242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3454.793115] Lustre: lustre-MDT0000: new disk, initializing [ 3454.827919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.839353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3456.411495] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3461.771280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3461.818707] Lustre: 140577:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3461.846502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3461.849334] Lustre: Skipped 1 previous similar message [ 3461.891493] Lustre: lustre-MDT0001: new disk, initializing [ 3461.942949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3461.961599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3461.968041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3463.763887] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3467.658548] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3469.406849] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3472.448897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3472.579342] Lustre: lustre-OST0000: new disk, initializing [ 3472.581595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3472.585870] Lustre: 142319:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3472.633156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.056366] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3478.543982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3478.547722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3478.596806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3478.880536] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3481.125484] Lustre: Failing over lustre-OST0000 [ 3481.163097] LustreError: 143011:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3481.215762] Lustre: server umount lustre-OST0000 complete [ 3494.612687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3494.728808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3494.734195] Lustre: Skipped 1 previous similar message [ 3494.741802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.749759] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3495.857341] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3496.930501] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3498.359359] Lustre: Failing over lustre-MDT0000 [ 3498.429368] LustreError: 143982:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3498.432129] LustreError: 143982:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3498.462235] Lustre: server umount lustre-MDT0000 complete [ 3499.424149] Lustre: 138803:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770687926/real 1770687926] req@000000000ee5b87f x1856700813436928/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1770687942 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3500.001777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.007974] Lustre: Skipped 1 previous similar message [ 3505.057334] LustreError: 140589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.063889] LustreError: 140589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3506.080176] Lustre: 138804:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770687932/real 1770687932] req@0000000068b61fd0 x1856700813438976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1770687948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3506.085535] LustreError: 140583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3506.090272] Lustre: 138804:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3510.177074] Lustre: 138806:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770687937/real 1770687937] req@0000000053f1abef x1856700813440128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1770687953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3510.241382] LustreError: 140589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.246933] LustreError: 140589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3511.612235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.669896] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3511.674364] Lustre: 140580:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.127@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3511.764781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3511.783124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.817425] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3513.410907] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3516.327738] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3516.903236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3516.911404] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3516.934900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3522.021599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.023559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3522.026735] Lustre: Skipped 1 previous similar message [ 3522.030885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3522.034968] Lustre: Skipped 1 previous similar message [ 3524.463118] LustreError: 144863:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3524.465948] LustreError: 144863:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3524.516462] Lustre: server umount lustre-OST0000 complete [ 3527.137775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.138624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3527.141538] Lustre: Skipped 1 previous similar message [ 3527.144788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3527.148314] LustreError: Skipped 1 previous similar message [ 3532.257844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.262106] Lustre: Skipped 2 previous similar messages [ 3532.309981] LustreError: 145066:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3532.313670] LustreError: 145066:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3532.374955] Lustre: server umount lustre-MDT0000 complete [ 3534.351452] LustreError: 142318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770687977 with bad export cookie 2288166112759401859 [ 3534.355078] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.359223] LustreError: 142318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3534.545223] Lustre: server umount lustre-MDT0001 complete [ 3539.562575] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3540.822635] Key type lgssc unregistered [ 3541.000431] LNet: 145798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3541.003473] LNetError: 145798:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3541.018614] LNet: Removed LNI 192.168.203.127@tcp [ 3541.389339] Key type .llcrypt unregistered [ 3541.391248] Key type ._llcrypt unregistered [ 3564.301042] Key type ._llcrypt registered [ 3564.302643] Key type .llcrypt registered [ 3564.351343] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3564.838846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3564.855172] alg: No test for adler32 (adler32-zlib) [ 3565.725925] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3565.813691] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3567.400907] Key type lgssc registered [ 3567.855538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3571.290706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3571.298065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.500503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.062489] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3577.760619] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3580.039982] LustreError: 147323:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3580.045150] LustreError: 147323:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3580.050196] LustreError: 147323:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3580.120225] LustreError: 147846:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3580.188404] Lustre: server umount lustre-MDT0000 complete [ 3586.481902] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 3590.695489] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3605.428333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.535547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3605.552842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3605.591457] Lustre: lustre-MDT0000: new disk, initializing [ 3605.626844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.636021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3607.275196] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3612.402172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3612.489213] Lustre: 150475:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3612.535618] LustreError: 150485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.543106] LustreError: 150485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3612.550099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3612.551810] Lustre: Skipped 1 previous similar message [ 3612.590075] Lustre: lustre-MDT0001: new disk, initializing [ 3612.638284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3612.660905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3612.666294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3614.205285] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3617.762068] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3619.243466] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3622.190415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3622.333751] Lustre: lustre-OST0000: new disk, initializing [ 3622.336044] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3622.340175] Lustre: 152212:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3622.381319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.121166] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3628.911705] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3631.124387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3631.130804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3631.200148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3637.559084] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3638.415750] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:48:01 (1770688081) [ 3753.952614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.952760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3753.956919] Lustre: Skipped 2 previous similar messages [ 3753.963545] Lustre: Skipped 2 previous similar messages [ 3757.255298] LustreError: 153973:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3757.261900] LustreError: 153973:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3757.314588] Lustre: server umount lustre-MDT0000 complete [ 3758.949955] LustreError: 150467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688202 with bad export cookie 4535153480826536586 [ 3758.953061] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3758.955043] LustreError: 150467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3759.072455] LustreError: 150480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.072715] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.083955] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3763.104678] LustreError: 150481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3763.104750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3765.062979] LustreError: 154176:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3765.065287] LustreError: 154176:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3765.121315] Lustre: server umount lustre-MDT0001 complete [ 3772.719113] LustreError: 154378:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3772.728160] LustreError: 154378:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3772.793129] Lustre: server umount lustre-OST0000 complete [ 3776.846712] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 3780.865519] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3795.869889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.970137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3795.984238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3796.030959] Lustre: lustre-MDT0000: new disk, initializing [ 3796.062731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.072586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3797.613537] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3802.740314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3802.784086] Lustre: 156700:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3802.807361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3802.810102] Lustre: Skipped 1 previous similar message [ 3802.847599] Lustre: lustre-MDT0001: new disk, initializing [ 3802.897503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3802.902244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3804.448025] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3808.038870] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3809.669081] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3812.801101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3812.924326] Lustre: lustre-OST0000: new disk, initializing [ 3812.926848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3812.930118] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3812.932796] Lustre: 158447:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3812.978523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3812.982891] Lustre: Skipped 1 previous similar message [ 3815.346367] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3818.479857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3818.484421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3818.503706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3819.021841] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3820.715224] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3820.813711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3822.576259] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3822.683123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3823.591081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.591938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.594499] LustreError: Skipped 1 previous similar message [ 3823.595626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.605416] Lustre: Skipped 1 previous similar message [ 3825.657981] LustreError: 159637:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3825.660778] LustreError: 159637:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3825.727385] Lustre: server umount lustre-OST0000 complete [ 3828.706356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3828.706356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.503661] Lustre: server umount lustre-MDT0000 complete [ 3833.824988] LustreError: 157485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.832140] LustreError: 157485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3834.966255] LustreError: 158444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688278 with bad export cookie 4535153480826695416 [ 3834.969660] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3834.970559] LustreError: 158444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3835.041276] LustreError: 160039:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3835.046260] LustreError: 160039:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3835.105954] Lustre: server umount lustre-MDT0001 complete [ 3838.345557] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:51:21 (1770688281) [ 3844.703994] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 3848.501954] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3863.616089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.726280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3863.742536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3863.785524] Lustre: lustre-MDT0000: new disk, initializing [ 3863.821471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.830568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3865.306356] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3870.186970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3870.240184] Lustre: 163013:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3870.364766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3872.098255] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3875.963814] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3877.674364] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3880.515970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3880.629761] Lustre: 164766:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3882.059798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3882.112654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3882.922124] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3886.881092] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3892.708571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3892.709156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3892.712786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3892.712794] Lustre: Skipped 3 previous similar messages [ 3892.714813] LustreError: Skipped 1 previous similar message [ 3892.719123] Lustre: Skipped 2 previous similar messages [ 3895.667083] LustreError: 165554:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3895.669984] LustreError: 165554:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3895.732453] Lustre: server umount lustre-OST0000 complete [ 3903.534305] Lustre: server umount lustre-MDT0000 complete [ 3905.062313] LustreError: 164765:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688348 with bad export cookie 4535153480826696284 [ 3905.065757] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3905.067780] LustreError: 164765:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3905.189697] Lustre: server umount lustre-MDT0001 complete [ 3908.617664] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:52:31 (1770688351) [ 3914.962827] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 3918.583869] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3932.752123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3932.864368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3932.879232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3932.881794] Lustre: Skipped 3 previous similar messages [ 3932.921924] Lustre: lustre-MDT0000: new disk, initializing [ 3932.924263] Lustre: Skipped 2 previous similar messages [ 3932.954468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.957407] Lustre: Skipped 2 previous similar messages [ 3932.966033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3932.969203] Lustre: Skipped 2 previous similar messages [ 3934.603924] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3939.626979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3939.680086] Lustre: 168930:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3939.799208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3941.335142] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3944.835452] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3946.315270] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3949.532913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3949.699948] Lustre: 170679:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3952.198538] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3955.751552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3955.920114] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3960.803855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.803955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3960.805302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3960.805309] Lustre: Skipped 5 previous similar messages [ 3960.807903] Lustre: Skipped 3 previous similar messages [ 3960.810465] LustreError: Skipped 1 previous similar message [ 3964.596203] LustreError: 171467:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3964.599742] LustreError: 171467:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3964.649977] Lustre: server umount lustre-OST0000 complete [ 3970.530122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3972.379454] Lustre: server umount lustre-MDT0000 complete [ 3973.992349] LustreError: 168921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688417 with bad export cookie 4535153480826697348 [ 3973.995502] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3973.998983] LustreError: 168921:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3974.134092] Lustre: server umount lustre-MDT0001 complete [ 3978.167355] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3979.398498] Key type lgssc unregistered [ 3979.548443] LNet: 172402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3979.551528] LNetError: 172402:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3979.560461] LNet: Removed LNI 192.168.203.127@tcp [ 3979.888389] Key type .llcrypt unregistered [ 3979.891167] Key type ._llcrypt unregistered [ 3989.943058] Key type ._llcrypt registered [ 3989.944883] Key type .llcrypt registered [ 3989.985986] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3990.424788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3990.443990] alg: No test for adler32 (adler32-zlib) [ 3991.314096] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 3991.419270] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3993.000143] Key type lgssc registered [ 3993.393512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3996.379487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3996.391241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.579361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.116267] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4002.413882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4002.553893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4004.105206] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4006.284438] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4007.743233] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4010.453331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4010.609644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.931115] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4015.299703] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4021.867784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4038.626381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.629334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.634786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.722156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.724106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.724932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.734257] Lustre: Skipped 1 previous similar message [ 4044.655996] LustreError: 176038:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4044.699852] Lustre: server umount lustre-OST0000 complete [ 4047.842525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.849591] Lustre: Skipped 1 previous similar message [ 4047.849754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4047.855703] Lustre: Skipped 1 previous similar message [ 4052.364925] LustreError: 176240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4052.366906] LustreError: 176240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4052.396633] Lustre: server umount lustre-MDT0000 complete [ 4052.961064] LustreError: 173774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.972247] LustreError: 173774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4053.945284] LustreError: 175389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688497 with bad export cookie 4645192862089378438 [ 4053.948740] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4053.951196] LustreError: 175389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4054.015858] LustreError: 176442:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4054.018310] LustreError: 176442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4054.070773] Lustre: server umount lustre-MDT0001 complete [ 4058.396581] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 4059.519841] Key type lgssc unregistered [ 4059.665229] LNet: 176973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4059.671267] LNetError: 176973:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4059.680326] LNet: Removed LNI 192.168.203.127@tcp [ 4060.027844] Key type .llcrypt unregistered [ 4060.028955] Key type ._llcrypt unregistered [ 4070.328163] Key type ._llcrypt registered [ 4070.329793] Key type .llcrypt registered [ 4070.376485] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4070.875064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4070.883874] alg: No test for adler32 (adler32-zlib) [ 4071.746952] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4071.845862] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 4073.440166] Key type lgssc registered [ 4073.835083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4076.808096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4076.815940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.998475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4079.565896] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4082.888320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4083.044321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4084.622617] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4086.991906] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4088.424340] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4091.020390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4091.150234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4093.235233] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4095.370267] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4103.466976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4108.770549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4108.770847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4108.773609] LustreError: Skipped 1 previous similar message [ 4108.779142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4108.783657] Lustre: Skipped 1 previous similar message [ 4111.280739] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 4113.892599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4113.895358] Lustre: Skipped 1 previous similar message [ 4116.389753] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 4121.509844] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 4121.512741] Lustre: Skipped 2 previous similar messages [ 4123.104116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4123.121954] LustreError: 180618:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4123.156798] Lustre: server umount lustre-OST0000 complete [ 4124.982762] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f0dd2fc2800.ost_server_uuid 50 [ 4125.566747] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f0dd2fc2800.ost_server_uuid in DISCONN state after 0 sec [ 4126.118194] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4139.489844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4139.489943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.498264] Lustre: Skipped 1 previous similar message [ 4139.500713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4139.502701] Lustre: Skipped 1 previous similar message [ 4142.205861] LustreError: 181015:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4142.207896] LustreError: 181015:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4142.230667] Lustre: server umount lustre-MDT0000 complete [ 4143.647795] LustreError: 178330:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688586 with bad export cookie 4524587813219863881 [ 4143.649492] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4143.652961] LustreError: 178330:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4143.732290] LustreError: 181216:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4143.734962] LustreError: 181216:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4143.797249] Lustre: server umount lustre-MDT0001 complete [ 4146.837632] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:56:29 (1770688589) [ 4149.193699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.360582] LustreError: 181885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.369390] LustreError: 181885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4149.392966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.813267] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4154.118228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4154.316490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4155.897875] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4157.884897] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4159.325207] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4161.962311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4164.020664] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4166.023865] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4168.585748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4168.675200] Lustre: lustre-OST0001: new disk, initializing [ 4168.677517] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4168.682601] Lustre: 184386:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4168.724274] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4168.726995] Lustre: Skipped 1 previous similar message [ 4170.809490] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4173.830445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4173.835115] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4173.863621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4174.145454] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4174.894903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4179.375114] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 4179.424924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4179.427624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.432047] Lustre: Skipped 1 previous similar message [ 4181.229123] LustreError: 185082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4181.231682] LustreError: 185082:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4181.246875] Lustre: server umount lustre-OST0000 complete [ 4183.247211] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4183.327763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4184.486546] LustreError: 184945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4184.687140] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4184.763640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4189.605691] LustreError: 185047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4189.611781] LustreError: 185047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4195.299481] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.299724] LustreError: 183478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.303549] Lustre: Skipped 2 previous similar messages [ 4195.309103] LustreError: 183478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4200.420038] LustreError: 183477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.426035] LustreError: 183477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4201.007164] LustreError: 185783:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4201.010601] LustreError: 185783:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4201.056333] Lustre: server umount lustre-OST0001 complete [ 4205.537957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4205.538316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.540506] LustreError: Skipped 1 previous similar message [ 4205.544471] Lustre: Skipped 1 previous similar message [ 4208.472254] Lustre: server umount lustre-MDT0000 complete [ 4209.741367] LustreError: 181864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688653 with bad export cookie 4524587813219864938 [ 4209.744017] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4209.745129] LustreError: 181864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4209.812133] LustreError: 186186:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4209.815019] LustreError: 186186:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4209.866066] Lustre: server umount lustre-MDT0001 complete [ 4217.715919] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:57:40 (1770688660) [ 4220.101843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.229890] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4220.246777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4220.328596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.695139] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4224.650604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4224.691616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4224.707903] Lustre: 187654:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4226.174264] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4228.172258] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4229.487598] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4231.949778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4232.022288] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4234.239301] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4236.120036] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4237.336262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4238.553876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4238.580311] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4240.730957] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4242.714890] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4243.941351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.942351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4243.945308] Lustre: Skipped 1 previous similar message [ 4243.949107] Lustre: Skipped 9 previous similar messages [ 4245.012033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4258.272721] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4258.292140] LustreError: 190828:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4258.294369] LustreError: 190828:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4258.316349] Lustre: server umount lustre-OST0000 complete [ 4260.326388] LustreError: 189985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.333877] LustreError: 189985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4270.563553] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.569115] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4275.167346] Lustre: server umount lustre-OST0001 complete [ 4282.657573] Lustre: server umount lustre-MDT0000 complete [ 4283.981124] LustreError: 188382:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688727 with bad export cookie 4524587813219866324 [ 4283.985117] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4283.985910] LustreError: 188382:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4284.109435] Lustre: server umount lustre-MDT0001 complete [ 4292.345510] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:58:55 (1770688735) [ 4297.767174] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4300.919911] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4313.060464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.161662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4313.173765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4313.213951] Lustre: lustre-MDT0000: new disk, initializing [ 4313.242304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.244940] Lustre: Skipped 3 previous similar messages [ 4313.251605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4314.618166] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4319.002875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4319.044037] Lustre: 195226:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4319.067144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4319.069569] Lustre: Skipped 1 previous similar message [ 4319.110719] Lustre: lustre-MDT0001: new disk, initializing [ 4319.150575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4319.154369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4320.421360] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4323.620931] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4324.914553] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4327.463684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4327.571638] Lustre: lustre-OST0000: new disk, initializing [ 4327.573439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4327.576331] Lustre: 196970:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4329.688626] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4332.927502] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4334.392334] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4337.147242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4337.151142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4337.185981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4337.531462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4338.992354] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4339.072202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4342.243920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.244335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4342.246514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4342.246520] Lustre: Skipped 15 previous similar messages [ 4342.250103] Lustre: Skipped 4 previous similar messages [ 4342.258534] LustreError: Skipped 1 previous similar message [ 4345.839100] LustreError: 198180:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4345.841463] LustreError: 198180:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4345.883527] Lustre: server umount lustre-OST0000 complete [ 4353.371174] Lustre: server umount lustre-MDT0000 complete [ 4354.760052] LustreError: 195218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688798 with bad export cookie 4524587813219868067 [ 4354.760725] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.765322] LustreError: 195218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4354.889322] Lustre: server umount lustre-MDT0001 complete [ 4357.839155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4357.999846] LustreError: 199060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.009237] LustreError: 199060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4359.331517] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4362.384085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4363.880154] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4365.792901] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4367.133895] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4369.735749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4371.880976] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4373.860647] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4375.397179] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4384.661362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 4386.092556] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4386.169231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4387.296791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4387.299870] LustreError: Skipped 1 previous similar message [ 4392.915875] Lustre: server umount lustre-OST0000 complete [ 4395.086120] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4396.458816] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4420.328397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4420.438372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.442229] Lustre: Skipped 5 previous similar messages [ 4422.388035] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4424.280326] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4424.842456] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4427.819091] LustreError: 203501:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4427.821680] LustreError: 203501:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4427.852244] Lustre: server umount lustre-OST0000 complete [ 4429.280830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4429.284246] LustreError: Skipped 1 previous similar message [ 4429.285927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.289703] Lustre: Skipped 4 previous similar messages [ 4431.187959] Lustre: server umount lustre-MDT0000 complete [ 4432.424312] LustreError: 199041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688875 with bad export cookie 4524587813219868935 [ 4432.428391] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4432.429890] LustreError: 199041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4432.557735] Lustre: server umount lustre-MDT0001 complete [ 4435.579545] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:01:18 (1770688878) [ 4437.978228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.187576] LustreError: 204576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.196164] LustreError: 204576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4439.599831] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4442.708954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4444.205468] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4446.027206] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4447.222197] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4449.572948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4451.645707] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4453.416079] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4454.793092] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4464.040765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 4465.397962] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4465.473644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4467.807237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4467.884850] Lustre: lustre-OST0001: new disk, initializing [ 4467.887148] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4467.889808] Lustre: 207637:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4469.928309] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4473.241232] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4474.638444] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4477.954870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4477.958711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4477.988378] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4478.818678] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4480.308192] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4480.392592] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4483.045130] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4483.047696] Lustre: Skipped 10 previous similar messages [ 4487.305862] Lustre: server umount lustre-OST0001 complete [ 4489.487502] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4489.559738] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4490.973226] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4491.056696] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4503.525914] LustreError: 206672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.531396] LustreError: 206672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4515.067410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4517.507918] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4519.547415] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4520.247482] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4520.427208] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4520.432305] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4520.433811] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4520.441913] Lustre: Skipped 1 previous similar message [ 4523.065559] Lustre: server umount lustre-OST0001 complete [ 4525.540583] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4525.543765] LustreError: Skipped 3 previous similar messages [ 4541.274654] Lustre: server umount lustre-OST0000 complete [ 4548.889670] Lustre: server umount lustre-MDT0000 complete [ 4550.148247] LustreError: 207647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688993 with bad export cookie 4524587813219869866 [ 4550.151450] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4550.152593] LustreError: 207647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4556.290624] LustreError: 211142:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4556.293898] LustreError: 211142:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 4556.349526] Lustre: server umount lustre-MDT0001 complete [ 4564.446537] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:03:27 (1770689007) [ 4566.786572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.881784] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4566.893812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4566.962857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4566.965344] Lustre: Skipped 5 previous similar messages [ 4568.268354] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4571.211688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4571.246313] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4571.257244] Lustre: 212611:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4572.664793] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4574.447735] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4575.653331] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4578.028401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4578.096140] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4580.052428] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4581.885722] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4590.912319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4590.936736] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4592.785433] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4594.485868] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4597.093746] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4598.523915] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4598.603332] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4599.919902] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4599.991525] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4601.232720] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0dc8c3e800.ost_server_uuid 50 [ 4601.732165] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f0dc8c3e800.ost_server_uuid in FULL state after 0 sec [ 4602.055706] Lustre: Permanently deactivating lustre-OST0001 [ 4602.060247] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4602.843509] Lustre: Permanently reactivating lustre-OST0001 [ 4606.946551] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4606.951863] Lustre: Skipped 9 previous similar messages [ 4609.859419] Lustre: server umount lustre-OST0001 complete [ 4625.376151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4625.430164] Lustre: server umount lustre-OST0000 complete [ 4632.801657] Lustre: server umount lustre-MDT0000 complete [ 4634.041313] LustreError: 216434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689077 with bad export cookie 4524587813219871140 [ 4634.044553] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4634.046942] LustreError: 216434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4634.159826] Lustre: server umount lustre-MDT0001 complete [ 4642.036663] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:04:45 (1770689085) [ 4644.656937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.753936] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4644.768231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4644.772772] Lustre: Skipped 5 previous similar messages [ 4646.189573] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4649.069732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4649.104593] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4649.116678] Lustre: 218713:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4650.466427] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4652.191383] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4653.387326] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4655.609604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4655.684271] Lustre: Permanently deactivating lustre-OST0000 [ 4655.688876] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4657.581531] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4659.395104] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4660.218284] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4661.850554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4663.791457] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4665.626532] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4665.942335] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4671.360472] Lustre: Permanently reactivating lustre-OST0000 [ 4674.430303] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4674.455242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4684.777143] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4684.780570] LustreError: Skipped 3 previous similar messages [ 4688.773935] Lustre: server umount lustre-OST0001 complete [ 4691.427919] LustreError: 220330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.433797] LustreError: 220330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 4696.281200] Lustre: server umount lustre-OST0000 complete [ 4704.023984] Lustre: server umount lustre-MDT0000 complete [ 4705.271298] LustreError: 218705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689148 with bad export cookie 4524587813219872771 [ 4705.273978] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4705.276771] LustreError: 218705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4705.403784] Lustre: server umount lustre-MDT0001 complete [ 4708.946208] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 4709.936349] Key type lgssc unregistered [ 4710.060381] LNet: 223110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4710.062878] LNetError: 223110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4710.070300] LNet: Removed LNI 192.168.203.127@tcp [ 4710.355722] Key type .llcrypt unregistered [ 4710.357139] Key type ._llcrypt unregistered [ 4718.921987] Key type ._llcrypt registered [ 4718.923441] Key type .llcrypt registered [ 4718.963133] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4719.246155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4719.261961] alg: No test for adler32 (adler32-zlib) [ 4720.112619] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4720.194833] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 4721.776113] Key type lgssc registered [ 4722.129463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4725.476318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4725.483143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.643165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.008023] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4730.853569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4730.893821] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4730.905572] Lustre: 224522:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4730.911303] Lustre: Permanently deactivating lustre-MDT0001 [ 4730.917704] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4731.028319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4732.389705] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4734.366671] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4735.706223] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4735.983106] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4735.986178] LustreError: 224565:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4735.990711] LustreError: 224565:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4735.994056] LustreError: 224565:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 1, failed: rc = -108 [ 4738.149989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4738.287128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.318393] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4741.348266] LustreError: 226613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.354535] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4741.355101] LustreError: 226613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4742.224036] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4744.828188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4744.910951] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4747.130254] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4749.318361] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4750.313291] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4751.337046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4751.338622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4756.310731] Lustre: Permanently reactivating lustre-MDT0001 [ 4756.316834] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4756.319359] Lustre: Skipped 2 previous similar messages [ 4763.567355] Lustre: lustre-MDT0001: Client 4fee8f98-5a5f-4517-a2f3-1de6f2a0ff48 (at 192.168.203.27@tcp) reconnecting [ 4765.703752] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4766.284650] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4768.000860] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4768.075394] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4769.495994] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4769.588884] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4770.445486] Lustre: Permanently deactivating lustre-MDT0001 [ 4770.451257] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4770.453921] Lustre: Skipped 2 previous similar messages [ 4775.935173] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4782.053530] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4782.056605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.062203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4783.074416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4783.075175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.077216] Lustre: Skipped 2 previous similar messages [ 4783.082585] Lustre: Skipped 2 previous similar messages [ 4786.306693] LustreError: 228596:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4786.340124] Lustre: server umount lustre-MDT0000 complete [ 4787.944228] LustreError: 224514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689231 with bad export cookie 12217805834490649677 [ 4787.947680] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4787.953473] LustreError: 224514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4788.034348] LustreError: 228797:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4788.036567] LustreError: 228797:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4788.105238] Lustre: server umount lustre-MDT0001 complete [ 4795.759087] LustreError: 228997:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4795.762163] LustreError: 228997:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4795.780321] Lustre: server umount lustre-OST0000 complete [ 4800.206331] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:07:23 (1770689243) [ 4804.056385] LustreError: 230002:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4804.059339] LustreError: 230002:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4804.109542] Lustre: server umount lustre-OST0001 complete [ 4806.172222] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4809.331144] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4820.924774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4821.008172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4821.011657] Lustre: Skipped 2 previous similar messages [ 4821.019763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4821.054268] Lustre: lustre-MDT0000: new disk, initializing [ 4821.075812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4821.082332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4822.306094] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4826.418202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4826.455250] Lustre: 232170:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4826.480971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4826.483173] Lustre: Skipped 1 previous similar message [ 4826.523342] Lustre: lustre-MDT0001: new disk, initializing [ 4826.564280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4826.567631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4827.839951] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4830.841636] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4831.992792] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4834.141413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4834.228980] Lustre: lustre-OST0000: new disk, initializing [ 4834.231755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4834.234786] Lustre: 233915:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4834.261625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4834.263710] Lustre: Skipped 1 previous similar message [ 4836.078637] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4839.162510] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4844.535752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4844.539691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4844.572113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4850.216359] LustreError: 234510:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4854.447816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4854.512616] Lustre: lustre-OST0001: new disk, initializing [ 4854.514392] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4854.516942] Lustre: 234936:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4854.542899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4856.274177] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4859.151813] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4860.304079] LustreError: 234510:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4860.310155] LustreError: 234510:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4863.996684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4864.000616] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4864.028278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4865.775441] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 4870.400131] LustreError: 234510:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4874.212226] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.213168] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4874.213929] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4874.213933] Lustre: Skipped 1 previous similar message [ 4874.216129] Lustre: Skipped 1 previous similar message [ 4874.221838] LustreError: Skipped 1 previous similar message [ 4877.102182] LustreError: 235676:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4877.105593] LustreError: 235676:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4877.125459] Lustre: server umount lustre-OST0001 complete [ 4879.332523] LustreError: 233896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.338898] LustreError: 233896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4884.452636] LustreError: 233899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.459249] LustreError: 233899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4889.571555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.571902] LustreError: 233898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.572786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4889.572793] Lustre: Skipped 2 previous similar messages [ 4889.577292] Lustre: Skipped 1 previous similar message [ 4889.586036] LustreError: 233898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4894.690358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4894.691716] LustreError: 233898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.692482] Lustre: Skipped 2 previous similar messages [ 4894.697858] LustreError: 233898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4903.392115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4903.411707] LustreError: 235903:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4903.414351] LustreError: 235903:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4903.448896] Lustre: server umount lustre-OST0000 complete [ 4904.929287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.929830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4904.932963] Lustre: Skipped 1 previous similar message [ 4904.935777] Lustre: Skipped 3 previous similar messages [ 4910.938659] Lustre: server umount lustre-MDT0000 complete [ 4912.151378] LustreError: 233914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689355 with bad export cookie 12217805834490652547 [ 4912.154864] LustreError: 233914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4912.155167] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4918.336346] Lustre: server umount lustre-MDT0001 complete [ 4921.784994] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 4922.794509] Key type lgssc unregistered [ 4922.919317] LNet: 236838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4922.922525] LNetError: 236838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4922.931314] LNet: Removed LNI 192.168.203.127@tcp [ 4923.222734] Key type .llcrypt unregistered [ 4923.224199] Key type ._llcrypt unregistered [ 4936.862627] Key type ._llcrypt registered [ 4936.863901] Key type .llcrypt registered [ 4936.898228] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4937.240590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4937.270886] alg: No test for adler32 (adler32-zlib) [ 4938.118707] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 4938.195416] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 4939.776113] Key type lgssc registered [ 4940.136768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4942.678678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4942.684397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.764477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4943.772104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4943.830400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.115254] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4947.904542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4947.936530] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4947.947993] Lustre: 239002:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4948.039051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4949.269089] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4951.014390] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4952.149336] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4954.249913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4954.314000] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4954.373540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.164227] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4957.880662] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4960.268159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5006.305144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5006.306761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.307994] LustreError: Skipped 1 previous similar message [ 5006.311894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5006.315126] Lustre: Skipped 1 previous similar message [ 5007.917070] LustreError: 241357:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5007.956841] Lustre: server umount lustre-OST0000 complete [ 5009.337956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5013.225655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5013.336319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5015.176526] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5016.953927] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5026.600977] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.605186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.605301] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5026.610845] Lustre: Skipped 1 previous similar message [ 5052.386175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5052.386889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.388308] LustreError: Skipped 1 previous similar message [ 5052.392287] Lustre: Skipped 1 previous similar message [ 5052.393085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5052.397541] Lustre: Skipped 1 previous similar message [ 5056.559171] LustreError: 243019:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5056.561505] LustreError: 243019:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5056.601397] Lustre: server umount lustre-OST0000 complete [ 5060.576879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5060.579327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.583647] Lustre: Skipped 1 previous similar message [ 5060.585888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.625074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.625655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.806692] LustreError: 243220:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5063.809875] LustreError: 243220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5063.838438] Lustre: server umount lustre-MDT0000 complete [ 5065.030069] LustreError: 238994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689508 with bad export cookie 16913947017819784285 [ 5065.033915] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5065.035356] LustreError: 238994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5065.142597] Lustre: server umount lustre-MDT0001 complete [ 5068.568431] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5069.508946] Key type lgssc unregistered [ 5069.636394] LNet: 243953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5069.639175] LNetError: 243953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5069.646268] LNet: Removed LNI 192.168.203.127@tcp [ 5069.932845] Key type .llcrypt unregistered [ 5069.934138] Key type ._llcrypt unregistered [ 5078.314801] Key type ._llcrypt registered [ 5078.315771] Key type .llcrypt registered [ 5078.349112] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5078.644167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5078.683677] alg: No test for adler32 (adler32-zlib) [ 5079.515293] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5079.585274] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5081.160149] Key type lgssc registered [ 5081.512964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5084.138958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5084.146697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5085.334966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5086.702349] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5089.584471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5089.707579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5091.004023] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5092.882686] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5094.092498] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5096.246500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5096.352293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.222455] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5100.006178] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5110.694733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5119.968864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5119.971617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.976405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5120.994202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5120.994492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5120.995001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5123.823120] LustreError: 248358:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5123.859768] Lustre: server umount lustre-OST0000 complete [ 5125.600887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5125.603700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.608309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5125.610445] Lustre: Skipped 1 previous similar message [ 5131.197863] LustreError: 248560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5131.199619] LustreError: 248560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5131.223853] Lustre: server umount lustre-MDT0000 complete [ 5131.233099] LustreError: 245320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.238138] LustreError: 245320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5132.411170] LustreError: 248271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689575 with bad export cookie 13533485200074354203 [ 5132.414157] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5132.415305] LustreError: 248271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5132.475190] LustreError: 248761:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5132.477140] LustreError: 248761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5132.526880] Lustre: server umount lustre-MDT0001 complete [ 5135.892668] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5136.934594] Key type lgssc unregistered [ 5137.061467] LNet: 249291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5137.064962] LNetError: 249291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5137.077064] LNet: Removed LNI 192.168.203.127@tcp [ 5137.383698] Key type .llcrypt unregistered [ 5137.384811] Key type ._llcrypt unregistered [ 5143.985186] Key type ._llcrypt registered [ 5143.985994] Key type .llcrypt registered [ 5144.021197] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5144.390549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5144.395614] alg: No test for adler32 (adler32-zlib) [ 5145.243219] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5145.320211] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5146.896158] Key type lgssc registered [ 5147.251764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5149.671037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5149.676595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5150.826334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5152.013460] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5154.726906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5154.834154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5156.016257] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5157.668507] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5158.833315] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5160.857267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5160.930346] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5160.961879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.738135] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5164.370775] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5165.094932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5175.268449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.268814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5175.269317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5175.271692] Lustre: Skipped 1 previous similar message [ 5179.885065] LustreError: 252859:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5179.914019] Lustre: server umount lustre-OST0000 complete [ 5185.506868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.507071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5185.508342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5185.508347] Lustre: Skipped 1 previous similar message [ 5185.511316] Lustre: Skipped 1 previous similar message [ 5185.513788] LustreError: Skipped 1 previous similar message [ 5187.199327] LustreError: 253061:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5187.201248] LustreError: 253061:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5187.228570] Lustre: server umount lustre-MDT0000 complete [ 5188.383141] LustreError: 252772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689631 with bad export cookie 9355544511899513095 [ 5188.386495] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5188.386805] LustreError: 252772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5188.452713] LustreError: 253263:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5188.454631] LustreError: 253263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5188.507140] Lustre: server umount lustre-MDT0001 complete [ 5192.349278] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5193.471352] Key type lgssc unregistered [ 5193.611387] LNet: 253793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5193.615210] LNetError: 253793:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5193.625404] LNet: Removed LNI 192.168.203.127@tcp [ 5193.970098] Key type .llcrypt unregistered [ 5193.973053] Key type ._llcrypt unregistered [ 5203.266703] Key type ._llcrypt registered [ 5203.267852] Key type .llcrypt registered [ 5203.304502] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5203.722451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5203.752442] alg: No test for adler32 (adler32-zlib) [ 5204.607906] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5204.693868] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5206.280170] Key type lgssc registered [ 5206.653885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5209.226669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5209.233322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.401497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.766628] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5214.634703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5214.755806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5216.092351] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5217.934661] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5219.157230] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5221.359770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5221.472973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.279408] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5225.113422] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5235.817873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5251.556682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5251.557377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.559348] LustreError: Skipped 1 previous similar message [ 5251.563922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5251.566434] Lustre: Skipped 1 previous similar message [ 5254.191101] LustreError: 258202:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5254.228659] Lustre: server umount lustre-OST0000 complete [ 5255.649162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5255.653966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.661542] Lustre: Skipped 1 previous similar message [ 5255.664753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5256.673400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5256.674158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.569997] LustreError: 258404:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5261.571795] LustreError: 258404:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5261.603116] Lustre: server umount lustre-MDT0000 complete [ 5261.793210] LustreError: 255168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.799882] LustreError: 255168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5262.867208] LustreError: 255147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689706 with bad export cookie 7821010988692982899 [ 5262.870451] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5262.871322] LustreError: 255147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5262.943042] LustreError: 258606:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5262.945378] LustreError: 258606:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5263.001699] Lustre: server umount lustre-MDT0001 complete [ 5266.679642] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5267.648345] Key type lgssc unregistered [ 5267.767402] LNet: 259138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5267.769620] LNetError: 259138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5267.777250] LNet: Removed LNI 192.168.203.127@tcp [ 5268.056197] Key type .llcrypt unregistered [ 5268.057739] Key type ._llcrypt unregistered [ 5275.001874] Key type ._llcrypt registered [ 5275.002797] Key type .llcrypt registered [ 5275.044245] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5275.459323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5275.513344] alg: No test for adler32 (adler32-zlib) [ 5276.372834] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5276.461286] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5278.048143] Key type lgssc registered [ 5278.385612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5280.837692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5280.844678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.998672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.142347] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5285.802182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5285.928591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.136141] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5288.822772] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5289.909699] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5291.993504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5292.107914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.876558] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5295.588333] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5307.495733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5317.088762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5317.091170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.097095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5318.115442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5318.115757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5318.115995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.125652] Lustre: Skipped 1 previous similar message [ 5320.814529] LustreError: 262760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5320.848430] Lustre: server umount lustre-OST0000 complete [ 5323.233426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.237930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5323.240489] Lustre: Skipped 1 previous similar message [ 5328.257183] LustreError: 262963:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5328.259011] LustreError: 262963:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5328.286955] Lustre: server umount lustre-MDT0000 complete [ 5328.354335] LustreError: 260342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.359547] LustreError: 260342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5329.588370] LustreError: 260297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689772 with bad export cookie 1327234007465031511 [ 5329.591122] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5329.591646] LustreError: 260297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5329.665961] LustreError: 263164:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5329.669492] LustreError: 263164:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5329.734593] Lustre: server umount lustre-MDT0001 complete [ 5333.252104] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5334.271398] Key type lgssc unregistered [ 5334.398342] LNet: 263695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5334.400475] LNetError: 263695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5334.409300] LNet: Removed LNI 192.168.203.127@tcp [ 5334.690730] Key type .llcrypt unregistered [ 5334.692455] Key type ._llcrypt unregistered [ 5338.245387] Key type ._llcrypt registered [ 5338.246701] Key type .llcrypt registered [ 5338.286341] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5343.812575] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 5349.746335] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5350.169731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5350.178753] alg: No test for adler32 (adler32-zlib) [ 5351.020891] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5351.097300] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5352.672159] Key type lgssc registered [ 5353.056661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5363.683186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5363.688955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.768214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5364.778365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5364.812873] Lustre: lustre-MDT0000: new disk, initializing [ 5364.836585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5364.843746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5366.001677] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5369.906460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5369.945709] Lustre: 267166:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5369.970094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5369.971919] Lustre: Skipped 1 previous similar message [ 5370.007766] Lustre: lustre-MDT0001: new disk, initializing [ 5370.037959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5370.049476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5370.052454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5371.255638] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5374.175698] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5375.282799] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5377.437912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5377.527701] Lustre: lustre-OST0000: new disk, initializing [ 5377.530137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5377.533355] Lustre: 268906:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5377.559342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.346640] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5380.281315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5380.285725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5380.313295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5382.568936] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5383.989252] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5384.055564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5385.444885] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5385.538705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5390.819952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5390.820510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5390.821185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5390.823265] LustreError: Skipped 1 previous similar message [ 5390.829816] Lustre: Skipped 1 previous similar message [ 5392.302092] LustreError: 270097:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5392.346739] Lustre: server umount lustre-OST0000 complete [ 5395.938142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5395.938201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5395.938875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5395.938878] Lustre: Skipped 1 previous similar message [ 5395.942417] Lustre: Skipped 1 previous similar message [ 5399.742021] LustreError: 270298:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5399.743591] LustreError: 270298:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5399.775704] Lustre: server umount lustre-MDT0000 complete [ 5400.917200] LustreError: 267157:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689844 with bad export cookie 1307963830065362011 [ 5400.919906] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5400.921129] LustreError: 267157:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5400.984771] LustreError: 270499:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5400.986262] LustreError: 270499:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5401.043852] Lustre: server umount lustre-MDT0001 complete [ 5403.617723] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:17:26 (1770689846) [ 5405.654698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.812919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.934805] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5409.489264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5410.828898] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5412.525250] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5413.641263] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.795644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5415.925958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.927800] Lustre: Skipped 1 previous similar message [ 5417.622266] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5419.265184] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5424.098919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.099275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5424.102536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5424.103862] LustreError: Skipped 1 previous similar message [ 5424.105128] Lustre: Skipped 1 previous similar message [ 5429.037120] LustreError: 273411:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5429.038903] LustreError: 273411:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5429.077123] Lustre: server umount lustre-OST0000 complete [ 5430.240801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5430.243313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.247196] Lustre: Skipped 1 previous similar message [ 5430.249701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.252812] Lustre: Skipped 1 previous similar message [ 5434.337222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5434.337726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.478648] LustreError: 273614:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5436.480675] LustreError: 273614:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5436.511267] Lustre: server umount lustre-MDT0000 complete [ 5437.729241] LustreError: 271149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689881 with bad export cookie 1307963830065362872 [ 5437.732450] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5437.732917] LustreError: 271149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5437.860482] Lustre: server umount lustre-MDT0001 complete [ 5441.208485] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5442.245806] Key type lgssc unregistered [ 5442.371327] LNet: 274347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5442.374093] LNetError: 274347:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5442.380271] LNet: Removed LNI 192.168.203.127@tcp [ 5442.666431] Key type .llcrypt unregistered [ 5442.667501] Key type ._llcrypt unregistered [ 5450.836384] Key type ._llcrypt registered [ 5450.837579] Key type .llcrypt registered [ 5450.874128] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 5456.455764] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5456.780448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5456.837349] alg: No test for adler32 (adler32-zlib) [ 5457.700553] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5457.779643] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5459.352147] Key type lgssc registered [ 5459.692471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5474.156282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5474.163402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.253751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5475.266273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5475.304926] Lustre: lustre-MDT0000: new disk, initializing [ 5475.330087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5475.337155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5476.685793] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5481.042745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5481.082284] Lustre: 278107:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5481.107772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5481.109969] Lustre: Skipped 1 previous similar message [ 5481.145711] Lustre: lustre-MDT0001: new disk, initializing [ 5481.178498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5481.192136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5481.195749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5482.579070] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5485.789875] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5486.999251] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5489.271193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5489.386867] Lustre: lustre-OST2710: new disk, initializing [ 5489.388892] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5489.392139] Lustre: 279847:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5489.430022] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5491.320991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5491.325660] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5491.362487] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5491.384365] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5494.414891] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5496.586837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5496.648044] Lustre: lustre-OST03e8: new disk, initializing [ 5496.650335] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5496.654343] Lustre: 280795:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5496.685398] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5498.474278] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5500.150519] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5500.669934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5500.672538] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5500.701361] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5502.731187] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5502.817384] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5504.166956] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5504.241698] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5505.414259] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5505.475861] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5506.701959] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5506.769404] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5511.140721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.141453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.145107] Lustre: Skipped 1 previous similar message [ 5511.148961] Lustre: Skipped 3 previous similar messages [ 5513.797354] LustreError: 282435:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5513.829553] Lustre: server umount lustre-MDT0000 complete [ 5515.079163] LustreError: 278098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770689958 with bad export cookie 17847516794065448951 [ 5515.082881] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5515.083858] LustreError: 278098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5515.150862] LustreError: 282636:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5515.153336] LustreError: 282636:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5515.204884] Lustre: server umount lustre-MDT0001 complete [ 5526.708084] LustreError: 282837:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5526.710386] LustreError: 282837:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5526.732696] Lustre: server umount lustre-OST2710 complete [ 5538.230216] LustreError: 283039:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5538.232937] LustreError: 283039:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5538.272638] Lustre: server umount lustre-OST03e8 complete [ 5544.377629] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 5547.033787] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5557.694687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.779403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5557.790853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5557.823789] Lustre: lustre-MDT0000: new disk, initializing [ 5557.845106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.852407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5559.004842] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5562.977648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5563.020168] Lustre: 285818:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5563.147286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5564.445224] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5567.565717] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5568.791937] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5571.015981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5571.128122] Lustre: lustre-OST0000: new disk, initializing [ 5571.129727] Lustre: Skipped 1 previous similar message [ 5571.131960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5571.135087] Lustre: Skipped 2 previous similar messages [ 5571.137775] Lustre: 287557:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5571.168511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.171866] Lustre: Skipped 1 previous similar message [ 5573.059255] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5576.022296] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5577.405919] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5577.720707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5577.723295] Lustre: Skipped 1 previous similar message [ 5577.725096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5577.752654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5578.499252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5579.758637] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5579.836078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5582.818483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5582.820316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.825516] Lustre: Skipped 2 previous similar messages [ 5582.827636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.829262] Lustre: Skipped 1 previous similar message [ 5586.543070] LustreError: 288754:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5586.545382] LustreError: 288754:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5586.580556] Lustre: server umount lustre-OST0000 complete [ 5587.937345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.937972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5587.941162] Lustre: Skipped 2 previous similar messages [ 5593.057809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5593.061306] Lustre: Skipped 2 previous similar messages [ 5598.177221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5598.179684] Lustre: Skipped 1 previous similar message [ 5602.272123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5602.305961] LustreError: 288956:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5602.308146] LustreError: 288956:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5602.338807] Lustre: server umount lustre-MDT0000 complete [ 5603.586386] LustreError: 285808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690046 with bad export cookie 17847516794065450624 [ 5603.589680] LustreError: 285808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5603.592506] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5603.707766] Lustre: server umount lustre-MDT0001 complete [ 5606.370313] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:20:49 (1770690049) [ 5606.854197] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5607.395461] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:20:50 (1770690050) [ 5608.418981] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5617.234371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.331849] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5617.339890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5617.400758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.826789] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5621.700341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5621.727366] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5621.735535] Lustre: 291254:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5622.973090] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5624.547782] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5625.563171] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5638.546481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5638.622315] Lustre: Denying initial registration attempt from nid 192.168.203.127@tcp, specified as failover [ 5638.625575] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5638.630127] LustreError: 292864:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -99 [ 5638.632374] LustreError: 292864:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5638.634588] LustreError: 292864:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5638.659299] Lustre: server umount lustre-OST0000 complete [ 5638.661486] LustreError: 292864:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5642.208768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5642.211816] LustreError: Skipped 1 previous similar message [ 5642.213065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.217537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5646.913095] LustreError: 293232:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5646.915084] LustreError: 293232:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5646.938359] Lustre: server umount lustre-MDT0000 complete [ 5648.158103] LustreError: 291244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690091 with bad export cookie 17847516794065451492 [ 5648.160672] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5648.161845] LustreError: 291244:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5648.275295] Lustre: server umount lustre-MDT0001 complete [ 5651.724844] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5652.642777] Key type lgssc unregistered [ 5652.763310] LNet: 293965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5652.765567] LNetError: 293965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5652.774277] LNet: Removed LNI 192.168.203.127@tcp [ 5653.069484] Key type .llcrypt unregistered [ 5653.070469] Key type ._llcrypt unregistered [ 5656.699693] Key type ._llcrypt registered [ 5656.700820] Key type .llcrypt registered [ 5656.736770] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5657.047757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5657.070051] alg: No test for adler32 (adler32-zlib) [ 5657.919735] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5657.999787] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5659.584103] Key type lgssc registered [ 5659.927333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5694.751993] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5697.576843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5697.584892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.676920] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5698.685665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5698.755472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5700.067762] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5702.821074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5702.852652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5702.862932] Lustre: 296675:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5702.963289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5704.253691] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5706.006672] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5707.225610] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5720.474620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5720.544520] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5720.610247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.501264] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5724.235540] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5724.954084] LustreError: 298941:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5724.985682] Lustre: server umount lustre-OST0000 complete [ 5728.737172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5728.740565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.745192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.786865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5730.787277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.356089] LustreError: 299142:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5732.357917] LustreError: 299142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5732.383143] Lustre: server umount lustre-MDT0000 complete [ 5733.599275] LustreError: 296667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690176 with bad export cookie 9880208137896437665 [ 5733.602322] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5733.605723] LustreError: 296667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5733.669155] LustreError: 299348:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 5733.671249] LustreError: 299348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5733.719396] Lustre: server umount lustre-MDT0001 complete [ 5737.294293] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5738.423108] Key type lgssc unregistered [ 5738.551446] LNet: 299878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5738.553705] LNetError: 299878:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5738.565370] LNet: Removed LNI 192.168.203.127@tcp [ 5738.880555] Key type .llcrypt unregistered [ 5738.881718] Key type ._llcrypt unregistered [ 5748.122705] Key type ._llcrypt registered [ 5748.123953] Key type .llcrypt registered [ 5748.163518] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5748.453789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5748.464729] alg: No test for adler32 (adler32-zlib) [ 5749.314244] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5749.388906] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5750.960196] Key type lgssc registered [ 5751.356193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5754.022719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5754.029721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.209639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.405284] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5759.147385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5759.276517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5760.611180] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5762.243858] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5763.460650] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5765.800473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5765.910838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5767.859278] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5769.649599] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5785.056784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5785.059393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.065080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5785.571344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.573560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5785.576413] Lustre: Skipped 1 previous similar message [ 5785.578233] Lustre: Skipped 1 previous similar message [ 5788.539761] LustreError: 303514:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5788.565698] Lustre: server umount lustre-MDT0000 complete [ 5789.826390] LustreError: 303517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690233 with bad export cookie 5713904933017957700 [ 5789.829963] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5789.831268] LustreError: 303517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5790.629906] LustreError: 303404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.637398] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [ 5790.637830] LustreError: 303404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5790.689985] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.720989] LustreError: 303404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.721306] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5794.729641] LustreError: 303404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5794.731515] Lustre: Skipped 1 previous similar message [ 5795.748672] LustreError: 303404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.967152] LustreError: 303720:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5795.969214] LustreError: 303720:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5796.027125] Lustre: server umount lustre-MDT0001 complete [ 5798.243355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5801.047635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.264582] LustreError: 300562:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000948f8f9 x1856703238790784/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5815.372845] LustreError: 302855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.397501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.418064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5816.772810] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5819.583182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5819.707262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.773573] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5820.965945] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5822.694280] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5824.870448] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5826.293779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5827.442661] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5828.017948] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5830.118718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.120600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5830.121605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5830.121614] Lustre: Skipped 1 previous similar message [ 5830.126762] Lustre: Skipped 1 previous similar message [ 5835.311085] LustreError: 306019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5835.312961] LustreError: 306019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5835.349817] Lustre: server umount lustre-OST0000 complete [ 5840.353934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5840.354186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5840.359527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.364299] Lustre: Skipped 4 previous similar messages [ 5842.685556] LustreError: 306222:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5842.688087] LustreError: 306222:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5842.709916] Lustre: server umount lustre-MDT0000 complete [ 5843.945955] LustreError: 301967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690287 with bad export cookie 5713904933017964854 [ 5843.948827] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5843.950031] LustreError: 301967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5844.065093] Lustre: server umount lustre-MDT0001 complete [ 5847.813262] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5848.779017] Key type lgssc unregistered [ 5848.901357] LNet: 306954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5848.903728] LNetError: 306954:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5848.913356] LNet: Removed LNI 192.168.203.127@tcp [ 5849.208787] Key type .llcrypt unregistered [ 5849.209868] Key type ._llcrypt unregistered [ 5858.480968] Key type ._llcrypt registered [ 5858.482209] Key type .llcrypt registered [ 5858.524865] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5858.958499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5858.966413] alg: No test for adler32 (adler32-zlib) [ 5859.813574] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5859.888331] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5861.464159] Key type lgssc registered [ 5861.810586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5864.322839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5864.328585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.490470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5866.743800] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5869.486807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5869.602442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5870.933994] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5872.687708] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5873.854849] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5878.946117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.946752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.950617] Lustre: Skipped 1 previous similar message [ 5878.953438] Lustre: Skipped 1 previous similar message [ 5880.955982] LustreError: 309712:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5880.985035] Lustre: server umount lustre-MDT0000 complete [ 5882.198309] LustreError: 309713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690325 with bad export cookie 419153009874363455 [ 5882.200554] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5882.201451] LustreError: 309713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5882.274631] LustreError: 309917:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5882.277209] LustreError: 309917:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5882.335436] Lustre: server umount lustre-MDT0001 complete [ 5887.145966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.248756] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5887.259830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5887.333311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5888.586309] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5891.545186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5891.580280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5891.593679] Lustre: 310685:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5891.699659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5893.031672] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5894.860537] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5896.103432] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5898.715540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5898.782512] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5898.786060] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5898.788883] LustreError: 312340:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 5898.791068] LustreError: 312340:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5898.793203] LustreError: 312340:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5898.813824] Lustre: server umount lustre-OST0000 complete [ 5898.815192] LustreError: 312340:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5901.584960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5901.648342] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5901.712405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.490503] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5905.280757] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5907.821568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5907.854165] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5907.883230] Lustre: lustre-OST0001: new disk, initializing [ 5907.886059] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5907.890225] Lustre: 313758:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5908.114179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5908.118204] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5908.143381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5908.167631] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5909.807735] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5911.529747] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5913.569954] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5913.570624] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.573625] LustreError: Skipped 1 previous similar message [ 5913.577675] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5913.580211] Lustre: Skipped 1 previous similar message [ 5914.478344] LustreError: 314450:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5914.480097] LustreError: 314450:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5914.499763] Lustre: server umount lustre-OST0001 complete [ 5918.690089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5918.693132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.693254] LustreError: 312804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.695675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.697573] Lustre: Skipped 2 previous similar messages [ 5918.702207] LustreError: 312804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5921.839079] LustreError: 314650:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5921.842108] LustreError: 314650:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5921.886127] Lustre: server umount lustre-OST0000 complete [ 5923.809664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5923.810224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5923.814309] Lustre: Skipped 1 previous similar message [ 5923.816308] Lustre: Skipped 1 previous similar message [ 5928.929641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5928.931812] Lustre: Skipped 2 previous similar messages [ 5929.284111] LustreError: 314851:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5929.285995] LustreError: 314851:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5929.316512] Lustre: server umount lustre-MDT0000 complete [ 5930.560451] LustreError: 314852:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690373 with bad export cookie 419153009874364001 [ 5930.563075] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5930.566407] LustreError: 314852:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5930.713145] Lustre: server umount lustre-MDT0001 complete [ 5934.401955] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5935.419306] Key type lgssc unregistered [ 5935.553317] LNet: 315584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5935.555728] LNetError: 315584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5935.562308] LNet: Removed LNI 192.168.203.127@tcp [ 5935.839895] Key type .llcrypt unregistered [ 5935.841277] Key type ._llcrypt unregistered [ 5966.418849] Key type ._llcrypt registered [ 5966.420549] Key type .llcrypt registered [ 5966.458453] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 5972.281659] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5972.730816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5972.738824] alg: No test for adler32 (adler32-zlib) [ 5973.589963] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 5973.668388] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5975.248166] Key type lgssc registered [ 5975.624930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5989.638395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5989.647892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.795587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5990.814473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5990.860507] Lustre: lustre-MDT0000: new disk, initializing [ 5990.918588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.929971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5992.820368] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5998.511547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5998.574088] Lustre: 320451:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5998.612017] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5998.615734] Lustre: Skipped 1 previous similar message [ 5998.657155] Lustre: lustre-MDT0001: new disk, initializing [ 5998.711784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5998.736755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5998.744437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6000.655675] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6004.615806] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6006.451181] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6009.878090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6010.015756] Lustre: lustre-OST0000: new disk, initializing [ 6010.018325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6010.022719] Lustre: 322195:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6010.061439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.392927] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6015.534449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6015.549396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6015.651310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6020.221711] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6022.581247] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6022.727729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6024.604025] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6024.704508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6026.211500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.213511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6026.214109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6026.220307] Lustre: Skipped 1 previous similar message [ 6026.222652] LustreError: Skipped 1 previous similar message [ 6031.332913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6031.335219] Lustre: Skipped 2 previous similar messages [ 6031.535129] LustreError: 323384:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6031.579858] Lustre: server umount lustre-OST0000 complete [ 6034.912980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6034.915475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.920446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6036.450465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.116907] LustreError: 323586:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6039.120173] LustreError: 323586:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6039.160322] Lustre: server umount lustre-MDT0000 complete [ 6040.888222] LustreError: 322194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690484 with bad export cookie 6576596873846094079 [ 6040.891153] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6040.892554] LustreError: 322194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6040.976129] LustreError: 323789:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6040.978478] LustreError: 323789:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6041.054415] Lustre: server umount lustre-MDT0001 complete [ 6044.240327] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:28:07 (1770690487) [ 6047.319557] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:28:10 (1770690490) [ 6050.722731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.930810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6052.289133] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6056.102297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6056.266980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6058.281899] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6060.926252] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6062.592916] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6065.343805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6065.503568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6068.064482] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6070.736849] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6079.807145] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6080.495399] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6081.161400] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6081.830125] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6082.579724] Lustre: Failing over lustre-MDT0000 [ 6082.682371] LustreError: 327231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6082.690451] LustreError: 327231:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6082.736158] Lustre: server umount lustre-MDT0000 complete [ 6086.113728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.121562] LustreError: 326950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.128949] LustreError: 326950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6091.233685] LustreError: 324800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.239953] LustreError: 324800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6095.885249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.934859] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.026888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.049058] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6096.805709] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6097.365100] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6101.479606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6101.485129] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6101.500472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6102.705973] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6103.290725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6105.200294] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6105.914105] Lustre: Failing over lustre-MDT0000 [ 6106.020294] LustreError: 328311:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6106.022618] LustreError: 328311:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6106.052749] Lustre: server umount lustre-MDT0000 complete [ 6106.593464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6106.595374] LustreError: 324794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.597967] Lustre: Skipped 4 previous similar messages [ 6106.602708] LustreError: 324794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6111.713390] LustreError: 324800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6111.721637] LustreError: 324800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6116.834088] LustreError: 324800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6116.840057] LustreError: 324800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6119.214750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.258457] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6119.366743] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6120.614048] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6122.408870] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6124.517877] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6124.519960] Lustre: Skipped 2 previous similar messages [ 6124.528697] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6124.545936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6125.893866] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6126.555175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6129.634645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.635845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6129.636349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6129.636355] Lustre: Skipped 2 previous similar messages [ 6129.639101] Lustre: Skipped 2 previous similar messages [ 6134.753285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6134.755795] Lustre: Skipped 4 previous similar messages [ 6135.040394] LustreError: 329336:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6135.042462] LustreError: 329336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6135.080300] Lustre: server umount lustre-MDT0000 complete [ 6136.513622] LustreError: 325518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690579 with bad export cookie 6576596873846096767 [ 6136.517316] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6136.518495] LustreError: 325518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6136.663719] Lustre: server umount lustre-MDT0001 complete [ 6148.641960] Lustre: server umount lustre-OST0000 complete [ 6154.684061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.852340] LustreError: 330565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6154.861305] LustreError: 330565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6154.886704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.889866] Lustre: Skipped 1 previous similar message [ 6156.585734] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6159.981397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6161.577652] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6163.584303] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6164.934209] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6167.589291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6169.858302] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6172.066877] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6180.007536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6185.441182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.441311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.444570] LustreError: Skipped 1 previous similar message [ 6185.451094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6185.454830] Lustre: Skipped 1 previous similar message [ 6186.286369] LustreError: 332828:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6186.288685] LustreError: 332828:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6186.328356] Lustre: server umount lustre-OST0000 complete [ 6189.723913] Lustre: server umount lustre-MDT0000 complete [ 6191.099101] LustreError: 332178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690634 with bad export cookie 6576596873846097474 [ 6191.102377] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6191.103341] LustreError: 332178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6191.224354] Lustre: server umount lustre-MDT0001 complete [ 6195.232586] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6196.409305] Key type lgssc unregistered [ 6196.555412] LNet: 333766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6196.558412] LNetError: 333766:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6196.572513] LNet: Removed LNI 192.168.203.127@tcp [ 6196.897967] Key type .llcrypt unregistered [ 6196.899641] Key type ._llcrypt unregistered [ 6207.454563] Key type ._llcrypt registered [ 6207.456043] Key type .llcrypt registered [ 6207.491231] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6207.906646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6207.914384] alg: No test for adler32 (adler32-zlib) [ 6208.778583] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6208.883446] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6210.488152] Key type lgssc registered [ 6210.948453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6213.846480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6213.854250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.040304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6216.601344] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6219.902961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6220.086982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6221.396138] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6223.355328] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6224.713612] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6227.306937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6227.448588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.558697] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6231.478361] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6240.747397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6250.983128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6250.984266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6250.984728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6250.989185] Lustre: Skipped 1 previous similar message [ 6253.453773] LustreError: 337501:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6253.486204] Lustre: server umount lustre-MDT0000 complete [ 6254.953483] LustreError: 335220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690698 with bad export cookie 12915370144214136472 [ 6254.956683] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6254.957554] LustreError: 335220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6255.035790] LustreError: 337705:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6255.037875] LustreError: 337705:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6255.097665] Lustre: server umount lustre-MDT0001 complete [ 6265.842781] LustreError: 337906:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6265.845082] LustreError: 337906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6265.889976] Lustre: server umount lustre-OST0000 complete [ 6270.749441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6270.895578] LustreError: 338634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.900994] LustreError: 338634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6270.918427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6272.251367] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6275.141899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6275.267859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6276.698534] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6278.894792] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6280.174208] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6282.536881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6284.907384] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6286.880436] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6295.977755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6306.273148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6306.276329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6306.277212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6306.277216] Lustre: Skipped 2 previous similar messages [ 6306.286546] Lustre: Skipped 1 previous similar message [ 6310.725451] LustreError: 341000:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6310.729754] LustreError: 341000:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6310.773149] Lustre: server umount lustre-MDT0000 complete [ 6311.393915] LustreError: 338634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.399845] LustreError: 338634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6312.273126] LustreError: 338613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690755 with bad export cookie 12915370144214137571 [ 6312.276423] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6312.278630] LustreError: 338613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6312.405227] Lustre: server umount lustre-MDT0001 complete [ 6324.275853] LustreError: 341403:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6324.278245] LustreError: 341403:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6324.330751] Lustre: server umount lustre-OST0000 complete [ 6329.807207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.977592] LustreError: 342230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.006793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.009049] Lustre: Skipped 1 previous similar message [ 6331.483585] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6334.454665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6335.989370] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6338.054535] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6339.568015] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6342.399692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6344.643942] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6346.574428] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6354.855548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6360.035001] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6360.035341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.036411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6360.036415] Lustre: Skipped 2 previous similar messages [ 6360.038272] LustreError: Skipped 1 previous similar message [ 6360.042614] Lustre: Skipped 2 previous similar messages [ 6364.336933] LustreError: 344491:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6364.339483] LustreError: 344491:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6364.381870] Lustre: server umount lustre-OST0000 complete [ 6365.665634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6365.667946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6365.672412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6365.674012] Lustre: Skipped 1 previous similar message [ 6370.273690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6370.274221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.801850] Lustre: server umount lustre-MDT0000 complete [ 6373.154192] LustreError: 342209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690816 with bad export cookie 12915370144214138579 [ 6373.156710] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.161108] LustreError: 342209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6373.300678] Lustre: server umount lustre-MDT0001 complete [ 6376.960222] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6377.986547] Key type lgssc unregistered [ 6378.126345] LNet: 345424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6378.129278] LNetError: 345424:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6378.140695] LNet: Removed LNI 192.168.203.127@tcp [ 6378.437989] Key type .llcrypt unregistered [ 6378.439057] Key type ._llcrypt unregistered [ 6388.179996] Key type ._llcrypt registered [ 6388.181553] Key type .llcrypt registered [ 6388.217981] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6388.645167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6388.663029] alg: No test for adler32 (adler32-zlib) [ 6389.519261] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6389.608862] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6391.192133] Key type lgssc registered [ 6391.565569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6394.251843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6394.257473] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.261395] LustreError: 346822:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6394.271073] LustreError: 346822:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6394.273334] LustreError: 346822:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6394.275800] LustreError: 346822:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6394.278747] LustreError: 346822:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6396.721945] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6396.724435] LustreError: 347218:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6396.733233] LustreError: 347218:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6396.735127] LustreError: 347218:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6396.737085] LustreError: 347218:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6396.739357] LustreError: 347218:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6403.350786] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6404.410217] Key type lgssc unregistered [ 6404.547398] LNet: 348200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6404.550203] LNetError: 348200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6404.558360] LNet: Removed LNI 192.168.203.127@tcp [ 6404.853481] Key type .llcrypt unregistered [ 6404.857927] Key type ._llcrypt unregistered [ 6411.432810] Key type ._llcrypt registered [ 6411.433745] Key type .llcrypt registered [ 6411.471742] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 6418.381486] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6418.712451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6418.726881] alg: No test for adler32 (adler32-zlib) [ 6419.587197] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6419.671810] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6421.248171] Key type lgssc registered [ 6421.631780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6433.028734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6433.036885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.133683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6434.146441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6434.183650] Lustre: lustre-MDT0000: new disk, initializing [ 6434.206244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6434.212343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6435.589257] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6439.777406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6439.821716] Lustre: 351292:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6439.845822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6439.848046] Lustre: Skipped 1 previous similar message [ 6439.882780] Lustre: lustre-MDT0001: new disk, initializing [ 6439.916405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6439.928571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6439.932229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6441.190952] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6444.411782] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6445.634129] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6448.066506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6448.165340] Lustre: lustre-OST0000: new disk, initializing [ 6448.167099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6448.170327] Lustre: 353033:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6448.197019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6450.110430] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6453.335187] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6454.267474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6454.271133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6454.307230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6454.696298] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6454.770604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6456.113041] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6456.176584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6459.366742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.367300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6459.367724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6459.371074] Lustre: Skipped 1 previous similar message [ 6459.373331] LustreError: Skipped 1 previous similar message [ 6462.894334] LustreError: 354223:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6462.933794] Lustre: server umount lustre-OST0000 complete [ 6464.481317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.481683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6464.486617] Lustre: Skipped 1 previous similar message [ 6464.488490] Lustre: Skipped 1 previous similar message [ 6469.601386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6469.603645] Lustre: Skipped 2 previous similar messages [ 6470.401902] LustreError: 354424:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6470.404497] LustreError: 354424:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6470.437347] Lustre: server umount lustre-MDT0000 complete [ 6471.685126] LustreError: 354426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690914 with bad export cookie 12801885190798642085 [ 6471.689756] LustreError: 354426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6471.689980] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6471.752028] LustreError: 354627:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6471.754508] LustreError: 354627:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6471.805503] Lustre: server umount lustre-MDT0001 complete [ 6474.627684] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6475.175270] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:35:18 (1770690918) [ 6477.422084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.599408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6478.857701] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6481.825720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6481.938145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6483.269204] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6485.010561] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6486.226770] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6488.508706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6490.438227] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6492.220614] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6494.636352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6494.698958] Lustre: lustre-OST0001: new disk, initializing [ 6494.701813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6494.705308] Lustre: 357838:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6494.737363] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6494.740727] Lustre: Skipped 1 previous similar message [ 6496.666192] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6499.712567] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6499.845379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6499.849877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6499.888759] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6503.589111] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6503.598230] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6503.601156] Lustre: Skipped 1 previous similar message [ 6507.631100] LustreError: 358536:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6507.633996] LustreError: 358536:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6507.653658] Lustre: server umount lustre-OST0001 complete [ 6509.029866] LustreError: 356935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6509.038858] LustreError: 356935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6514.151929] LustreError: 356934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.159807] LustreError: 356934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6519.268325] LustreError: 358537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6519.274118] LustreError: 358537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6523.360878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6523.363933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.367719] Lustre: Skipped 1 previous similar message [ 6523.369554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6523.371113] Lustre: Skipped 1 previous similar message [ 6524.387024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6524.388102] LustreError: 356935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.396394] LustreError: 356935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6525.741966] LustreError: 358759:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6525.744259] LustreError: 358759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6525.777882] Lustre: server umount lustre-OST0000 complete [ 6527.968835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6527.971804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.975914] Lustre: Skipped 1 previous similar message [ 6533.222640] Lustre: server umount lustre-MDT0000 complete [ 6534.470490] LustreError: 356956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770690977 with bad export cookie 12801885190798642953 [ 6534.473897] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6534.475165] LustreError: 356956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6540.606123] LustreError: 359164:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6540.607953] LustreError: 359164:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6540.660377] Lustre: server umount lustre-MDT0001 complete [ 6544.270812] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6545.262795] Key type lgssc unregistered [ 6545.374323] LNet: 359697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6545.378045] LNetError: 359697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6545.385338] LNet: Removed LNI 192.168.203.127@tcp [ 6545.675279] Key type .llcrypt unregistered [ 6545.676442] Key type ._llcrypt unregistered [ 6564.198750] Key type ._llcrypt registered [ 6564.199901] Key type .llcrypt registered [ 6564.235489] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6564.560811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6564.567787] alg: No test for adler32 (adler32-zlib) [ 6565.435661] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6565.531502] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6567.120203] Key type lgssc registered [ 6567.520430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6570.285227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6570.291932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.386380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6571.395041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6571.463664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6572.835821] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6575.842949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6575.878411] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6575.892569] Lustre: 362452:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6575.996803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6577.345812] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6579.171877] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6580.448934] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6581.217760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6581.221491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.226574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6584.801126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6584.801539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.325942] LustreError: 363817:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6587.348514] Lustre: server umount lustre-MDT0000 complete [ 6588.645080] LustreError: 363177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691031 with bad export cookie 13715719663806628807 [ 6588.648441] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6588.648946] LustreError: 363177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6588.713271] LustreError: 364018:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6588.715670] LustreError: 364018:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6588.762317] Lustre: server umount lustre-MDT0001 complete [ 6591.914440] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:37:15 (1770691035) [ 6594.083379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.252549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6595.561128] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6598.578746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6598.695421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6600.084174] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6602.096316] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6603.459223] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6606.047469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6606.116421] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6606.202751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6608.260946] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6610.080246] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6613.813856] Lustre: Permanently deactivating lustre-OST0000 [ 6613.819161] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6621.682506] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6622.415854] LustreError: 367477:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6622.420159] LustreError: 367477:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6622.422917] LustreError: 367477:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6623.129392] LustreError: 367525:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6623.132755] LustreError: 367525:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6629.871927] LustreError: 367622:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6629.874382] LustreError: 367622:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6629.916476] Lustre: server umount lustre-OST0000 complete [ 6630.846441] LustreError: 367774:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 6630.849234] LustreError: 367774:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6631.905119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.905620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6631.910189] Lustre: Skipped 1 previous similar message [ 6631.912824] Lustre: Skipped 1 previous similar message [ 6637.025114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6637.028967] Lustre: Skipped 2 previous similar messages [ 6641.124651] LustreError: 367871:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6641.127577] LustreError: 367871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6641.154596] Lustre: server umount lustre-MDT0000 complete [ 6642.145815] LustreError: 364735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6642.151813] LustreError: 364735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6642.424123] LustreError: 364758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691085 with bad export cookie 13715719663806629374 [ 6642.426630] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6642.428401] LustreError: 364758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6651.875727] LustreError: 368073:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6651.877844] LustreError: 368073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6651.930526] Lustre: server umount lustre-MDT0001 complete [ 6655.166482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.874681] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6658.035155] LustreError: 368915:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6660.600476] Lustre: 369253:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.127@tcp [ 6661.311578] LustreError: 369350:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6661.314166] LustreError: 369350:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6661.345683] Lustre: server umount MGS complete [ 6665.227094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.373709] LustreError: 369976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6665.378593] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6665.383948] Lustre: Skipped 1 previous similar message [ 6665.402793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6666.689815] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6669.648288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6669.746310] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6671.031617] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6672.792991] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6674.018569] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6674.641643] Lustre: Permanently reactivating lustre-OST0000 [ 6674.646794] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6674.649235] Lustre: Skipped 2 previous similar messages [ 6701.024121] LustreError: 371578:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6703.983555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6704.134848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6704.137824] Lustre: Skipped 1 previous similar message [ 6706.138728] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6708.026499] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6720.483404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6720.487500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.493112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6721.508374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.873092] LustreError: 373353:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6725.875949] LustreError: 373353:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6725.917565] Lustre: server umount lustre-OST0000 complete [ 6731.232839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6731.235713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.240092] Lustre: Skipped 1 previous similar message [ 6731.242119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6731.244656] Lustre: Skipped 2 previous similar messages [ 6733.526075] Lustre: server umount lustre-MDT0000 complete [ 6734.901354] LustreError: 369956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691178 with bad export cookie 13715719663806630410 [ 6734.905255] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6734.908314] LustreError: 369956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6735.049599] Lustre: server umount lustre-MDT0001 complete [ 6738.835611] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6739.896685] Key type lgssc unregistered [ 6740.045246] LNet: 374291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6740.048632] LNetError: 374291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6740.057631] LNet: Removed LNI 192.168.203.127@tcp [ 6740.383170] Key type .llcrypt unregistered [ 6740.384766] Key type ._llcrypt unregistered [ 6747.324832] Key type ._llcrypt registered [ 6747.326354] Key type .llcrypt registered [ 6747.371271] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 6753.831277] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6754.135900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6754.142090] alg: No test for adler32 (adler32-zlib) [ 6754.994410] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6755.079159] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6756.656177] Key type lgssc registered [ 6757.035964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6769.402846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6769.421834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.543952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6770.555976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6770.591827] Lustre: lustre-MDT0000: new disk, initializing [ 6770.633591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.643694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6772.082447] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6776.407477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6776.451072] Lustre: 377384:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6776.475962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6776.478599] Lustre: Skipped 1 previous similar message [ 6776.519833] Lustre: lustre-MDT0001: new disk, initializing [ 6776.557594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6776.571794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6776.575517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6777.899396] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6781.092357] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6782.307468] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6784.644329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6784.730385] Lustre: lustre-OST0000: new disk, initializing [ 6784.732338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6784.734791] Lustre: 379123:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6784.758601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6786.235890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6786.239294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6786.282524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6786.666066] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6789.845994] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6791.171244] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6791.239911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6792.717361] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6792.805301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6796.769733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6796.771997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.772534] LustreError: Skipped 1 previous similar message [ 6796.773433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6796.777383] Lustre: Skipped 1 previous similar message [ 6799.536129] LustreError: 380313:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6799.575502] Lustre: server umount lustre-OST0000 complete [ 6801.889535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.890938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6801.893371] Lustre: Skipped 1 previous similar message [ 6801.895185] Lustre: Skipped 1 previous similar message [ 6807.009078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6807.011234] Lustre: Skipped 2 previous similar messages [ 6812.129078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6812.131360] Lustre: Skipped 1 previous similar message [ 6815.200150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6815.236306] LustreError: 380514:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6815.239029] LustreError: 380514:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6815.264500] Lustre: server umount lustre-MDT0000 complete [ 6816.611261] LustreError: 378161:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691259 with bad export cookie 17354202105497563237 [ 6816.614512] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6816.615677] LustreError: 378161:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6816.687864] LustreError: 380717:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6816.690671] LustreError: 380717:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6816.761306] Lustre: server umount lustre-MDT0001 complete [ 6819.886748] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6820.449633] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:41:03 (1770691263) [ 6822.833709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6823.003831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6824.258592] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6827.056898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6827.177905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6828.429194] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6830.194189] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6831.379619] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6833.661133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6835.697901] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6837.608416] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6838.282916] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6849.008879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6852.576745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6852.579027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.582901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6853.602170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6856.559154] LustreError: 383885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6856.561736] LustreError: 383885:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6856.604962] Lustre: server umount lustre-OST0000 complete [ 6858.208882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6858.211567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.216010] Lustre: Skipped 1 previous similar message [ 6863.841141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6863.843714] Lustre: Skipped 5 previous similar messages [ 6864.062183] LustreError: 384089:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6864.064467] LustreError: 384089:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6864.087266] Lustre: server umount lustre-MDT0000 complete [ 6865.304106] LustreError: 382150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691308 with bad export cookie 17354202105497564105 [ 6865.307133] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6865.307633] LustreError: 382150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6865.425255] Lustre: server umount lustre-MDT0001 complete [ 6869.002912] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6870.116292] Key type lgssc unregistered [ 6870.242377] LNet: 384822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6870.244790] LNetError: 384822:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6870.253326] LNet: Removed LNI 192.168.203.127@tcp [ 6870.565847] Key type .llcrypt unregistered [ 6870.566829] Key type ._llcrypt unregistered [ 6878.531058] Key type ._llcrypt registered [ 6878.532358] Key type .llcrypt registered [ 6878.579259] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6878.702351] Key type .llcrypt unregistered [ 6878.703537] Key type ._llcrypt unregistered [ 6885.596585] Key type ._llcrypt registered [ 6885.597997] Key type .llcrypt registered [ 6885.637782] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6886.056044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6886.065340] alg: No test for adler32 (adler32-zlib) [ 6886.935278] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6887.048726] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6888.632153] Key type lgssc registered [ 6889.025156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6891.797346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6891.804168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.968958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.458503] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6897.686018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6897.861282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.030894] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6901.991749] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6903.009605] LustreError: 386970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.015505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6903.017377] LustreError: 386970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6904.537786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6904.676476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6906.145913] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6915.046092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6915.046567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6915.050608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6915.054352] Lustre: Skipped 1 previous similar message [ 6919.663079] LustreError: 389021:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6919.696348] Lustre: server umount lustre-OST0000 complete [ 6923.746109] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6923.748682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.753259] Lustre: Skipped 1 previous similar message [ 6923.755130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6925.280896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6925.281465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.034118] LustreError: 389223:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6927.036019] LustreError: 389223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6927.062880] Lustre: server umount lustre-MDT0000 complete [ 6928.242227] LustreError: 386951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691371 with bad export cookie 6226489953580665475 [ 6928.245017] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6928.246759] LustreError: 386951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6928.313173] LustreError: 389424:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6928.314856] LustreError: 389424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6928.363316] Lustre: server umount lustre-MDT0001 complete [ 6931.674590] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6932.620208] Key type lgssc unregistered [ 6932.761320] LNet: 389955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6932.764054] LNetError: 389955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6932.772272] LNet: Removed LNI 192.168.203.127@tcp [ 6933.057794] Key type .llcrypt unregistered [ 6933.058830] Key type ._llcrypt unregistered [ 6942.002649] Key type ._llcrypt registered [ 6942.003711] Key type .llcrypt registered [ 6942.039903] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6942.386796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6942.430731] alg: No test for adler32 (adler32-zlib) [ 6943.281896] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 6943.359529] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6944.944135] Key type lgssc registered [ 6945.332787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6947.726311] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6947.731461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6963.168263] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6968.288183] LustreError: 391302:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.127@tcp: failed processing log, type 4: rc = -110 [ 6994.912155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.693853] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6998.396611] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7000.548167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7000.701142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.727357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7001.904578] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7004.644100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7004.772038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7005.969574] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7007.577847] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7008.740331] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7016.416818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7016.419346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7016.423442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7019.491073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7019.492073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7021.551659] LustreError: 393575:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7021.593172] Lustre: server umount lustre-OST0000 complete [ 7024.610592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.615168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7024.616850] Lustre: Skipped 2 previous similar messages [ 7028.990915] LustreError: 393776:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7028.992470] LustreError: 393776:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7029.021921] Lustre: server umount lustre-MDT0000 complete [ 7029.728851] LustreError: 392226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.734707] LustreError: 392226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7030.245993] LustreError: 391310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691473 with bad export cookie 10524984409143195475 [ 7030.249648] LustreError: 391310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7030.249894] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7030.318514] LustreError: 393980:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7030.320822] LustreError: 393980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7030.368796] Lustre: server umount lustre-MDT0001 complete [ 7034.094669] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7035.108841] Key type lgssc unregistered [ 7035.233358] LNet: 394511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7035.235895] LNetError: 394511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7035.249384] LNet: Removed LNI 192.168.203.127@tcp [ 7035.531990] Key type .llcrypt unregistered [ 7035.533177] Key type ._llcrypt unregistered [ 7044.427996] Key type ._llcrypt registered [ 7044.429545] Key type .llcrypt registered [ 7044.476064] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7044.900679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7044.960905] alg: No test for adler32 (adler32-zlib) [ 7045.823178] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 7045.909650] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7047.488162] Key type lgssc registered [ 7047.873388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7050.524667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7050.531303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.699431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.022783] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7055.926900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7056.047851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7057.247350] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7058.958497] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7060.073680] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7062.240020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7062.362678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7064.183428] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7065.918927] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7076.713463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7086.506607] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 7087.074460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7087.075444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.076808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.076815] Lustre: Skipped 1 previous similar message [ 7087.082945] Lustre: Skipped 2 previous similar messages [ 7088.251466] LustreError: 398144:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7088.275881] Lustre: server umount lustre-MDT0000 complete [ 7088.961376] LustreError: 395864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.27@tcp arrived at 1770691532 with bad export cookie 6388066339357860162 [ 7088.967059] LustreError: 395864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7089.577105] LustreError: 395864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691532 with bad export cookie 6388066339357859511 [ 7089.580686] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7089.581853] LustreError: 395864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7092.193747] LustreError: 395885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.194153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7092.194647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.195643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7092.195650] Lustre: Skipped 1 previous similar message [ 7092.199052] LustreError: 395885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7094.240900] LustreError: 395885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.241437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7094.244851] LustreError: 395885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7095.662987] LustreError: 398346:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7095.665401] LustreError: 398346:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7095.703010] Lustre: server umount lustre-OST0000 complete [ 7097.878820] LustreError: 398699:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7097.881430] LustreError: 398699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7097.939041] Lustre: server umount lustre-MDT0001 complete [ 7101.240751] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7102.176834] Key type lgssc unregistered [ 7102.299350] LNet: 399231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7102.302550] LNetError: 399231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7102.312299] LNet: Removed LNI 192.168.203.127@tcp [ 7102.582570] Key type .llcrypt unregistered [ 7102.583630] Key type ._llcrypt unregistered [ 7111.182844] Key type ._llcrypt registered [ 7111.183911] Key type .llcrypt registered [ 7111.219371] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7111.612970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7111.635196] alg: No test for adler32 (adler32-zlib) [ 7112.477932] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 7112.554098] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7114.128169] Key type lgssc registered [ 7114.489497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7117.056672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7117.063310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7118.218283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7119.496032] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7122.389890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7122.515172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7123.797117] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7125.604244] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7126.819908] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7129.089307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7129.221412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.041876] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7132.822581] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7143.591643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7148.000845] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7148.003065] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.007244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7149.027618] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.028084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7149.033010] Lustre: Skipped 1 previous similar message [ 7151.359682] LustreError: 402864:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7151.384960] Lustre: server umount lustre-MDT0001 complete [ 7154.146100] LustreError: 400603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.147140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7154.147522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.148973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7154.148980] Lustre: Skipped 1 previous similar message [ 7154.150480] LustreError: 400603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7159.216111] LustreError: 403067:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7159.217924] LustreError: 403067:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7159.262272] Lustre: server umount lustre-OST0000 complete [ 7159.266981] LustreError: 400603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7160.598988] LustreError: 403268:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7160.600884] LustreError: 403268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7160.662759] Lustre: server umount lustre-MDT0000 complete [ 7165.510779] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7166.549456] Key type lgssc unregistered [ 7166.674521] LNet: 403951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7166.678687] LNetError: 403951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7166.690360] LNet: Removed LNI 192.168.203.127@tcp [ 7167.000482] Key type .llcrypt unregistered [ 7167.001694] Key type ._llcrypt unregistered [ 7175.305515] Key type ._llcrypt registered [ 7175.306593] Key type .llcrypt registered [ 7175.345105] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 7181.485255] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7181.771725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7181.786059] alg: No test for adler32 (adler32-zlib) [ 7182.649282] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 7182.740871] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7184.328161] Key type lgssc registered [ 7184.692887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7197.029187] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7202.331975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7202.336976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.437430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7203.452713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7203.492431] Lustre: lustre-MDT0000: new disk, initializing [ 7203.517107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.524793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7204.864341] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7210.503259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7210.613224] Lustre: lustre-OST0000: new disk, initializing [ 7210.615580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7210.617459] Lustre: Skipped 1 previous similar message [ 7210.619730] Lustre: 408884:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7210.643347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7211.970313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7211.973616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7211.985058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7212.581422] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7215.715069] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7221.299050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7221.339019] Lustre: 407884:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7221.364495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7221.400931] Lustre: lustre-MDT0001: new disk, initializing [ 7221.441391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7221.453921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7221.457800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7222.688745] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7226.158556] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7226.229338] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7227.522886] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7228.611524] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 7232.941137] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [ 7232.994352] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7232.994726] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.349550] LustreError: 411299:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7236.373227] Lustre: server umount lustre-MDT0001 complete [ 7243.235659] LustreError: 407889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.239445] LustreError: 407889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7248.355201] LustreError: 407889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7248.360505] LustreError: 407889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7252.960982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7252.963798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.967819] Lustre: Skipped 1 previous similar message [ 7252.969821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7252.972452] Lustre: Skipped 2 previous similar messages [ 7253.475178] LustreError: 410112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.767090] LustreError: 411571:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7254.770177] LustreError: 411571:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7254.815538] Lustre: server umount lustre-OST0000 complete [ 7262.465616] LustreError: 411773:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7262.468135] LustreError: 411773:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7262.533061] Lustre: server umount lustre-MDT0000 complete [ 7267.481377] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7268.514781] Key type lgssc unregistered [ 7268.649307] LNet: 412457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7268.651825] LNetError: 412457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7268.659280] LNet: Removed LNI 192.168.203.127@tcp [ 7268.966390] Key type .llcrypt unregistered [ 7268.968058] Key type ._llcrypt unregistered [ 7281.504664] Key type ._llcrypt registered [ 7281.506022] Key type .llcrypt registered [ 7281.546159] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7281.661313] Key type .llcrypt unregistered [ 7281.662430] Key type ._llcrypt unregistered [ 7288.373397] Key type ._llcrypt registered [ 7288.374785] Key type .llcrypt registered [ 7288.413621] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7288.770455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7288.780802] alg: No test for adler32 (adler32-zlib) [ 7289.639395] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 7289.722874] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7291.304194] Key type lgssc registered [ 7291.670074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7300.540069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7300.548745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.649289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7301.661355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7301.702925] Lustre: lustre-MDT0000: new disk, initializing [ 7301.729609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7301.736933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7303.055078] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7307.233968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7307.273936] Lustre: 415764:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7307.296477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7307.298911] Lustre: Skipped 1 previous similar message [ 7307.339498] Lustre: lustre-MDT0001: new disk, initializing [ 7307.372595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7307.383789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7307.389134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7308.669838] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7311.774423] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7312.984567] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7315.320242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7315.445734] Lustre: lustre-OST0000: new disk, initializing [ 7315.447643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7315.450070] Lustre: 417503:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7315.486133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.398180] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7320.565979] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7322.618273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7322.622629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7322.656202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7332.834953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.838663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.838948] Lustre: Skipped 2 previous similar messages [ 7332.840772] Lustre: Skipped 1 previous similar message [ 7334.530572] LustreError: 418198:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7334.559294] Lustre: server umount lustre-MDT0000 complete [ 7335.824362] LustreError: 415756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691779 with bad export cookie 3674722737978783398 [ 7335.827876] LustreError: 415756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7335.831045] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7335.903692] LustreError: 418399:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7335.905618] LustreError: 418399:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7335.962616] Lustre: server umount lustre-MDT0001 complete [ 7347.381424] LustreError: 418600:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7347.383631] LustreError: 418600:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7347.431511] Lustre: server umount lustre-OST0000 complete [ 7350.985517] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:49:54 (1770691794) [ 7353.553217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.728614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7355.008213] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7357.854318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7357.978467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7359.271952] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7361.199185] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7362.511784] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7375.754888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7375.838256] Lustre: Found index 0 for lustre-OST0000, updating log [ 7375.897701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7377.707656] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7378.982503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7379.463381] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7383.714195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7383.714848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7386.098558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7386.101858] Lustre: Skipped 1 previous similar message [ 7386.732105] LustreError: 421661:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7386.735474] LustreError: 421661:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7386.769417] Lustre: server umount lustre-OST0000 complete [ 7402.337932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.338497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7402.343816] Lustre: Skipped 1 previous similar message [ 7402.346371] Lustre: Skipped 1 previous similar message [ 7405.374738] LustreError: 421923:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7405.377231] LustreError: 421923:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7405.405850] Lustre: server umount lustre-MDT0000 complete [ 7406.664518] LustreError: 419397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691849 with bad export cookie 3674722737978784777 [ 7406.667827] LustreError: 419397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7406.668572] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7406.794520] Lustre: server umount lustre-MDT0001 complete [ 7409.520348] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:50:52 (1770691852) [ 7437.216316] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp [ 7989.374654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.529867] LustreError: 425479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.535585] LustreError: 425479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7989.552625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.828235] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7993.548323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7994.897575] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7996.749280] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7997.919124] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8011.506047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8011.607705] Lustre: Found index 0 for lustre-OST0000, updating log [ 8013.978809] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8015.734957] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8015.847048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 8019.555915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.564867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8019.568739] Lustre: Skipped 3 previous similar messages [ 8020.581226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8022.573101] LustreError: 427784:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8022.576885] LustreError: 427784:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8022.613481] Lustre: server umount lustre-OST0000 complete [ 8024.545470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8024.550090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.558793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8025.570728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.593418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8026.596013] Lustre: Skipped 3 previous similar messages [ 8030.689883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8030.692789] Lustre: Skipped 7 previous similar messages [ 8038.368081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8038.427185] Lustre: server umount lustre-MDT0000 complete [ 8038.881586] LustreError: 425479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8038.889832] LustreError: 425479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8039.906067] LustreError: 425479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.910642] LustreError: 425479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8040.069914] LustreError: 425460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770692483 with bad export cookie 3674722737978785834 [ 8040.075452] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8040.077179] LustreError: 425460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8040.182513] LustreError: 428207:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8040.186711] LustreError: 428207:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8040.255808] Lustre: server umount lustre-MDT0001 complete [ 8052.640308] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp [ 8057.634774] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8060.813916] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8072.398953] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:01:55 (1770692515) [ 8078.223744] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8079.194900] Key type lgssc unregistered [ 8079.327907] LNet: 431755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8079.332264] LNetError: 431755:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8144.870302] LNet: Removed LNI 192.168.203.127@tcp [ 8145.182462] Key type .llcrypt unregistered [ 8145.183623] Key type ._llcrypt unregistered [ 8151.658902] Key type ._llcrypt registered [ 8151.660689] Key type .llcrypt registered [ 8151.696078] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8152.032326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8152.050698] alg: No test for adler32 (adler32-zlib) [ 8152.904906] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 8152.985199] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8154.568161] Key type lgssc registered [ 8154.907690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8156.374728] LNet: Added LNI 192.168.203.127@tcp100 [8/256/0/180] [ 8156.645520] LNet: Added LNI 192.168.203.127@tcp101 [8/256/0/180] [ 8156.915356] LNet: Added LNI 192.168.203.127@tcp102 [8/256/0/180] [ 8157.192669] LNet: Added LNI 192.168.203.127@tcp103 [8/256/0/180] [ 8157.482556] LNet: Added LNI 192.168.203.127@tcp104 [8/256/0/180] [ 8157.770668] LNet: Added LNI 192.168.203.127@tcp105 [8/256/0/180] [ 8158.099696] LNet: Added LNI 192.168.203.127@tcp106 [8/256/0/180] [ 8158.422344] LNet: Added LNI 192.168.203.127@tcp107 [8/256/0/180] [ 8158.865976] LNet: Added LNI 192.168.203.127@tcp108 [8/256/0/180] [ 8159.310701] LNet: Added LNI 192.168.203.127@tcp109 [8/256/0/180] [ 8159.714281] LNet: Added LNI 192.168.203.127@tcp110 [8/256/0/180] [ 8160.122188] LNet: Added LNI 192.168.203.127@tcp111 [8/256/0/180] [ 8160.438750] LNet: Added LNI 192.168.203.127@tcp112 [8/256/0/180] [ 8160.756699] LNet: Added LNI 192.168.203.127@tcp113 [8/256/0/180] [ 8161.062162] LNet: Added LNI 192.168.203.127@tcp114 [8/256/0/180] [ 8161.388348] LNet: Added LNI 192.168.203.127@tcp115 [8/256/0/180] [ 8161.661031] LNet: Added LNI 192.168.203.127@tcp116 [8/256/0/180] [ 8161.926869] LNet: Added LNI 192.168.203.127@tcp117 [8/256/0/180] [ 8162.208181] LNet: Added LNI 192.168.203.127@tcp118 [8/256/0/180] [ 8162.507647] LNet: Added LNI 192.168.203.127@tcp119 [8/256/0/180] [ 8162.828997] LNet: Added LNI 192.168.203.127@tcp120 [8/256/0/180] [ 8163.179594] LNet: Added LNI 192.168.203.127@tcp121 [8/256/0/180] [ 8163.555559] LNet: Added LNI 192.168.203.127@tcp122 [8/256/0/180] [ 8163.836073] LNet: Added LNI 192.168.203.127@tcp123 [8/256/0/180] [ 8164.108130] LNet: Added LNI 192.168.203.127@tcp124 [8/256/0/180] [ 8164.392727] LNet: Added LNI 192.168.203.127@tcp125 [8/256/0/180] [ 8164.721517] LNet: Added LNI 192.168.203.127@tcp126 [8/256/0/180] [ 8165.055496] LNet: Added LNI 192.168.203.127@tcp127 [8/256/0/180] [ 8165.372972] LNet: Added LNI 192.168.203.127@tcp128 [8/256/0/180] [ 8165.663599] LNet: Added LNI 192.168.203.127@tcp129 [8/256/0/180] [ 8165.994574] LNet: Added LNI 192.168.203.127@tcp130 [8/256/0/180] [ 8166.292354] LNet: Added LNI 192.168.203.127@tcp131 [8/256/0/180] [ 8166.570967] LNet: Added LNI 192.168.203.127@tcp132 [8/256/0/180] [ 8166.851417] LNet: Added LNI 192.168.203.127@tcp133 [8/256/0/180] [ 8167.137785] LNet: Added LNI 192.168.203.127@tcp134 [8/256/0/180] [ 8167.418463] LNet: Added LNI 192.168.203.127@tcp135 [8/256/0/180] [ 8169.693916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8169.702656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.747805] Lustre: 434731:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [ 8170.811092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8170.821115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8170.861125] Lustre: lustre-MDT0000: new disk, initializing [ 8170.890788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8170.898193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8172.145917] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8176.187848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8176.231202] Lustre: 434748:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8176.254256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8176.256291] Lustre: Skipped 1 previous similar message [ 8176.293217] Lustre: lustre-MDT0001: new disk, initializing [ 8176.305078] Lustre: 435529:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [ 8176.308212] Lustre: 435529:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8176.344421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8176.355682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8176.358420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8177.571643] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8178.225449] Lustre: 435877:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [ 8178.228278] Lustre: 435877:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8180.631190] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8181.907358] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8184.194528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8184.309921] Lustre: lustre-OST0000: new disk, initializing [ 8184.312294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8184.315322] Lustre: 436492:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8184.344908] Lustre: 436497:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [ 8184.347565] Lustre: 436497:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 8184.362549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8186.225407] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8187.005832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8187.008965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8187.054318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8189.279972] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8192.482188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8192.483485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.484112] LustreError: Skipped 1 previous similar message [ 8192.484693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8192.487008] Lustre: Skipped 1 previous similar message [ 8192.490368] Lustre: Skipped 1 previous similar message [ 8197.602703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8197.604704] Lustre: Skipped 1 previous similar message [ 8202.724208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8202.727656] Lustre: Skipped 1 previous similar message [ 8206.304246] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8206.325672] LustreError: 437332:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8206.363891] Lustre: server umount lustre-OST0000 complete [ 8207.841174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8207.846483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8212.449255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8212.453046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8212.459603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8213.697526] LustreError: 437534:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8213.699469] LustreError: 437534:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8213.729872] Lustre: server umount lustre-MDT0000 complete [ 8215.036182] LustreError: 434741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770692658 with bad export cookie 16214403559452780979 [ 8215.042990] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8215.044403] LustreError: 434741:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8215.112920] LustreError: 437736:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8215.116388] LustreError: 437736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8215.176307] Lustre: server umount lustre-MDT0001 complete [ 8218.820240] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8219.830484] Key type lgssc unregistered [ 8219.988878] LNet: 438275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8219.992594] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8219.995828] LNet: Removed LNI 192.168.203.127@tcp [ 8221.024232] LNet: Removed LNI 192.168.203.127@tcp100 [ 8221.026939] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8221.031609] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8221.036357] LNet: Removed LNI 192.168.203.127@tcp101 [ 8222.048174] LNet: Removed LNI 192.168.203.127@tcp102 [ 8222.049631] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8222.052603] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8222.055871] LNet: Removed LNI 192.168.203.127@tcp103 [ 8223.072166] LNet: Removed LNI 192.168.203.127@tcp104 [ 8224.096167] LNet: Removed LNI 192.168.203.127@tcp105 [ 8224.097768] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8224.100432] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8224.103110] LNet: Removed LNI 192.168.203.127@tcp106 [ 8225.120162] LNet: Removed LNI 192.168.203.127@tcp107 [ 8226.144158] LNet: Removed LNI 192.168.203.127@tcp108 [ 8227.168116] LNet: Removed LNI 192.168.203.127@tcp109 [ 8228.192131] LNet: Removed LNI 192.168.203.127@tcp110 [ 8228.193561] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8228.196595] LNetError: 438275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8228.200226] LNet: Removed LNI 192.168.203.127@tcp111 [ 8229.216128] LNet: Removed LNI 192.168.203.127@tcp112 [ 8230.240132] LNet: Removed LNI 192.168.203.127@tcp113 [ 8230.241551] LNet: Removed LNI 192.168.203.127@tcp114 [ 8230.242811] LNet: Removed LNI 192.168.203.127@tcp115 [ 8230.243982] LNet: Removed LNI 192.168.203.127@tcp116 [ 8230.245190] LNet: Removed LNI 192.168.203.127@tcp117 [ 8230.246275] LNet: Removed LNI 192.168.203.127@tcp118 [ 8230.247512] LNet: Removed LNI 192.168.203.127@tcp119 [ 8230.248713] LNet: Removed LNI 192.168.203.127@tcp120 [ 8230.249890] LNet: Removed LNI 192.168.203.127@tcp121 [ 8230.251150] LNet: Removed LNI 192.168.203.127@tcp122 [ 8230.252384] LNet: Removed LNI 192.168.203.127@tcp123 [ 8230.253520] LNet: Removed LNI 192.168.203.127@tcp124 [ 8230.254702] LNet: Removed LNI 192.168.203.127@tcp125 [ 8230.255934] LNet: Removed LNI 192.168.203.127@tcp126 [ 8230.257181] LNet: Removed LNI 192.168.203.127@tcp127 [ 8230.258349] LNet: Removed LNI 192.168.203.127@tcp128 [ 8230.259492] LNet: Removed LNI 192.168.203.127@tcp129 [ 8230.260881] LNet: Removed LNI 192.168.203.127@tcp130 [ 8230.262135] LNet: Removed LNI 192.168.203.127@tcp131 [ 8230.263342] LNet: Removed LNI 192.168.203.127@tcp132 [ 8230.264533] LNet: Removed LNI 192.168.203.127@tcp133 [ 8230.265894] LNet: Removed LNI 192.168.203.127@tcp134 [ 8230.273660] LNet: Removed LNI 192.168.203.127@tcp135 [ 8230.560470] Key type .llcrypt unregistered [ 8230.561533] Key type ._llcrypt unregistered [ 8237.891930] Key type ._llcrypt registered [ 8237.893938] Key type .llcrypt registered [ 8237.933236] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8244.289233] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8244.838716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8244.868668] alg: No test for adler32 (adler32-zlib) [ 8245.735311] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 8245.829736] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8247.424161] Key type lgssc registered [ 8247.798735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8259.247882] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:05:02 (1770692702) [ 8265.254651] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8266.314759] Key type lgssc unregistered [ 8266.439331] LNet: 442276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8266.441699] LNetError: 442276:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8266.448279] LNet: Removed LNI 192.168.203.127@tcp [ 8266.722653] Key type .llcrypt unregistered [ 8266.723723] Key type ._llcrypt unregistered [ 8273.355742] Key type ._llcrypt registered [ 8273.356662] Key type .llcrypt registered [ 8273.389943] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8273.735462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8273.781383] alg: No test for adler32 (adler32-zlib) [ 8274.631122] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 8274.714869] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8276.304197] Key type lgssc registered [ 8276.706387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8280.008062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8280.013172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8281.110222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8281.121272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8281.158124] Lustre: lustre-MDT0000: new disk, initializing [ 8281.181106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8281.187708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8282.408586] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8287.205057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8287.250203] Lustre: 443446:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8287.275604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8287.277907] Lustre: Skipped 1 previous similar message [ 8287.318784] Lustre: lustre-MDT0001: new disk, initializing [ 8287.349449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8287.360281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8287.363467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8288.587507] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8291.832204] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8293.090643] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8295.455572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8295.545546] Lustre: lustre-OST0000: new disk, initializing [ 8295.547342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8295.550649] Lustre: 445188:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8295.576310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.087266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8297.090962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8297.126346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8297.419127] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8300.631591] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8301.814290] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing lnet_if_list [ 8303.370645] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8303.380494] LNet: Added LNI 192.168.203.127@tcp10 [8/256/0/180] [ 8305.267614] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8305.277985] LNet: Added LNI 192.168.203.127@tcp11 [8/256/0/180] [ 8306.597486] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8306.607334] LNet: Added LNI 192.168.203.127@tcp12 [8/256/0/180] [ 8307.857132] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8307.867786] LNet: Added LNI 192.168.203.127@tcp13 [8/256/0/180] [ 8309.361226] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8309.383566] LNet: Added LNI 192.168.203.127@tcp14 [8/256/0/180] [ 8311.022232] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8311.039708] LNet: Added LNI 192.168.203.127@tcp15 [8/256/0/180] [ 8312.523259] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8312.533950] LNet: Added LNI 192.168.203.127@tcp16 [8/256/0/180] [ 8314.015685] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8314.025762] LNet: Added LNI 192.168.203.127@tcp17 [8/256/0/180] [ 8315.612173] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8315.626675] LNet: Added LNI 192.168.203.127@tcp18 [8/256/0/180] [ 8317.133126] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8317.146296] LNet: Added LNI 192.168.203.127@tcp19 [8/256/0/180] [ 8318.772152] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8318.782469] LNet: Added LNI 192.168.203.127@tcp20 [8/256/0/180] [ 8320.272478] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8320.283506] LNet: Added LNI 192.168.203.127@tcp21 [8/256/0/180] [ 8322.106763] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8322.117860] LNet: Added LNI 192.168.203.127@tcp22 [8/256/0/180] [ 8323.824871] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8323.835806] LNet: Added LNI 192.168.203.127@tcp23 [8/256/0/180] [ 8325.600626] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8325.617241] LNet: Added LNI 192.168.203.127@tcp24 [8/256/0/180] [ 8327.322156] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8327.334128] LNet: Added LNI 192.168.203.127@tcp25 [8/256/0/180] [ 8329.156889] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8329.169638] LNet: Added LNI 192.168.203.127@tcp26 [8/256/0/180] [ 8331.132749] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8331.149685] LNet: Added LNI 192.168.203.127@tcp27 [8/256/0/180] [ 8331.157569] LNetError: 442709:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [ 8331.161801] LNetError: Protocol error connecting to 192.168.203.127@tcp26 at host 192.168.203.127:988: is it running a compatible version of Lustre? [ 8333.080280] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8333.091055] LNet: Added LNI 192.168.203.127@tcp28 [8/256/0/180] [ 8333.095514] LNetError: 442706:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [ 8333.099068] LNetError: Protocol error connecting to 192.168.203.127@tcp26 at host 192.168.203.127:988: is it running a compatible version of Lustre? [ 8335.228287] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8335.241182] LNet: Added LNI 192.168.203.127@tcp29 [8/256/0/180] [ 8335.245833] LNetError: 442707:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [ 8335.248674] LNetError: Protocol error connecting to 192.168.203.127@tcp28 at host 192.168.203.127:988: is it running a compatible version of Lustre? [ 8336.160195] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp28 [ 8337.228437] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8337.238343] LNet: Added LNI 192.168.203.127@tcp30 [8/256/0/180] [ 8338.209042] LNetError: 442707:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [ 8338.212913] LNetError: 442707:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 8338.215733] LNetError: Protocol error connecting to 192.168.203.127@tcp28 at host 192.168.203.127:988: is it running a compatible version of Lustre? [ 8338.218439] LNetError: Skipped 3 previous similar messages [ 8339.029438] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8339.038958] LNet: Added LNI 192.168.203.127@tcp31 [8/256/0/180] [ 8340.777417] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8340.787054] LNet: Added LNI 192.168.203.127@tcp32 [8/256/0/180] [ 8342.306374] LNetError: 442709:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [ 8342.309338] LNetError: 442709:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 8342.311492] LNetError: Protocol error connecting to 192.168.203.127@tcp28 at host 192.168.203.127:988: is it running a compatible version of Lustre? [ 8342.314613] LNetError: Skipped 3 previous similar messages [ 8342.620178] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8342.629859] LNet: Added LNI 192.168.203.127@tcp33 [8/256/0/180] [ 8344.539226] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8344.549143] LNet: Added LNI 192.168.203.127@tcp34 [8/256/0/180] [ 8346.399323] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8346.408537] LNet: Added LNI 192.168.203.127@tcp35 [8/256/0/180] [ 8348.345819] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8348.359924] LNet: Added LNI 192.168.203.127@tcp36 [8/256/0/180] [ 8350.356535] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8350.365528] LNet: Added LNI 192.168.203.127@tcp37 [8/256/0/180] [ 8350.368795] LNetError: 442708:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [ 8350.371638] LNetError: 442708:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 14 previous similar messages [ 8350.374099] LNetError: Protocol error connecting to 192.168.203.127@tcp35 at host 192.168.203.127:988: is it running a compatible version of Lustre? [ 8350.377140] LNetError: Skipped 14 previous similar messages [ 8352.435312] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8352.445209] LNet: Added LNI 192.168.203.127@tcp38 [8/256/0/180] [ 8354.502185] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8354.511623] LNet: Added LNI 192.168.203.127@tcp39 [8/256/0/180] [ 8356.427801] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8356.436427] LNet: Added LNI 192.168.203.127@tcp40 [8/256/0/180] [ 8358.383944] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8358.393451] LNet: Added LNI 192.168.203.127@tcp41 [8/256/0/180] [ 8360.405493] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8360.416202] LNet: Added LNI 192.168.203.127@tcp42 [8/256/0/180] [ 8362.453982] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8362.467877] LNet: Added LNI 192.168.203.127@tcp43 [8/256/0/180] [ 8364.574072] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8364.583644] LNet: Added LNI 192.168.203.127@tcp44 [8/256/0/180] [ 8366.849277] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8366.859447] LNet: Added LNI 192.168.203.127@tcp45 [8/256/0/180] [ 8366.862848] LNetError: 442706:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [ 8366.865988] LNetError: 442706:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 32 previous similar messages [ 8366.868446] LNetError: Protocol error connecting to 192.168.203.127@tcp43 at host 192.168.203.127:988: is it running a compatible version of Lustre? [ 8366.870966] LNetError: Skipped 32 previous similar messages [ 8369.115828] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8369.129190] LNet: Added LNI 192.168.203.127@tcp46 [8/256/0/180] [ 8371.369962] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8371.381772] LNet: Added LNI 192.168.203.127@tcp47 [8/256/0/180] [ 8373.699119] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8373.708866] LNet: Added LNI 192.168.203.127@tcp48 [8/256/0/180] [ 8376.105804] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8376.118879] LNet: Added LNI 192.168.203.127@tcp49 [8/256/0/180] [ 8378.641939] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8378.652056] LNet: Added LNI 192.168.203.127@tcp50 [8/256/0/180] [ 8389.603956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.604627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8389.605058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8389.607718] Lustre: Skipped 1 previous similar message [ 8389.626200] LustreError: Skipped 1 previous similar message [ 8391.790510] LustreError: 467616:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8391.829424] Lustre: server umount lustre-OST0000 complete [ 8394.721611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.722321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8394.725766] Lustre: Skipped 1 previous similar message [ 8394.727736] Lustre: Skipped 1 previous similar message [ 8396.576129] LNet: 6 local NIs in recovery (showing 6): 192.168.203.127@tcp48, 192.168.203.127@tcp39, 192.168.203.127@tcp38, 192.168.203.127@tcp33, 192.168.203.127@tcp29, 192.168.203.127@tcp28 [ 8399.168163] LustreError: 467817:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8399.169883] LustreError: 467817:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8399.196297] Lustre: server umount lustre-MDT0000 complete [ 8399.649087] LNetError: 442706:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [ 8399.651721] LNetError: 442706:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 45 previous similar messages [ 8399.655589] LNetError: Protocol error connecting to 192.168.203.127@tcp29 at host 192.168.203.127:988: is it running a compatible version of Lustre? [ 8399.658372] LNetError: Skipped 45 previous similar messages [ 8400.385153] LustreError: 444224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770692843 with bad export cookie 16618094563490087512 [ 8400.387926] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8400.388365] LustreError: 444224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8400.458263] LustreError: 468019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8400.460132] LustreError: 468019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8400.510957] Lustre: server umount lustre-MDT0001 complete [ 8404.182386] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8405.137859] Key type lgssc unregistered [ 8405.297999] LNet: 468557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8405.306285] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8405.313662] LNet: Removed LNI 192.168.203.127@tcp [ 8406.368163] LNet: Removed LNI 192.168.203.127@tcp10 [ 8406.372641] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8406.380652] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8406.389450] LNet: Removed LNI 192.168.203.127@tcp11 [ 8407.456151] LNet: Removed LNI 192.168.203.127@tcp12 [ 8407.457409] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8407.459304] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8407.461299] LNet: Removed LNI 192.168.203.127@tcp13 [ 8408.480168] LNet: Removed LNI 192.168.203.127@tcp14 [ 8409.504158] LNet: Removed LNI 192.168.203.127@tcp15 [ 8409.505500] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8409.507507] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8409.510463] LNet: Removed LNI 192.168.203.127@tcp16 [ 8410.528101] LNet: Removed LNI 192.168.203.127@tcp17 [ 8411.552099] LNet: Removed LNI 192.168.203.127@tcp18 [ 8412.576105] LNet: Removed LNI 192.168.203.127@tcp19 [ 8413.600147] LNet: Removed LNI 192.168.203.127@tcp20 [ 8413.602149] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8413.604241] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8413.606453] LNet: Removed LNI 192.168.203.127@tcp21 [ 8414.624115] LNet: Removed LNI 192.168.203.127@tcp22 [ 8415.648125] LNet: Removed LNI 192.168.203.127@tcp23 [ 8416.672173] LNet: Removed LNI 192.168.203.127@tcp24 [ 8417.696123] LNet: Removed LNI 192.168.203.127@tcp25 [ 8418.720176] LNet: Removed LNI 192.168.203.127@tcp26 [ 8419.744104] LNet: Removed LNI 192.168.203.127@tcp27 [ 8419.745285] LNet: Removed LNI 192.168.203.127@tcp28 [ 8419.746394] LNet: Removed LNI 192.168.203.127@tcp29 [ 8420.768130] LNet: Removed LNI 192.168.203.127@tcp30 [ 8421.792155] LNet: Removed LNI 192.168.203.127@tcp31 [ 8421.793422] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8421.795486] LNetError: 468557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 10 previous similar messages [ 8421.797747] LNet: Removed LNI 192.168.203.127@tcp32 [ 8421.798940] LNet: Removed LNI 192.168.203.127@tcp33 [ 8422.816124] LNet: Removed LNI 192.168.203.127@tcp34 [ 8423.840127] LNet: Removed LNI 192.168.203.127@tcp35 [ 8424.864135] LNet: Removed LNI 192.168.203.127@tcp36 [ 8425.888102] LNet: Removed LNI 192.168.203.127@tcp37 [ 8425.889360] LNet: Removed LNI 192.168.203.127@tcp38 [ 8425.890447] LNet: Removed LNI 192.168.203.127@tcp39 [ 8426.912148] LNet: Removed LNI 192.168.203.127@tcp40 [ 8427.936138] LNet: Removed LNI 192.168.203.127@tcp41 [ 8428.960138] LNet: Removed LNI 192.168.203.127@tcp42 [ 8429.984128] LNet: Removed LNI 192.168.203.127@tcp43 [ 8431.008192] LNet: Removed LNI 192.168.203.127@tcp44 [ 8432.032155] LNet: Removed LNI 192.168.203.127@tcp45 [ 8433.056181] LNet: Removed LNI 192.168.203.127@tcp46 [ 8434.080126] LNet: Removed LNI 192.168.203.127@tcp47 [ 8434.081327] LNet: Removed LNI 192.168.203.127@tcp48 [ 8435.104121] LNet: Removed LNI 192.168.203.127@tcp49 [ 8435.111400] LNet: Removed LNI 192.168.203.127@tcp50 [ 8435.370668] Key type .llcrypt unregistered [ 8435.371825] Key type ._llcrypt unregistered [ 8442.783338] Key type ._llcrypt registered [ 8442.784387] Key type .llcrypt registered [ 8442.820428] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8443.206463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8443.239583] alg: No test for adler32 (adler32-zlib) [ 8444.092482] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 8444.174494] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8445.752157] Key type lgssc registered [ 8446.110058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8448.630183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8448.637978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.787224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.071417] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8453.836109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8453.950318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8455.114471] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8456.694302] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8457.750865] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8459.828200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8459.943157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8461.685258] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8463.420862] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8468.073434] loop0: detected capacity change from 0 to 204800000 [ 8468.144988] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8468.148607] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8468.154647] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8470.454640] loop0: detected capacity change from 0 to 204800000 [ 8470.573046] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8470.577423] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8470.582332] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8472.523441] loop0: detected capacity change from 0 to 204800000 [ 8474.434843] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8474.499563] Lustre: fs15246-OST0000: new disk, initializing [ 8474.502781] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8474.507184] Lustre: 473037:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8474.534250] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8476.738691] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8481.379558] loop1: detected capacity change from 0 to 204800000 [ 8483.805834] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8483.864603] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8483.903864] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8483.952283] Lustre: fs15246-MDT0000: new disk, initializing [ 8484.025627] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.039725] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8484.108846] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8485.153930] Lustre: 474141:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000fabb5eb6 x1856706064708736/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1770692944 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8485.485552] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8494.228606] LustreError: 474141:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8494.237156] LustreError: 474141:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8494.245698] LustreError: 474141:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8498.153513] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8498.163259] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.347617] LustreError: 475264:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8500.370850] Lustre: server umount fs15246-MDT0000 complete [ 8503.268457] LustreError: 469778:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8503.272976] LustreError: 469778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8503.546088] LustreError: 475765:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 8503.548369] LustreError: 475765:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8503.564981] Lustre: server umount fs15246-OST0000 complete [ 8507.360924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8507.363297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.367315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8508.389320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8508.390818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8513.073705] LustreError: 476266:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8513.077787] LustreError: 476266:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8513.126271] Lustre: server umount lustre-OST0000 complete [ 8515.552906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8515.556289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.560417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.562641] Lustre: Skipped 1 previous similar message [ 8520.714320] LustreError: 476468:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8520.717200] LustreError: 476468:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8520.742648] Lustre: server umount lustre-MDT0000 complete [ 8522.052342] LustreError: 474135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770692965 with bad export cookie 13200521233631234144 [ 8522.055818] LustreError: 474135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8522.056195] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.180166] Lustre: server umount lustre-MDT0001 complete [ 8525.839603] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8526.860065] Key type lgssc unregistered [ 8526.996372] LNet: 477199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8526.999456] LNetError: 477199:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8527.009331] LNet: Removed LNI 192.168.203.127@tcp [ 8527.306103] Key type .llcrypt unregistered [ 8527.307932] Key type ._llcrypt unregistered [ 8534.797185] Key type ._llcrypt registered [ 8534.798232] Key type .llcrypt registered [ 8534.838994] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8535.160879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8535.167415] alg: No test for adler32 (adler32-zlib) [ 8536.008973] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 8536.100496] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8537.680137] Key type lgssc registered [ 8538.034927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8551.416551] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8554.462554] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8566.464089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8566.469475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8567.566430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8567.577611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8567.624281] Lustre: lustre-MDT0000: new disk, initializing [ 8567.650377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8567.660821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8568.992578] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8573.689411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8573.736045] Lustre: 481581:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8573.772441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8573.775224] Lustre: Skipped 1 previous similar message [ 8573.821192] Lustre: lustre-MDT0001: new disk, initializing [ 8573.859881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8573.876433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8573.880284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8575.234911] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8578.489881] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8579.790076] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8582.652144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8582.819321] Lustre: lustre-OST0000: new disk, initializing [ 8582.823084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8582.828467] Lustre: 483321:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8582.881438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8583.949404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8583.956178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8584.017679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8585.408736] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8588.888726] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8590.354967] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8590.427343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8591.792228] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8591.874277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8594.405257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8594.405735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.406983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8594.409323] LustreError: Skipped 1 previous similar message [ 8594.413196] Lustre: Skipped 1 previous similar message [ 8598.639170] LustreError: 484511:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8598.680506] Lustre: server umount lustre-OST0000 complete [ 8604.643598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8604.643781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.644978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8604.644988] Lustre: Skipped 1 previous similar message [ 8604.661834] Lustre: Skipped 1 previous similar message [ 8609.761113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8609.762934] Lustre: Skipped 2 previous similar messages [ 8613.857600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8613.859844] Lustre: Skipped 1 previous similar message [ 8614.368089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8614.399391] LustreError: 484714:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8614.401129] LustreError: 484714:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8614.428243] Lustre: server umount lustre-MDT0000 complete [ 8615.666180] LustreError: 481571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770693058 with bad export cookie 1637355277676963352 [ 8615.670439] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8615.671131] LustreError: 481571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8615.739184] LustreError: 484916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8615.741441] LustreError: 484916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8615.793726] Lustre: server umount lustre-MDT0001 complete [ 8624.074742] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8627.963167] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8638.210825] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8351 sec ========== 22:11:21 (1770693081) [ 8638.753780] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:11:21 (1770693081) === [ 8639.872368] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:11:23 (1770693083) === [ 8653.550245] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8654.636962] Key type lgssc unregistered [ 8654.770328] LNet: 489754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8654.773449] LNetError: 489754:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8654.785450] LNet: Removed LNI 192.168.203.127@tcp [ 8655.136463] Key type .llcrypt unregistered [ 8655.138390] Key type ._llcrypt unregistered