[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473054609 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2866568K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003303] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008297] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.012149] LSM: Security Framework initializing [ 0.014024] Yama: becoming mindful. [ 0.015041] SELinux: Initializing. [ 0.016090] *** VALIDATE selinux *** [ 0.024555] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029240] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032120] *** VALIDATE tmpfs *** [ 0.033481] *** VALIDATE proc *** [ 0.035248] *** VALIDATE cgroup *** [ 0.036012] *** VALIDATE cgroup2 *** [ 0.037295] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041030] Spectre V2 : User space: Vulnerable [ 0.043006] Speculative Store Bypass: Vulnerable [ 0.046154] debug: unmapping init [mem 0xffffffffa1259000-0xffffffffa1260fff] [ 0.048195] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049763] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050029] ... version: 2 [ 0.051017] ... bit width: 48 [ 0.052014] ... generic registers: 4 [ 0.053027] ... value mask: 0000ffffffffffff [ 0.054018] ... max period: 00007fffffffffff [ 0.055017] ... fixed-purpose events: 3 [ 0.056023] ... event mask: 000000070000000f [ 0.057334] rcu: Hierarchical SRCU implementation. [ 0.059564] smp: Bringing up secondary CPUs ... [ 0.060591] x86: Booting SMP configuration: [ 0.061031] .... node #0, CPUs: #1 #2 #3 [ 0.065011] smp: Brought up 1 node, 4 CPUs [ 0.067015] smpboot: Max logical packages: 1 [ 0.068011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144029] node 0 deferred pages initialised in 73ms [ 0.147135] devtmpfs: initialized [ 0.148237] x86/mm: Memory block size: 128MB [ 0.150855] gcov: version magic: 0x41383552 [ 0.152197] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154294] pinctrl core: initialized pinctrl subsystem [ 0.155238] [ 0.155854] ************************************************************* [ 0.156017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157011] ** ** [ 0.158023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159020] ** ** [ 0.160016] ** This means that this kernel is built to expose internal ** [ 0.161018] ** IOMMU data structures, which may compromise security on ** [ 0.162014] ** your system. ** [ 0.163016] ** ** [ 0.164016] ** If you see this message and you are not debugging the ** [ 0.165017] ** kernel, report this immediately to your vendor! ** [ 0.166013] ** ** [ 0.167013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168013] ************************************************************* [ 0.169874] NET: Registered protocol family 16 [ 0.170441] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174043] cpuidle: using governor menu [ 0.176906] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180477] PCI: Using configuration type 1 for base access [ 0.182133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195080] cryptd: max_cpu_qlen set to 1000 [ 0.198354] ACPI: Added _OSI(Module Device) [ 0.200014] ACPI: Added _OSI(Processor Device) [ 0.200875] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205220] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210079] ACPI: Interpreter enabled [ 0.211065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213013] ACPI: Using IOAPIC for interrupt routing [ 0.215101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.230000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241448] acpiphp: Slot [2] registered [ 0.243182] acpiphp: Slot [5] registered [ 0.244149] acpiphp: Slot [6] registered [ 0.246120] acpiphp: Slot [7] registered [ 0.247108] acpiphp: Slot [8] registered [ 0.249116] acpiphp: Slot [9] registered [ 0.250132] acpiphp: Slot [10] registered [ 0.252143] acpiphp: Slot [3] registered [ 0.253101] acpiphp: Slot [4] registered [ 0.255134] acpiphp: Slot [11] registered [ 0.257185] acpiphp: Slot [12] registered [ 0.258134] acpiphp: Slot [13] registered [ 0.260163] acpiphp: Slot [14] registered [ 0.261100] acpiphp: Slot [15] registered [ 0.262161] acpiphp: Slot [16] registered [ 0.264116] acpiphp: Slot [17] registered [ 0.265140] acpiphp: Slot [18] registered [ 0.267085] acpiphp: Slot [19] registered [ 0.269122] acpiphp: Slot [20] registered [ 0.271117] acpiphp: Slot [21] registered [ 0.273120] acpiphp: Slot [22] registered [ 0.275135] acpiphp: Slot [23] registered [ 0.277114] acpiphp: Slot [24] registered [ 0.279142] acpiphp: Slot [25] registered [ 0.281117] acpiphp: Slot [26] registered [ 0.283117] acpiphp: Slot [27] registered [ 0.286128] acpiphp: Slot [28] registered [ 0.288186] acpiphp: Slot [29] registered [ 0.290124] acpiphp: Slot [30] registered [ 0.292148] acpiphp: Slot [31] registered [ 0.293063] PCI host bridge to bus 0000:00 [ 0.295027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.304021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.308168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309687] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312954] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319964] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.323061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.331011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.333475] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.337920] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344708] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.359644] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.369308] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.377020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.387017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.409019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422423] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.457021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.465455] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.473022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.479020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.515428] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.521015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.542020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.556040] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.566017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.577018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.615028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.627566] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.637018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.645018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.678022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.689954] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.692360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.694342] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.696432] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.700334] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.705072] iommu: Default domain type: Passthrough [ 0.707270] SCSI subsystem initialized [ 0.709155] ACPI: bus type USB registered [ 0.711102] usbcore: registered new interface driver usbfs [ 0.712168] usbcore: registered new interface driver hub [ 0.714089] usbcore: registered new device driver usb [ 0.716190] pps_core: LinuxPPS API ver. 1 registered [ 0.719011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.721064] PTP clock support registered [ 0.724097] EDAC MC: Ver: 3.0.0 [ 0.726102] PCI: Using ACPI for IRQ routing [ 0.727655] NetLabel: Initializing [ 0.728009] NetLabel: domain hash size = 128 [ 0.730009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.731074] NetLabel: unlabeled traffic allowed by default [ 0.733172] vgaarb: loaded [ 0.734223] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.736067] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741298] clocksource: Switched to clocksource kvm-clock [ 0.836719] VFS: Disk quotas dquot_6.6.0 [ 0.838308] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.840929] *** VALIDATE ramfs *** [ 0.842327] *** VALIDATE hugetlbfs *** [ 0.844045] pnp: PnP ACPI init [ 0.846570] pnp: PnP ACPI: found 6 devices [ 0.872050] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.875500] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.877758] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.879672] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.881719] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.883536] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.886098] NET: Registered protocol family 2 [ 0.888117] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.892031] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.895065] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.900760] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.903962] TCP: Hash tables configured (established 65536 bind 65536) [ 0.906505] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.909323] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.911991] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914723] NET: Registered protocol family 1 [ 0.918616] RPC: Registered named UNIX socket transport module. [ 0.921031] RPC: Registered udp transport module. [ 0.922835] RPC: Registered tcp transport module. [ 0.924593] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.927104] NET: Registered protocol family 44 [ 0.928793] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.931079] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.933442] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.935902] PCI: CLS 0 bytes, default 64 [ 0.939157] Unpacking initramfs... [ 2.319422] debug: unmapping init [mem 0xffff9dac7cc54000-0xffff9dac7ffbffff] [ 2.322048] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.323689] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.325438] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.787298] Initialise system trusted keyrings [ 2.788953] Key type blacklist registered [ 2.790733] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.799977] zbud: loaded [ 2.803738] *** VALIDATE nfs *** [ 2.804670] *** VALIDATE nfs4 *** [ 2.805751] pstore: using deflate compression [ 2.809407] Platform Keyring initialized [ 2.895551] NET: Registered protocol family 38 [ 2.897781] Key type asymmetric registered [ 2.899460] Asymmetric key parser 'x509' registered [ 2.901676] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.904325] io scheduler mq-deadline registered [ 2.905854] io scheduler kyber registered [ 2.907128] io scheduler bfq registered [ 2.908696] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.911018] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.913282] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.915450] ACPI: Power Button [PWRF] [ 2.919205] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.924630] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.933904] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.942138] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.956598] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.982196] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.010690] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.015840] Non-volatile memory driver v1.3 [ 3.016986] Linux agpgart interface v0.103 [ 3.042450] virtio_blk virtio1: [vda] 149392 512-byte logical blocks (76.5 MB/72.9 MiB) [ 3.045879] vda: detected capacity change from 0 to 76488704 [ 3.062155] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.065041] vdb: detected capacity change from 0 to 1073741824 [ 3.078664] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.081833] vdc: detected capacity change from 0 to 2621440000 [ 3.095684] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.098566] vdd: detected capacity change from 0 to 2621440000 [ 3.112778] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.115433] vde: detected capacity change from 0 to 4294967296 [ 3.127465] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.129762] vdf: detected capacity change from 0 to 4294967296 [ 3.137784] libphy: Fixed MDIO Bus: probed [ 3.142472] usbcore: registered new interface driver usbserial_generic [ 3.144396] usbserial: USB Serial support registered for generic [ 3.146211] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.149552] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.151090] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.152892] mousedev: PS/2 mouse device common for all mice [ 3.155609] rtc_cmos 00:05: RTC can wake from S4 [ 3.155833] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.159738] rtc_cmos 00:05: registered as rtc0 [ 3.162512] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.162573] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.165394] intel_pstate: CPU model not supported [ 3.171124] hid: raw HID events driver (C) Jiri Kosina [ 3.172404] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.173304] usbcore: registered new interface driver usbhid [ 3.177887] usbhid: USB HID core driver [ 3.179430] drop_monitor: Initializing network drop monitor service [ 3.182102] Initializing XFRM netlink socket [ 3.183676] NET: Registered protocol family 10 [ 3.186274] Segment Routing with IPv6 [ 3.187815] NET: Registered protocol family 17 [ 3.189830] mpls_gso: MPLS GSO support [ 3.193759] RAS: Correctable Errors collector initialized. [ 3.195288] AVX version of gcm_enc/dec engaged. [ 3.196488] AES CTR mode by8 optimization enabled [ 3.248574] sched_clock: Marking stable (3248556023, 0)->(4171948660, -923392637) [ 3.251698] registered taskstats version 1 [ 3.253435] Loading compiled-in X.509 certificates [ 3.256326] zswap: loaded using pool lzo/zbud [ 3.273107] Key type big_key registered [ 3.284378] Key type encrypted registered [ 3.286049] ima: No TPM chip found, activating TPM-bypass! [ 3.287841] ima: Allocated hash algorithm: sha1 [ 3.289395] ima: No architecture policies found [ 3.291225] evm: Initialising EVM extended attributes: [ 3.292966] evm: security.selinux [ 3.294062] evm: security.ima [ 3.295119] evm: security.capability [ 3.296401] evm: HMAC attrs: 0x1 [ 3.298232] rtc_cmos 00:05: setting system clock to 2026-08-10 22:29:08 UTC (1786400948) [ 3.303258] debug: unmapping init [mem 0xffffffffa2203000-0xffffffffa23fffff] [ 3.305973] debug: unmapping init [mem 0xffffffffa0f82000-0xffffffffa1258fff] [ 3.314048] Write protecting the kernel read-only data: 28672k [ 3.316740] debug: unmapping init [mem 0xffffffff9f603000-0xffffffff9f7fffff] [ 3.319319] debug: unmapping init [mem 0xffffffff9ff14000-0xffffffff9fffffff] [ 3.343167] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.351588] systemd[1]: Detected virtualization kvm. [ 3.353446] systemd[1]: Detected architecture x86-64. [ 3.355632] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.382828] systemd[1]: No hostname configured. [ 3.384590] systemd[1]: Set hostname to . [ 3.386637] random: systemd: uninitialized urandom read (16 bytes read) [ 3.389264] systemd[1]: Initializing machine ID from random generator. [ 3.431143] random: ln: uninitialized urandom read (6 bytes read) [ 3.536984] random: systemd: uninitialized urandom read (16 bytes read) [ 3.539467] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.544254] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.549568] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.241759] device-mapper: uevent: version 1.0.3 [ 4.243715] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.898909] random: fast init done [ 4.902821] virtio_net virtio0 ens2: renamed from eth0 [ 4.932765] scsi host0: ata_piix [ 5.001075] scsi host1: ata_piix [ 5.012294] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.014853] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.717112] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.802072] random: crng init done [ 9.803417] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.244229] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.426921] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.707162] SELinux: Disabled at runtime. [ 11.773402] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.780174] systemd[1]: Detected virtualization kvm. [ 11.781551] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.305712] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.308454] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.312970] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.318202] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.323303] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.332193] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.337672] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ 12.402405] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.827411] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.203500] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.239662] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.418168] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.432609] EDAC sbridge: Ver: 1.1.2 [ 15.219405] Key type dns_resolver registered [ 15.517971] NFS: Registering the id_resolver key type [ 15.519903] Key type id_resolver registered [ 15.521248] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg152-server login: [ 45.383129] hrtimer: interrupt took 3137133 ns [ 47.307892] libcfs: loading out-of-tree module taints kernel. [ 47.395597] Key type ._llcrypt registered [ 47.398272] Key type .llcrypt registered [ 47.538236] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 70.226227] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 72.424722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.452649] alg: No test for adler32 (adler32-zlib) [ 74.014736] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 75.039544] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 76.855502] Key type lgssc registered [ 78.811916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.528924] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.742320] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 163.936640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.959849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.232704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.288851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.396706] Lustre: lustre-MDT0000: new disk, initializing [ 165.474615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.487428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 170.542119] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.175716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.281169] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.305782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.309785] Lustre: Skipped 1 previous similar message [ 184.388255] Lustre: lustre-MDT0001: new disk, initializing [ 184.452783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.502198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.513752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 188.648492] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.788849] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 205.351861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.005752] Lustre: lustre-OST0000: new disk, initializing [ 206.015745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.035898] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.189131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.012580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.026892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.138966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 214.534851] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.736937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.931945] Lustre: lustre-OST0001: new disk, initializing [ 229.935421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.944131] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.023347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.088975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.107079] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.166874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.918680] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.062449] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 259.556813] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.820355] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing check_logdir /tmp/testlogs/ [ 272.340752] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing yml_node [ 277.123429] Lustre: DEBUG MARKER: Client: 2.17.54.108 [ 279.597865] Lustre: DEBUG MARKER: MDS: 2.17.54.108 [ 281.994042] Lustre: DEBUG MARKER: OSS: 2.17.54.108 [ 283.513693] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 18:33:46 EDT 2026 [ 301.422343] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 303.089908] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 321.505733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.510772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.522034] Lustre: Skipped 3 previous similar messages [ 321.532524] Lustre: Skipped 3 previous similar messages [ 326.626237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.628945] Lustre: Skipped 3 previous similar messages [ 327.099523] Lustre: server umount lustre-MDT0000 complete [ 331.558955] LustreError: 7458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401276 with bad export cookie 13819043840883175237 [ 331.560659] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.579993] LustreError: 7458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 331.747054] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.758254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.864358] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.864983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.896111] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 336.905231] Lustre: Skipped 2 previous similar messages [ 338.170494] Lustre: server umount lustre-MDT0001 complete [ 342.509941] Lustre: server umount lustre-OST0000 complete [ 346.723401] Lustre: server umount lustre-OST0001 complete [ 354.069259] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 365.174488] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 408.421854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.781419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 408.823398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 408.908913] Lustre: lustre-MDT0000: new disk, initializing [ 408.990450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.016560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 414.262992] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 426.113931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.297196] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 426.413134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 426.423418] Lustre: Skipped 1 previous similar message [ 426.534946] Lustre: lustre-MDT0001: new disk, initializing [ 426.691669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.744944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.762124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 431.286815] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 440.650730] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 447.427027] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 454.856350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.413662] Lustre: lustre-OST0000: new disk, initializing [ 455.418505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 455.430386] Lustre: 17043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.542464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.776765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 456.793613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 456.948214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 463.221425] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 472.721829] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.846616] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 478.145270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 483.829906] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 484.103256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 487.904873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.906314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.914906] LustreError: Skipped 1 previous similar message [ 487.920056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.941176] Lustre: Skipped 2 previous similar messages [ 487.965246] Lustre: Skipped 1 previous similar message [ 492.418201] Lustre: server umount lustre-OST0000 complete [ 498.160845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.184319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.186368] Lustre: Skipped 1 previous similar message [ 498.207781] Lustre: Skipped 1 previous similar message [ 504.415208] Lustre: server umount lustre-MDT0000 complete [ 508.405865] LustreError: 15084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.446386] LustreError: 15084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 509.633523] LustreError: 15879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401454 with bad export cookie 13819043840883176763 [ 509.645882] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.650117] LustreError: 15879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 510.278956] Lustre: server umount lustre-MDT0001 complete [ 515.588397] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:37:37 (1786401457) [ 518.803651] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 520.720352] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:37:43 (1786401463) [ 528.378201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.852684] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.966390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.991898] LustreError: 19454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.010320] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 539.111373] LustreError: 19453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.562687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.648977] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 562.547641] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 568.174329] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.056073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 585.429086] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 594.844195] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 609.762034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 609.770064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.799038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.798741] Lustre: Skipped 1 previous similar message [ 609.808336] Lustre: Skipped 2 previous similar messages [ 615.786961] Lustre: server umount lustre-OST0000 complete [ 623.100978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.134674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.729435] Lustre: server umount lustre-MDT0000 complete [ 630.242987] LustreError: 19454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.272116] LustreError: 19454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 631.735630] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401576 with bad export cookie 13819043840883177645 [ 631.740937] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.748047] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 632.223497] Lustre: server umount lustre-MDT0001 complete [ 645.125244] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 648.755557] Key type lgssc unregistered [ 649.083694] LNet: 23009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.092707] LNetError: 23009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.105338] LNet: Removed LNI 192.168.201.152@tcp [ 650.049184] Key type .llcrypt unregistered [ 650.053304] Key type ._llcrypt unregistered [ 678.883534] Key type ._llcrypt registered [ 678.886244] Key type .llcrypt registered [ 679.019748] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 680.301908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 680.606648] alg: No test for adler32 (adler32-zlib) [ 681.608961] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 681.810574] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 683.447248] Key type lgssc registered [ 684.477519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 695.199248] loop: module loaded [ 695.335461] loop0: detected capacity change from 0 to 204800000 [ 695.634718] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.651741] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.010164] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.791322] loop0: detected capacity change from 0 to 204800000 [ 704.073448] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 704.091305] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 704.120321] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.573406] loop0: detected capacity change from 0 to 204800000 [ 715.906525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 715.930498] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.281950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 717.357897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 717.429401] Lustre: lustre-MDT0000: new disk, initializing [ 717.496317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.519814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 722.799603] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 733.709447] loop1: detected capacity change from 0 to 204800000 [ 740.242290] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 740.569447] Lustre: lustre-OST07c6: new disk, initializing [ 740.574478] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 740.580534] Lustre: Skipped 1 previous similar message [ 740.588812] Lustre: 26204:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 740.678962] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 747.734304] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 748.231272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 748.241242] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 748.285524] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 753.495537] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 753.501830] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 753.513518] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 753.529341] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 753.537253] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 753.544835] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 768.155361] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 768.164298] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 768.170459] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.178220] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 768.184242] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 768.193561] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.753768] Lustre: 26212:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 772.769076] Lustre: 26212:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 772.785122] Lustre: 26212:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 772.802943] Lustre: 26212:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.815831] Lustre: 26212:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.823758] Lustre: 26212:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.833604] Lustre: 26212:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 772.851806] Lustre: 26212:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.870529] Lustre: 26212:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.881670] Lustre: 26212:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.886735] Lustre: 26212:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.895074] Lustre: 26212:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.190443] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 776.203565] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.213855] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 776.222856] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.241911] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.244593] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.262252] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 776.269507] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.276185] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.282569] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.288392] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.292996] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.072068] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 781.097410] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 781.110092] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 781.120021] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.134189] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.153519] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.163882] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 781.184891] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.216821] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 781.234439] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.254025] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 781.264964] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.491675] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 792.495239] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 792.502706] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 792.508901] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.515833] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 792.523094] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.529591] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 792.539744] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.544705] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 792.550436] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.556959] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 792.571383] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 840.161193] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.179563] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 845.403977] Lustre: server umount lustre-OST07c6 complete [ 855.687589] Lustre: server umount lustre-MDT0000 complete [ 877.850719] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 883.827980] Key type lgssc unregistered [ 884.124704] LNet: 28759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.144964] LNetError: 28759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.205380] LNet: Removed LNI 192.168.201.152@tcp [ 885.894955] Key type .llcrypt unregistered [ 885.912522] Key type ._llcrypt unregistered [ 907.908695] Key type ._llcrypt registered [ 907.911589] Key type .llcrypt registered [ 907.991843] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 909.546206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.580520] alg: No test for adler32 (adler32-zlib) [ 910.835648] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 911.224747] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 912.927699] Key type lgssc registered [ 914.363850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.942979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.967330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.547316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.308869] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 933.889061] LustreError: 30034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.910345] LustreError: 30034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 938.295251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.673376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 944.269549] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 952.266486] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 958.092350] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 965.139027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 965.473520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.608073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 971.709236] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 979.414835] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 998.368402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 998.377653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.401971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 998.420945] Lustre: Skipped 1 previous similar message [ 998.446160] Lustre: Skipped 1 previous similar message [ 999.619877] Lustre: server umount lustre-OST0000 complete [ 1005.024559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1005.041389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.063479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.617448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.624822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.728858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1013.743996] Lustre: Skipped 2 previous similar messages [ 1017.824691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.840428] Lustre: Skipped 1 previous similar message [ 1018.335455] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1018.821987] Lustre: server umount lustre-MDT0000 complete [ 1022.837262] LustreError: 30016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786401968 with bad export cookie 9725983186567617258 [ 1022.839506] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1022.844345] LustreError: 30016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1023.291573] Lustre: server umount lustre-MDT0001 complete [ 1034.299548] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1037.013415] Key type lgssc unregistered [ 1037.247891] LNet: 33770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.256247] LNetError: 33770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.278309] LNet: Removed LNI 192.168.201.152@tcp [ 1038.158591] Key type .llcrypt unregistered [ 1038.161174] Key type ._llcrypt unregistered [ 1061.148591] Key type ._llcrypt registered [ 1061.151588] Key type .llcrypt registered [ 1061.307343] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1063.118825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1063.155292] alg: No test for adler32 (adler32-zlib) [ 1064.209287] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 1064.431332] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1066.087632] Key type lgssc registered [ 1067.233591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1074.980874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1075.011342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.500924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.930153] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1085.923660] LustreError: 35041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.933937] LustreError: 35041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1089.695533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.910572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1093.339881] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1100.151223] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1106.393686] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1114.114984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.632577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.818920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1121.807864] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1130.196283] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1141.224548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.232650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.235606] LustreError: Skipped 1 previous similar message [ 1141.236301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.252749] Lustre: Skipped 1 previous similar message [ 1146.250181] Lustre: server umount lustre-OST0000 complete [ 1151.457657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.467657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1151.468089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.475690] Lustre: Skipped 1 previous similar message [ 1151.479033] Lustre: Skipped 1 previous similar message [ 1156.583042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.589040] Lustre: Skipped 2 previous similar messages [ 1156.861787] Lustre: server umount lustre-MDT0000 complete [ 1161.148887] LustreError: 35023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402106 with bad export cookie 13599924272313262585 [ 1161.159039] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.178063] LustreError: 35023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1161.541186] Lustre: server umount lustre-MDT0001 complete [ 1174.843440] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1177.756447] Key type lgssc unregistered [ 1177.979464] LNet: 38576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.987171] LNetError: 38576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.050694] LNet: Removed LNI 192.168.201.152@tcp [ 1180.109347] Key type .llcrypt unregistered [ 1180.112150] Key type ._llcrypt unregistered [ 1204.912465] Key type ._llcrypt registered [ 1204.914670] Key type .llcrypt registered [ 1205.028452] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1206.563438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.609546] alg: No test for adler32 (adler32-zlib) [ 1207.842397] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 1208.085119] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1209.759164] Key type lgssc registered [ 1210.982993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1220.921411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1220.932883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.532675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.866227] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1231.855793] LustreError: 39993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.864177] LustreError: 39993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1236.966807] LustreError: 39994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.398530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.772691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1244.131710] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1251.875680] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.684802] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1264.805161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1265.296101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.441373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1272.690085] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1280.441147] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1293.646525] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1294.321821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.323929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.334208] Lustre: Skipped 2 previous similar messages [ 1294.343202] Lustre: Skipped 1 previous similar message [ 1296.707795] Lustre: server umount lustre-MDT0000 complete [ 1298.726903] LustreError: 39990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.967154] LustreError: 39974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402246 with bad export cookie 16977453390204758929 [ 1300.997687] LustreError: 39974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1301.002655] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.846533] LustreError: 39988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.867796] LustreError: 39988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1303.871595] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 1303.887050] Lustre: Skipped 1 previous similar message [ 1304.545991] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.970664] LustreError: 39989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.970751] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 1308.988961] LustreError: 39989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1309.028809] Lustre: Skipped 2 previous similar messages [ 1314.091639] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 1314.109580] Lustre: Skipped 2 previous similar messages [ 1315.295476] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1315.630570] Lustre: server umount lustre-MDT0001 complete [ 1331.370476] Lustre: server umount lustre-OST0000 complete [ 1351.748523] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1354.285666] Key type lgssc unregistered [ 1354.538544] LNet: 43856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.544996] LNetError: 43856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.562913] LNet: Removed LNI 192.168.201.152@tcp [ 1355.449217] Key type .llcrypt unregistered [ 1355.453926] Key type ._llcrypt unregistered [ 1379.791534] Key type ._llcrypt registered [ 1379.794622] Key type .llcrypt registered [ 1379.922186] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1381.206562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.384797] alg: No test for adler32 (adler32-zlib) [ 1382.423634] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 1382.732587] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1384.399159] Key type lgssc registered [ 1385.753816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1394.186772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1394.220610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.880126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.982329] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1406.441368] LustreError: 45273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.466870] LustreError: 45273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1410.683996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.046469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1416.334469] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1424.960982] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1430.784138] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1437.361341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1437.854251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.965588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1444.472351] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1454.339881] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1466.694875] Lustre: lustre-OST0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1466.849668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.853710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.864605] LustreError: Skipped 1 previous similar message [ 1466.919649] Lustre: Skipped 1 previous similar message [ 1470.013602] Lustre: server umount lustre-OST0000 complete [ 1492.449739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.458761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.464822] Lustre: Skipped 1 previous similar message [ 1492.477926] Lustre: Skipped 3 previous similar messages [ 1494.858924] Lustre: server umount lustre-MDT0000 complete [ 1497.568234] LustreError: 45273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.598312] LustreError: 45273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1499.605195] LustreError: 45254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402444 with bad export cookie 4949109024210290131 [ 1499.615095] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1506.133961] Lustre: server umount lustre-MDT0001 complete [ 1519.686664] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1523.350627] Key type lgssc unregistered [ 1523.651967] LNet: 48981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.657982] LNetError: 48981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.672425] LNet: Removed LNI 192.168.201.152@tcp [ 1524.567921] Key type .llcrypt unregistered [ 1524.570532] Key type ._llcrypt unregistered [ 1554.171357] Key type ._llcrypt registered [ 1554.174609] Key type .llcrypt registered [ 1554.341645] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1555.996943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1556.118331] alg: No test for adler32 (adler32-zlib) [ 1557.389953] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 1557.753365] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1559.488227] Key type lgssc registered [ 1560.880580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.716171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1570.738265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.404917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.767426] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1582.564149] LustreError: 50398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.584856] LustreError: 50398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1586.341236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.648061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1591.145797] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1599.099517] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1605.812043] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1613.248044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1613.759070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.818834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1621.313239] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1629.638670] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1636.348367] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1641.212621] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1653.549738] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1018h56m35s [ 1655.370939] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1658.164576] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1658.335941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1658.344297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.240608] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1659.249851] Lustre: Skipped 3 previous similar messages [ 1663.937612] Lustre: server umount lustre-MDT0000 complete [ 1664.303700] LustreError: 50394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.510431] LustreError: 50398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.529284] LustreError: 50398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1666.710438] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1670.467110] LustreError: 50393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1672.508402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.633383] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.853136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1672.921559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1673.908570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1678.001757] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1681.836927] Lustre: DEBUG MARKER: Wait for df (36582) ... [ 1683.372581] Lustre: DEBUG MARKER: done [ 1687.523035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.531456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.543945] Lustre: Skipped 2 previous similar messages [ 1687.546109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.552896] Lustre: Skipped 3 previous similar messages [ 1688.556350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.569883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.573584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1689.584831] Lustre: Skipped 2 previous similar messages [ 1694.704549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.715266] Lustre: Skipped 1 previous similar message [ 1701.343469] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1701.579347] Lustre: server umount lustre-OST0000 complete [ 1710.053280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1710.067138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.070681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1710.070690] Lustre: Skipped 2 previous similar messages [ 1710.116765] Lustre: Skipped 1 previous similar message [ 1711.705068] Lustre: server umount lustre-MDT0000 complete [ 1716.384855] LustreError: 50380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402661 with bad export cookie 6563324088631116403 [ 1716.386095] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.410579] LustreError: 50380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1716.902926] Lustre: server umount lustre-MDT0001 complete [ 1728.734322] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1731.504503] Key type lgssc unregistered [ 1731.730568] LNet: 55327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.737958] LNetError: 55327:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.750691] LNet: Removed LNI 192.168.201.152@tcp [ 1732.634229] Key type .llcrypt unregistered [ 1732.636598] Key type ._llcrypt unregistered [ 1778.676806] Key type ._llcrypt registered [ 1778.678336] Key type .llcrypt registered [ 1778.831718] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1780.406191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1780.417023] alg: No test for adler32 (adler32-zlib) [ 1781.431656] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 1781.823507] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1783.519280] Key type lgssc registered [ 1785.356943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.343305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.372923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.730529] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1795.767553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1795.990380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.689990] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1811.558307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.665627] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.699104] Lustre: 57536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1811.799786] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1811.967275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1817.120131] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1825.651783] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1831.961871] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1838.963924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1839.130189] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1839.313063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.084980] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1849.082346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1855.347338] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1862.762937] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1019h00m05s [ 1864.731648] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1873.768509] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1877.054418] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1877.134178] LustreError: 57543:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1897.330698] Lustre: lustre-MDT0000: Client 8f3d8f5a-7d39-4d63-87be-55ac13be3add (at 192.168.201.52@tcp) reconnecting [ 1917.208363] LustreError: 57543:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1920.620987] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1924.158829] Lustre: DEBUG MARKER: done [ 1936.362163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1936.370262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.371579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1936.406220] LustreError: Skipped 1 previous similar message [ 1938.531664] Lustre: server umount lustre-OST0000 complete [ 1945.056270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.062659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.079987] Lustre: Skipped 1 previous similar message [ 1945.086799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.094864] Lustre: Skipped 1 previous similar message [ 1950.080455] Lustre: server umount lustre-MDT0000 complete [ 1954.997368] LustreError: 57528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786402900 with bad export cookie 4062094983557905061 [ 1955.002297] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.010604] LustreError: 57528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1955.744425] Lustre: server umount lustre-MDT0001 complete [ 1969.377925] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1972.501471] Key type lgssc unregistered [ 1972.834197] LNet: 61883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.841165] LNetError: 61883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.852520] LNet: Removed LNI 192.168.201.152@tcp [ 1974.097707] Key type .llcrypt unregistered [ 1974.101361] Key type ._llcrypt unregistered [ 2035.024192] Key type ._llcrypt registered [ 2035.029891] Key type .llcrypt registered [ 2035.210592] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2037.075378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2037.128141] alg: No test for adler32 (adler32-zlib) [ 2038.315717] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 2038.634507] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2040.384779] Key type lgssc registered [ 2042.295725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2047.149648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2047.192763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.598931] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2048.634132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2048.860810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.177368] Lustre: Failing over lustre-MDT0000 [ 2050.708583] Lustre: server umount lustre-MDT0000 complete [ 2057.673691] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:03:20 (1786403000) [ 2066.211925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.887457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.393692] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2084.659627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.762237] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2084.804459] Lustre: 64685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2085.037526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.094623] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2099.826046] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2107.074620] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2115.645601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2115.861594] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2116.164609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.391755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2125.340662] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2134.191943] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2140.445501] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2147.301600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.304685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2147.331776] Lustre: Skipped 2 previous similar messages [ 2147.337710] Lustre: Skipped 1 previous similar message [ 2152.418597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2152.432206] Lustre: Skipped 3 previous similar messages [ 2153.401487] Lustre: server umount lustre-MDT0000 complete [ 2157.609534] LustreError: 65432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403102 with bad export cookie 15422790224021374240 [ 2157.617736] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2157.619344] LustreError: 65432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2157.953763] Lustre: server umount lustre-MDT0001 complete [ 2162.216770] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2169.649962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2178.020550] Lustre: 63607:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786403107/real 1786403107] req@000000009c53eb08 x1873179294345600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786403123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2178.072399] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.073525] Lustre: 63608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786403112/real 1786403112] req@00000000f8a5901c x1873179294345856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786403128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2184.279405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.280292] Lustre: 63608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786403118/real 1786403118] req@00000000a795b36e x1873179294345984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786403134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2194.399631] Lustre: 63606:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786403123/real 1786403123] req@0000000098c24d1f x1873179294346112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786403139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2202.663446] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd608c1fecc294e0a [ 2202.685725] Lustre: MGC192.168.201.152@tcp: Connection restored to 0@lo (at 0@lo) [ 2203.391176] LustreError: 66536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.416148] LustreError: 66536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2203.563552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.754771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2204.581842] LustreError: 68461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.606716] LustreError: 68461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2209.773550] LustreError: 68460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.804270] LustreError: 68460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2210.479790] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2211.841245] LustreError: 68780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.864666] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2211.878245] LustreError: 68780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2216.939221] LustreError: 68459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.964272] LustreError: 68459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2221.257699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.704354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.565463] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2227.174372] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2235.056343] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2242.347445] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2252.257940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2252.292886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.340952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2257.243984] Lustre: server umount lustre-MDT0000 complete [ 2257.890847] LustreError: 68459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.908097] LustreError: 68459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2262.333117] LustreError: 64676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403207 with bad export cookie 15422790224021376522 [ 2262.339853] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.349173] LustreError: 64676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2262.868062] Lustre: server umount lustre-MDT0001 complete [ 2269.903277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2279.393446] Lustre: 63608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786403208/real 1786403208] req@00000000dd46a817 x1873179294374912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786403224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2279.443569] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.463750] Lustre: Skipped 2 previous similar messages [ 2280.207193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.136474] LustreError: 69196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.252506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.351132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2289.262334] Lustre: 63608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786403217/real 1786403217] req@000000002791e540 x1873179294375424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786403233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2289.300542] Lustre: 63608:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2293.549173] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2304.104476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.522578] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2310.046909] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2318.121649] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2324.883873] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2334.179171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.180333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.203366] Lustre: Skipped 2 previous similar messages [ 2334.228919] Lustre: Skipped 4 previous similar messages [ 2339.298665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2339.314925] Lustre: Skipped 2 previous similar messages [ 2339.771970] Lustre: server umount lustre-MDT0000 complete [ 2344.423746] LustreError: 71142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.458639] LustreError: 71142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2344.481460] LustreError: 64678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403289 with bad export cookie 15422790224021377901 [ 2344.508079] LustreError: 64678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2344.510160] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2345.084948] Lustre: server umount lustre-MDT0001 complete [ 2353.072816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2357.688650] Lustre: DEBUG MARKER: files compared the same [ 2370.552389] Lustre: server umount lustre-OST0000 complete [ 2397.346592] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2401.126599] Key type lgssc unregistered [ 2401.545336] LNet: 74446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.560504] LNetError: 74446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2401.577805] LNet: Removed LNI 192.168.201.152@tcp [ 2402.629948] Key type .llcrypt unregistered [ 2402.633687] Key type ._llcrypt unregistered [ 2432.791824] Key type ._llcrypt registered [ 2432.795264] Key type .llcrypt registered [ 2432.901068] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2434.770450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2434.818467] alg: No test for adler32 (adler32-zlib) [ 2436.229605] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 2436.630566] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2438.407300] Key type lgssc registered [ 2440.118481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2449.566742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2449.582583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.213064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.317386] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2461.664055] LustreError: 75864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.697402] LustreError: 75864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2466.786347] LustreError: 75863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.025421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.565118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2473.852459] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2482.373479] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2489.148318] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2498.982737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.406664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.296604] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2519.332871] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2520.881618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2534.368557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.375465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.398976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.419430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2536.429497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.430624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.460473] Lustre: Skipped 1 previous similar message [ 2540.727567] Lustre: server umount lustre-OST0000 complete [ 2545.632090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2545.642720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.661190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.782031] LustreError: 75864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.792334] Lustre: server umount lustre-MDT0000 complete [ 2556.899048] LustreError: 77876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.921403] LustreError: 77876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2558.589831] LustreError: 75844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403503 with bad export cookie 10768925341912341881 [ 2558.617097] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.144974] Lustre: server umount lustre-MDT0001 complete [ 2572.379786] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2576.239853] Key type lgssc unregistered [ 2576.573861] LNet: 79417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.582953] LNetError: 79417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2577.642073] LNet: Removed LNI 192.168.201.152@tcp [ 2579.113734] Key type .llcrypt unregistered [ 2579.117470] Key type ._llcrypt unregistered [ 2612.866720] Key type ._llcrypt registered [ 2612.872849] Key type .llcrypt registered [ 2613.095977] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2615.747978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.111258] alg: No test for adler32 (adler32-zlib) [ 2617.453500] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 2618.020286] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2619.767617] Key type lgssc registered [ 2621.077793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2630.894683] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2630.934511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2646.623537] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2651.744268] LustreError: 80811:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 2678.431444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.820711] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2693.243648] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2702.073260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.177744] Lustre: *** cfs_fail_loc=706, val=0*** [ 2702.644373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.731344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2707.893692] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2707.936914] LustreError: 80836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.953301] LustreError: 80836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2713.064319] LustreError: 80837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.089776] LustreError: 80837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2717.680168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.288510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2723.619662] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2732.216426] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2738.085564] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2743.776398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2743.777633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.792525] LustreError: Skipped 1 previous similar message [ 2743.803100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2743.807831] Lustre: Skipped 1 previous similar message [ 2743.826210] Lustre: Skipped 1 previous similar message [ 2746.640094] Lustre: server umount lustre-OST0000 complete [ 2754.016852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2754.023087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.043497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.070522] Lustre: Skipped 1 previous similar message [ 2754.089338] Lustre: Skipped 1 previous similar message [ 2757.398636] Lustre: server umount lustre-MDT0000 complete [ 2759.136829] LustreError: 81897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.171667] LustreError: 81897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2761.321540] LustreError: 80819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786403706 with bad export cookie 15423240905608207137 [ 2761.331721] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2761.335941] LustreError: 80819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2761.724404] Lustre: server umount lustre-MDT0001 complete [ 2773.943395] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2776.599233] Key type lgssc unregistered [ 2776.915028] LNet: 84428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2776.921176] LNetError: 84428:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2777.961811] LNet: Removed LNI 192.168.201.152@tcp [ 2778.850444] Key type .llcrypt unregistered [ 2778.853624] Key type ._llcrypt unregistered [ 2809.167496] Key type ._llcrypt registered [ 2809.169047] Key type .llcrypt registered [ 2809.302764] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2811.237870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2811.255242] alg: No test for adler32 (adler32-zlib) [ 2812.241391] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 2812.448614] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2814.119362] Key type lgssc registered [ 2815.245603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2826.259673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2826.311629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.231755] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2842.124802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.908973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2848.225327] LustreError: 86549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.256961] LustreError: 86549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2848.832748] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2853.348627] LustreError: 86547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.382065] LustreError: 86547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2858.487940] LustreError: 86549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.521410] LustreError: 86549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2858.732539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.269851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.635363] LustreError: 86553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.664033] LustreError: 86553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2867.123206] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2868.778178] LustreError: 86549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.822558] LustreError: 86549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2878.961975] LustreError: 86552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.997318] LustreError: 86552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2879.494632] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2879.513053] Lustre: MGS: Client 1f21471f-5629-4c36-ad3d-b3eddb4e6519 (at 0@lo) reconnecting [ 2879.542623] LustreError: 86563:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2879.791336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.906958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2885.511657] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2905.568654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.577423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.597267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2905.607258] Lustre: Skipped 1 previous similar message [ 2908.188688] Lustre: server umount lustre-OST0000 complete [ 2915.808272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2915.809928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.812815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2915.858785] Lustre: Skipped 2 previous similar messages [ 2919.238900] Lustre: server umount lustre-MDT0000 complete [ 2920.928710] LustreError: 88093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.945629] LustreError: 88093:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2924.063482] Lustre: server umount lustre-MDT0001 complete [ 2928.616658] Lustre: server umount MGS complete [ 2941.283952] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:18:03 (1786403883) [ 2978.553973] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 2988.563896] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3031.056368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.761568] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3046.620264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.984738] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3047.046053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3047.193516] Lustre: lustre-MDT0001: new disk, initializing [ 3047.359224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.403617] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3047.414387] Lustre: 93744:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3047.423933] LustreError: 93744:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3052.854804] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3065.646300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.889718] Lustre: lustre-OST0000: new disk, initializing [ 3065.893367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3065.898329] Lustre: 94510:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3065.962264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.926420] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3074.723061] Lustre: 94890:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001602143b x1873180105944064/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786404035 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3083.192346] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3090.742379] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.753312] Lustre: MGS: Client fb807abb-3e07-43b0-a719-74b584aa6911 (at 0@lo) reconnecting [ 3090.810383] Lustre: 93013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3090.823978] LustreError: 93730:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3090.862704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3090.944519] Lustre: lustre-MDT0000: new disk, initializing [ 3091.123053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.151309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3091.181936] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3092.257687] Lustre: 95579:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000092c36198 x1873180105951232/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786404053 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3096.902660] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3097.505532] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3101.381781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3101.410951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3101.611572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3124.201622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.225928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3124.234906] Lustre: Skipped 2 previous similar messages [ 3129.353514] Lustre: server umount lustre-OST0000 complete [ 3134.434645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.446186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.450045] Lustre: Skipped 2 previous similar messages [ 3134.469216] Lustre: Skipped 1 previous similar message [ 3139.554288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.572645] Lustre: Skipped 1 previous similar message [ 3148.255648] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3148.376954] Lustre: server umount lustre-MDT0000 complete [ 3149.816362] LustreError: 93714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.862817] LustreError: 93714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3159.465627] Lustre: server umount lustre-MDT0001 complete [ 3163.643958] Lustre: server umount MGS complete [ 3177.557488] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:22:00 (1786404120) [ 3199.256966] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3202.819928] Key type lgssc unregistered [ 3203.361910] LNet: 98078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3203.369037] LNetError: 98078:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3203.382378] LNet: Removed LNI 192.168.201.152@tcp [ 3204.478297] Key type .llcrypt unregistered [ 3204.480665] Key type ._llcrypt unregistered [ 3225.732529] Key type ._llcrypt registered [ 3225.734760] Key type .llcrypt registered [ 3225.848362] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3227.426265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3227.445681] alg: No test for adler32 (adler32-zlib) [ 3228.588402] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 3228.846537] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3230.536168] Key type lgssc registered [ 3231.984335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3241.934235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3241.952379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.663039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.659814] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3253.229624] LustreError: 99316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3253.271515] LustreError: 99316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3258.340488] LustreError: 99317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.119952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.731558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3265.348436] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3274.035782] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3280.070531] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3287.265219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3287.728372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3288.782444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3294.441779] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3302.966116] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3314.679269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3314.684537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.692611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.693402] LustreError: Skipped 1 previous similar message [ 3314.757469] Lustre: Skipped 1 previous similar message [ 3318.391825] Lustre: server umount lustre-OST0000 complete [ 3324.897543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.906837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.938042] Lustre: Skipped 1 previous similar message [ 3324.968168] Lustre: Skipped 2 previous similar messages [ 3330.034533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3330.039388] Lustre: Skipped 1 previous similar message [ 3334.181075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.194015] Lustre: Skipped 1 previous similar message [ 3338.207135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3338.419249] Lustre: server umount lustre-MDT0000 complete [ 3339.236771] LustreError: 99311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.265931] LustreError: 99311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3341.979309] LustreError: 99297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404287 with bad export cookie 289069483517143427 [ 3341.984772] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.996513] LustreError: 99297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3342.770420] Lustre: server umount lustre-MDT0001 complete [ 3354.553757] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3359.018740] Key type lgssc unregistered [ 3359.616869] LNet: 102853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3359.639963] LNetError: 102853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3359.662745] LNet: Removed LNI 192.168.201.152@tcp [ 3360.856994] Key type .llcrypt unregistered [ 3360.862408] Key type ._llcrypt unregistered [ 3380.038279] Key type ._llcrypt registered [ 3380.041742] Key type .llcrypt registered [ 3380.178326] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3382.162796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3382.222224] alg: No test for adler32 (adler32-zlib) [ 3383.377455] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 3383.713870] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3385.399396] Key type lgssc registered [ 3387.077442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3394.068523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3394.076534] LustreError: 103934:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3394.102533] LustreError: 103938:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3394.108868] LustreError: 103934:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 3394.116413] LustreError: 103934:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3394.125747] LustreError: 103934:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3394.134254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.939513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.405152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.716557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3411.032609] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3416.522888] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing lsmod [ 3419.512510] LustreError: 105139:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3419.619935] LustreError: 105146:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3419.630062] LustreError: 105139:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4893 [ 3419.645182] LustreError: 105139:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3419.661608] LustreError: 105139:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3419.676311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.383958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.442215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3426.273388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3426.285176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.299184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.786138] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.800038] Lustre: Skipped 1 previous similar message [ 3426.805780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.810089] Lustre: Skipped 1 previous similar message [ 3429.924354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.935541] Lustre: Skipped 2 previous similar messages [ 3430.512698] Lustre: server umount lustre-MDT0000 complete [ 3434.981030] LustreError: 104408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.003757] LustreError: 104408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3435.174470] LustreError: 103951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404380 with bad export cookie 13303075187377647502 [ 3435.181715] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.190712] LustreError: 103951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3436.128358] LustreError: 104407:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002f565a18 x1873180704659200/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3436.407941] Lustre: server umount lustre-MDT0001 complete [ 3450.340598] Lustre: server umount lustre-OST0000 complete [ 3460.805393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.568564] LustreError: 106210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.658433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3466.722140] LustreError: 106211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.390285] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3470.819423] LustreError: 106210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.940286] LustreError: 106211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3478.417732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.906418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3484.665364] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3493.884707] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3500.399764] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3509.601485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3510.289541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.886446] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3527.920580] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3531.837890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3541.984585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3542.004343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.023302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.034263] Lustre: Skipped 1 previous similar message [ 3542.057644] Lustre: Skipped 1 previous similar message [ 3547.106438] LustreError: 108031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3547.434963] Lustre: server umount lustre-OST0000 complete [ 3555.810867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3555.827230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.852259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.920107] Lustre: server umount lustre-MDT0000 complete [ 3563.392598] LustreError: 106190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404508 with bad export cookie 13303075187377648426 [ 3563.400882] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.411440] LustreError: 106190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3564.105654] Lustre: server umount lustre-MDT0001 complete [ 3577.833870] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3581.456225] Key type lgssc unregistered [ 3581.888794] LNet: 109768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3581.906830] LNetError: 109768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3581.915953] LNet: Removed LNI 192.168.201.152@tcp [ 3582.882722] Key type .llcrypt unregistered [ 3582.885688] Key type ._llcrypt unregistered [ 3607.541874] Key type ._llcrypt registered [ 3607.545070] Key type .llcrypt registered [ 3607.729950] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3609.210768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3609.231885] alg: No test for adler32 (adler32-zlib) [ 3610.314592] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 3610.607874] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3612.303204] Key type lgssc registered [ 3613.746956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3622.409620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3622.423672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.960323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3628.946658] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3634.145955] LustreError: 111189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3634.206250] LustreError: 111189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3637.644845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.098448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3642.336370] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3651.309444] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3657.964397] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3664.059399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3664.454088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.639115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3671.637785] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3682.687405] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3689.126744] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3694.564188] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3700.710692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3700.731529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.775536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.780801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.784384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.840330] Lustre: server umount lustre-OST0000 complete [ 3714.016735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.020346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3714.052332] Lustre: Skipped 1 previous similar message [ 3714.091230] Lustre: Skipped 2 previous similar messages [ 3716.946946] Lustre: server umount lustre-MDT0000 complete [ 3719.150062] LustreError: 113875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3719.172028] LustreError: 113875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3721.504713] LustreError: 113682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404666 with bad export cookie 6566254883479731197 [ 3721.514677] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.530216] LustreError: 113682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3722.172047] Lustre: server umount lustre-MDT0001 complete [ 3735.259424] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3739.068430] Key type lgssc unregistered [ 3739.565547] LNet: 114831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3739.573596] LNetError: 114831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3739.589314] LNet: Removed LNI 192.168.201.152@tcp [ 3740.789430] Key type .llcrypt unregistered [ 3740.790995] Key type ._llcrypt unregistered [ 3762.972262] Key type ._llcrypt registered [ 3762.974603] Key type .llcrypt registered [ 3763.114648] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3764.935352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3765.258986] alg: No test for adler32 (adler32-zlib) [ 3766.478456] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 3766.819251] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3768.455222] Key type lgssc registered [ 3769.570662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3778.527693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3778.549291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.175705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3785.334870] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3790.307291] LustreError: 116055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3790.332446] LustreError: 116055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3793.663622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.074326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3798.167550] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3807.416868] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3814.635308] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3822.861175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.551622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3825.622121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3831.366313] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3840.108890] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3849.198067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.204359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3849.219974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3851.068837] Lustre: server umount lustre-OST0000 complete [ 3855.844795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3855.851850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.861696] Lustre: Skipped 1 previous similar message [ 3855.868752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3855.871462] Lustre: Skipped 1 previous similar message [ 3859.425090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3859.425613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.163671] Lustre: server umount lustre-MDT0000 complete [ 3864.560580] LustreError: 116048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3864.581586] LustreError: 116048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3865.134081] LustreError: 116034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786404810 with bad export cookie 8353956316497276818 [ 3865.143620] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3865.688413] Lustre: server umount lustre-MDT0001 complete [ 3880.216259] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3885.071886] Key type lgssc unregistered [ 3885.710751] LNet: 119590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3885.716234] LNetError: 119590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3885.766757] LNet: Removed LNI 192.168.201.152@tcp [ 3887.181213] Key type .llcrypt unregistered [ 3887.186789] Key type ._llcrypt unregistered [ 3917.030277] Key type ._llcrypt registered [ 3917.041926] Key type .llcrypt registered [ 3917.199776] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3918.981202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3919.111518] alg: No test for adler32 (adler32-zlib) [ 3920.864473] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 3921.351499] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3923.223577] Key type lgssc registered [ 3924.787188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3935.297171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3935.328594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.994868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.570328] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3947.515092] LustreError: 121011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.545043] LustreError: 121011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3952.614849] LustreError: 121012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.664740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3953.122102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3957.850698] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3965.682833] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3972.048848] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3979.664529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3980.076168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.176169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3987.260931] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3995.556911] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4006.820363] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4011.832121] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4011.838107] Lustre: Skipped 2 previous similar messages [ 4012.158240] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4012.227113] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4017.379452] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4017.392393] Lustre: Skipped 2 previous similar messages [ 4020.614219] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4020.618216] Lustre: Skipped 1 previous similar message [ 4030.627139] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4030.634306] Lustre: Skipped 2 previous similar messages [ 4036.714494] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.52@tcp 192.168.0.[2,10]@tcp [ 4041.827314] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.52/24@tcp in log lustre-MDT0000 [ 4041.832681] Lustre: Skipped 2 previous similar messages [ 4046.919625] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.52/24@tcp [ 4046.932421] Lustre: Skipped 1 previous similar message [ 4055.027336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.027875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4055.052377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4055.060640] Lustre: Skipped 1 previous similar message [ 4056.665219] Lustre: server umount lustre-OST0000 complete [ 4060.645960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4060.659113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.682547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4060.691096] Lustre: Skipped 1 previous similar message [ 4065.257632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.258448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4065.276925] Lustre: Skipped 1 previous similar message [ 4066.752791] Lustre: server umount lustre-MDT0000 complete [ 4070.368229] LustreError: 121011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.393777] LustreError: 121011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4072.511915] LustreError: 120992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405017 with bad export cookie 2551754313733876587 [ 4072.516299] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.532316] LustreError: 120992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4073.818667] Lustre: server umount lustre-MDT0001 complete [ 4091.582628] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4095.014295] Key type lgssc unregistered [ 4095.225591] LNet: 125979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4095.232403] LNetError: 125979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4095.246988] LNet: Removed LNI 192.168.201.152@tcp [ 4096.037801] Key type .llcrypt unregistered [ 4096.039721] Key type ._llcrypt unregistered [ 4129.149634] Key type ._llcrypt registered [ 4129.151625] Key type .llcrypt registered [ 4129.283255] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4130.944821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4130.977357] alg: No test for adler32 (adler32-zlib) [ 4131.991193] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 4132.466700] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4134.223184] Key type lgssc registered [ 4136.072633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4147.242555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4147.294125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.338930] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4149.423213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.428989] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.52/24@tcp [ 4155.092644] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4158.950900] LustreError: 127552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4158.969947] LustreError: 127552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4164.076290] LustreError: 127553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.547031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.826527] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4164.861453] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.52/24@tcp [ 4164.917463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.714520] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4180.067065] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4186.574604] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4193.500348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.877581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.924961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4201.554249] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4211.032625] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4224.995986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.999056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4225.010877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.017344] Lustre: Skipped 1 previous similar message [ 4228.891582] Lustre: server umount lustre-OST0000 complete [ 4235.241534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.260044] Lustre: Skipped 1 previous similar message [ 4235.268157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.775603] Lustre: server umount lustre-MDT0000 complete [ 4240.371522] LustreError: 127553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.400032] LustreError: 127553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4245.192930] LustreError: 127533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405190 with bad export cookie 12902180372622554520 [ 4245.210360] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4245.230836] LustreError: 127533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4245.570067] Lustre: server umount lustre-MDT0001 complete [ 4258.111445] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4261.831592] Key type lgssc unregistered [ 4262.112765] LNet: 131143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4262.130852] LNetError: 131143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4263.146152] LNet: Removed LNI 192.168.201.152@tcp [ 4265.196898] Key type .llcrypt unregistered [ 4265.198195] Key type ._llcrypt unregistered [ 4296.835951] Key type ._llcrypt registered [ 4296.841295] Key type .llcrypt registered [ 4296.995449] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 4314.972148] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4316.174325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4316.338068] alg: No test for adler32 (adler32-zlib) [ 4317.482156] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 4317.842676] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4319.592350] Key type lgssc registered [ 4321.062361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4358.938042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4358.990855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.177882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4360.212243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4360.321173] Lustre: lustre-MDT0000: new disk, initializing [ 4360.394238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.414546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4365.917921] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4377.211390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.312953] Lustre: 134563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4377.358232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4377.364966] Lustre: Skipped 1 previous similar message [ 4377.414282] Lustre: lustre-MDT0001: new disk, initializing [ 4377.493299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4377.542725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4377.550772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4382.753383] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4394.272251] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4400.405978] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4407.631779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4408.122512] Lustre: lustre-OST0000: new disk, initializing [ 4408.129728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4408.140402] Lustre: 136534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4408.273879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4414.058814] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4416.098949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4416.125099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4416.229252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4425.673887] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4431.214753] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4431.554154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4437.808326] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4438.192473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4442.090294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4442.094250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.095919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4442.171185] Lustre: Skipped 1 previous similar message [ 4446.911983] Lustre: server umount lustre-OST0000 complete [ 4452.320213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4452.324416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.353644] Lustre: Skipped 1 previous similar message [ 4452.359083] Lustre: Skipped 1 previous similar message [ 4457.441037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.459671] Lustre: Skipped 1 previous similar message [ 4457.814351] Lustre: server umount lustre-MDT0000 complete [ 4462.862973] LustreError: 136533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405408 with bad export cookie 14077556547919593401 [ 4462.869328] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4462.878856] LustreError: 136533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4463.306140] Lustre: server umount lustre-MDT0001 complete [ 4472.487209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.946653] LustreError: 138750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.965151] LustreError: 138750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4473.033474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.789723] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4478.446662] LustreError: 138751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.555590] LustreError: 138750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.941644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.396759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4492.908817] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4502.428603] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4510.113759] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4517.818212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4518.170107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4525.864305] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4534.568751] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4539.848602] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4540.091524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4545.426792] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4545.731901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4555.083516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4555.374393] Lustre: lustre-OST0001: new disk, initializing [ 4555.380105] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4555.390779] Lustre: 142122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4555.540255] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4564.005974] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4564.584149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4564.597932] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4564.719254] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4573.424949] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4573.720203] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4579.527287] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4579.809845] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4586.961565] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e004412f800.ost_server_uuid 50 [ 4588.913415] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e004412f800.ost_server_uuid in IDLE state after 0 sec [ 4600.803132] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4600.816031] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.816121] LustreError: Skipped 1 previous similar message [ 4600.816681] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4600.839090] Lustre: Skipped 1 previous similar message [ 4605.577383] Lustre: server umount lustre-OST0001 complete [ 4605.948124] LustreError: 140572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.976803] LustreError: 140572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4609.504898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4609.507200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.513832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4609.517523] Lustre: Skipped 1 previous similar message [ 4611.055762] LustreError: 140572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4611.077626] LustreError: 140572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4621.288294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4621.294361] LustreError: 140574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.303952] Lustre: Skipped 5 previous similar messages [ 4621.339269] LustreError: 140574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4623.327151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4623.752189] Lustre: server umount lustre-OST0000 complete [ 4630.497348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4630.508696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.518867] Lustre: Skipped 1 previous similar message [ 4634.735865] Lustre: server umount lustre-MDT0000 complete [ 4639.159769] LustreError: 138732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405584 with bad export cookie 14077556547919594290 [ 4639.170601] LustreError: 138732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4639.177816] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4639.453940] Lustre: server umount lustre-MDT0001 complete [ 4660.009100] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4662.889165] Key type lgssc unregistered [ 4663.195612] LNet: 145131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4663.210480] LNetError: 145131:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4663.240189] LNet: Removed LNI 192.168.201.152@tcp [ 4664.328600] Key type .llcrypt unregistered [ 4664.331069] Key type ._llcrypt unregistered [ 4708.216875] Key type ._llcrypt registered [ 4708.218774] Key type .llcrypt registered [ 4708.308653] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 4726.668935] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4728.680414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4728.816770] alg: No test for adler32 (adler32-zlib) [ 4730.304905] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 4730.765288] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4732.551450] Key type lgssc registered [ 4734.732148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4771.126783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4771.196916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.548236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4772.568771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4772.656586] Lustre: lustre-MDT0000: new disk, initializing [ 4772.767746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.791319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4778.251401] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4789.683311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.900193] Lustre: 149290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4790.005916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4790.011246] Lustre: Skipped 1 previous similar message [ 4790.149430] Lustre: lustre-MDT0001: new disk, initializing [ 4790.242357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4790.290214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4790.315537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4795.541753] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4805.450138] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4811.541495] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4819.564035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.947374] Lustre: lustre-OST0000: new disk, initializing [ 4819.950621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4819.961203] Lustre: 151260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4820.095081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.784559] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4828.779805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4828.800696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4829.001073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4838.586086] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4844.151593] Lustre: Failing over lustre-OST0000 [ 4844.442641] Lustre: server umount lustre-OST0000 complete [ 4859.874325] Lustre: 147516:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786405789/real 1786405789] req@000000008c28be34 x1873182117103744/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786405805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4859.913895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.888520] Lustre: 147514:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786405789/real 1786405789] req@000000004aec7a4b x1873182117103872/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786405805 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4860.939950] Lustre: 147514:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4861.965139] Lustre: 147516:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786405791/real 1786405791] req@00000000e989c9d9 x1873182117104128/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786405807 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4865.113160] Lustre: 147514:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786405794/real 1786405794] req@0000000083d120b1 x1873182117105024/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786405810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4866.239962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4866.728663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4866.763899] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4869.863924] Lustre: 147515:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786405799/real 1786405799] req@00000000afb1ab9c x1873182117106176/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786405815 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4869.894032] Lustre: 147515:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4870.830172] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4873.051333] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4873.054406] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4874.235168] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4878.811441] Lustre: Failing over lustre-MDT0000 [ 4879.107471] Lustre: server umount lustre-MDT0000 complete [ 4881.385103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.406156] Lustre: Skipped 1 previous similar message [ 4886.503985] LustreError: 149302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4886.522105] LustreError: 149302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4888.364210] LustreError: 149296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4891.622286] LustreError: 149301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.739606] LustreError: 149296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.768673] LustreError: 149296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4897.759236] Lustre: 147513:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786405826/real 1786405826] req@00000000d9d41b5c x1873182117117824/t0(0) o400->MGC192.168.201.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786405842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4897.803592] Lustre: 147513:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4897.813608] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4899.278218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.882107] LustreError: 149302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.905038] LustreError: 149302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4908.000625] LustreError: 147512:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ade88c26 x1873182117123328/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4908.293711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4908.351773] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4909.864307] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4912.262686] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4913.640731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4913.646627] Lustre: Skipped 1 previous similar message [ 4913.674614] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4913.745725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4923.878156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4923.889733] Lustre: Skipped 3 previous similar messages [ 4923.891068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4923.899851] Lustre: Skipped 1 previous similar message [ 4928.994992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4929.007860] Lustre: Skipped 1 previous similar message [ 4934.124243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4935.135169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4935.436585] Lustre: server umount lustre-OST0000 complete [ 4944.352804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4944.354300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.365603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4944.391958] Lustre: Skipped 1 previous similar message [ 4944.398761] Lustre: Skipped 1 previous similar message [ 4945.908957] Lustre: server umount lustre-MDT0000 complete [ 4949.473342] LustreError: 149296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.484243] LustreError: 149296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4950.691482] LustreError: 150085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786405895 with bad export cookie 7339721009989627217 [ 4950.716675] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.719756] LustreError: 150085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4951.021439] Lustre: server umount lustre-MDT0001 complete [ 4965.429796] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4970.508974] Key type lgssc unregistered [ 4971.038756] LNet: 154922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4971.050096] LNetError: 154922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4971.086281] LNet: Removed LNI 192.168.201.152@tcp [ 4972.543988] Key type .llcrypt unregistered [ 4972.546022] Key type ._llcrypt unregistered [ 5012.728490] Key type ._llcrypt registered [ 5012.730813] Key type .llcrypt registered [ 5012.843846] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5013.940592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5013.964458] alg: No test for adler32 (adler32-zlib) [ 5015.385647] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 5015.811265] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5017.535187] Key type lgssc registered [ 5019.149466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5027.798731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5027.812684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5029.217327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.900799] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5039.604693] LustreError: 156444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5039.624828] LustreError: 156444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5043.728346] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5044.706753] LustreError: 156445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.687168] LustreError: 156473:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5047.700886] LustreError: 156473:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5047.709175] LustreError: 156473:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 5048.032923] Lustre: server umount lustre-MDT0000 complete [ 5063.894367] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5074.777978] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5111.577059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.884805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5111.925175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5112.004772] Lustre: lustre-MDT0000: new disk, initializing [ 5112.082567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.103824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5117.056532] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5128.125080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.284727] Lustre: 159775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5128.354300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5128.358542] Lustre: Skipped 1 previous similar message [ 5128.499533] Lustre: lustre-MDT0001: new disk, initializing [ 5128.666959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5128.729311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5128.738922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5133.398157] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5143.706289] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5149.847464] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5157.524278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5157.776523] Lustre: lustre-OST0000: new disk, initializing [ 5157.780174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5157.786643] Lustre: 161741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5157.862299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.193673] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5165.201382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5165.233892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5165.394836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5175.973988] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5180.180823] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5182.417153] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:55:24 (1786406124) [ 5420.512373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5420.524668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.542307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.537594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.548810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.562818] Lustre: Skipped 1 previous similar message [ 5421.579561] Lustre: Skipped 2 previous similar messages [ 5426.658816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5431.777266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5431.800348] Lustre: Skipped 4 previous similar messages [ 5434.335132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5434.594861] Lustre: server umount lustre-MDT0000 complete [ 5436.898500] LustreError: 162099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.931575] LustreError: 162099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5438.651679] LustreError: 159766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406383 with bad export cookie 13095498616920748787 [ 5438.653063] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5438.667406] LustreError: 159766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5442.016875] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.020178] LustreError: 159780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.026567] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5445.310836] Lustre: server umount lustre-MDT0001 complete [ 5456.079311] Lustre: server umount lustre-OST0000 complete [ 5467.671614] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5484.036391] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5527.562595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5528.039131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5528.098614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5528.237632] Lustre: lustre-MDT0000: new disk, initializing [ 5528.371203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.407768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5533.897803] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5544.363274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.435570] Lustre: 166446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5544.464255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5544.467596] Lustre: Skipped 1 previous similar message [ 5544.515632] Lustre: lustre-MDT0001: new disk, initializing [ 5544.562772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5544.587960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5544.599623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5548.313546] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5558.614769] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5565.747331] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5572.983346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5573.344885] Lustre: lustre-OST0000: new disk, initializing [ 5573.348983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5573.355910] Lustre: 168415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5573.456411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.658920] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5582.420387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5582.441668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5582.589796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5593.568564] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5599.379641] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5599.609765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5604.678787] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5604.915475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5608.417671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5608.442847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.446388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5608.456842] Lustre: Skipped 1 previous similar message [ 5608.475357] Lustre: Skipped 2 previous similar messages [ 5621.215237] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5621.468632] Lustre: server umount lustre-OST0000 complete [ 5626.863147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5626.872596] LustreError: Skipped 1 previous similar message [ 5626.878776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.893038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5626.899148] Lustre: Skipped 4 previous similar messages [ 5632.479602] Lustre: server umount lustre-MDT0000 complete [ 5634.023655] LustreError: 166455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5634.047988] LustreError: 166455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5636.567565] LustreError: 166440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406581 with bad export cookie 13095498616920907617 [ 5636.578826] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.586548] LustreError: 166440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5637.280066] Lustre: server umount lustre-MDT0001 complete [ 5647.380577] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:03:10 (1786406590) [ 5663.246385] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5679.329820] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5724.143444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5724.425075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5724.539626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5724.686289] Lustre: lustre-MDT0000: new disk, initializing [ 5724.813507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.862672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5731.549515] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5743.591553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.769815] Lustre: 173197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5744.214434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5750.035618] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5761.708775] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5769.156585] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5777.437961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5777.814643] Lustre: lustre-OST0000: new disk, initializing [ 5777.817136] Lustre: Skipped 1 previous similar message [ 5777.819255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5777.822843] Lustre: Skipped 2 previous similar messages [ 5777.826909] Lustre: 175178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5785.157441] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5785.169156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5785.177865] Lustre: Skipped 1 previous similar message [ 5785.180363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5785.293631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5799.069951] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5811.170481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5811.195871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.226475] Lustre: Skipped 1 previous similar message [ 5811.241904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5811.266487] Lustre: Skipped 3 previous similar messages [ 5813.450825] Lustre: server umount lustre-OST0000 complete [ 5820.909302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5820.934216] LustreError: Skipped 1 previous similar message [ 5825.890746] Lustre: server umount lustre-MDT0000 complete [ 5826.539195] LustreError: 173204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.567836] LustreError: 173204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5831.095390] LustreError: 173190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406776 with bad export cookie 13095498616920908499 [ 5831.107043] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5831.117428] LustreError: 173190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5831.977424] Lustre: server umount lustre-MDT0001 complete [ 5842.073366] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:06:24 (1786406784) [ 5865.640986] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5879.879219] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5924.382978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.626430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5924.655709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5924.775892] Lustre: lustre-MDT0000: new disk, initializing [ 5924.901407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.909753] Lustre: Skipped 2 previous similar messages [ 5924.925502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5930.338478] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5941.777615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.957051] Lustre: 179516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5942.413437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5948.322866] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5958.619336] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5965.244392] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5972.743753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5973.030112] Lustre: 181493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5979.976342] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5980.258898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5980.381712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5989.098921] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6001.124407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.125223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6001.144458] Lustre: Skipped 3 previous similar messages [ 6001.146775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6001.161174] LustreError: Skipped 1 previous similar message [ 6001.176901] Lustre: Skipped 4 previous similar messages [ 6002.595647] Lustre: server umount lustre-OST0000 complete [ 6013.308823] Lustre: server umount lustre-MDT0000 complete [ 6016.486728] LustreError: 179524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6016.554535] LustreError: 179524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6018.519542] LustreError: 179508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786406963 with bad export cookie 13095498616920909584 [ 6018.522364] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6018.530499] LustreError: 179508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6018.941109] Lustre: server umount lustre-MDT0001 complete [ 6033.535299] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6037.429546] Key type lgssc unregistered [ 6037.870866] LNet: 183349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6037.879483] LNetError: 183349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6037.909966] LNet: Removed LNI 192.168.201.152@tcp [ 6038.992283] Key type .llcrypt unregistered [ 6038.995749] Key type ._llcrypt unregistered [ 6071.335256] Key type ._llcrypt registered [ 6071.343835] Key type .llcrypt registered [ 6071.518312] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6073.174219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6073.561124] alg: No test for adler32 (adler32-zlib) [ 6074.752522] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 6075.034512] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6076.775329] Key type lgssc registered [ 6078.177706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6087.443328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6087.478712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.085574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.662119] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6099.425133] LustreError: 184768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.455606] LustreError: 184768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6104.569726] LustreError: 184769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.947683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.479877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6110.681456] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6119.962441] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6126.159730] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6133.185205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6133.713994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.719048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6140.440857] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6148.049833] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6170.596975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6170.604563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6170.621629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6171.621078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6171.625077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.635081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6176.373711] Lustre: server umount lustre-OST0000 complete [ 6181.857856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.863050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6181.879197] Lustre: Skipped 1 previous similar message [ 6181.892340] Lustre: Skipped 2 previous similar messages [ 6186.781092] Lustre: server umount lustre-MDT0000 complete [ 6186.981662] LustreError: 184763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.000304] LustreError: 184763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6190.689864] LustreError: 187576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407135 with bad export cookie 14402359373291901303 [ 6190.706920] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6191.095128] Lustre: server umount lustre-MDT0001 complete [ 6202.495884] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 6205.199813] Key type lgssc unregistered [ 6205.601542] LNet: 188328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6205.610894] LNetError: 188328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6205.627087] LNet: Removed LNI 192.168.201.152@tcp [ 6206.680527] Key type .llcrypt unregistered [ 6206.683035] Key type ._llcrypt unregistered [ 6233.363866] Key type ._llcrypt registered [ 6233.366379] Key type .llcrypt registered [ 6233.464533] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6235.370703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6235.505228] alg: No test for adler32 (adler32-zlib) [ 6236.581724] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 6236.796051] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 6238.439251] Key type lgssc registered [ 6239.626513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6247.980325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6248.005896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.595474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.277628] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6259.172335] LustreError: 189749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6259.185851] LustreError: 189749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6264.290142] LustreError: 189748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.050859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.788843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6271.315359] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6279.320927] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6285.704557] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6292.755898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6293.239950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.343569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6302.129334] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6310.175238] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6317.892067] Lustre: lustre-OST0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 6319.075480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.080184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.095218] Lustre: Skipped 1 previous similar message [ 6319.112152] Lustre: Skipped 1 previous similar message [ 6322.994429] Lustre: lustre-OST0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 6323.516434] Lustre: server umount lustre-OST0000 complete [ 6332.493649] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e005970a800.ost_server_uuid 50 [ 6334.551960] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6349.795685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.804351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6349.813089] Lustre: Skipped 1 previous similar message [ 6349.829523] Lustre: Skipped 1 previous similar message [ 6352.195051] Lustre: server umount lustre-MDT0000 complete [ 6352.800600] LustreError: 189744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.816034] LustreError: 189744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6357.195871] LustreError: 189731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407302 with bad export cookie 1450577868659209474 [ 6357.208289] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6357.217945] LustreError: 189731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6357.842620] Lustre: server umount lustre-MDT0001 complete [ 6367.982087] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:15:10 (1786407310) [ 6375.079182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.525961] LustreError: 193664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.639276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.270871] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6381.026328] LustreError: 193665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6388.508972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6388.784792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6393.759632] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6402.616565] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6409.006493] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6416.384847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6416.760254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6419.839501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6423.916398] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6431.414411] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6439.198860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6439.441924] Lustre: lustre-OST0001: new disk, initializing [ 6439.445509] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6439.451919] Lustre: 196506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6439.538611] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6447.326442] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6448.267472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6448.279342] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6448.405794] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6457.388739] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6460.897988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6460.908349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.940484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6463.969854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6463.990717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6466.714309] Lustre: server umount lustre-OST0000 complete [ 6468.402523] LustreError: 195489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6468.420291] LustreError: 195489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6473.516636] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6473.713446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6478.966565] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6479.162079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6484.457846] LustreError: 195489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.478550] LustreError: 195489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6494.692202] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6494.693930] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6494.695402] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6494.695410] Lustre: Skipped 3 previous similar messages [ 6494.728136] LustreError: Skipped 1 previous similar message [ 6499.534261] Lustre: server umount lustre-OST0001 complete [ 6504.928421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6504.946089] Lustre: Skipped 2 previous similar messages [ 6509.863913] Lustre: server umount lustre-MDT0000 complete [ 6513.272499] LustreError: 196505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407458 with bad export cookie 1450577868659210545 [ 6513.276580] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6513.280047] LustreError: 196505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6513.798737] Lustre: server umount lustre-MDT0001 complete [ 6536.156345] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:17:58 (1786407478) [ 6542.551302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.910942] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6542.949124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6543.144488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.559634] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6555.401905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6555.462394] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6555.493879] Lustre: 199951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6560.172777] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6568.199397] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6573.827596] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6580.279336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6580.482087] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6587.606446] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6588.060516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6595.064066] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6603.126036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6603.235289] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6610.432966] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6611.126623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6618.869141] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6623.716678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6623.724369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.744236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6623.756683] Lustre: Skipped 3 previous similar messages [ 6628.026974] Lustre: server umount lustre-OST0000 complete [ 6629.672614] LustreError: 201799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.695329] LustreError: 201799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6646.754751] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6646.764249] LustreError: Skipped 1 previous similar message [ 6648.523254] Lustre: server umount lustre-OST0001 complete [ 6657.505548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.506219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6657.519276] Lustre: Skipped 4 previous similar messages [ 6657.532630] Lustre: Skipped 9 previous similar messages [ 6659.649766] Lustre: server umount lustre-MDT0000 complete [ 6663.576521] LustreError: 199942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407608 with bad export cookie 1450577868659211952 [ 6663.582161] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6663.584450] LustreError: 199942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6664.196790] Lustre: server umount lustre-MDT0001 complete [ 6689.202038] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:20:31 (1786407631) [ 6708.238371] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 6720.130326] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6757.178473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.385565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6757.410861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6757.482557] Lustre: lustre-MDT0000: new disk, initializing [ 6757.677139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6757.680380] Lustre: Skipped 3 previous similar messages [ 6757.710345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6762.755385] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6773.748736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6773.885556] Lustre: 208047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6773.956174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6773.964323] Lustre: Skipped 1 previous similar message [ 6774.040979] Lustre: lustre-MDT0001: new disk, initializing [ 6774.167536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6774.187867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6778.909987] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6788.238122] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6794.090788] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6801.837852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6802.143054] Lustre: lustre-OST0000: new disk, initializing [ 6802.147528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6802.156380] Lustre: 210012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6809.237356] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6812.272684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6812.285215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6812.479361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6821.662219] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6827.893284] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6828.257995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6834.706611] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6835.203625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6838.240900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6838.242035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6838.244578] LustreError: Skipped 3 previous similar messages [ 6838.258549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6838.272211] Lustre: Skipped 1 previous similar message [ 6844.575703] Lustre: server umount lustre-OST0000 complete [ 6863.329654] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6863.551791] Lustre: server umount lustre-MDT0000 complete [ 6865.889585] LustreError: 208053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.909408] LustreError: 208053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6867.642983] LustreError: 210023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407812 with bad export cookie 1450577868659213737 [ 6867.647820] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.656953] LustreError: 210023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6868.133911] Lustre: server umount lustre-MDT0001 complete [ 6877.127645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.209349] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6892.844230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.353380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6893.365449] Lustre: Skipped 3 previous similar messages [ 6898.747318] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6907.095687] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6914.703810] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6922.004530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6928.991298] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6937.326892] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6942.483529] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6942.746131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6947.878568] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6948.208685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6954.470667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6954.487976] LustreError: Skipped 2 previous similar messages [ 6956.881868] Lustre: server umount lustre-OST0000 complete [ 6965.095569] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6970.661943] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7000.904656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7009.238048] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7017.599284] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7019.351538] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7019.764841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7019.778424] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7019.782740] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7019.799885] Lustre: Skipped 1 previous similar message [ 7025.122162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7025.140299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7025.149202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7025.157853] Lustre: Skipped 6 previous similar messages [ 7025.175313] Lustre: Skipped 11 previous similar messages [ 7029.995620] Lustre: server umount lustre-OST0000 complete [ 7040.487571] LustreError: 212233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7040.514942] LustreError: 212233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7040.664388] Lustre: server umount lustre-MDT0000 complete [ 7045.237196] LustreError: 212214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786407990 with bad export cookie 1450577868659214619 [ 7045.242472] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7045.263111] LustreError: 212214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7045.734461] Lustre: server umount lustre-MDT0001 complete [ 7055.341618] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:26:37 (1786407997) [ 7063.471260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.461192] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7080.560859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.925092] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7093.610608] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7099.700775] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7106.686720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7114.315813] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7122.679540] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7128.151982] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7128.358363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7134.279853] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7134.572366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7143.215767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7143.434266] Lustre: lustre-OST0001: new disk, initializing [ 7143.438735] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7143.445557] Lustre: 221575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7150.747828] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7151.206888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7151.220929] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7151.341126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7160.880538] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7166.404477] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7166.658623] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7172.371587] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7172.609521] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7177.211502] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7177.220960] LustreError: Skipped 1 previous similar message [ 7181.295780] Lustre: server umount lustre-OST0001 complete [ 7188.628771] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7188.823798] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7193.977340] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7194.194897] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7224.235196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7224.555889] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7224.570418] Lustre: Skipped 6 previous similar messages [ 7229.935588] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7229.956475] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7229.958568] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7230.806150] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7239.455916] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7241.545251] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7249.867397] Lustre: server umount lustre-OST0001 complete [ 7274.640557] Lustre: server umount lustre-OST0000 complete [ 7286.452665] Lustre: server umount lustre-MDT0000 complete [ 7290.617085] LustreError: 218182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408235 with bad export cookie 1450577868659215585 [ 7290.626370] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7290.630356] LustreError: 218182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7297.273814] Lustre: server umount lustre-MDT0001 complete [ 7323.604437] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:31:06 (1786408266) [ 7330.916543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.323167] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7331.377626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7336.712295] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7346.679205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7346.875803] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7346.915159] Lustre: 226820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7352.797588] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7361.534409] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7367.924707] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7374.784612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.971568] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7381.842965] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7389.458255] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7399.041051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.109969] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7406.492814] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7414.121499] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7415.678432] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7420.668126] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7420.867667] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7425.641931] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7425.835416] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7432.152748] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e00451cf000.ost_server_uuid 50 [ 7433.746614] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e00451cf000.ost_server_uuid in IDLE state after 0 sec [ 7434.627630] Lustre: Permanently deactivating lustre-OST0001 [ 7434.650784] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7438.138131] Lustre: Permanently reactivating lustre-OST0001 [ 7442.400448] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7442.416134] LustreError: Skipped 4 previous similar messages [ 7442.422178] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.438717] Lustre: Skipped 10 previous similar messages [ 7442.454757] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7442.461967] Lustre: Skipped 12 previous similar messages [ 7448.347924] Lustre: server umount lustre-OST0001 complete [ 7453.157294] LustreError: 230064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.193458] LustreError: 230064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 7459.053256] Lustre: server umount lustre-OST0000 complete [ 7477.216080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7477.489951] Lustre: server umount lustre-MDT0000 complete [ 7481.792704] LustreError: 226814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408426 with bad export cookie 1450577868659216873 [ 7481.798845] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.802504] LustreError: 226814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7482.231176] Lustre: server umount lustre-MDT0001 complete [ 7509.593310] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:34:12 (1786408452) [ 7518.955975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.308294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7519.351593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7519.363910] Lustre: Skipped 5 previous similar messages [ 7525.338245] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7535.231551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.326579] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7535.373667] Lustre: 233519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7540.642101] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7548.326970] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7554.904478] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7562.310498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.553643] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7562.611058] Lustre: Permanently deactivating lustre-OST0000 [ 7562.643756] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7569.970665] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7572.538936] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7579.968907] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7588.435628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7588.557232] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7595.241759] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7596.849987] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7604.980230] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7606.808589] Lustre: Permanently reactivating lustre-OST0000 [ 7606.826689] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7606.840161] Lustre: Skipped 2 previous similar messages [ 7616.683269] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 00000000d7d2ff73, cur 1786408561 deadline 1786408547 last 1786408517 [ 7616.683330] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7616.709263] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7616.729761] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7616.748610] Lustre: Skipped 1 previous similar message [ 7616.851797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7631.860153] Lustre: server umount lustre-OST0001 complete [ 7642.950382] Lustre: server umount lustre-OST0000 complete [ 7653.957682] Lustre: server umount lustre-MDT0000 complete [ 7657.876869] LustreError: 233512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408603 with bad export cookie 1450577868659218525 [ 7657.899781] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7657.904194] LustreError: 233512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7658.347898] Lustre: server umount lustre-MDT0001 complete [ 7670.263276] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 7673.698332] Key type lgssc unregistered [ 7674.042082] LNet: 238400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7674.051192] LNetError: 238400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7674.078616] LNet: Removed LNI 192.168.201.152@tcp [ 7675.129939] Key type .llcrypt unregistered [ 7675.132365] Key type ._llcrypt unregistered [ 7702.099238] Key type ._llcrypt registered [ 7702.101547] Key type .llcrypt registered [ 7702.224903] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7703.460401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7703.617835] alg: No test for adler32 (adler32-zlib) [ 7705.049967] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 7705.362893] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 7707.090645] Key type lgssc registered [ 7708.619318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7720.640596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7720.687087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7722.435661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.147761] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7732.708185] LustreError: 239867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7732.735950] LustreError: 239867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7737.825279] LustreError: 239868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.394700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.550810] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7738.589869] Lustre: 239856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7738.612189] Lustre: Permanently deactivating lustre-MDT0001 [ 7738.633195] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7739.082688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7745.516320] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7747.658187] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7754.831169] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7761.671213] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7770.039518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7770.680550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.516309] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7784.037984] LustreError: 241709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.052445] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7785.617844] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7789.545938] LustreError: 241711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.622861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7793.054588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7793.311764] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7798.789898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7798.794885] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7799.989916] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7808.791767] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7812.706726] Lustre: Permanently reactivating lustre-MDT0001 [ 7812.757989] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7812.772963] Lustre: Skipped 2 previous similar messages [ 7815.816935] Lustre: lustre-MDT0001: Client 47a74a98-143f-4c37-a153-0c183334ae5e (at 192.168.201.52@tcp) reconnecting [ 7818.425869] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.446194] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7818.461499] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7818.478580] LustreError: 243458:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7818.481408] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7818.493607] LustreError: 243458:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7818.493730] LustreError: 243458:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.152@tcp: cfg command failed: rc = -108 [ 7818.493752] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7818.493752] [ 7818.494807] LustreError: 239859:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7824.319850] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7825.611240] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7830.728306] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7831.016598] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7836.847576] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7837.248517] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7840.328669] Lustre: Permanently deactivating lustre-MDT0001 [ 7840.367833] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7840.377572] Lustre: Skipped 2 previous similar messages [ 7849.640966] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7860.197631] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.204800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7860.221404] Lustre: Skipped 1 previous similar message [ 7865.314086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7865.339201] Lustre: Skipped 6 previous similar messages [ 7866.546802] Lustre: server umount lustre-MDT0000 complete [ 7870.457046] LustreError: 239868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.483844] LustreError: 239868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7871.095157] LustreError: 239849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786408816 with bad export cookie 1563092744279364997 [ 7871.097113] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7871.119301] LustreError: 239849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7871.855990] Lustre: server umount lustre-MDT0001 complete [ 7878.368711] Lustre: server umount lustre-OST0000 complete [ 7890.309590] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:40:33 (1786408833) [ 7903.016459] Lustre: server umount lustre-OST0001 complete [ 7913.391547] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 7928.675243] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7972.788588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.140828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7973.147479] Lustre: Skipped 2 previous similar messages [ 7973.174869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7973.272824] Lustre: lustre-MDT0000: new disk, initializing [ 7973.360511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7973.375609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7978.333368] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7988.812178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.949899] Lustre: 248201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7989.003682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7989.007746] Lustre: Skipped 1 previous similar message [ 7989.133127] Lustre: lustre-MDT0001: new disk, initializing [ 7989.259850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7989.291946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7989.300645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7994.510871] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8004.590230] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8011.639414] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8019.983559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8020.564381] Lustre: lustre-OST0000: new disk, initializing [ 8020.607109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8020.614475] Lustre: 250171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8020.804584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8028.835155] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8030.299784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8030.321922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8030.461985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8040.078848] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8042.177516] LustreError: 248209:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8052.191254] LustreError: 248209:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8052.232249] LustreError: 248208:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8052.298439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8052.594658] Lustre: lustre-OST0001: new disk, initializing [ 8052.607049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8052.622830] Lustre: 251284:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8052.854288] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8060.601316] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8062.072378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8062.084586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8062.184806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8062.336063] LustreError: 248208:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8072.226810] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8074.017162] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8077.795095] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8077.801043] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.806448] Lustre: Skipped 2 previous similar messages [ 8077.809398] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8077.816133] Lustre: Skipped 1 previous similar message [ 8082.801314] Lustre: server umount lustre-OST0001 complete [ 8082.922157] LustreError: 250156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8082.941799] LustreError: 250156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8103.393164] LustreError: 250595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.404835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8103.435637] LustreError: 250595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 8103.469132] Lustre: Skipped 1 previous similar message [ 8103.488519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.503801] Lustre: Skipped 1 previous similar message [ 8108.516667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8108.527593] Lustre: Skipped 1 previous similar message [ 8109.617831] Lustre: server umount lustre-OST0000 complete [ 8117.219625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8117.227972] LustreError: Skipped 1 previous similar message [ 8117.241478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8117.272454] Lustre: Skipped 1 previous similar message [ 8117.283317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8121.193709] Lustre: server umount lustre-MDT0000 complete [ 8125.342706] LustreError: 248192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409070 with bad export cookie 1563092744279367811 [ 8125.355551] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8125.361310] LustreError: 248192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8131.729626] Lustre: server umount lustre-MDT0001 complete [ 8144.635186] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8149.194176] Key type lgssc unregistered [ 8149.620856] LNet: 253326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8149.629875] LNetError: 253326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8149.641161] LNet: Removed LNI 192.168.201.152@tcp [ 8150.978568] Key type .llcrypt unregistered [ 8150.981365] Key type ._llcrypt unregistered [ 8197.810467] Key type ._llcrypt registered [ 8197.813186] Key type .llcrypt registered [ 8198.039434] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8199.309598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8199.499203] alg: No test for adler32 (adler32-zlib) [ 8200.726063] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 8200.930497] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8202.615307] Key type lgssc registered [ 8204.071600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8214.171127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8214.188702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.578943] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8215.619592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8215.829598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8221.444052] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8232.705243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.797877] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8232.838274] Lustre: 255538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8233.372921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8239.210313] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8249.900238] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8258.375519] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8266.913794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8267.252425] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8267.538262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8275.260534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8275.966561] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8286.094805] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8347.107789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8347.113453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8347.117786] LustreError: Skipped 1 previous similar message [ 8347.134266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8348.800359] Lustre: server umount lustre-OST0000 complete [ 8352.756206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8367.996201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8368.362835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8374.826344] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8383.661446] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8387.759935] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8387.777972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8387.788739] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8387.822788] Lustre: Skipped 1 previous similar message [ 8429.024466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8429.038664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.047945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8429.053730] Lustre: Skipped 2 previous similar messages [ 8429.077800] Lustre: Skipped 2 previous similar messages [ 8430.313631] Lustre: server umount lustre-OST0000 complete [ 8438.241801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8438.251721] LustreError: Skipped 1 previous similar message [ 8438.260376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8438.279277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8441.171683] Lustre: server umount lustre-MDT0000 complete [ 8445.219103] LustreError: 257398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409390 with bad export cookie 14676215797951016642 [ 8445.222850] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8445.233043] LustreError: 257398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8445.715602] Lustre: server umount lustre-MDT0001 complete [ 8457.960332] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8461.242464] Key type lgssc unregistered [ 8461.728894] LNet: 260961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8461.742964] LNetError: 260961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8461.757925] LNet: Removed LNI 192.168.201.152@tcp [ 8463.066548] Key type .llcrypt unregistered [ 8463.069236] Key type ._llcrypt unregistered [ 8490.459409] Key type ._llcrypt registered [ 8490.461568] Key type .llcrypt registered [ 8490.619349] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8492.190445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8492.349756] alg: No test for adler32 (adler32-zlib) [ 8493.665546] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 8494.003948] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8495.775290] Key type lgssc registered [ 8497.382407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8506.927515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8506.952465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.454036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8513.249421] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8518.627874] LustreError: 262379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8518.644688] LustreError: 262379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8522.232666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.711640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8527.799335] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8536.291671] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8542.886344] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8550.281358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8550.642036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.756938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8557.639118] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8565.550457] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8592.869654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.871415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8592.892814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8592.909113] Lustre: Skipped 1 previous similar message [ 8597.996215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8598.004444] Lustre: Skipped 1 previous similar message [ 8598.893494] Lustre: server umount lustre-OST0000 complete [ 8603.112785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8603.135130] Lustre: Skipped 1 previous similar message [ 8603.140358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8603.144914] Lustre: Skipped 1 previous similar message [ 8608.224643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8608.231567] Lustre: Skipped 1 previous similar message [ 8609.560147] Lustre: server umount lustre-MDT0000 complete [ 8613.347573] LustreError: 263108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.365186] LustreError: 263108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8613.879552] LustreError: 262361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409559 with bad export cookie 1450067170002070974 [ 8613.884282] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8613.889962] LustreError: 262361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8614.237792] Lustre: server umount lustre-MDT0001 complete [ 8625.940209] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8628.912714] Key type lgssc unregistered [ 8629.238217] LNet: 266687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8629.246500] LNetError: 266687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8630.330210] LNet: Removed LNI 192.168.201.152@tcp [ 8631.259840] Key type .llcrypt unregistered [ 8631.265660] Key type ._llcrypt unregistered [ 8651.464563] Key type ._llcrypt registered [ 8651.466519] Key type .llcrypt registered [ 8651.560169] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8652.769419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8652.985078] alg: No test for adler32 (adler32-zlib) [ 8654.106843] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 8654.292531] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8655.943426] Key type lgssc registered [ 8657.458924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8666.054253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8666.072427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.563961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8671.435917] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8676.837354] LustreError: 267910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.864558] LustreError: 267910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8679.487769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.767030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8685.297790] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8698.797445] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8710.522141] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8724.863779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8725.557386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8734.082074] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8736.910955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8745.230853] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8762.856235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8762.875464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8762.892890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8764.920956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8764.924109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8764.937200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.975060] Lustre: Skipped 1 previous similar message [ 8767.425459] Lustre: server umount lustre-OST0000 complete [ 8772.076132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8772.088401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8772.106488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8775.149986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8775.151478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.262356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.268570] Lustre: Skipped 2 previous similar messages [ 8786.399686] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8786.715824] Lustre: server umount lustre-MDT0000 complete [ 8790.504621] LustreError: 267911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.526461] LustreError: 267911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8791.047799] LustreError: 270842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409736 with bad export cookie 9005676852995902240 [ 8791.050575] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8791.056124] LustreError: 270842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8791.402791] Lustre: server umount lustre-MDT0001 complete [ 8803.974810] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8806.966147] Key type lgssc unregistered [ 8807.307619] LNet: 271594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8807.323874] LNetError: 271594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8807.344837] LNet: Removed LNI 192.168.201.152@tcp [ 8808.560885] Key type .llcrypt unregistered [ 8808.563295] Key type ._llcrypt unregistered [ 8835.001232] Key type ._llcrypt registered [ 8835.007334] Key type .llcrypt registered [ 8835.185769] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8837.138594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8837.164033] alg: No test for adler32 (adler32-zlib) [ 8838.254827] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 8838.526976] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8840.231257] Key type lgssc registered [ 8841.563689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8850.747905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8850.768719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.278523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.903582] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8862.692538] LustreError: 273011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8862.706841] LustreError: 273011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8865.910477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.191024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8870.522085] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8878.665201] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8886.260185] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8894.508775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8894.879176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.973755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8902.575836] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8911.453714] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8936.931034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.933638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8936.957134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8936.972038] Lustre: Skipped 1 previous similar message [ 8942.052849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.063097] Lustre: Skipped 1 previous similar message [ 8943.063244] Lustre: server umount lustre-OST0000 complete [ 8947.172183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.174963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8947.191827] Lustre: Skipped 2 previous similar messages [ 8952.289701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8952.293568] Lustre: Skipped 1 previous similar message [ 8953.475145] Lustre: server umount lustre-MDT0000 complete [ 8957.408351] LustreError: 273012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.440095] LustreError: 273012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8958.178107] LustreError: 272991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786409903 with bad export cookie 17789348068510510704 [ 8958.188938] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8958.194113] LustreError: 272991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8958.693825] Lustre: server umount lustre-MDT0001 complete [ 8972.095399] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8975.358703] Key type lgssc unregistered [ 8975.644504] LNet: 277318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8975.654148] LNetError: 277318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8975.675855] LNet: Removed LNI 192.168.201.152@tcp [ 8976.678265] Key type .llcrypt unregistered [ 8976.681671] Key type ._llcrypt unregistered [ 8997.330476] Key type ._llcrypt registered [ 8997.333282] Key type .llcrypt registered [ 8997.455637] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8999.116335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8999.175366] alg: No test for adler32 (adler32-zlib) [ 9000.459851] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 9000.837252] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 9002.527169] Key type lgssc registered [ 9003.902556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9013.223674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9013.261059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.915611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9020.668879] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9025.508847] LustreError: 278555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.520634] LustreError: 278555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9030.634949] LustreError: 278557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9030.736985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.248986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9037.455541] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9046.195603] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9053.364783] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9061.786245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9062.139715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.386193] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9078.010440] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9085.678952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9101.300710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9101.312270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9101.331437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9102.308782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9102.320597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.361765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9106.405955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9106.418186] Lustre: Skipped 1 previous similar message [ 9107.683777] Lustre: server umount lustre-OST0000 complete [ 9113.057846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9113.068255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.079952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9116.642214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9118.515669] Lustre: server umount lustre-MDT0000 complete [ 9121.769332] LustreError: 278549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9121.798969] LustreError: 278549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9123.953755] LustreError: 278520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410069 with bad export cookie 3084975346446839486 [ 9123.974694] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9124.533532] Lustre: server umount lustre-MDT0001 complete [ 9139.976413] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 9143.665585] Key type lgssc unregistered [ 9144.024369] LNet: 282290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9144.038028] LNetError: 282290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9144.067867] LNet: Removed LNI 192.168.201.152@tcp [ 9145.255942] Key type .llcrypt unregistered [ 9145.258288] Key type ._llcrypt unregistered [ 9161.262664] Key type ._llcrypt registered [ 9161.267422] Key type .llcrypt registered [ 9161.488932] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9181.136613] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 9200.107938] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9201.222530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9201.323032] alg: No test for adler32 (adler32-zlib) [ 9202.399646] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 9202.645509] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 9204.383202] Key type lgssc registered [ 9205.552971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9243.222658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9243.260555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.565475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9244.593555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9244.673430] Lustre: lustre-MDT0000: new disk, initializing [ 9244.739911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9244.755421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9250.034832] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9261.433556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.626043] Lustre: 285856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9261.715102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9261.719644] Lustre: Skipped 1 previous similar message [ 9261.831401] Lustre: lustre-MDT0001: new disk, initializing [ 9261.834078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9261.939347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9261.978573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9261.993889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9267.116536] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9278.361224] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9286.694824] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9295.178362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9295.710614] Lustre: lustre-OST0000: new disk, initializing [ 9295.718931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9295.724638] Lustre: 287828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9295.911633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.580385] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9305.161123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9305.177941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9305.336333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9314.185322] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9319.870629] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9320.186859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9325.627728] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9325.924686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9331.183173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9331.189674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9331.195096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9331.216336] Lustre: Skipped 1 previous similar message [ 9334.424186] Lustre: server umount lustre-OST0000 complete [ 9338.850570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9338.859975] LustreError: Skipped 1 previous similar message [ 9338.866675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.886413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.890711] Lustre: Skipped 1 previous similar message [ 9341.409076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9341.410967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9344.803609] Lustre: server umount lustre-MDT0000 complete [ 9348.785685] LustreError: 285848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410293 with bad export cookie 16397367715290445595 [ 9348.803463] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9348.803878] LustreError: 285848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9349.202525] Lustre: server umount lustre-MDT0001 complete [ 9358.140670] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:05:01 (1786410301) [ 9365.547908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9366.265478] LustreError: 290238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.295571] LustreError: 290238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9366.386720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9371.625937] LustreError: 290239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.906032] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9376.735995] LustreError: 290238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9381.540833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9381.858956] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9381.864179] Lustre: Skipped 1 previous similar message [ 9381.985264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9387.389805] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9396.559964] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9404.052365] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9413.300684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9413.649508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9421.745476] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9430.661636] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9446.882950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9446.886232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9446.898082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9446.911719] Lustre: Skipped 1 previous similar message [ 9452.701859] Lustre: server umount lustre-OST0000 complete [ 9458.657542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9458.669757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9463.694637] Lustre: server umount lustre-MDT0000 complete [ 9467.368196] LustreError: 290239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9467.396155] LustreError: 290239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9468.136440] LustreError: 290220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410413 with bad export cookie 16397367715290446477 [ 9468.177559] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9468.628461] Lustre: server umount lustre-MDT0001 complete [ 9481.679309] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 9485.321821] Key type lgssc unregistered [ 9485.656356] LNet: 293800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9485.664365] LNetError: 293800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9485.676914] LNet: Removed LNI 192.168.201.152@tcp [ 9486.640333] Key type .llcrypt unregistered [ 9486.641796] Key type ._llcrypt unregistered [ 9519.775544] Key type ._llcrypt registered [ 9519.777789] Key type .llcrypt registered [ 9519.939947] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 9537.119479] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9538.331873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9538.426940] alg: No test for adler32 (adler32-zlib) [ 9539.644561] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [ 9539.965260] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 9541.711206] Key type lgssc registered [ 9543.387371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9588.654508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9588.672268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.070853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9590.141482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9590.244450] Lustre: lustre-MDT0000: new disk, initializing [ 9590.309055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.325429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9595.453610] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9607.016179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.218909] Lustre: 297629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9607.261784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9607.265108] Lustre: Skipped 1 previous similar message [ 9607.347872] Lustre: lustre-MDT0001: new disk, initializing [ 9607.453471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9607.480707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9607.493971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9612.829546] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9624.803377] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9633.687601] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9641.667259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9642.014587] Lustre: lustre-OST2710: new disk, initializing [ 9642.023608] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9642.030670] Lustre: 299598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9642.156994] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9649.823165] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9650.756276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9650.769085] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9650.921703] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9660.118487] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9667.422524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9667.648371] Lustre: lustre-OST03e8: new disk, initializing [ 9667.656141] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9667.666946] Lustre: 300664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9667.792186] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9674.878431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9674.898104] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9675.093736] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9675.154838] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9683.448561] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9690.841775] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9691.114874] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9696.107496] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9696.402789] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9701.281517] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9701.535772] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9707.068584] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9707.389567] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9714.767034] Lustre: server umount lustre-MDT0000 complete [ 9715.172865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9715.176056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.219878] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.252381] Lustre: Skipped 2 previous similar messages [ 9718.940934] LustreError: 297621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410664 with bad export cookie 13832171432984592015 [ 9718.942603] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9718.952930] LustreError: 297621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9719.364678] Lustre: server umount lustre-MDT0001 complete [ 9733.925540] Lustre: server umount lustre-OST2710 complete [ 9736.800099] Lustre: 295425:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786410666/real 1786410666] req@0000000090e247da x1873187160790400/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786410682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9736.855237] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9739.904686] Lustre: server umount lustre-OST03e8 complete [ 9770.215897] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 9782.371138] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9820.377716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9820.583459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9820.613400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9820.706962] Lustre: lustre-MDT0000: new disk, initializing [ 9820.794637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9820.814354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9825.582637] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9837.096062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9837.373163] Lustre: 305914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9837.480300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9837.488383] Lustre: Skipped 1 previous similar message [ 9837.589258] Lustre: lustre-MDT0001: new disk, initializing [ 9837.661994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9837.683494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9837.694564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9842.701466] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9853.293746] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9859.004870] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9866.334584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9866.527778] Lustre: lustre-OST0000: new disk, initializing [ 9866.531888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9866.536233] Lustre: 307883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9866.607779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9873.717938] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9874.566321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9874.587808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9874.698434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9882.838868] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9888.518971] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9888.831455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9894.418524] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9894.673113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9900.517866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9900.533479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9900.560212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9900.562556] LustreError: Skipped 1 previous similar message [ 9900.592104] Lustre: Skipped 1 previous similar message [ 9903.359750] Lustre: server umount lustre-OST0000 complete [ 9909.222273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9909.240375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9909.267164] Lustre: Skipped 1 previous similar message [ 9909.282328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9910.752911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9914.196339] Lustre: server umount lustre-MDT0000 complete [ 9915.874943] LustreError: 305921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9915.902497] LustreError: 305921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9917.842093] LustreError: 307882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410863 with bad export cookie 13832171432984593695 [ 9917.847897] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9917.853659] LustreError: 307882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9918.214303] Lustre: server umount lustre-MDT0001 complete [ 9926.695737] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:14:29 (1786410869) [ 9928.350350] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9930.354420] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:14:33 (1786410873) [ 9934.158240] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9960.064901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.250443] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9960.261751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9960.363949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.476884] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9973.501616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.607400] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9973.663800] Lustre: 311739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9979.136760] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9999.743953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10000.058142] Lustre: Denying initial registration attempt from nid 192.168.201.152@tcp, specified as failover [10000.084504] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10000.101888] LustreError: 313192:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10000.118461] LustreError: 313192:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10000.130755] LustreError: 313192:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10000.334528] Lustre: server umount lustre-OST0000 complete [10000.339562] LustreError: 313192:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [10010.086981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10010.098123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.122022] Lustre: Skipped 1 previous similar message [10010.132803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10010.148214] Lustre: Skipped 1 previous similar message [10017.761578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10017.786204] Lustre: Skipped 3 previous similar messages [10023.391140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10023.726735] Lustre: server umount lustre-MDT0000 complete [10027.930051] LustreError: 311733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786410973 with bad export cookie 13832171432984594570 [10027.945559] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10028.229235] Lustre: server umount lustre-MDT0001 complete [10041.334613] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10044.964988] Key type lgssc unregistered [10045.436236] LNet: 314315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10045.442670] LNetError: 314315:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10045.480230] LNet: Removed LNI 192.168.201.152@tcp [10046.684254] Key type .llcrypt unregistered [10046.686321] Key type ._llcrypt unregistered [10060.107686] Key type ._llcrypt registered [10060.112540] Key type .llcrypt registered [10060.227945] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10061.299908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10061.316033] alg: No test for adler32 (adler32-zlib) [10062.409724] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [10062.646741] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10064.312733] Key type lgssc registered [10065.539394] Lustre: Echo OBD driver; http://www.lustre.org/ [10112.084625] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10121.126799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10121.153043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10122.335125] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10122.366472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10122.554034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10126.748702] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10133.539614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.612517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10133.638796] Lustre: 317081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10133.850652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10136.825217] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10142.589610] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10148.731595] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10169.069030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10169.388986] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10169.585770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10175.493949] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10183.143931] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10185.699866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10185.706052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10185.718948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10186.216486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10186.227213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.257162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10186.264932] Lustre: Skipped 1 previous similar message [10191.336983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10191.361419] Lustre: Skipped 1 previous similar message [10191.839785] Lustre: server umount lustre-OST0000 complete [10196.449209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10196.458175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10196.472811] Lustre: Skipped 1 previous similar message [10201.571062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10201.576086] Lustre: Skipped 2 previous similar messages [10210.273742] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10210.564947] Lustre: server umount lustre-MDT0000 complete [10214.774536] LustreError: 317074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411159 with bad export cookie 13970175410265823791 [10214.777669] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10214.783300] LustreError: 317074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10215.147507] Lustre: server umount lustre-MDT0001 complete [10227.146463] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10230.775933] Key type lgssc unregistered [10231.142825] LNet: 320656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10231.148963] LNetError: 320656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10231.167793] LNet: Removed LNI 192.168.201.152@tcp [10232.366162] Key type .llcrypt unregistered [10232.368310] Key type ._llcrypt unregistered [10258.598992] Key type ._llcrypt registered [10258.606583] Key type .llcrypt registered [10258.735660] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10259.953053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10259.988074] alg: No test for adler32 (adler32-zlib) [10260.965272] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [10261.180060] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10262.890321] Key type lgssc registered [10264.155755] Lustre: Echo OBD driver; http://www.lustre.org/ [10272.725336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10272.775799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10274.419024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10279.320797] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10284.069339] LustreError: 322073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.093659] LustreError: 322073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10289.124018] LustreError: 322074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10289.689961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10289.906456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10293.107715] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10301.311338] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10307.023371] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10316.085246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10316.638347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10323.713145] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10331.125465] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10345.448639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.453580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10345.460519] Lustre: Skipped 1 previous similar message [10345.487583] Lustre: Skipped 2 previous similar messages [10348.365754] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [10350.561611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10350.566908] Lustre: Skipped 2 previous similar messages [10353.443966] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [10358.588585] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [10358.603138] Lustre: Skipped 3 previous similar messages [10359.263220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10359.541550] Lustre: server umount lustre-MDT0000 complete [10360.812854] LustreError: 324068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.838895] LustreError: 324068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10363.688308] LustreError: 322068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10364.657924] LustreError: 322055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411309 with bad export cookie 3618262981547311906 [10364.685695] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10365.938822] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10365.951809] Lustre: Skipped 1 previous similar message [10368.815368] LustreError: 322070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10368.841703] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [10368.841713] Lustre: Skipped 1 previous similar message [10368.854283] LustreError: 322070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10371.219455] Lustre: server umount lustre-MDT0001 complete [10378.575268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10388.511662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10389.471857] LustreError: 321384:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009c89c48f x1873187916785408/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10389.835240] LustreError: 323905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10389.856904] LustreError: 323905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10389.931505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10390.012881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10395.201118] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10399.145103] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10404.574144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10405.045036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10410.478363] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10410.586499] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10418.766165] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10420.783134] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10425.901897] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10427.726973] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10436.065459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10436.069480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10436.087789] LustreError: Skipped 1 previous similar message [10436.098362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10436.117472] Lustre: Skipped 1 previous similar message [10436.122425] Lustre: Skipped 1 previous similar message [10440.434252] Lustre: server umount lustre-OST0000 complete [10445.792145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10445.804684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10451.131495] Lustre: server umount lustre-MDT0000 complete [10451.429244] LustreError: 326398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.443374] LustreError: 326398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10454.677899] LustreError: 322056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411399 with bad export cookie 3618262981547319522 [10454.678996] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10454.692374] LustreError: 322056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10455.156512] Lustre: server umount lustre-MDT0001 complete [10467.632766] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10471.962592] Key type lgssc unregistered [10472.233508] LNet: 328363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10472.261232] LNetError: 328363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10472.276256] LNet: Removed LNI 192.168.201.152@tcp [10473.558183] Key type .llcrypt unregistered [10473.563442] Key type ._llcrypt unregistered [10501.369600] Key type ._llcrypt registered [10501.374517] Key type .llcrypt registered [10501.576312] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10503.002427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10503.052183] alg: No test for adler32 (adler32-zlib) [10504.218412] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [10504.586217] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10506.271202] Key type lgssc registered [10508.263606] Lustre: Echo OBD driver; http://www.lustre.org/ [10517.767635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10517.804140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.713111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10526.284273] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10529.257098] LustreError: 329781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10529.281696] LustreError: 329781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10534.369445] LustreError: 329780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10537.733533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10538.174996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10544.209435] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10550.668388] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10557.824656] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10563.553711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10563.565388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10563.605839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10564.070012] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10564.106954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10564.116937] Lustre: Skipped 1 previous similar message [10565.154895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10565.160982] Lustre: Skipped 1 previous similar message [10568.536362] Lustre: server umount lustre-MDT0000 complete [10570.211891] LustreError: 329776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.238818] LustreError: 329776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10574.190319] LustreError: 329761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411519 with bad export cookie 12627211093177031528 [10574.214148] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10574.774540] Lustre: server umount lustre-MDT0001 complete [10592.316625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.893853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10592.963403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10593.488919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10599.529593] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10610.413881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.804823] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10610.911844] Lustre: 332371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10611.554840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10616.727933] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10624.387301] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10630.976441] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10643.238476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10643.365082] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10643.370172] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10643.373204] LustreError: 334252:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10643.375725] LustreError: 334252:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10643.378791] LustreError: 334252:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10643.422610] Lustre: server umount lustre-OST0000 complete [10643.427437] LustreError: 334252:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [10654.844261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10655.187835] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10655.531331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10663.147273] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10663.230128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10672.693630] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10681.829460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10681.997081] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10682.092524] Lustre: lustre-OST0001: new disk, initializing [10682.097313] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10682.114984] Lustre: 335794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10682.233975] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10688.067686] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10691.167247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10691.182711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10691.321736] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10697.851912] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10706.913347] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10706.914793] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.930189] LustreError: Skipped 1 previous similar message [10706.932078] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10706.968848] Lustre: Skipped 1 previous similar message [10708.421536] Lustre: server umount lustre-OST0001 complete [10712.055934] LustreError: 336269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10712.078345] LustreError: 336269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10714.593442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10714.594939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10714.599915] LustreError: Skipped 1 previous similar message [10714.611467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10714.628836] Lustre: Skipped 1 previous similar message [10717.176392] LustreError: 334719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10717.214835] LustreError: 334719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10719.149910] Lustre: server umount lustre-OST0000 complete [10727.398692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10727.413629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10727.428024] Lustre: Skipped 2 previous similar messages [10727.460607] Lustre: Skipped 4 previous similar messages [10731.739258] Lustre: server umount lustre-MDT0000 complete [10732.513901] LustreError: 332381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10732.549443] LustreError: 332381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10736.438733] LustreError: 335793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411681 with bad export cookie 12627211093177032074 [10736.441207] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10736.457315] LustreError: 335793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10736.722838] Lustre: server umount lustre-MDT0001 complete [10748.699489] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10751.470767] Key type lgssc unregistered [10751.923503] LNet: 337755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10751.928309] LNetError: 337755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10753.017378] LNet: Removed LNI 192.168.201.152@tcp [10754.834307] Key type .llcrypt unregistered [10754.841316] Key type ._llcrypt unregistered [10837.385460] Key type ._llcrypt registered [10837.387730] Key type .llcrypt registered [10837.517705] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [10854.397558] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10856.047833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10856.065814] alg: No test for adler32 (adler32-zlib) [10857.184757] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [10857.598705] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10859.311552] Key type lgssc registered [10861.189145] Lustre: Echo OBD driver; http://www.lustre.org/ [10896.643061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10896.681302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10897.965799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10897.991173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10898.077246] Lustre: lustre-MDT0000: new disk, initializing [10898.159659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10898.176553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10902.846373] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10914.268044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.451036] Lustre: 342694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10914.513323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10914.518330] Lustre: Skipped 1 previous similar message [10914.577550] Lustre: lustre-MDT0001: new disk, initializing [10914.691349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10914.731582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10914.754829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10919.457988] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10931.057534] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10936.980972] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10943.890077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10944.273563] Lustre: lustre-OST0000: new disk, initializing [10944.278757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10944.284833] Lustre: 344660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10944.438063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10950.274158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10950.289486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10950.452673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10952.421505] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10962.847207] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10967.602770] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10967.860763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10972.596962] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10972.884588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10976.224709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10976.236292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10976.244951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10976.271644] Lustre: Skipped 1 previous similar message [10981.587174] Lustre: server umount lustre-OST0000 complete [10986.466056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10986.473173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10986.477390] Lustre: Skipped 1 previous similar message [10986.501818] Lustre: Skipped 2 previous similar messages [10991.588579] LustreError: 343830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10991.613216] LustreError: 343830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10991.831154] Lustre: server umount lustre-MDT0000 complete [10995.810582] LustreError: 342684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786411941 with bad export cookie 7443971860708702278 [10995.829814] LustreError: 342684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10995.831762] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10996.234164] Lustre: server umount lustre-MDT0001 complete [11006.022633] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:32:28 (1786411948) [11014.356549] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:32:36 (1786411956) [11025.070932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.642336] LustreError: 347417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11025.686939] LustreError: 347417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11025.813949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11031.012108] LustreError: 347418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11031.572555] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11035.109837] LustreError: 347417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.227849] LustreError: 347418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11041.041546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11041.554097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11046.759870] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11054.802634] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11060.787274] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11069.559449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.812161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11077.088470] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11084.695730] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11095.258531] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11098.020195] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11099.961898] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11102.188595] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11104.554559] Lustre: Failing over lustre-MDT0000 [11105.010978] Lustre: server umount lustre-MDT0000 complete [11106.795623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.828763] LustreError: 347417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11106.869832] LustreError: 347417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11122.150489] Lustre: 340919:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786412051/real 1786412051] req@000000000f086798 x1873188541768064/t0(0) o400->MGC192.168.201.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786412067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11122.197494] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11123.507834] LustreError: 348406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11123.534981] LustreError: 348406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [11124.266527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11132.731437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11132.847134] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11134.767807] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11138.069365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11138.118920] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11138.210282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11139.386776] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11150.622150] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11152.547498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11156.065309] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11159.118522] Lustre: Failing over lustre-MDT0000 [11159.407494] Lustre: server umount lustre-MDT0000 complete [11160.425790] LustreError: 347413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.435739] LustreError: 347413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [11163.619765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11163.624473] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11163.639883] LustreError: Skipped 1 previous similar message [11163.677046] Lustre: Skipped 4 previous similar messages [11178.976427] Lustre: 340918:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786412108/real 1786412108] req@000000002a36f548 x1873188541790336/t0(0) o400->MGC192.168.201.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786412124 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11178.998465] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11179.849333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11189.711503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11189.774170] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11192.107749] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11194.897936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11194.916397] Lustre: Skipped 2 previous similar messages [11194.929557] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11194.993475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11195.094655] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11205.400890] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11207.380105] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11215.328877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11215.333970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11215.340661] Lustre: Skipped 1 previous similar message [11215.351886] Lustre: Skipped 2 previous similar messages [11220.474535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11220.478933] Lustre: Skipped 2 previous similar messages [11221.182848] Lustre: server umount lustre-MDT0000 complete [11225.308753] LustreError: 347398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412170 with bad export cookie 7443971860708705001 [11225.313381] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11225.328048] LustreError: 347398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11225.571086] LustreError: 348406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11225.580618] LustreError: 348406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [11225.611526] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11225.624930] Lustre: Skipped 1 previous similar message [11225.932919] Lustre: server umount lustre-MDT0001 complete [11233.297887] Lustre: server umount lustre-OST0000 complete [11251.756426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11257.840964] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11270.911117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11271.474394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11271.483718] Lustre: Skipped 1 previous similar message [11277.255385] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11284.564562] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11290.417434] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11297.565089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11304.338964] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11312.039927] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11322.495806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11332.065407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11332.080056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11332.093513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11343.327622] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11343.348742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11343.359093] Lustre: Skipped 5 previous similar messages [11343.457472] Lustre: server umount lustre-OST0000 complete [11362.271146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11362.645903] Lustre: server umount lustre-MDT0000 complete [11363.808496] LustreError: 353756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11363.824737] LustreError: 353756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11366.347373] LustreError: 353742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412311 with bad export cookie 7443971860708705736 [11366.381355] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11366.823798] Lustre: server umount lustre-MDT0001 complete [11379.684779] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11383.033728] Key type lgssc unregistered [11383.322582] LNet: 357320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11383.335506] LNetError: 357320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11383.367487] LNet: Removed LNI 192.168.201.152@tcp [11384.192509] Key type .llcrypt unregistered [11384.196581] Key type ._llcrypt unregistered [11414.139586] Key type ._llcrypt registered [11414.141907] Key type .llcrypt registered [11414.328816] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11415.430345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11415.582286] alg: No test for adler32 (adler32-zlib) [11416.708815] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [11416.948526] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [11418.631413] Key type lgssc registered [11420.321212] Lustre: Echo OBD driver; http://www.lustre.org/ [11429.183876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11429.223185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.761406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11435.553301] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11440.102060] LustreError: 358836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.137944] LustreError: 358836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11444.283784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11444.761367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11450.658492] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11459.531578] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11465.969681] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11472.011289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11472.367829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11475.459463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11478.950795] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11487.253071] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11505.122653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11505.130983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11505.135923] Lustre: Skipped 1 previous similar message [11505.144811] Lustre: Skipped 2 previous similar messages [11510.258209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11510.266142] Lustre: Skipped 2 previous similar messages [11515.370441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11517.408652] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11517.760693] Lustre: server umount lustre-MDT0000 complete [11520.496893] LustreError: 358837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11520.529804] LustreError: 358837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11522.702922] LustreError: 358817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412467 with bad export cookie 10351436250352162024 [11522.725687] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11523.252694] Lustre: server umount lustre-MDT0001 complete [11537.530470] Lustre: server umount lustre-OST0000 complete [11551.012040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11551.673388] LustreError: 362570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.774727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11556.836548] LustreError: 362571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.375138] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11560.931277] LustreError: 362570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11571.087503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11571.169672] LustreError: 362571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11571.199597] LustreError: 362571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11571.475326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11578.136181] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11587.769775] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11594.097444] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11601.818745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11602.239121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11609.683284] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11619.645544] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11621.689214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11642.336720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11642.340125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11642.353897] Lustre: Skipped 2 previous similar messages [11642.360524] Lustre: Skipped 4 previous similar messages [11645.401722] Lustre: server umount lustre-MDT0000 complete [11647.456631] LustreError: 362571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11647.476597] LustreError: 362571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11648.749969] LustreError: 362550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412593 with bad export cookie 10351436250352163137 [11648.772237] LustreError: 362550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11648.773139] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11649.248374] Lustre: server umount lustre-MDT0001 complete [11663.324321] Lustre: server umount lustre-OST0000 complete [11680.981171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11681.473191] LustreError: 366486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.517313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11686.932258] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11696.786990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.281684] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11711.653307] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11718.310778] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11725.434436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11725.732610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11725.745505] Lustre: Skipped 1 previous similar message [11726.708029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11732.387731] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11742.593977] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11752.417047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11752.418665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11752.438934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11752.471859] Lustre: Skipped 2 previous similar messages [11755.651432] Lustre: server umount lustre-OST0000 complete [11762.657720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11762.660472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11762.666301] Lustre: Skipped 1 previous similar message [11762.675728] Lustre: Skipped 2 previous similar messages [11766.473763] Lustre: server umount lustre-MDT0000 complete [11767.781688] LustreError: 366486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.798091] LustreError: 366486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11770.393051] LustreError: 366467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786412715 with bad export cookie 10351436250352164159 [11770.405260] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11770.417696] LustreError: 366467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11770.779663] Lustre: server umount lustre-MDT0001 complete [11782.369717] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11785.230970] Key type lgssc unregistered [11785.571944] LNet: 370021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11785.580254] LNetError: 370021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11785.595957] LNet: Removed LNI 192.168.201.152@tcp [11786.418689] Key type .llcrypt unregistered [11786.420740] Key type ._llcrypt unregistered [11811.142135] Key type ._llcrypt registered [11811.143962] Key type .llcrypt registered [11811.231575] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11811.999835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11812.104164] alg: No test for adler32 (adler32-zlib) [11813.105329] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [11813.302573] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [11814.999255] Key type lgssc registered [11816.282658] Lustre: Echo OBD driver; http://www.lustre.org/ [11826.089882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11826.135093] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11826.150809] LustreError: 371461:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11826.176353] LustreError: 371461:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11826.181138] LustreError: 371461:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11826.185376] LustreError: 371461:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11826.190851] LustreError: 371461:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11834.735791] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11834.739418] LustreError: 371859:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11834.753555] LustreError: 371859:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11834.761824] LustreError: 371859:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11834.768524] LustreError: 371859:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11834.773232] LustreError: 371859:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11857.476735] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11861.094438] Key type lgssc unregistered [11861.438338] LNet: 372863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11861.463682] LNetError: 372863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11861.477702] LNet: Removed LNI 192.168.201.152@tcp [11862.818685] Key type .llcrypt unregistered [11862.825881] Key type ._llcrypt unregistered [11885.990243] Key type ._llcrypt registered [11885.993557] Key type .llcrypt registered [11886.119774] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [11902.773691] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11903.869975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11903.889667] alg: No test for adler32 (adler32-zlib) [11905.067605] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [11905.394568] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [11907.130756] Key type lgssc registered [11908.934630] Lustre: Echo OBD driver; http://www.lustre.org/ [11950.668499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11950.735232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11952.057854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11952.106863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11952.253514] Lustre: lustre-MDT0000: new disk, initializing [11952.421517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11952.449791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11957.816175] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11969.448597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11969.571886] Lustre: 376023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11969.621628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11969.625133] Lustre: Skipped 1 previous similar message [11969.700955] Lustre: lustre-MDT0001: new disk, initializing [11969.802245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11969.835566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11969.847561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11974.886796] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11985.322757] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11991.190319] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11997.866912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11998.168380] Lustre: lustre-OST0000: new disk, initializing [11998.171682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11998.181517] Lustre: 377994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11998.284657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12005.424801] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12005.452937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12005.465052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12005.621908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12017.335818] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12024.209627] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12024.615569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12030.468288] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12030.716536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12036.578432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12036.581256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12036.600691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12036.608336] Lustre: Skipped 1 previous similar message [12039.638019] Lustre: server umount lustre-OST0000 complete [12046.821102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12046.821236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.823351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.823357] Lustre: Skipped 1 previous similar message [12046.846451] Lustre: Skipped 1 previous similar message [12051.937512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12051.941515] Lustre: Skipped 2 previous similar messages [12056.039865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12056.060106] Lustre: Skipped 1 previous similar message [12059.103148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12059.457348] Lustre: server umount lustre-MDT0000 complete [12064.164722] LustreError: 376016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413009 with bad export cookie 17981185193922763961 [12064.166222] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12064.190938] LustreError: 376016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12065.181297] Lustre: server umount lustre-MDT0001 complete [12075.255735] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12077.070522] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:50:19 (1786413019) [12084.184081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.736390] LustreError: 380457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12084.758961] LustreError: 380457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12084.817752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12089.830417] LustreError: 380458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.931452] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12093.925514] LustreError: 380457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12099.043064] LustreError: 380458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12099.495834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12100.017164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.147625] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12113.776576] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12120.031415] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12127.809539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12128.102558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12134.304462] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12143.802334] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12152.810717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12153.145513] Lustre: lustre-OST0001: new disk, initializing [12153.150717] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12153.163370] Lustre: 383294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12153.274580] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12159.074561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12159.088996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12159.218612] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12161.024804] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12170.733204] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12174.818284] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12174.834888] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12174.836428] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12174.864794] LustreError: Skipped 1 previous similar message [12179.980698] LustreError: 383559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12180.029941] LustreError: 383559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12180.118975] Lustre: server umount lustre-OST0001 complete [12190.193956] LustreError: 382278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12190.210139] LustreError: 382278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12200.422854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.424715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12200.445415] Lustre: Skipped 2 previous similar messages [12200.455264] Lustre: Skipped 3 previous similar messages [12203.441660] Lustre: server umount lustre-OST0000 complete [12207.593857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12207.602817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12213.459209] Lustre: server umount lustre-MDT0000 complete [12215.778873] LustreError: 381440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12215.795392] LustreError: 381440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12217.775924] LustreError: 380440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413162 with bad export cookie 17981185193922764843 [12217.788505] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12217.789538] LustreError: 380440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12232.159173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12232.473930] Lustre: server umount lustre-MDT0001 complete [12245.899956] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12249.172895] Key type lgssc unregistered [12249.639083] LNet: 385286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12249.653388] LNetError: 385286:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12250.728929] LNet: Removed LNI 192.168.201.152@tcp [12251.864298] Key type .llcrypt unregistered [12251.872789] Key type ._llcrypt unregistered [12305.992287] Key type ._llcrypt registered [12305.994805] Key type .llcrypt registered [12306.107176] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12307.773695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12307.833123] alg: No test for adler32 (adler32-zlib) [12309.014427] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [12309.265628] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [12310.999226] Key type lgssc registered [12312.309621] Lustre: Echo OBD driver; http://www.lustre.org/ [12321.382769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12321.398767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12322.671663] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12322.684799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12322.817711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12327.303406] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12336.613086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12336.699239] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12336.721672] Lustre: 388086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12336.969713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12342.409755] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12349.754841] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12355.554627] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12357.600561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12357.603437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12357.612831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12358.626113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12358.626780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12363.820795] Lustre: server umount lustre-MDT0000 complete [12367.719627] LustreError: 388827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413312 with bad export cookie 13504848339756984515 [12367.730967] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12367.731535] LustreError: 388827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12368.275988] Lustre: server umount lustre-MDT0001 complete [12378.331707] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:55:20 (1786413320) [12385.854603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12386.493127] LustreError: 390600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.510250] LustreError: 390600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12386.584517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12391.613774] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12391.919560] LustreError: 390601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12397.026499] LustreError: 390600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12400.551737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12401.084722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12405.962514] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12413.490162] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12419.182176] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12426.202263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12426.424742] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12426.628273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12433.966710] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12441.352378] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12448.831757] Lustre: Permanently deactivating lustre-OST0000 [12448.858699] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12458.555704] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12459.785013] LustreError: 393636:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12459.794524] LustreError: 393636:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12459.801060] LustreError: 393636:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12463.083613] LustreError: 393684:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12463.088327] LustreError: 393684:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12471.049725] Lustre: server umount lustre-OST0000 complete [12473.592451] LustreError: 393934:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12473.606460] LustreError: 393934:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12477.920179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12477.923627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.932776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12477.936763] Lustre: Skipped 1 previous similar message [12483.553727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12483.567178] Lustre: Skipped 3 previous similar messages [12485.695345] Lustre: server umount lustre-MDT0000 complete [12488.676861] LustreError: 391326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12488.704955] LustreError: 391326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12489.339781] LustreError: 390580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413434 with bad export cookie 13504848339756985096 [12489.340879] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12489.349048] LustreError: 390580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12499.211410] Lustre: server umount lustre-MDT0001 complete [12507.824072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12513.383683] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12516.303455] LustreError: 395093:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12523.007202] Lustre: 395432:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.152@tcp [12524.668792] Lustre: server umount MGS complete [12536.287090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.681924] LustreError: 396157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12536.698356] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12536.702906] Lustre: Skipped 1 previous similar message [12536.776773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12540.711351] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12547.042634] LustreError: 396158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12547.067335] LustreError: 396158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12549.947113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12550.118386] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12554.731977] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12563.219810] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12569.582801] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12571.682755] Lustre: Permanently reactivating lustre-OST0000 [12571.700560] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12571.706422] Lustre: Skipped 2 previous similar messages [12598.758228] LustreError: 397938:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12606.013046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12606.683150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12606.698797] Lustre: Skipped 1 previous similar message [12613.691055] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12621.405631] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12629.482044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12629.488709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12629.509553] Lustre: Skipped 1 previous similar message [12629.512115] LustreError: 399128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12629.712575] Lustre: server umount lustre-OST0000 complete [12635.621777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12635.630860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12635.640621] Lustre: Skipped 1 previous similar message [12635.652402] Lustre: Skipped 1 previous similar message [12643.867733] Lustre: server umount lustre-MDT0000 complete [12647.758329] LustreError: 397732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413592 with bad export cookie 13504848339756986146 [12647.770602] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12647.771897] LustreError: 397732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12648.361767] Lustre: server umount lustre-MDT0001 complete [12661.830271] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12664.661620] Key type lgssc unregistered [12665.143988] LNet: 400473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12665.170837] LNetError: 400473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12665.194472] LNet: Removed LNI 192.168.201.152@tcp [12666.477248] Key type .llcrypt unregistered [12666.478877] Key type ._llcrypt unregistered [12687.073764] Key type ._llcrypt registered [12687.078258] Key type .llcrypt registered [12687.234802] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [12705.549936] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12708.703033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12708.732124] alg: No test for adler32 (adler32-zlib) [12710.051793] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [12710.451198] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [12712.231112] Key type lgssc registered [12713.654741] Lustre: Echo OBD driver; http://www.lustre.org/ [12750.884036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12750.920175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.336413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12752.422660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12752.648967] Lustre: lustre-MDT0000: new disk, initializing [12752.799788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12752.815770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12759.659434] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12773.045660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12773.163304] Lustre: 403642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12773.219868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12773.222290] Lustre: Skipped 1 previous similar message [12773.273892] Lustre: lustre-MDT0001: new disk, initializing [12773.359746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12773.410769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12773.427028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12778.754361] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12790.797285] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12798.732581] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12806.156600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12806.546188] Lustre: lustre-OST0000: new disk, initializing [12806.548258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12806.556120] Lustre: 405610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12806.707752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12813.839332] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12816.483656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12816.490933] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12816.633231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12824.786788] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12830.347535] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12830.657720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12835.035852] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12835.391252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12837.369815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12837.381604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12837.398441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12837.408639] Lustre: Skipped 1 previous similar message [12837.435549] Lustre: Skipped 1 previous similar message [12842.467700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12842.474924] Lustre: Skipped 1 previous similar message [12843.526816] Lustre: server umount lustre-OST0000 complete [12850.148722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12850.186726] LustreError: Skipped 1 previous similar message [12850.197858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12850.249529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12852.706975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12852.707534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12854.488606] Lustre: server umount lustre-MDT0000 complete [12858.771941] LustreError: 403634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413803 with bad export cookie 7610786219942477640 [12858.789753] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12859.147485] Lustre: server umount lustre-MDT0001 complete [12870.073959] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12872.611612] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:03:34 (1786413814) [12880.477292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12881.199192] LustreError: 408076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12881.216024] LustreError: 408076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12881.282299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12886.499335] LustreError: 408077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12888.085814] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12891.618261] LustreError: 408076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12896.753961] LustreError: 408077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12897.538245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12897.865196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12902.400662] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12908.515946] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12914.953661] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12923.148664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12923.430359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12930.888253] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12939.176558] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12941.429315] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12944.986550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12952.039492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12952.045621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12952.060581] LustreError: Skipped 1 previous similar message [12952.071528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12952.087326] Lustre: Skipped 2 previous similar messages [12956.073141] Lustre: server umount lustre-OST0000 complete [12962.274163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12962.294821] Lustre: Skipped 1 previous similar message [12962.311994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12966.908258] Lustre: server umount lustre-MDT0000 complete [12967.392689] LustreError: 409508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12967.403975] LustreError: 409508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12970.915627] LustreError: 408056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786413916 with bad export cookie 7610786219942478522 [12970.926081] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12970.930078] LustreError: 408056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12971.431829] Lustre: server umount lustre-MDT0001 complete [12984.207654] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12986.992728] Key type lgssc unregistered [12987.390871] LNet: 411802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12987.401199] LNetError: 411802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12988.455299] LNet: Removed LNI 192.168.201.152@tcp [12989.652327] Key type .llcrypt unregistered [12989.654342] Key type ._llcrypt unregistered [13015.003695] Key type ._llcrypt registered [13015.011924] Key type .llcrypt registered [13015.186355] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13015.617698] Key type .llcrypt unregistered [13015.622464] Key type ._llcrypt unregistered [13036.971564] Key type ._llcrypt registered [13036.973059] Key type .llcrypt registered [13037.158204] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13039.006521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13039.111801] alg: No test for adler32 (adler32-zlib) [13040.312432] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [13040.656285] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [13042.439373] Key type lgssc registered [13043.909335] Lustre: Echo OBD driver; http://www.lustre.org/ [13053.601245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13053.629358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13055.231114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13060.807902] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13065.703177] LustreError: 414018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13065.726223] LustreError: 414018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13070.724585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13070.822026] LustreError: 414019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.351261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13076.449880] LustreError: 414018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13076.468945] LustreError: 414018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13079.362308] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13080.552965] LustreError: 414019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13080.628869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13085.670402] LustreError: 414012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13085.694206] LustreError: 414012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13086.271275] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13095.665607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13095.912516] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13095.931525] Lustre: Skipped 1 previous similar message [13096.253868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13101.688854] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13111.305379] Lustre: server umount lustre-OST0000 complete [13116.899764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13116.905521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13116.907888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13122.017496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13122.031494] Lustre: Skipped 2 previous similar messages [13125.264268] Lustre: server umount lustre-MDT0000 complete [13127.138167] Lustre: 413328:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786414056/real 1786414056] req@0000000085b43c8d x1873190830762496/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786414072 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13127.143496] LustreError: 414018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13127.166463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13127.175863] LustreError: 414018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13127.206228] Lustre: Skipped 1 previous similar message [13129.933524] LustreError: 413999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414075 with bad export cookie 17750542625105602548 [13129.948379] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13130.230362] Lustre: server umount lustre-MDT0001 complete [13142.222200] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13145.449921] Key type lgssc unregistered [13145.820749] LNet: 417166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13145.833516] LNetError: 417166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13145.847675] LNet: Removed LNI 192.168.201.152@tcp [13146.907795] Key type .llcrypt unregistered [13146.913820] Key type ._llcrypt unregistered [13175.332780] Key type ._llcrypt registered [13175.335470] Key type .llcrypt registered [13175.505448] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13176.692177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13176.751377] alg: No test for adler32 (adler32-zlib) [13177.881747] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [13178.261287] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [13179.991245] Key type lgssc registered [13181.682735] Lustre: Echo OBD driver; http://www.lustre.org/ [13191.304924] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13191.346058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13206.943599] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13212.063428] LustreError: 418560:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [13238.815307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13245.131140] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13253.846809] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13262.905955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13263.639120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13263.725257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13267.791629] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13268.969136] LustreError: 418585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13268.989927] LustreError: 418585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13274.102558] LustreError: 419597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13274.145367] LustreError: 419597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13279.208781] LustreError: 419598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13279.236843] LustreError: 419598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13279.806249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13280.319294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13285.829362] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13293.649912] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13299.780587] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13309.922709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13309.950408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13309.974772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13310.968482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13310.974526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13310.983933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13314.759527] Lustre: server umount lustre-OST0000 complete [13321.193463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13321.203750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13321.208882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13321.220566] Lustre: Skipped 1 previous similar message [13321.238580] Lustre: Skipped 2 previous similar messages [13324.933817] Lustre: server umount lustre-MDT0000 complete [13326.306906] LustreError: 419602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13326.317712] LustreError: 419602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13328.822785] LustreError: 418567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414274 with bad export cookie 12155077039027596520 [13328.827088] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13328.831770] LustreError: 418567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13329.085579] Lustre: server umount lustre-MDT0001 complete [13340.936598] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13344.019320] Key type lgssc unregistered [13344.291555] LNet: 422128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13344.296902] LNetError: 422128:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13344.311326] LNet: Removed LNI 192.168.201.152@tcp [13345.255397] Key type .llcrypt unregistered [13345.263321] Key type ._llcrypt unregistered [13372.882044] Key type ._llcrypt registered [13372.884373] Key type .llcrypt registered [13372.999287] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13373.922258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13373.934194] alg: No test for adler32 (adler32-zlib) [13374.904551] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [13375.021124] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [13376.631278] Key type lgssc registered [13378.600168] Lustre: Echo OBD driver; http://www.lustre.org/ [13387.854500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13387.868096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13389.401557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13395.971451] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13399.538748] LustreError: 423548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13399.557938] LustreError: 423548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13404.649778] LustreError: 423547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13406.278164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13406.824060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13412.842680] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13422.186966] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13428.295783] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13435.880521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13436.267622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13443.734698] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13452.345764] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13459.845509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13468.133281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13468.148096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13468.170625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13469.523177] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [13470.178806] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13473.751288] Lustre: server umount lustre-MDT0000 complete [13474.617106] LustreError: 425388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13477.835419] LustreError: 423528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.52@tcp arrived at 1786414423 with bad export cookie 7879396551428981898 [13480.424368] LustreError: 424275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13480.455294] LustreError: 424275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13481.134965] LustreError: 423527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786414426 with bad export cookie 7879396551428981233 [13481.163070] LustreError: 423527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13481.171303] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13482.466533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13482.479663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13482.509349] Lustre: Skipped 1 previous similar message [13482.513411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13482.517262] Lustre: Skipped 3 previous similar messages [13485.537355] LustreError: 423548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13485.542406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13485.571105] LustreError: 423548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13487.533642] Lustre: server umount lustre-OST0000 complete [13496.800253] LustreError: 423542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13496.817620] LustreError: 423542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13497.573739] Lustre: server umount lustre-MDT0001 complete [13506.509705] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13509.619805] Key type lgssc unregistered [13509.925961] LNet: 427257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13509.936283] LNetError: 427257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13509.965244] LNet: Removed LNI 192.168.201.152@tcp [13511.616329] Key type .llcrypt unregistered [13511.619199] Key type ._llcrypt unregistered [13544.934479] Key type ._llcrypt registered [13544.940795] Key type .llcrypt registered [13545.110756] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13546.390521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13546.400554] alg: No test for adler32 (adler32-zlib) [13547.757616] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [13548.061372] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [13549.823301] Key type lgssc registered [13551.544621] Lustre: Echo OBD driver; http://www.lustre.org/ [13562.304544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13562.322694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13563.806588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13568.428743] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13573.089360] LustreError: 428676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13573.095224] LustreError: 428676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13577.465303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13578.285292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13584.000483] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13592.576982] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13598.495592] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13607.493240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13607.923349] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [13608.104062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13613.563600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13616.287866] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13626.009184] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13633.812124] Lustre: server umount lustre-MDT0001 complete [13633.889519] LustreError: 431187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13634.532080] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13634.547565] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13635.075104] LustreError: 429404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13635.077636] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13635.103741] LustreError: 429404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13640.166787] LustreError: 428677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13640.197410] LustreError: 428677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13644.258158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13644.261612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13644.273521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13645.282559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13645.283956] LustreError: 428676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13647.534770] Lustre: server umount lustre-OST0000 complete [13651.898374] Lustre: server umount lustre-MDT0000 complete [13671.246308] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13675.781967] Key type lgssc unregistered [13676.277141] LNet: 432361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13676.287120] LNetError: 432361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13676.307168] LNet: Removed LNI 192.168.201.152@tcp [13678.152844] Key type .llcrypt unregistered [13678.156361] Key type ._llcrypt unregistered [13711.528212] Key type ._llcrypt registered [13711.529767] Key type .llcrypt registered [13711.692831] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [13731.526595] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13732.857279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13732.887991] alg: No test for adler32 (adler32-zlib) [13734.342293] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [13734.852647] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [13736.623228] Key type lgssc registered [13738.545855] Lustre: Echo OBD driver; http://www.lustre.org/ [13789.507814] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13813.368435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13813.434428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13814.871416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13814.945418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13815.066504] Lustre: lustre-MDT0000: new disk, initializing [13815.142362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13815.160227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13820.571123] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13837.948524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13838.294122] Lustre: lustre-OST0000: new disk, initializing [13838.296790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13838.302457] Lustre: Skipped 1 previous similar message [13838.310838] Lustre: 437404:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13838.409751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13845.345964] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13847.691418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13847.701472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13847.731288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13855.297453] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13870.707568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13870.863909] Lustre: 436384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13870.937071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13870.985235] Lustre: lustre-MDT0001: new disk, initializing [13871.165378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13871.224063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13871.242354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13877.819289] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13888.516135] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13888.751671] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13893.702880] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13893.953767] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13899.595441] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [13900.265359] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13900.270269] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13900.318602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13904.699298] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [13904.715514] Lustre: Skipped 1 previous similar message [13905.889696] Lustre: server umount lustre-MDT0001 complete [13915.635251] LustreError: 436394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13915.662043] LustreError: 436394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13920.739814] LustreError: 436397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13920.768120] LustreError: 436397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13925.858795] LustreError: 436398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13925.860792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13925.906994] Lustre: Skipped 1 previous similar message [13925.910896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13925.917573] Lustre: Skipped 2 previous similar messages [13930.894243] Lustre: server umount lustre-OST0000 complete [13930.979737] LustreError: 436397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13941.722095] Lustre: server umount lustre-MDT0000 complete [13957.503680] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13960.931638] Key type lgssc unregistered [13961.346855] LNet: 441155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13961.357967] LNetError: 441155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13961.385400] LNet: Removed LNI 192.168.201.152@tcp [13962.534993] Key type .llcrypt unregistered [13962.537870] Key type ._llcrypt unregistered [14009.724860] Key type ._llcrypt registered [14009.731443] Key type .llcrypt registered [14009.965838] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [14010.298742] Key type .llcrypt unregistered [14010.300437] Key type ._llcrypt unregistered [14029.240816] Key type ._llcrypt registered [14029.242630] Key type .llcrypt registered [14029.405635] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [14030.905937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14031.110595] alg: No test for adler32 (adler32-zlib) [14032.190644] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [14032.614798] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [14034.279418] Key type lgssc registered [14035.531228] Lustre: Echo OBD driver; http://www.lustre.org/ [14065.213414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14065.233591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14066.463469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14066.487382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14066.577481] Lustre: lustre-MDT0000: new disk, initializing [14066.649961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14066.667385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14071.648385] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14082.771213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14082.969145] Lustre: 444536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14083.029310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14083.033160] Lustre: Skipped 1 previous similar message [14083.104720] Lustre: lustre-MDT0001: new disk, initializing [14083.233370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14083.271541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14083.283769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14087.897443] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14098.140370] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14104.641924] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14112.348102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14112.672825] Lustre: lustre-OST0000: new disk, initializing [14112.677545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14112.688365] Lustre: 446504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14112.772040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14120.227956] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14121.591295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14121.600000] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14121.828633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14130.560441] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14139.363052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14139.374849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14139.394335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14142.434955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14142.436127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14142.448128] Lustre: Skipped 1 previous similar message [14142.479360] Lustre: Skipped 2 previous similar messages [14144.188357] Lustre: server umount lustre-MDT0000 complete [14148.851539] LustreError: 444527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415094 with bad export cookie 6431572572424894265 [14148.854711] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14148.859703] LustreError: 444527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14149.595482] Lustre: server umount lustre-MDT0001 complete [14165.643635] Lustre: server umount lustre-OST0000 complete [14178.484328] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:25:21 (1786415121) [14187.436074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14187.889852] LustreError: 448535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14187.916590] LustreError: 448535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14188.041892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14193.121884] LustreError: 448536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14193.143472] LustreError: 448536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14194.285938] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14198.242141] LustreError: 448535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14203.362390] LustreError: 448536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14204.508172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14205.069241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14211.021794] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14219.860590] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14227.034344] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14247.475337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14247.755067] Lustre: Found index 0 for lustre-OST0000, updating log [14248.066811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14255.985245] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14259.329573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14264.886619] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14275.040467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14275.049576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14275.062670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14275.074449] Lustre: Skipped 1 previous similar message [14275.085959] Lustre: Skipped 1 previous similar message [14277.961345] Lustre: server umount lustre-OST0000 complete [14297.058464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14297.066612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14297.092473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14297.101583] Lustre: Skipped 2 previous similar messages [14300.253969] Lustre: server umount lustre-MDT0000 complete [14302.688818] LustreError: 448531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14302.723035] LustreError: 448531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14304.469175] LustreError: 448517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415249 with bad export cookie 6431572572424895658 [14304.485748] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14304.490688] LustreError: 448517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14304.942523] Lustre: server umount lustre-MDT0001 complete [14308.127150] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [14314.619579] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:27:37 (1786415257) [14943.226281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14943.866186] LustreError: 454947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14943.895596] LustreError: 454947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14943.991414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14949.345025] LustreError: 454948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14950.132081] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14954.465517] LustreError: 454947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14960.608659] LustreError: 454948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14960.622621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14960.634031] LustreError: 454948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14961.130914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14966.669245] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14975.182758] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14981.476599] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14999.327718] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [15002.474768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15002.810922] Lustre: Found index 0 for lustre-OST0000, updating log [15003.308065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15009.968529] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15017.528106] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15020.004258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15020.021138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15020.042894] Lustre: Skipped 1 previous similar message [15020.051838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15020.065642] Lustre: Skipped 1 previous similar message [15025.925479] Lustre: server umount lustre-OST0000 complete [15030.628527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15030.653286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15031.713693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15031.724415] Lustre: Skipped 2 previous similar messages [15033.763232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15033.770383] Lustre: Skipped 3 previous similar messages [15036.230425] Lustre: server umount lustre-MDT0000 complete [15036.836246] LustreError: 454943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15036.865052] LustreError: 454943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15041.715889] LustreError: 455681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786415986 with bad export cookie 6431572572424896743 [15041.726041] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15041.733748] LustreError: 455681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15042.056441] Lustre: server umount lustre-MDT0001 complete [15059.745372] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [15076.886689] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [15088.529559] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [15131.651102] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:41:14 (1786416074) [15152.797458] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [15155.704781] Key type lgssc unregistered [15155.994636] LNet: 461565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15156.002388] LNetError: 461565:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15263.540198] LNet: Removed LNI 192.168.201.152@tcp [15264.820192] Key type .llcrypt unregistered [15264.821980] Key type ._llcrypt unregistered [15286.712523] Key type ._llcrypt registered [15286.717244] Key type .llcrypt registered [15286.907341] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [15288.794978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15288.825841] alg: No test for adler32 (adler32-zlib) [15289.951222] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [15290.291451] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [15292.016586] Key type lgssc registered [15293.258591] Lustre: Echo OBD driver; http://www.lustre.org/ [15297.991713] LNet: Added LNI 192.168.201.152@tcp100 [8/256/0/180] [15299.091921] LNet: Added LNI 192.168.201.152@tcp101 [8/256/0/180] [15299.910701] LNet: Added LNI 192.168.201.152@tcp102 [8/256/0/180] [15300.879431] LNet: Added LNI 192.168.201.152@tcp103 [8/256/0/180] [15301.809642] LNet: Added LNI 192.168.201.152@tcp104 [8/256/0/180] [15302.843933] LNet: Added LNI 192.168.201.152@tcp105 [8/256/0/180] [15303.876111] LNet: Added LNI 192.168.201.152@tcp106 [8/256/0/180] [15304.901551] LNet: Added LNI 192.168.201.152@tcp107 [8/256/0/180] [15306.032657] LNet: Added LNI 192.168.201.152@tcp108 [8/256/0/180] [15306.849320] LNet: Added LNI 192.168.201.152@tcp109 [8/256/0/180] [15307.942536] LNet: Added LNI 192.168.201.152@tcp110 [8/256/0/180] [15309.098104] LNet: Added LNI 192.168.201.152@tcp111 [8/256/0/180] [15310.199634] LNet: Added LNI 192.168.201.152@tcp112 [8/256/0/180] [15311.236841] LNet: Added LNI 192.168.201.152@tcp113 [8/256/0/180] [15312.091652] LNet: Added LNI 192.168.201.152@tcp114 [8/256/0/180] [15312.885461] LNet: Added LNI 192.168.201.152@tcp115 [8/256/0/180] [15313.768254] LNet: Added LNI 192.168.201.152@tcp116 [8/256/0/180] [15314.783099] LNet: Added LNI 192.168.201.152@tcp117 [8/256/0/180] [15315.833022] LNet: Added LNI 192.168.201.152@tcp118 [8/256/0/180] [15317.080877] LNet: Added LNI 192.168.201.152@tcp119 [8/256/0/180] [15318.080466] LNet: Added LNI 192.168.201.152@tcp120 [8/256/0/180] [15319.003773] LNet: Added LNI 192.168.201.152@tcp121 [8/256/0/180] [15320.054197] LNet: Added LNI 192.168.201.152@tcp122 [8/256/0/180] [15321.010903] LNet: Added LNI 192.168.201.152@tcp123 [8/256/0/180] [15321.964078] LNet: Added LNI 192.168.201.152@tcp124 [8/256/0/180] [15322.924109] LNet: Added LNI 192.168.201.152@tcp125 [8/256/0/180] [15323.965205] LNet: Added LNI 192.168.201.152@tcp126 [8/256/0/180] [15324.803501] LNet: Added LNI 192.168.201.152@tcp127 [8/256/0/180] [15325.875772] LNet: Added LNI 192.168.201.152@tcp128 [8/256/0/180] [15326.989153] LNet: Added LNI 192.168.201.152@tcp129 [8/256/0/180] [15327.884770] LNet: Added LNI 192.168.201.152@tcp130 [8/256/0/180] [15328.860438] LNet: Added LNI 192.168.201.152@tcp131 [8/256/0/180] [15330.057817] LNet: Added LNI 192.168.201.152@tcp132 [8/256/0/180] [15331.023521] LNet: Added LNI 192.168.201.152@tcp133 [8/256/0/180] [15332.164038] LNet: Added LNI 192.168.201.152@tcp134 [8/256/0/180] [15333.352448] LNet: Added LNI 192.168.201.152@tcp135 [8/256/0/180] [15341.330950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15341.382953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15342.529381] Lustre: 464597:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [15342.638668] LNetError: 464618:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [15342.657657] LNetError: Protocol error connecting to 192.168.201.152@tcp119 at host 192.168.201.152:988: is it running a compatible version of Lustre? [15342.674464] LNetError: Skipped 3 previous similar messages [15342.848547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15342.926265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15343.084769] Lustre: lustre-MDT0000: new disk, initializing [15343.159383] Lustre: 464654:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [15343.163573] Lustre: 464654:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15343.193540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15343.257594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15343.583982] LNet: 5 local NIs in recovery (showing 5): 192.168.201.152@tcp125, 192.168.201.152@tcp126, 192.168.201.152@tcp127, 192.168.201.152@tcp129, 192.168.201.152@tcp119 [15343.585407] LNetError: 464623:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [15343.613721] LNetError: 464623:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [15343.627893] LNetError: Protocol error connecting to 192.168.201.152@tcp116 at host 192.168.201.152:988: is it running a compatible version of Lustre? [15343.643185] LNetError: Skipped 7 previous similar messages [15349.277577] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15362.244742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15362.415197] Lustre: 464614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15362.481151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15362.484163] Lustre: Skipped 1 previous similar message [15362.565186] Lustre: lustre-MDT0001: new disk, initializing [15362.643329] Lustre: 465416:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [15362.649542] Lustre: 465416:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15362.836057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15362.888290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15362.901909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15369.003238] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15369.836063] Lustre: 465702:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [15369.852551] Lustre: 465702:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15372.258274] LNetError: 464618:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [15372.271619] LNetError: Protocol error connecting to 192.168.201.152@tcp118 at host 192.168.201.152:988: is it running a compatible version of Lustre? [15374.319660] LNetError: 464617:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [15374.334817] LNetError: Protocol error connecting to 192.168.201.152@tcp121 at host 192.168.201.152:988: is it running a compatible version of Lustre? [15378.402360] LNetError: 464628:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [15378.419675] LNetError: 464628:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15378.442889] LNetError: Protocol error connecting to 192.168.201.152@tcp114 at host 192.168.201.152:988: is it running a compatible version of Lustre? [15378.457131] LNetError: Skipped 2 previous similar messages [15380.703822] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15386.927540] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15387.619110] LNetError: 462056:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [15387.632059] LNetError: 462056:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [15387.664180] LNetError: Protocol error connecting to 192.168.201.152@tcp121 at host 192.168.201.152:988: is it running a compatible version of Lustre? [15387.673665] LNetError: Skipped 4 previous similar messages [15395.677416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15396.302078] Lustre: lustre-OST0000: new disk, initializing [15396.307958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15396.320162] Lustre: 466592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15396.474670] Lustre: 466597:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [15396.482590] Lustre: 466597:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15396.563443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15405.026977] LNetError: 462056:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [15405.052510] LNetError: 462056:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15405.061894] LNetError: Protocol error connecting to 192.168.201.152@tcp114 at host 192.168.201.152:988: is it running a compatible version of Lustre? [15405.086660] LNetError: Skipped 10 previous similar messages [15406.107267] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15406.151473] Lustre: 466945:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [15406.161304] Lustre: 466945:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15406.254839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15406.268774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15406.518154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15417.608410] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15427.047116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15427.048213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15427.058312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15427.073697] Lustre: Skipped 1 previous similar message [15427.099705] Lustre: Skipped 1 previous similar message [15432.164020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15432.185351] Lustre: Skipped 1 previous similar message [15437.284277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15437.287749] Lustre: Skipped 1 previous similar message [15438.816812] LNetError: 462056:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [15438.835693] LNetError: 462056:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15438.845522] LNetError: Protocol error connecting to 192.168.201.152@tcp121 at host 192.168.201.152:988: is it running a compatible version of Lustre? [15438.874326] LNetError: Skipped 10 previous similar messages [15441.375260] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15441.568667] Lustre: server umount lustre-OST0000 complete [15447.549888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15447.581982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15451.738168] Lustre: server umount lustre-MDT0000 complete [15455.530794] LustreError: 464605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786416400 with bad export cookie 6114144587227667600 [15455.534808] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15455.538657] LustreError: 464605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15455.800075] Lustre: server umount lustre-MDT0001 complete [15466.311379] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [15468.973321] Key type lgssc unregistered [15469.293374] LNet: 468506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15469.305355] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15470.367200] LNet: Removed LNI 192.168.201.152@tcp [15470.371327] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15470.379877] LNet: Removed LNI 192.168.201.152@tcp100 [15471.391173] LNet: Removed LNI 192.168.201.152@tcp101 [15471.398134] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15471.404442] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15471.409276] LNet: Removed LNI 192.168.201.152@tcp102 [15472.416591] LNet: Removed LNI 192.168.201.152@tcp103 [15473.439182] LNet: Removed LNI 192.168.201.152@tcp104 [15473.471385] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15473.484296] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15474.529497] LNet: Removed LNI 192.168.201.152@tcp105 [15475.551878] LNet: Removed LNI 192.168.201.152@tcp106 [15476.575770] LNet: Removed LNI 192.168.201.152@tcp107 [15477.599217] LNet: Removed LNI 192.168.201.152@tcp108 [15477.610639] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15477.620025] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [15478.687660] LNet: Removed LNI 192.168.201.152@tcp109 [15479.712655] LNet: Removed LNI 192.168.201.152@tcp110 [15480.736073] LNet: Removed LNI 192.168.201.152@tcp111 [15481.761736] LNet: Removed LNI 192.168.201.152@tcp112 [15482.783276] LNet: Removed LNI 192.168.201.152@tcp113 [15483.807440] LNet: Removed LNI 192.168.201.152@tcp114 [15484.831245] LNet: Removed LNI 192.168.201.152@tcp115 [15485.855925] LNet: Removed LNI 192.168.201.152@tcp116 [15485.862420] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15485.881267] LNetError: 468506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [15485.896229] LNet: Removed LNI 192.168.201.152@tcp117 [15486.946138] LNet: Removed LNI 192.168.201.152@tcp118 [15487.967265] LNet: Removed LNI 192.168.201.152@tcp119 [15488.991191] LNet: Removed LNI 192.168.201.152@tcp120 [15490.015533] LNet: Removed LNI 192.168.201.152@tcp121 [15491.040159] LNet: Removed LNI 192.168.201.152@tcp122 [15492.127228] LNet: Removed LNI 192.168.201.152@tcp123 [15493.152107] LNet: Removed LNI 192.168.201.152@tcp124 [15494.175206] LNet: Removed LNI 192.168.201.152@tcp125 [15495.199193] LNet: Removed LNI 192.168.201.152@tcp126 [15496.223547] LNet: Removed LNI 192.168.201.152@tcp127 [15497.248940] LNet: Removed LNI 192.168.201.152@tcp128 [15498.271291] LNet: Removed LNI 192.168.201.152@tcp129 [15498.275820] LNet: Removed LNI 192.168.201.152@tcp130 [15498.279229] LNet: Removed LNI 192.168.201.152@tcp131 [15498.283785] LNet: Removed LNI 192.168.201.152@tcp132 [15498.291250] LNet: Removed LNI 192.168.201.152@tcp133 [15498.294310] LNet: Removed LNI 192.168.201.152@tcp134 [15498.306100] LNet: Removed LNI 192.168.201.152@tcp135 [15499.154553] Key type .llcrypt unregistered [15499.157093] Key type ._llcrypt unregistered [15527.947400] Key type ._llcrypt registered [15527.951928] Key type .llcrypt registered [15528.126307] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [15546.781617] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [15548.536236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15548.831625] alg: No test for adler32 (adler32-zlib) [15549.947972] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [15550.327250] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [15552.143211] Key type lgssc registered [15554.130769] Lustre: Echo OBD driver; http://www.lustre.org/ [15593.928389] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:48:56 (1786416536) [15614.397553] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [15617.377531] Key type lgssc unregistered [15617.699451] LNet: 472598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15617.706903] LNetError: 472598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15617.729898] LNet: Removed LNI 192.168.201.152@tcp [15618.697352] Key type .llcrypt unregistered [15618.700171] Key type ._llcrypt unregistered [15638.048625] Key type ._llcrypt registered [15638.050712] Key type .llcrypt registered [15638.170609] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [15639.395685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15639.504357] alg: No test for adler32 (adler32-zlib) [15640.682166] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [15640.998600] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [15642.727558] Key type lgssc registered [15644.053247] Lustre: Echo OBD driver; http://www.lustre.org/ [15656.416920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15656.449916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15657.775369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15657.826388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15657.993494] Lustre: lustre-MDT0000: new disk, initializing [15658.197247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15658.289525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15663.866594] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15676.005336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15676.145184] Lustre: 473821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15676.209644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15676.212665] Lustre: Skipped 1 previous similar message [15676.313184] Lustre: lustre-MDT0001: new disk, initializing [15676.454930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15676.499199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15676.512807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15682.276542] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15692.091257] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15699.181713] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15707.252809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15707.597805] Lustre: lustre-OST0000: new disk, initializing [15707.606780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15707.615603] Lustre: 475789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15707.714814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15715.387228] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15716.449543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15716.462509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15716.734584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15726.321803] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15731.698741] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing lnet_if_list [15737.516433] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15737.588623] LNet: Added LNI 192.168.201.152@tcp10 [8/256/0/180] [15744.947502] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15745.010191] LNet: Added LNI 192.168.201.152@tcp11 [8/256/0/180] [15750.508459] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15750.573274] LNet: Added LNI 192.168.201.152@tcp12 [8/256/0/180] [15755.217902] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15755.249561] LNet: Added LNI 192.168.201.152@tcp13 [8/256/0/180] [15760.108328] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15760.129039] LNet: Added LNI 192.168.201.152@tcp14 [8/256/0/180] [15765.175722] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15765.230988] LNet: Added LNI 192.168.201.152@tcp15 [8/256/0/180] [15770.904889] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15770.955583] LNet: Added LNI 192.168.201.152@tcp16 [8/256/0/180] [15776.393235] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15776.419325] LNet: Added LNI 192.168.201.152@tcp17 [8/256/0/180] [15782.816450] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15782.875679] LNet: Added LNI 192.168.201.152@tcp18 [8/256/0/180] [15791.882305] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15791.944719] LNet: Added LNI 192.168.201.152@tcp19 [8/256/0/180] [15799.136387] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15799.170269] LNet: Added LNI 192.168.201.152@tcp20 [8/256/0/180] [15805.852311] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15805.880179] LNet: Added LNI 192.168.201.152@tcp21 [8/256/0/180] [15813.917729] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15813.973758] LNet: Added LNI 192.168.201.152@tcp22 [8/256/0/180] [15821.051312] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15821.091043] LNet: Added LNI 192.168.201.152@tcp23 [8/256/0/180] [15828.911958] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15828.954309] LNet: Added LNI 192.168.201.152@tcp24 [8/256/0/180] [15835.566903] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15835.608296] LNet: Added LNI 192.168.201.152@tcp25 [8/256/0/180] [15842.812637] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15842.852830] LNet: Added LNI 192.168.201.152@tcp26 [8/256/0/180] [15849.994306] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15850.022383] LNet: Added LNI 192.168.201.152@tcp27 [8/256/0/180] [15857.940200] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15857.994934] LNet: Added LNI 192.168.201.152@tcp28 [8/256/0/180] [15865.728051] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15865.755811] LNet: Added LNI 192.168.201.152@tcp29 [8/256/0/180] [15873.272147] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15873.303212] LNet: Added LNI 192.168.201.152@tcp30 [8/256/0/180] [15881.901767] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15881.933759] LNet: Added LNI 192.168.201.152@tcp31 [8/256/0/180] [15889.886687] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15889.922336] LNet: Added LNI 192.168.201.152@tcp32 [8/256/0/180] [15897.233582] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15897.253469] LNet: Added LNI 192.168.201.152@tcp33 [8/256/0/180] [15904.457065] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15904.498657] LNet: Added LNI 192.168.201.152@tcp34 [8/256/0/180] [15912.940905] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15912.968633] LNet: Added LNI 192.168.201.152@tcp35 [8/256/0/180] [15923.954414] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15923.982385] LNet: Added LNI 192.168.201.152@tcp36 [8/256/0/180] [15936.551398] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15936.590539] LNet: Added LNI 192.168.201.152@tcp37 [8/256/0/180] [15944.120731] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15944.174916] LNet: Added LNI 192.168.201.152@tcp38 [8/256/0/180] [15953.211608] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15953.267035] LNet: Added LNI 192.168.201.152@tcp39 [8/256/0/180] [15962.317958] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15962.349687] LNet: Added LNI 192.168.201.152@tcp40 [8/256/0/180] [15970.179191] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15970.211599] LNet: Added LNI 192.168.201.152@tcp41 [8/256/0/180] [15977.498307] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15977.535236] LNet: Added LNI 192.168.201.152@tcp42 [8/256/0/180] [15986.409978] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15986.458575] LNet: Added LNI 192.168.201.152@tcp43 [8/256/0/180] [15996.500716] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15996.549892] LNet: Added LNI 192.168.201.152@tcp44 [8/256/0/180] [16006.731125] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16006.785775] LNet: Added LNI 192.168.201.152@tcp45 [8/256/0/180] [16016.309710] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16016.349988] LNet: Added LNI 192.168.201.152@tcp46 [8/256/0/180] [16025.488624] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16025.520904] LNet: Added LNI 192.168.201.152@tcp47 [8/256/0/180] [16036.009656] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16036.074043] LNet: Added LNI 192.168.201.152@tcp48 [8/256/0/180] [16045.554834] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16045.646506] LNet: Added LNI 192.168.201.152@tcp49 [8/256/0/180] [16056.027392] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16056.060505] LNet: Added LNI 192.168.201.152@tcp50 [8/256/0/180] [16075.745556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16075.750269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16075.766592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16078.305255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16078.317573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16078.340238] Lustre: Skipped 1 previous similar message [16081.638856] Lustre: server umount lustre-MDT0000 complete [16085.605884] LustreError: 473815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417030 with bad export cookie 11108041138815363607 [16085.612740] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16085.626806] LustreError: 473815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16086.065136] Lustre: server umount lustre-MDT0001 complete [16103.903415] Lustre: 473095:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417033/real 1786417033] req@00000000acf89867 x1873193558274432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16103.941852] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16105.024797] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16105.951163] Lustre: 473092:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417035/real 1786417035] req@00000000fbd62c78 x1873193558274560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417051 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16109.025296] Lustre: 473093:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417038/real 1786417038] req@000000008c9eee42 x1873193558274688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16111.135404] Lustre: 473095:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417040/real 1786417040] req@00000000331c3234 x1873193558274944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417056 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16114.935729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16117.343242] Lustre: 473095:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786417046/real 1786417046] req@0000000042c08b77 x1873193558275072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786417062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16130.593044] LustreError: 473091:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fa742a23 x1873193558275840/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16130.911697] LustreError: 475782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16130.932905] LustreError: 475782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [16131.021724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16133.027523] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16133.035769] LustreError: 475783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16134.060112] LustreError: 475783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16135.955405] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16139.235108] LustreError: 475782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16144.354456] LustreError: 475783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16144.751735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16145.202389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16149.260655] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16149.318627] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16156.646295] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16163.483104] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16187.365757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16187.380811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16187.398509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16187.412078] Lustre: Skipped 2 previous similar messages [16190.449805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16190.454630] Lustre: Skipped 1 previous similar message [16195.552951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16195.564696] Lustre: Skipped 1 previous similar message [16201.183254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16201.370602] Lustre: server umount lustre-OST0000 complete [16205.792593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16205.797634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16205.809768] Lustre: Skipped 2 previous similar messages [16205.823839] Lustre: Skipped 3 previous similar messages [16211.913577] Lustre: server umount lustre-MDT0000 complete [16215.960362] LustreError: 473815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417161 with bad export cookie 11108041138815366253 [16215.971012] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16215.990427] LustreError: 473815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16216.576717] Lustre: server umount lustre-MDT0001 complete [16229.087431] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [16231.813183] Key type lgssc unregistered [16232.195245] LNet: 502845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16232.228902] LNetError: 502845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16232.239391] LNet: Removed LNI 192.168.201.152@tcp [16232.249309] LNet: Removed LNI 192.168.201.152@tcp10 [16232.251330] LNet: Removed LNI 192.168.201.152@tcp11 [16232.261935] LNet: Removed LNI 192.168.201.152@tcp12 [16232.266915] LNet: Removed LNI 192.168.201.152@tcp13 [16232.274249] LNet: Removed LNI 192.168.201.152@tcp14 [16232.279818] LNet: Removed LNI 192.168.201.152@tcp15 [16232.290856] LNet: Removed LNI 192.168.201.152@tcp16 [16232.293545] LNet: Removed LNI 192.168.201.152@tcp17 [16232.299637] LNet: Removed LNI 192.168.201.152@tcp18 [16232.303799] LNet: Removed LNI 192.168.201.152@tcp19 [16232.306532] LNet: Removed LNI 192.168.201.152@tcp20 [16232.309296] LNet: Removed LNI 192.168.201.152@tcp21 [16232.314259] LNet: Removed LNI 192.168.201.152@tcp22 [16232.316758] LNet: Removed LNI 192.168.201.152@tcp23 [16232.319244] LNet: Removed LNI 192.168.201.152@tcp24 [16232.321512] LNet: Removed LNI 192.168.201.152@tcp25 [16232.323938] LNet: Removed LNI 192.168.201.152@tcp26 [16232.327534] LNet: Removed LNI 192.168.201.152@tcp27 [16232.330408] LNet: Removed LNI 192.168.201.152@tcp28 [16232.332805] LNet: Removed LNI 192.168.201.152@tcp29 [16232.335497] LNet: Removed LNI 192.168.201.152@tcp30 [16232.337961] LNet: Removed LNI 192.168.201.152@tcp31 [16232.340540] LNet: Removed LNI 192.168.201.152@tcp32 [16232.343432] LNet: Removed LNI 192.168.201.152@tcp33 [16232.345880] LNet: Removed LNI 192.168.201.152@tcp34 [16232.348420] LNet: Removed LNI 192.168.201.152@tcp35 [16232.354782] LNet: Removed LNI 192.168.201.152@tcp36 [16232.360695] LNet: Removed LNI 192.168.201.152@tcp37 [16232.363569] LNet: Removed LNI 192.168.201.152@tcp38 [16232.368472] LNet: Removed LNI 192.168.201.152@tcp39 [16232.373070] LNet: Removed LNI 192.168.201.152@tcp40 [16232.377118] LNet: Removed LNI 192.168.201.152@tcp41 [16232.382256] LNet: Removed LNI 192.168.201.152@tcp42 [16232.389258] LNet: Removed LNI 192.168.201.152@tcp43 [16232.393723] LNet: Removed LNI 192.168.201.152@tcp44 [16232.397300] LNet: Removed LNI 192.168.201.152@tcp45 [16232.408286] LNet: Removed LNI 192.168.201.152@tcp46 [16232.410245] LNet: Removed LNI 192.168.201.152@tcp47 [16232.415828] LNet: Removed LNI 192.168.201.152@tcp48 [16232.422269] LNet: Removed LNI 192.168.201.152@tcp49 [16232.431394] LNet: Removed LNI 192.168.201.152@tcp50 [16233.208582] Key type .llcrypt unregistered [16233.210579] Key type ._llcrypt unregistered [16253.922346] Key type ._llcrypt registered [16253.926172] Key type .llcrypt registered [16254.126457] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [16271.388222] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16272.880194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16272.927850] alg: No test for adler32 (adler32-zlib) [16274.018973] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [16274.227061] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [16275.936859] Key type lgssc registered [16277.479777] Lustre: Echo OBD driver; http://www.lustre.org/ [16316.467212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16316.492252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16317.846154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16317.866638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16317.934468] Lustre: lustre-MDT0000: new disk, initializing [16318.022101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16318.055599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16322.535666] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16334.280619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16334.458424] Lustre: 506058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16334.578281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16334.583251] Lustre: Skipped 1 previous similar message [16334.696408] Lustre: lustre-MDT0001: new disk, initializing [16334.824154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16334.885122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16334.903940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16339.980597] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16349.329516] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16357.092795] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16366.534468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16366.785855] Lustre: *** cfs_fail_loc=90f, val=0*** [16366.880699] Lustre: lustre-OST0000: new disk, initializing [16366.885849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16366.893695] Lustre: 508073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16366.986118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16373.346321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16373.355678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16373.506655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16374.039849] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16384.426652] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16394.209403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16394.215235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16394.224336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16394.235732] Lustre: Skipped 1 previous similar message [16394.252039] Lustre: Skipped 1 previous similar message [16398.302628] Lustre: server umount lustre-OST0000 complete [16404.455339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16404.465815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16404.470604] Lustre: Skipped 1 previous similar message [16408.395626] Lustre: server umount lustre-MDT0000 complete [16411.920849] LustreError: 506048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417357 with bad export cookie 5258067676996344199 [16411.942448] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16411.948624] LustreError: 506048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16412.638770] Lustre: server umount lustre-MDT0001 complete [16423.680880] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [16426.603156] Key type lgssc unregistered [16426.872584] LNet: 509979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16426.877685] LNetError: 509979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16426.894025] LNet: Removed LNI 192.168.201.152@tcp [16427.795540] Key type .llcrypt unregistered [16427.798283] Key type ._llcrypt unregistered [16446.684734] Key type ._llcrypt registered [16446.688125] Key type .llcrypt registered [16446.788622] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16448.037032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16448.060519] alg: No test for adler32 (adler32-zlib) [16449.217411] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [16449.521920] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [16451.215248] Key type lgssc registered [16453.296567] Lustre: Echo OBD driver; http://www.lustre.org/ [16462.606972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16462.621557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16464.280173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16469.422289] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16474.596984] LustreError: 511250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16474.625467] LustreError: 511250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [16479.445629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16479.712749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16484.315035] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16491.576254] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16497.418285] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16504.993492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16505.463402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16512.387772] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16520.867865] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16535.060794] loop0: detected capacity change from 0 to 204800000 [16535.370785] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16535.390886] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16535.406237] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16542.411395] loop0: detected capacity change from 0 to 204800000 [16542.624169] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16542.637359] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16542.670886] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16548.269677] loop0: detected capacity change from 0 to 204800000 [16553.884707] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16554.050654] Lustre: fs15246-OST0000: new disk, initializing [16554.054452] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16554.063353] Lustre: 514844:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16554.176992] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16561.029058] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16570.844992] loop1: detected capacity change from 0 to 204800000 [16576.844084] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16576.970751] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16577.030566] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16577.105296] Lustre: fs15246-MDT0000: new disk, initializing [16577.321490] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16577.360912] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16577.516172] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16578.594525] Lustre: 515967:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d8041daf x1873194405379200/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786417539 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16582.455322] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16587.689419] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16587.697692] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16587.724830] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16609.763801] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16609.780276] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16611.027111] Lustre: server umount fs15246-MDT0000 complete [16614.890647] LustreError: 511246:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16619.175504] Lustre: server umount fs15246-OST0000 complete [16630.242900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16630.248793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16630.263654] Lustre: Skipped 1 previous similar message [16630.278996] Lustre: Skipped 1 previous similar message [16635.363310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16635.377224] Lustre: Skipped 1 previous similar message [16636.211222] Lustre: server umount lustre-OST0000 complete [16640.481029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16640.496779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16640.506734] Lustre: Skipped 1 previous similar message [16640.526559] Lustre: Skipped 1 previous similar message [16645.605425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16645.617349] Lustre: Skipped 1 previous similar message [16646.425045] Lustre: server umount lustre-MDT0000 complete [16649.928675] LustreError: 511230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417595 with bad export cookie 2306484862201566683 [16649.932390] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16649.948512] LustreError: 511230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16650.396219] Lustre: server umount lustre-MDT0001 complete [16661.180080] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [16664.396950] Key type lgssc unregistered [16664.817060] LNet: 519064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16664.833203] LNetError: 519064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16664.854338] LNet: Removed LNI 192.168.201.152@tcp [16665.914663] Key type .llcrypt unregistered [16665.917642] Key type ._llcrypt unregistered [16686.296507] Key type ._llcrypt registered [16686.304188] Key type .llcrypt registered [16686.483978] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16687.678927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16687.699791] alg: No test for adler32 (adler32-zlib) [16688.742473] Lustre: Lustre: Build Version: 2.17.54_108_g0ecde8c [16689.046262] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [16690.711442] Key type lgssc registered [16692.414694] Lustre: Echo OBD driver; http://www.lustre.org/ [16735.201323] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [16747.063470] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16786.393810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16786.435611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16787.861501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16787.897458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16787.978521] Lustre: lustre-MDT0000: new disk, initializing [16788.068582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16788.094549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16792.719421] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16804.033350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16804.269046] Lustre: 523533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16804.321456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16804.324817] Lustre: Skipped 1 previous similar message [16804.400504] Lustre: lustre-MDT0001: new disk, initializing [16804.514279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16804.599811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16804.627258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16809.706967] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16819.429230] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16826.579144] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16835.007051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16835.611762] Lustre: lustre-OST0000: new disk, initializing [16835.617242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16835.636761] Lustre: 525504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16835.796539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16843.765389] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16844.934560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16844.943518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16845.101332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16854.622970] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16859.748232] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16859.987993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16865.034117] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16865.272096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16870.891633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16870.893831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16870.897151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16870.920306] Lustre: Skipped 1 previous similar message [16870.955669] LustreError: Skipped 1 previous similar message [16876.014681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16876.025677] Lustre: Skipped 2 previous similar messages [16881.125960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16881.146119] Lustre: Skipped 1 previous similar message [16882.148040] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16882.703545] Lustre: server umount lustre-OST0000 complete [16891.361403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16891.367903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16891.389333] Lustre: Skipped 1 previous similar message [16891.408808] Lustre: Skipped 1 previous similar message [16893.773937] Lustre: server umount lustre-MDT0000 complete [16898.836099] LustreError: 523525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786417844 with bad export cookie 8072392528342023737 [16898.852259] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16898.852701] LustreError: 523525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16899.163392] Lustre: server umount lustre-MDT0001 complete [16928.371869] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [16938.553860] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16969.662117] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16685 sec ========= 23:11:52 (1786417912) [16971.845340] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:11:54 (1786417914) === [16976.762411] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:11:59 (1786417919) === [17023.511684] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [17027.011825] Key type lgssc unregistered [17027.331884] LNet: 532155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17027.340505] LNetError: 532155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [17027.373775] LNet: Removed LNI 192.168.201.152@tcp [17028.454797] Key type .llcrypt unregistered [17028.458659] Key type ._llcrypt unregistered